Evaluate Models - Part 3 - Persistence Model for Forecasting

Establishing a baseline is essential on any time series forecasting problem. A baseline in performance gives you an idea of how well all other models will actually perform on your problem. In this tutorial, you will discover how to develop a persistence forecast that you can use to calculate a baseline level of performance on a time series dataset with Python. After completing this tutorial, you will know:

• The importance of calculating a baseline of performance on time series forecast problems.
• How to develop a persistence model from scratch in Python.
• How to evaluate the forecast from a persistence model and use it to establish a baseline in performance.

A. Forecast Performance Baseline

A baseline in forecast performance provides a point of comparison. It is a point of reference for all other modeling techniques on your problem. If a model achieves performance at or below the baseline, the technique should be fixed or abandoned. The technique used to generate a forecast to calculate the baseline performance must be easy to implement and naive of problem-specific details. Before you can establish a performance baseline on your forecast problem, you must develop a test harness. 

This is comprised of:

1. The dataset you intend to use to train and evaluate models.
2. The resampling technique you intend to use to estimate the performance of the technique (e.g. train/test split).
3. The performance measure you intend to use to evaluate forecasts (e.g. root mean squared error).

Once prepared, you then need to select a naive technique that you can use to make a forecast and calculate the baseline performance. The goal is to get a baseline performance on your time series forecast problem as quickly as possible so that you can get to work better understanding the dataset and developing more advanced models. Three properties of a good technique for making a baseline forecast are:

• Simple: A method that requires little or no training or intelligence.
• Fast: A method that is fast to implement and computationally trivial to make a prediction.
• Repeatable: A method that is deterministic, meaning that it produces an expected output given the same input.

B. Persistence Algorithm

The persistence algorithm uses the value at the current time step (t) to predict the expected outcome at the next time step (t+1). This satisfies the three above conditions for a baseline forecast. To make this concrete, we will look at how to develop a persistence model and use it to establish a baseline performance for a simple univariate time series problem.

The simplest forecast that we can make is to forecast that what happened in the previous time step will be the same as what will happen in the next time step. This is called the naive forecast or the persistence forecast model.

C. Persistence Algorithm Steps

A persistence model can be implemented easily in Python. We will break this tutorial down into 5 steps:

1. Transform the univariate dataset into a supervised learning problem.

2. Establish the train and test datasets for the test harness.

3. Define the persistence model.

4. Make a forecast and establish a baseline performance.

5. Review the complete example and plot the output.

Step 1: Define the Supervised Learning Problem

The first step is to load the dataset and create a lagged representation. That is, given the observation at t, predict the observation at t+1.

# Create lagged dataset

values = DataFrame(series.values)

dataframe = concat([values.shift(1), values], axis=1)

dataframe.columns = ['t', 't+1']

print(dataframe.head(5))

-----Result-----

t t+1

0 NaN 266.0

1 266.0 145.9

2 145.9 183.1

3 183.1 119.3

4 119.3 180.3

Step 2: Train and Test Sets

The next step is to separate the dataset into train and test sets. We will keep the first 66% of the observations for training and the remaining 34% for evaluation. During the split, we are careful to exclude the first row of data with the NaN value. No training is required in this case; it’s just habit. Each of the train and test sets are then split into the input and output variables.

Step 3: Persistence Algorithm

We can define our persistence model as a function that returns the value provided as input. For example, if the t value of 266.0 was provided, then this is returned as the prediction, whereas the actual real or expected value happens to be 145.9.

# persistence model

def model_persistence(x):

    return x

Step 4: Make and Evaluate Forecast

Now we can evaluate this model on the test dataset. We do this using the walk-forward validation method. No model training or retraining is required, so in essence, we step through the test dataset time step by time step and get predictions. Once predictions are made for each time step in the test dataset, they are compared to the expected values and a Root Mean Squared Error (RMSE) score is calculated.

# walk-forward validation
predictions = list()
for x in test_X:
    yhat = model_persistence(x)
    predictions.append(yhat)
rmse = sqrt(mean_squared_error(test_y, predictions))
print('Test RMSE: %.3f' % rmse)

Step 5: Complete Example

# evaluate a persistence forecast model

from pandas import read_csv

from pandas import datetime

from pandas import DataFrame

from pandas import concat

from matplotlib import pyplot

from sklearn.metrics import mean_squared_error

from math import sqrt

# load dataset

def parser(x):

return datetime.strptime('190'+x, '%Y-%m')

series = read_csv('shampoo-sales.csv', header=0, index_col=0, parse_dates=True, squeeze=True, date_parser=parser)

# create lagged dataset

values = DataFrame(series.values)

dataframe = concat([values.shift(1), values], axis=1)

dataframe.columns = ['t', 't+1']

print(dataframe.head(5))

# split into train and test sets

X = dataframe.values

train_size = int(len(X) * 0.66)

train, test = X[1:train_size], X[train_size:]

train_X, train_y = train[:,0], train[:,1]

test_X, test_y = test[:,0], test[:,1]

# persistence model

def model_persistence(x):

return x

# walk-forward validation

predictions = list()

for x in test_X:

yhat = model_persistence(x)

predictions.append(yhat)

rmse = sqrt(mean_squared_error(test_y, predictions))

print('Test RMSE: %.3f' % rmse)

# plot predictions and expected results

pyplot.plot(train_y)

pyplot.plot([None for i in train_y] + [x for x in test_y])

pyplot.plot([None for i in train_y] + [x for x in predictions])

pyplot.show()

-----Result-----

Test RMSE: 133.156

Line plot of the persistence forecast for the Shampoo Sales dataset showing the
training set (blue), test set (green) and predictions (red)

We have seen an example of the persistence model developed from scratch for the Shampoo Sales problem. The persistence algorithm is naive. It is often called the naive forecast. It assumes nothing about the specifics of the time series problem to which it is applied. This is what makes it so easy to understand and so quick to implement and evaluate.