1 Introduction

Now it’s time to take time series forecasting a step further. In this post, I explain the use of different neural networks to effectively predict time series. Here, we will focus on one target variable and use its time history to calculate future values.

For this post the dataset Metro_Interstate_Traffic_Volume from the statistic platform “Kaggle” was used. You can download it from my “GitHub Repository”.

2 Import the libraries and the data

import pandas as pd
import numpy as np

from sklearn import metrics
from sklearn import preprocessing
import tensorflow as tf

import matplotlib.pyplot as plt

df = pd.read_csv('Metro_Interstate_Traffic_Volume.csv')

print(df.shape)
df.head()

The variable ‘traffic_volume’ will be our target variable here.

3 Definition of required functions

def mean_absolute_percentage_error_func(y_true, y_pred):
    '''
    Calculate the mean absolute percentage error as a metric for evaluation
    
    Args:
        y_true (float64): Y values for the dependent variable (test part), numpy array of floats 
        y_pred (float64): Predicted values for the dependen variable (test parrt), numpy array of floats
    
    Returns:
        Mean absolute percentage error 
    '''    
    y_true, y_pred = np.array(y_true), np.array(y_pred)
    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100

def timeseries_evaluation_metrics_func(y_true, y_pred):
    '''
    Calculate the following evaluation metrics:
        - MSE
        - MAE
        - RMSE
        - MAPE
        - R²
    
    Args:
        y_true (float64): Y values for the dependent variable (test part), numpy array of floats 
        y_pred (float64): Predicted values for the dependen variable (test parrt), numpy array of floats
    
    Returns:
        MSE, MAE, RMSE, MAPE and R² 
    '''    
    print('Evaluation metric results: ')
    print(f'MSE is : {metrics.mean_squared_error(y_true, y_pred)}')
    print(f'MAE is : {metrics.mean_absolute_error(y_true, y_pred)}')
    print(f'RMSE is : {np.sqrt(metrics.mean_squared_error(y_true, y_pred))}')
    print(f'MAPE is : {mean_absolute_percentage_error_func(y_true, y_pred)}')
    print(f'R2 is : {metrics.r2_score(y_true, y_pred)}',end='\n\n')

def univariate_data_prep_func(dataset, start, end, window, horizon):
    '''
    Prepare univariate data that is suitable for a time series
    
    Args:
        dataset (float64): Scaled values for the dependent variable, numpy array of floats 
        start (int): Start point of range, integer
        end (int): End point of range, integer
        window (int): Number of units to be viewed per step, integer
        horizon (int): Number of units to be predicted, integer
    
    Returns:
        X (float64): Generated X-values for each step, numpy array of floats
        y (float64): Generated y-values for each step, numpy array of floats
    '''   
    X = []
    y = []

    start = start + window
    if end is None:
        end = len(dataset) - horizon

    for i in range(start, end):
        indicesx = range(i-window, i)
        X.append(np.reshape(dataset[indicesx], (window, 1)))
        indicesy = range(i,i+horizon)
        y.append(dataset[indicesy])
    return np.array(X), np.array(y)

4 Data pre-processing

4.1 Drop Duplicates

df = df.drop_duplicates(subset=['date_time'], keep=False)

df.shape

4.2 Generate Test Set

test_data = df['traffic_volume'].tail(10)

df = df.drop(df['traffic_volume'].tail(10).index)

df.shape

4.3 Define Target Variable

uni_data = df['traffic_volume']
uni_data.index = df['date_time']
uni_data.head()

4.4 Scaling

uni_data = uni_data.values
scaler_x = preprocessing.MinMaxScaler()
x_scaled = scaler_x.fit_transform(uni_data.reshape(-1, 1))

4.5 Train-Validation Split

In order to add the data to a neural network for training, they must be processed accordingly. There are two ways to do this:

Single Step Style
Horizon Style

Single Step Style

Single Step time-series forecasting is a technique where the model is exposed to one window of data at a time, such as days, weeks, months, years … and attempts to predict the next consecutive step. For example: Data is at the daily level. The model is shown the first window from the 1st to the 90th day (i.e. three months of data) and predicts the 91st day’s value. Then the next iteration (the 2nd to 91st day) for training it tries to predict the 92nd day.

Horizon Style

Horizon style time-series forecasting is a technique where the model is exposed to one window of data at a time, such as days, weeks, months, years … and attempts to predict the next n consecutive steps. For example: Data is at the daily level. The model is shown the first window from the 1st to the 90th day (i.e. three months of data) and predicts the values for the 91st to 101st days. Then the next iteration (the 2nd to 91st day) for training it tries to predict the 92nd to 102nd days.

I will use both variants in the following neural networks.

4.5.1 for Single Step Style (sss)

In the examples where I use single-step forecasting, I want to train the model on the last 48 hours and then try to predict the values for the 49th hour. Therefore horizon_sss = 1.

univar_hist_window_sss = 48
horizon_sss = 1
# 35120 observations in total
# 30000 should be part of the training (5120 validation)
train_split_sss = 30000

x_train_uni_sss, y_train_uni_sss = univariate_data_prep_func(x_scaled, 0, train_split_sss, 
                                                             univar_hist_window_sss, horizon_sss)

x_val_uni_sss, y_val_uni_sss = univariate_data_prep_func(x_scaled, train_split_sss, None, 
                                                         univar_hist_window_sss, horizon_sss)

print ('Length of first Single Window:')
print (len(x_train_uni_sss[0]))
print()
print ('Target horizon:')
print (y_train_uni_sss[0])

4.5.2 for Horizon Style (hs)

In the examples where I use the horizon style, I want to train the model on the last 48 hours and then try to predict the values for the next 10 hours. Therefore horizon_hs = 10.

univar_hist_window_hs = 48
horizon_hs = 10
# see comments of section 4.5.1
train_split_hs = 30000

x_train_uni_hs, y_train_uni_hs = univariate_data_prep_func(x_scaled, 0, train_split_hs, 
                                                           univar_hist_window_hs, horizon_hs)

x_val_uni_hs, y_val_uni_hs = univariate_data_prep_func(x_scaled, train_split_hs, None, 
                                                       univar_hist_window_hs, horizon_hs)

print ('Length of first Single Window:')
print (len(x_train_uni_hs[0]))
print()
print ('Target horizon:')
print (y_train_uni_hs[0])

4.6 Prepare training and test data using tf

To prepare the training and validation data I use the tf.data function. This allows a much faster and more efficient training of the neural networks.

4.6.1 for Single Step Style (sss)

BATCH_SIZE_sss = 256
BUFFER_SIZE_sss = 150

train_univariate_sss = tf.data.Dataset.from_tensor_slices((x_train_uni_sss, y_train_uni_sss))
train_univariate_sss = train_univariate_sss.cache().shuffle(BUFFER_SIZE_sss).batch(BATCH_SIZE_sss).repeat()

validation_univariate_sss = tf.data.Dataset.from_tensor_slices((x_val_uni_sss, y_val_uni_sss))
validation_univariate_sss = validation_univariate_sss.batch(BATCH_SIZE_sss).repeat()

4.6.2 for Horizon Style (hs)

BATCH_SIZE_hs = 256
BUFFER_SIZE_hs = 150

train_univariate_hs = tf.data.Dataset.from_tensor_slices((x_train_uni_hs, y_train_uni_hs))
train_univariate_hs = train_univariate_hs.cache().shuffle(BUFFER_SIZE_hs).batch(BATCH_SIZE_hs).repeat()

validation_univariate_hs = tf.data.Dataset.from_tensor_slices((x_val_uni_hs, y_val_uni_hs))
validation_univariate_hs = validation_univariate_hs.batch(BATCH_SIZE_hs).repeat()

5 Neural Networks

To save me more lines of code later, I’ll set a few parameters for the model training at this point:

n_steps_per_epoch = 117
n_validation_steps = 20
n_epochs = 100

This is how I calculated the selected values:

n_steps_per_epoch: len(training_df) / batch_size [30000/256]
n_validation_steps: len(validation_df) / batch_size [5120/256]

5.1 LSTM

5.1.1 Single Step Style

Define Layer Structure

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(100, input_shape=x_train_uni_sss.shape[-2:],return_sequences=True),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.LSTM(units=50,return_sequences=False),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(units=horizon_sss)])

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/lstm_model_sss.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_sss, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_sss, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_lstm_model_sss = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_sss)
test_history = test_horizon.values

result = []
# Define Forecast length here
window_len = len(test_data)
test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))

for i in range(1, window_len+1): 
    test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))
    
    # Inserting the model
    predicted_results = trained_lstm_model_sss.predict(test_scaled)
    
    print(f'predicted : {predicted_results}')
    result.append(predicted_results[0])
    test_scaled = np.append(test_scaled[:,1:],[[predicted_results]])

result_inv_trans = scaler_x.inverse_transform(result)
result_inv_trans

timeseries_evaluation_metrics_func(test_data, result_inv_trans)

rmse_lstm_model_sss = np.sqrt(metrics.mean_squared_error(test_data, result_inv_trans))

plt.plot(list(test_data))
plt.plot(list(result_inv_trans))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.1.2 Horizon Style

Define Layer Structure

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(100, input_shape=x_train_uni_hs.shape[-2:],return_sequences=True),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.LSTM(units=50,return_sequences=False),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(units=horizon_hs)])

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/lstm_model_hs.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_hs, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_hs, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_lstm_model_hs = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_hs)
test_history = test_horizon.values


test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))
test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))

# Inserting the model
predicted_results = trained_lstm_model_hs.predict(test_scaled)
predicted_results

predicted_inv_trans = scaler_x.inverse_transform(predicted_results)
predicted_inv_trans

timeseries_evaluation_metrics_func(test_data, predicted_inv_trans[0])

rmse_lstm_model_hs = np.sqrt(metrics.mean_squared_error(test_data, predicted_inv_trans[0]))

plt.plot(list(test_data))
plt.plot(list(predicted_inv_trans[0]))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.2 Bidirectional LSTM

5.2.1 Single Step Style

Define Layer Structure

model = tf.keras.models.Sequential([
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(100, return_sequences=True), 
                                  input_shape=x_train_uni_sss.shape[-2:]),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(50)),
    tf.keras.layers.Dense(20, activation='softmax'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(units=horizon_sss)])

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/bi_lstm_model_sss.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_sss, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_sss, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_bi_lstm_model_sss = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_sss)
test_history = test_horizon.values

result = []
# Define Forecast length here
window_len = len(test_data)
test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))

for i in range(1, window_len+1):
    
    test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))
    
    # Inserting the model
    predicted_results = trained_bi_lstm_model_sss.predict(test_scaled)
    
    print(f'predicted : {predicted_results}')
    result.append(predicted_results[0])
    test_scaled = np.append(test_scaled[:,1:],[[predicted_results]])

result_inv_trans = scaler_x.inverse_transform(result)
result_inv_trans

timeseries_evaluation_metrics_func(test_data, result_inv_trans)

rmse_bi_lstm_model_sss = np.sqrt(metrics.mean_squared_error(test_data, result_inv_trans))

plt.plot(list(test_data))
plt.plot(list(result_inv_trans))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.2.2 Horizon Style

Define Layer Structure

model = tf.keras.models.Sequential([
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(100, return_sequences=True), 
                                  input_shape=x_train_uni_hs.shape[-2:]),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(50)),
    tf.keras.layers.Dense(20, activation='softmax'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(units=horizon_hs)])

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/bi_lstm_model_hs.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_hs, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_hs, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_bi_lstm_model_hs = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_hs)
test_history = test_horizon.values


test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))
test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))

# Inserting the model
predicted_results = trained_bi_lstm_model_hs.predict(test_scaled)
predicted_results

predicted_inv_trans = scaler_x.inverse_transform(predicted_results)
predicted_inv_trans

timeseries_evaluation_metrics_func(test_data, predicted_inv_trans[0])

rmse_bi_lstm_model_hs = np.sqrt(metrics.mean_squared_error(test_data, predicted_inv_trans[0]))

plt.plot(list(test_data))
plt.plot(list(predicted_inv_trans[0]))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.3 GRU

5.3.1 Single Step Style

Define Layer Structure

model = tf.keras.models.Sequential([
    tf.keras.layers.GRU(100, input_shape=x_train_uni_sss.shape[-2:],return_sequences=True),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.GRU(units=50,return_sequences=False),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(units=horizon_sss)])

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/gru_model_sss.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_sss, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_sss, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_gru_model_sss = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_sss)
test_history = test_horizon.values

result = []
# Define Forecast length here
window_len = len(test_data)
test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))

for i in range(1, window_len+1):
    
    test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))
    
    # Inserting the model
    predicted_results = trained_gru_model_sss.predict(test_scaled)
    
    print(f'predicted : {predicted_results}')
    result.append(predicted_results[0])
    test_scaled = np.append(test_scaled[:,1:],[[predicted_results]])

result_inv_trans = scaler_x.inverse_transform(result)
result_inv_trans

timeseries_evaluation_metrics_func(test_data, result_inv_trans)

rmse_gru_model_sss = np.sqrt(metrics.mean_squared_error(test_data, result_inv_trans))

plt.plot(list(test_data))
plt.plot(list(result_inv_trans))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.3.2 Horizon Style

Define Layer Structure

model = tf.keras.models.Sequential([
    tf.keras.layers.GRU(100, input_shape=x_train_uni_hs.shape[-2:],return_sequences=True),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.GRU(units=50,return_sequences=False),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(units=horizon_hs)])

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/gru_model_hs.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_hs, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_hs, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_gru_model_hs = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_hs)
test_history = test_horizon.values


test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))
test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))

# Inserting the model
predicted_results = trained_gru_model_hs.predict(test_scaled)
predicted_results

predicted_inv_trans = scaler_x.inverse_transform(predicted_results)
predicted_inv_trans

timeseries_evaluation_metrics_func(test_data, predicted_inv_trans[0])

rmse_gru_model_hs = np.sqrt(metrics.mean_squared_error(test_data, predicted_inv_trans[0]))

plt.plot(list(test_data))
plt.plot(list(predicted_inv_trans[0]))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.4 Encoder Decoder LSTM

5.4.1 Single Step Style

Define Layer Structure

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(100, input_shape=x_train_uni_sss.shape[-2:], return_sequences=True),
    tf.keras.layers.LSTM(units=50,return_sequences=True),
    tf.keras.layers.LSTM(units=15),
    tf.keras.layers.RepeatVector(y_train_uni_sss.shape[1]), 
    tf.keras.layers.LSTM(units=100,return_sequences=True),
    tf.keras.layers.LSTM(units=50,return_sequences=True),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(units=horizon_sss))])

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/ed_lstm_model_sss.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_sss, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_sss, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_ed_lstm_model_sss = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_sss)
test_history = test_horizon.values

result = []
# Define Forecast length here
window_len = len(test_data)
test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))

for i in range(1, window_len+1):
    
    test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))
    
    # Inserting the model
    predicted_results = trained_ed_lstm_model_sss.predict(test_scaled)
    
    print(f'predicted : {predicted_results}')
    result.append(predicted_results[0])
    test_scaled = np.append(test_scaled[:,1:],[[predicted_results]])

result_inv_trans = scaler_x.inverse_transform(np.array(result).reshape(-1,1))
result_inv_trans

timeseries_evaluation_metrics_func(test_data, result_inv_trans)

rmse_ed_lstm_model_sss = np.sqrt(metrics.mean_squared_error(test_data, result_inv_trans))

plt.plot(list(test_data))
plt.plot(list(result_inv_trans))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.4.2 Horizon Style

Define Layer Structure

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(100, input_shape=x_train_uni_hs.shape[-2:], return_sequences=True),
    tf.keras.layers.LSTM(units=50,return_sequences=True),
    tf.keras.layers.LSTM(units=15),
    tf.keras.layers.RepeatVector(y_train_uni_hs.shape[1]), 
    tf.keras.layers.LSTM(units=100,return_sequences=True),
    tf.keras.layers.LSTM(units=50,return_sequences=True),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(units=1))])

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/ed_lstm_model_hs.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_hs, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_hs, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_ed_lstm_model_hs = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_hs)
test_history = test_horizon.values


test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))
test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))

# Inserting the model
predicted_results = trained_ed_lstm_model_hs.predict(test_scaled)
predicted_results

predicted_inv_trans = scaler_x.inverse_transform(predicted_results[0])
predicted_inv_trans

timeseries_evaluation_metrics_func(test_data, predicted_inv_trans)

rmse_ed_lstm_model_hs = np.sqrt(metrics.mean_squared_error(test_data, predicted_inv_trans))

plt.plot(list(test_data))
plt.plot(list(predicted_inv_trans))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.5 CNN

5.5.1 Single Step Style

Define Layer Structure

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=64, kernel_size=3, activation='relu', 
                                 input_shape=(x_train_uni_sss.shape[1], x_train_uni_sss.shape[2])))
model.add(tf.keras.layers.MaxPool1D(pool_size=2))
model.add(tf.keras.layers.Dropout(0.2))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(30, activation='relu'))
model.add(tf.keras.layers.Dropout(0.2))
model.add(tf.keras.layers.Dense(units=horizon_sss))

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/cnn_model_sss.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_sss, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_sss, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_cnn_model_sss = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_sss)
test_history = test_horizon.values

result = []
# Define Forecast length here
window_len = len(test_data)
test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))

for i in range(1, window_len+1):
    
    test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))
    
    # Inserting the model
    predicted_results = trained_cnn_model_sss.predict(test_scaled)
    
    print(f'predicted : {predicted_results}')
    result.append(predicted_results[0])
    test_scaled = np.append(test_scaled[:,1:],[[predicted_results]])

result_inv_trans = scaler_x.inverse_transform(result)
result_inv_trans

timeseries_evaluation_metrics_func(test_data, result_inv_trans)

rmse_cnn_model_sss = np.sqrt(metrics.mean_squared_error(test_data, result_inv_trans))

plt.plot(list(test_data))
plt.plot(list(result_inv_trans))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

5.5.2 Horizon Style

Define Layer Structure

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=64, kernel_size=3, activation='relu', 
                                 input_shape=(x_train_uni_hs.shape[1], x_train_uni_hs.shape[2])))
model.add(tf.keras.layers.MaxPool1D(pool_size=2))
model.add(tf.keras.layers.Dropout(0.2))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(30, activation='relu'))
model.add(tf.keras.layers.Dropout(0.2))
model.add(tf.keras.layers.Dense(units=horizon_hs))

model.compile(loss='mse',
              optimizer='adam')

Fit the model

model_path = 'model/cnn_model_hs.h5'

keras_callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                                    min_delta=0, patience=10, 
                                                    verbose=1, mode='min'),
                   tf.keras.callbacks.ModelCheckpoint(model_path,monitor='val_loss', 
                                                      save_best_only=True, 
                                                      mode='min', verbose=0)]

history = model.fit(train_univariate_hs, epochs=n_epochs, steps_per_epoch=n_steps_per_epoch,
                    validation_data=validation_univariate_hs, validation_steps=n_validation_steps, verbose =1,
                    callbacks = keras_callbacks)

Validate the model

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(loss) + 1)


plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Test the model

trained_cnn_model_hs = tf.keras.models.load_model(model_path)

df_temp = df['traffic_volume']
test_horizon = df_temp.tail(univar_hist_window_hs)
test_history = test_horizon.values


test_scaled = scaler_x.fit_transform(test_history.reshape(-1, 1))
test_scaled = test_scaled.reshape((1, test_scaled.shape[0], 1))

# Inserting the model
predicted_results = trained_cnn_model_hs.predict(test_scaled)
predicted_results

predicted_inv_trans = scaler_x.inverse_transform(predicted_results)
predicted_inv_trans

timeseries_evaluation_metrics_func(test_data, predicted_inv_trans[0])

rmse_cnn_model_hs = np.sqrt(metrics.mean_squared_error(test_data, predicted_inv_trans[0]))

plt.plot(list(test_data))
plt.plot(list(predicted_inv_trans[0]))
plt.title("Actual vs Predicted")
plt.ylabel("Traffic volume")
plt.legend(('Actual','predicted'))
plt.show()

6 Get the Best Model

Now we want to know which model performed best:

column_names = ["Model", "RMSE"]
df = pd.DataFrame(columns = column_names)

rmse_lstm_model_sss_df = pd.DataFrame([('lstm_model_sss', rmse_lstm_model_sss)], columns=column_names)
df = df.append(rmse_lstm_model_sss_df)

rmse_lstm_model_hs_df = pd.DataFrame([('lstm_model_hs', rmse_lstm_model_hs)], columns=column_names)
df = df.append(rmse_lstm_model_hs_df)

rmse_bi_lstm_model_sss_df = pd.DataFrame([('bi_lstm_model_sss', rmse_bi_lstm_model_sss)], columns=column_names)
df = df.append(rmse_bi_lstm_model_sss_df)

rmse_bi_lstm_model_hs_df = pd.DataFrame([('bi_lstm_model_hs', rmse_bi_lstm_model_hs)], columns=column_names)
df = df.append(rmse_bi_lstm_model_hs_df)

rmse_gru_model_sss_df = pd.DataFrame([('gru_model_sss', rmse_gru_model_sss)], columns=column_names)
df = df.append(rmse_gru_model_sss_df)

rmse_gru_model_hs_df = pd.DataFrame([('gru_model_hs', rmse_gru_model_hs)], columns=column_names)
df = df.append(rmse_gru_model_hs_df)

rmse_ed_lstm_model_sss_df = pd.DataFrame([('ed_lstm_model_sss', rmse_ed_lstm_model_sss)], columns=column_names)
df = df.append(rmse_ed_lstm_model_sss_df)

rmse_ed_lstm_model_hs_df = pd.DataFrame([('ed_lstm_model_hs', rmse_ed_lstm_model_hs)], columns=column_names)
df = df.append(rmse_ed_lstm_model_hs_df)

rmse_cnn_model_sss_df = pd.DataFrame([('cnn_model_sss', rmse_cnn_model_sss)], columns=column_names)
df = df.append(rmse_cnn_model_sss_df)

rmse_cnn_model_hs_df = pd.DataFrame([('cnn_model_hs', rmse_cnn_model_hs)], columns=column_names)
df = df.append(rmse_cnn_model_hs_df)

df

best_model = df.sort_values(by='RMSE', ascending=True)
best_model

As we can see, there are already serious performance differences in the different neural networks. It is therefore always worthwhile to use different networks to see which one fits best. The best fit can then be further optimized.

7 Conclusion

In this post I showed how to make time series predictions using neural networks. I have applied:

LSTM
Bidirectional LSTM
GRU
Encoder Decoder LSTM
CNN

References

The content of this post was inspired by:

Kaggle: Time Series Analysis using LSTM Keras from Hassan Amin

Chollet, F. (2018). Deep learning with Python (Vol. 361). New York: Manning.

Vishwas, B. V., & Patel, A. (2020). Hands-on Time Series Analysis with Python. New York: Apress. DOI: 10.1007/978-1-4842-5992-4

Medium: Time Series Forecast Using Deep Learning from Rajaram Suryanarayanan

Time Series Analysis - Neural Networks for Univariate Time Series

1 Introduction

2 Import the libraries and the data

3 Definition of required functions

4 Data pre-processing

4.1 Drop Duplicates

4.2 Generate Test Set

4.3 Define Target Variable

4.4 Scaling

4.5 Train-Validation Split

4.5.1 for Single Step Style (sss)

4.5.2 for Horizon Style (hs)

4.6 Prepare training and test data using tf

4.6.1 for Single Step Style (sss)

4.6.2 for Horizon Style (hs)

5 Neural Networks

5.1 LSTM

5.1.1 Single Step Style

5.1.2 Horizon Style

5.2 Bidirectional LSTM

5.2.1 Single Step Style

5.2.2 Horizon Style

5.3 GRU

5.3.1 Single Step Style

5.3.2 Horizon Style

5.4 Encoder Decoder LSTM

5.4.1 Single Step Style

5.4.2 Horizon Style

5.5 CNN

5.5.1 Single Step Style

5.5.2 Horizon Style

6 Get the Best Model

7 Conclusion