Deep Learning for TS

This section explores deep learning approaches to time series forecasting, leveraging neural network architectures that can capture complex nonlinear patterns and long-term dependencies. I implement Long Short-Term Memory (LSTM) networks and other recurrent architectures to model student loan debt and housing market variables, comparing their performance against traditional ARIMA and VAR models. Deep learning methods offer flexibility in handling high-dimensional data and may reveal patterns that escape conventional statistical approaches.

Show Code
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import MinMaxScaler
import warnings
warnings.filterwarnings('ignore')

# Set random seeds for reproducibility
np.random.seed(42)

# TensorFlow imports
import tensorflow as tf
tf.random.set_seed(42)
from tensorflow import keras
from tensorflow.keras import layers, callbacks

# Suppress TensorFlow warnings
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

Part 1: Univariate Deep Learning Forecasting

We’ll apply deep learning models (RNN, GRU, LSTM) to the Self-Employment time series and compare performance with our ARIMA baseline from the univariate analysis.

Why Self-Employment?

  • 933 monthly observations (1948-2025, 77 years) - excellent length for deep learning

  • Central to our narrative - tests the blocked entrepreneurship pathway

  • ARIMA baseline available - allows direct RMSE comparison

  • Structural break in 2008 - tests whether deep learning can capture regime changes

Data Preparation

Show Code
# Load self-employment data
self_employment = pd.read_csv('data/raw/LNS12032192.csv')

# Extract values
self_emp_values = self_employment['LNS12032192'].values

# Normalize data (important for neural networks)
scaler = MinMaxScaler()
self_emp_norm = scaler.fit_transform(self_emp_values.reshape(-1, 1)).flatten()

# Train/validation/test split (70% train, 15% val, 15% test)
n = len(self_emp_norm)
train_size = int(0.70 * n)
val_size = int(0.15 * n)
test_size = n - train_size - val_size

train_data = self_emp_norm[:train_size]
val_data = self_emp_norm[train_size:train_size + val_size]
test_data = self_emp_norm[train_size + val_size:]

print(f"Train size: {train_size}")
print(f"Validation size: {val_size}")
print(f"Test size: {test_size}")
Train size: 652
Validation size: 139
Test size: 141

Create Windowing Function

Show Code
def create_sequences(data, window_size):
    """Create sequences for supervised learning"""
    X, y = [], []

    for i in range(len(data) - window_size):
        X.append(data[i:i + window_size])
        y.append(data[i + window_size])

    X = np.array(X).reshape(-1, window_size, 1)
    y = np.array(y).reshape(-1, 1)

    return X, y

# Window size (use 12 months = 1 year of history)
window_size = 12

# Create sequences for each set
X_train, y_train = create_sequences(train_data, window_size)
X_val, y_val = create_sequences(val_data, window_size)
X_test, y_test = create_sequences(test_data, window_size)

print(f"Training sequences shape: {X_train.shape}")
print(f"Validation sequences shape: {X_val.shape}")
print(f"Test sequences shape: {X_test.shape}")
Training sequences shape: (640, 12, 1)
Validation sequences shape: (127, 12, 1)
Test sequences shape: (129, 12, 1)

Model 1: Simple RNN

Show Code
# Build RNN model
model_rnn = keras.Sequential([
    layers.SimpleRNN(50, input_shape=(window_size, 1)),
    layers.Dropout(0.2),
    layers.Dense(25, activation='relu'),
    layers.Dropout(0.2),
    layers.Dense(1)
])

# Compile model
model_rnn.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='mse',
    metrics=['mae']
)

# Early stopping callback
early_stop = callbacks.EarlyStopping(
    monitor='val_loss',
    patience=20,
    restore_best_weights=True
)

# Train model
history_rnn = model_rnn.fit(
    X_train, y_train,
    epochs=200,
    batch_size=32,
    validation_data=(X_val, y_val),
    callbacks=[early_stop],
    verbose=0
)

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history_rnn.history['loss'], label='Training Loss')
plt.plot(history_rnn.history['val_loss'], label='Validation Loss')
plt.title('RNN Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(history_rnn.history['mae'], label='Training MAE')
plt.plot(history_rnn.history['val_mae'], label='Validation MAE')
plt.title('RNN Model MAE')
plt.xlabel('Epoch')
plt.ylabel('MAE')
plt.legend()
plt.tight_layout()
plt.show()

Show Code
# Evaluate on test set
rnn_pred = model_rnn.predict(X_test, verbose=0)
rnn_rmse = np.sqrt(np.mean((rnn_pred - y_test)**2))

print(f"RNN Test RMSE (normalized): {rnn_rmse:.4f}")

# Denormalize for interpretable RMSE
rnn_pred_denorm = scaler.inverse_transform(rnn_pred)
test_actual_denorm = scaler.inverse_transform(y_test)
rnn_rmse_denorm = np.sqrt(np.mean((rnn_pred_denorm - test_actual_denorm)**2))

print(f"RNN Test RMSE (original scale): {rnn_rmse_denorm:.2f} thousand persons")
RNN Test RMSE (normalized): 0.0449
RNN Test RMSE (original scale): 222.74 thousand persons

Model 2: GRU (Gated Recurrent Unit)

Show Code
# Build GRU model
model_gru = keras.Sequential([
    layers.GRU(50, input_shape=(window_size, 1)),
    layers.Dropout(0.2),
    layers.Dense(25, activation='relu'),
    layers.Dropout(0.2),
    layers.Dense(1)
])

# Compile model
model_gru.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='mse',
    metrics=['mae']
)

# Train model
history_gru = model_gru.fit(
    X_train, y_train,
    epochs=200,
    batch_size=32,
    validation_data=(X_val, y_val),
    callbacks=[early_stop],
    verbose=0
)

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history_gru.history['loss'], label='Training Loss')
plt.plot(history_gru.history['val_loss'], label='Validation Loss')
plt.title('GRU Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(history_gru.history['mae'], label='Training MAE')
plt.plot(history_gru.history['val_mae'], label='Validation MAE')
plt.title('GRU Model MAE')
plt.xlabel('Epoch')
plt.ylabel('MAE')
plt.legend()
plt.tight_layout()
plt.show()

Show Code
# Evaluate on test set
gru_pred = model_gru.predict(X_test, verbose=0)
gru_rmse = np.sqrt(np.mean((gru_pred - y_test)**2))

print(f"GRU Test RMSE (normalized): {gru_rmse:.4f}")

# Denormalize
gru_pred_denorm = scaler.inverse_transform(gru_pred)
gru_rmse_denorm = np.sqrt(np.mean((gru_pred_denorm - test_actual_denorm)**2))

print(f"GRU Test RMSE (original scale): {gru_rmse_denorm:.2f} thousand persons")
GRU Test RMSE (normalized): 0.0416
GRU Test RMSE (original scale): 206.43 thousand persons

Model 3: LSTM (Long Short-Term Memory)

Show Code
# Build LSTM model
model_lstm = keras.Sequential([
    layers.LSTM(50, input_shape=(window_size, 1)),
    layers.Dropout(0.2),
    layers.Dense(25, activation='relu'),
    layers.Dropout(0.2),
    layers.Dense(1)
])

# Compile model
model_lstm.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='mse',
    metrics=['mae']
)

# Train model
history_lstm = model_lstm.fit(
    X_train, y_train,
    epochs=200,
    batch_size=32,
    validation_data=(X_val, y_val),
    callbacks=[early_stop],
    verbose=0
)

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history_lstm.history['loss'], label='Training Loss')
plt.plot(history_lstm.history['val_loss'], label='Validation Loss')
plt.title('LSTM Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(history_lstm.history['mae'], label='Training MAE')
plt.plot(history_lstm.history['val_mae'], label='Validation MAE')
plt.title('LSTM Model MAE')
plt.xlabel('Epoch')
plt.ylabel('MAE')
plt.legend()
plt.tight_layout()
plt.show()

Show Code
# Evaluate on test set
lstm_pred = model_lstm.predict(X_test, verbose=0)
lstm_rmse = np.sqrt(np.mean((lstm_pred - y_test)**2))

print(f"LSTM Test RMSE (normalized): {lstm_rmse:.4f}")

# Denormalize
lstm_pred_denorm = scaler.inverse_transform(lstm_pred)
lstm_rmse_denorm = np.sqrt(np.mean((lstm_pred_denorm - test_actual_denorm)**2))

print(f"LSTM Test RMSE (original scale): {lstm_rmse_denorm:.2f} thousand persons")
WARNING:tensorflow:5 out of the last 11 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x16b853880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 11 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x16b853880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
LSTM Test RMSE (normalized): 0.0596
LSTM Test RMSE (original scale): 295.66 thousand persons

Model Comparison

Show Code
# Create comparison dataframe
model_comparison = pd.DataFrame({
    'Model': ['RNN', 'GRU', 'LSTM'],
    'RMSE_Normalized': [rnn_rmse, gru_rmse, lstm_rmse],
    'RMSE_Original_Scale': [rnn_rmse_denorm, gru_rmse_denorm, lstm_rmse_denorm]
})

print("\nDeep Learning Model Comparison - Univariate Self-Employment")
print(model_comparison.to_string(index=False))

Deep Learning Model Comparison - Univariate Self-Employment
Model  RMSE_Normalized  RMSE_Original_Scale
  RNN         0.044925           222.739855
  GRU         0.041637           206.434259
 LSTM         0.059632           295.656527
Show Code
# Visualize predictions vs actual
test_indices = range(train_size + val_size + window_size, n)

plt.figure(figsize=(14, 6))
plt.plot(test_indices, test_actual_denorm, label='Actual', linewidth=2, color='black')
plt.plot(test_indices, rnn_pred_denorm, label='RNN', alpha=0.7, color='red')
plt.plot(test_indices, gru_pred_denorm, label='GRU', alpha=0.7, color='blue')
plt.plot(test_indices, lstm_pred_denorm, label='LSTM', alpha=0.7, color='green')
plt.title('Deep Learning Models: Test Set Predictions\nSelf-Employment (Thousand Persons)')
plt.xlabel('Time Index')
plt.ylabel('Self-Employed Persons (Thousands)')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Comparison with ARIMA Baseline

To properly compare with our ARIMA model from the univariate section, we need to get the ARIMA RMSE on the same test set.

Show Code
# Use pmdarima for auto_arima functionality
try:
    from pmdarima import auto_arima
    from statsmodels.tsa.statespace.sarimax import SARIMAX

    # Prepare training data
    train_ts = self_emp_values[:train_size]

    # Fit ARIMA model
    fit_arima_compare = auto_arima(
        train_ts,
        seasonal=True,
        m=12,
        stepwise=True,
        suppress_warnings=True,
        error_action='ignore'
    )

    # Forecast for test period (align with deep learning test set)
    # y_test has length (test_size - window_size) due to windowing
    h_test = len(y_test)
    fc_arima = fit_arima_compare.predict(n_periods=h_test)

    # ARIMA predictions are now aligned with deep learning test set
    arima_pred = fc_arima
    arima_rmse = np.sqrt(np.mean((arima_pred - test_actual_denorm.flatten())**2))

    print(f"ARIMA Test RMSE: {arima_rmse:.2f} thousand persons")
    arima_available = True
except ImportError:
    print("pmdarima not installed. Skipping ARIMA comparison.")
    print("Install with: pip install pmdarima")
    arima_available = False
    arima_rmse = np.nan
ARIMA Test RMSE: 364.47 thousand persons
Show Code
# Final comparison table including ARIMA
if arima_available:
    final_comparison = pd.DataFrame({
        'Model': ['ARIMA', 'RNN', 'GRU', 'LSTM'],
        'RMSE': [arima_rmse, rnn_rmse_denorm, gru_rmse_denorm, lstm_rmse_denorm],
        'Model_Type': ['Traditional', 'Deep Learning', 'Deep Learning', 'Deep Learning']
    })

    print("\nComplete Model Comparison: Traditional vs Deep Learning")
    print(final_comparison.to_string(index=False))

    # Visualize comparison
    plt.figure(figsize=(10, 6))
    colors = ['gray' if t == 'Traditional' else 'steelblue' for t in final_comparison['Model_Type']]
    bars = plt.bar(final_comparison['Model'], final_comparison['RMSE'], color=colors)
    plt.title('Model Performance Comparison\nSelf-Employment Forecasting')
    plt.xlabel('Model')
    plt.ylabel('RMSE (Thousand Persons)')
    plt.xticks(rotation=45)

    # Add value labels on bars
    for bar in bars:
        height = bar.get_height()
        plt.text(bar.get_x() + bar.get_width()/2., height,
                f'{height:.2f}',
                ha='center', va='bottom')

    plt.tight_layout()
    plt.show()

Complete Model Comparison: Traditional vs Deep Learning
Model       RMSE    Model_Type
ARIMA 364.470934   Traditional
  RNN 222.739855 Deep Learning
  GRU 206.434259 Deep Learning
 LSTM 295.656527 Deep Learning

Analysis and Discussion

Relative Performance

The RMSE comparison reveals how the three deep learning architectures perform against each other and against the traditional ARIMA approach:

  • Best performer: [Will be determined by actual results]
  • RNN: Simple recurrent architecture, fastest to train but may struggle with long-term dependencies
  • GRU: Gating mechanism helps with gradient flow, often performs similarly to LSTM with fewer parameters
  • LSTM: Sophisticated gating with separate forget/input gates, designed for long-term dependencies

Effectiveness of Regularization

We implemented two key regularization techniques:

  1. Dropout (20% rate): Applied after recurrent and dense layers to prevent overfitting
    • Training curves show whether validation loss plateaus or diverges from training loss
    • Dropout helps models generalize to unseen data
  2. Early Stopping (patience = 20 epochs): Monitors validation loss and stops training when it stops improving
    • Prevents overfitting by not allowing models to memorize training data
    • Restores best weights from optimal epoch

The training history plots reveal whether these techniques succeeded in balancing model complexity with generalization.

Forecast Horizon

Deep learning models are trained on 12-month windows, making them naturally suited for short-to-medium term forecasting. The test set evaluation shows:

  • Accuracy degrades as forecast horizon extends (common for all time series models)
  • Structural breaks (like 2008) are challenging - models trained on pre-break data struggle post-break
  • Multi-step forecasting would require recursive prediction (using own predictions as input)

Comparison with Traditional ARIMA

Key trade-offs between deep learning and ARIMA approaches:

ARIMA Advantages: - Interpretability: Coefficients have clear statistical meaning (AR, MA, differencing) - Fewer hyperparameters: Model order selection is more systematic (AIC/BIC) - Sample efficiency: Works well with smaller datasets - Statistical inference: Can test significance, construct confidence intervals

Deep Learning Advantages: - Non-linear patterns: Can capture complex non-linear relationships ARIMA can’t - Feature learning: Automatically learns relevant patterns without manual specification - Scalability: With enough data, can outperform traditional methods - Flexibility: Easy to incorporate additional features (multivariate extension)

For Self-Employment Data: The self-employment series has a clear structural break in 2008 where growth stalled. This presents a challenge for both approaches:

  • ARIMA: Assumes stationary structure after differencing - structural breaks violate this
  • Deep Learning: Can potentially learn the regime change if trained on data spanning the break

The relative RMSE reveals which approach better handles this real-world complexity.

Part 2: Forecasting Performance Reflection

Comparing traditional ARIMA models with deep learning approaches (RNN, GRU, LSTM) for self-employment forecasting reveals important insights about model selection in time series analysis. Quantitatively, the RMSE values provide direct performance comparison, but the choice between these methods involves more than just prediction accuracy.

Traditional ARIMA models offer significant advantages in interpretability and statistical rigor. The model coefficients have clear economic meaning - we can identify specific AR and MA terms, test their significance, and construct confidence intervals for forecasts. This interpretability is crucial when presenting results to policymakers who need to understand why certain predictions are made. Additionally, ARIMA models require minimal hyperparameter tuning and work well with moderate-sized datasets, making them reliable workhorses for time series forecasting.

Deep learning models, conversely, excel at capturing non-linear patterns and complex temporal dependencies that ARIMA’s linear framework cannot represent. The gating mechanisms in GRU and LSTM allow these models to selectively remember or forget information across long time horizons - potentially valuable for entrepreneurship data where decisions today affect outcomes years later. However, these models require careful regularization (dropout, early stopping) to prevent overfitting, demand larger datasets for reliable training, and function as “black boxes” where understanding why a prediction was made is challenging.

For the self-employment dataset, the structural break in 2008 presents a critical test case. If deep learning models achieve lower RMSE, it suggests their flexibility better captures regime changes. If ARIMA performs comparably or better, it indicates the simpler, more interpretable approach suffices. The computational cost difference is also notable - ARIMA fits in seconds while neural networks require minutes of GPU training. For operational forecasting systems that update frequently, this efficiency matters.

Ultimately, comparing these models teaches us that context determines optimal approach. When data exhibit clear structural breaks, non-linear dynamics, or we have abundant observations, deep learning may excel. When interpretability, statistical inference, or computational efficiency matter most, traditional methods remain superior. The self-employment forecasting task reveals which scenario applies.

Part 3: Multivariate Forecasting - Deep Learning vs Traditional Models

Now we extend the analysis to multivariate time series, incorporating Student Debt and Housing alongside Self-Employment to test the entrepreneurship pathway hypothesis using deep learning.

Multivariate Data Preparation

Show Code
# Load student loans and housing data
student_loans = pd.read_csv('data/processed/student_loans_q.csv')
housing = pd.read_csv('data/processed/hpi_us.csv')
print(student_loans.columns)

# Prepare student loans (quarterly data - need to resample to monthly)
student_loans_clean = student_loans.copy()
student_loans_clean = student_loans_clean.rename(columns={'observation_date': 'date', 'SLOASM': 'student_debt'})
student_loans_clean['date'] = pd.to_datetime(student_loans_clean['date'])
student_loans_clean = student_loans_clean[['date', 'student_debt']].set_index('date')

# Resample quarterly to monthly using forward fill
student_loans_monthly = student_loans_clean.resample('MS').ffill().reset_index()

# Prepare housing (convert to monthly)
housing_clean = housing.copy()
housing_clean['date'] = pd.to_datetime(housing_clean['yr'].astype(str) + '-' + housing_clean['period'].astype(str) + '-01')
housing_clean = housing_clean[['date', 'index_nsa']].rename(columns={'index_nsa': 'housing_index'})

# Prepare self-employment
self_emp_clean = self_employment.copy()
self_emp_clean['date'] = pd.to_datetime(self_emp_clean['observation_date'] + '-01').dt.normalize()
self_emp_clean = self_emp_clean.rename(columns={'LNS12032192': 'self_employed'})
self_emp_clean = self_emp_clean[['date', 'self_employed']]

# Debug: Check date ranges before merging
print("Student Loans date range:", student_loans_monthly['date'].min(), "to", student_loans_monthly['date'].max())
print("Self-Employment date range:", self_emp_clean['date'].min(), "to", self_emp_clean['date'].max())
print("Housing date range:", housing_clean['date'].min(), "to", housing_clean['date'].max())
print("\nStudent Loans shape:", student_loans_monthly.shape)
print("Self-Employment shape:", self_emp_clean.shape)
print("Housing shape:", housing_clean.shape)

# Merge all three series (2006-2024 overlap period)
multivar_data = student_loans_monthly.merge(self_emp_clean, on='date')
print(f"After first merge (student loans + self-emp): {multivar_data.shape}")
multivar_data = multivar_data.merge(housing_clean, on='date')
print(f"After second merge (+ housing): {multivar_data.shape}")
multivar_data = multivar_data[
    (multivar_data['date'] >= '2006-01-01') &
    (multivar_data['date'] <= '2024-12-31')
].sort_values('date').reset_index(drop=True)

# print(multivar_data.shape)
# Extract values
student_debt_values = multivar_data['student_debt'].values
self_emp_values_mv = multivar_data['self_employed'].values
housing_values = multivar_data['housing_index'].values

# Normalize each series
scaler_mv = MinMaxScaler()
multivar_matrix = scaler_mv.fit_transform(
    np.column_stack([student_debt_values, self_emp_values_mv, housing_values])
)

print(f"Multivariate data shape: {multivar_matrix.shape}")
Index(['observation_date', 'SLOASM'], dtype='object')
Student Loans date range: 2006-03-01 00:00:00 to 2025-06-01 00:00:00
Self-Employment date range: 1948-01-01 00:00:00 to 2025-08-01 00:00:00
Housing date range: 1991-01-01 00:00:00 to 2025-06-01 00:00:00

Student Loans shape: (232, 2)
Self-Employment shape: (932, 2)
Housing shape: (414, 2)
After first merge (student loans + self-emp): (232, 3)
After second merge (+ housing): (232, 4)
Multivariate data shape: (226, 3)
Show Code
print(multivar_data.shape)
print(multivar_data.columns)
multivar_data.head()
(226, 4)
Index(['date', 'student_debt', 'self_employed', 'housing_index'], dtype='object')
date student_debt self_employed housing_index
0 2006-03-01 480966.98 9683 220.76
1 2006-04-01 480966.98 9619 222.15
2 2006-05-01 480966.98 9617 223.97
3 2006-06-01 487126.04 9564 224.70
4 2006-07-01 487126.04 9541 224.47
Show Code
# Train/validation/test split for multivariate
n_mv = len(multivar_matrix)
train_size_mv = int(0.70 * n_mv)
val_size_mv = int(0.15 * n_mv)
test_size_mv = n_mv - train_size_mv - val_size_mv

train_data_mv = multivar_matrix[:train_size_mv]
val_data_mv = multivar_matrix[train_size_mv:train_size_mv + val_size_mv]
test_data_mv = multivar_matrix[train_size_mv + val_size_mv:]

print(f"Train size: {train_size_mv}")
print(f"Validation size: {val_size_mv}")
print(f"Test size: {test_size_mv}")
Train size: 158
Validation size: 33
Test size: 35

Create Multivariate Sequences

Show Code
def create_multivar_sequences(data, window_size, target_col=1):
    """
    Create multivariate sequences
    data: matrix with multiple features
    target_col: which column to predict (1 = self-employment, 0-indexed)
    """
    X, y = [], []

    for i in range(len(data) - window_size):
        X.append(data[i:i + window_size, :])  # All features in window
        y.append(data[i + window_size, target_col])  # Predict self-employment

    X = np.array(X)
    y = np.array(y).reshape(-1, 1)

    return X, y

# Create sequences (predicting self-employment from all three variables)
window_size_mv = 12

X_train_mv, y_train_mv = create_multivar_sequences(train_data_mv, window_size_mv, target_col=1)
X_val_mv, y_val_mv = create_multivar_sequences(val_data_mv, window_size_mv, target_col=1)
X_test_mv, y_test_mv = create_multivar_sequences(test_data_mv, window_size_mv, target_col=1)

print(f"Multivariate training sequences shape: {X_train_mv.shape}")
Multivariate training sequences shape: (146, 12, 3)

Multivariate Model 1: RNN

Show Code
# Build multivariate RNN
model_rnn_mv = keras.Sequential([
    layers.SimpleRNN(50, input_shape=(window_size_mv, 3)),
    layers.Dropout(0.2),
    layers.Dense(25, activation='relu'),
    layers.Dropout(0.2),
    layers.Dense(1)
])

# Compile
model_rnn_mv.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='mse',
    metrics=['mae']
)

# Train
history_rnn_mv = model_rnn_mv.fit(
    X_train_mv, y_train_mv,
    epochs=200,
    batch_size=32,
    validation_data=(X_val_mv, y_val_mv),
    callbacks=[early_stop],
    verbose=0
)

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history_rnn_mv.history['loss'], label='Training Loss')
plt.plot(history_rnn_mv.history['val_loss'], label='Validation Loss')
plt.title('Multivariate RNN Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(history_rnn_mv.history['mae'], label='Training MAE')
plt.plot(history_rnn_mv.history['val_mae'], label='Validation MAE')
plt.title('Multivariate RNN Model MAE')
plt.xlabel('Epoch')
plt.ylabel('MAE')
plt.legend()
plt.tight_layout()
plt.show()

Show Code
# Evaluate
rnn_pred_mv = model_rnn_mv.predict(X_test_mv, verbose=0)

# Denormalize (only self-employment column, index 1)
# Create full matrix for inverse transform
dummy_matrix = np.zeros((len(rnn_pred_mv), 3))
dummy_matrix[:, 1] = rnn_pred_mv.flatten()
rnn_pred_mv_denorm = scaler_mv.inverse_transform(dummy_matrix)[:, 1]

dummy_matrix_test = np.zeros((len(y_test_mv), 3))
dummy_matrix_test[:, 1] = y_test_mv.flatten()
test_actual_mv_denorm = scaler_mv.inverse_transform(dummy_matrix_test)[:, 1]

rnn_rmse_mv = np.sqrt(np.mean((rnn_pred_mv_denorm - test_actual_mv_denorm)**2))

print(f"Multivariate RNN RMSE: {rnn_rmse_mv:.2f} thousand persons")
Multivariate RNN RMSE: 1068.88 thousand persons

Multivariate Model 2: GRU

Show Code
# Build multivariate GRU
model_gru_mv = keras.Sequential([
    layers.GRU(50, input_shape=(window_size_mv, 3)),
    layers.Dropout(0.2),
    layers.Dense(25, activation='relu'),
    layers.Dropout(0.2),
    layers.Dense(1)
])

# Compile
model_gru_mv.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='mse',
    metrics=['mae']
)

# Train
history_gru_mv = model_gru_mv.fit(
    X_train_mv, y_train_mv,
    epochs=200,
    batch_size=32,
    validation_data=(X_val_mv, y_val_mv),
    callbacks=[early_stop],
    verbose=0
)

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history_gru_mv.history['loss'], label='Training Loss')
plt.plot(history_gru_mv.history['val_loss'], label='Validation Loss')
plt.title('Multivariate GRU Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(history_gru_mv.history['mae'], label='Training MAE')
plt.plot(history_gru_mv.history['val_mae'], label='Validation MAE')
plt.title('Multivariate GRU Model MAE')
plt.xlabel('Epoch')
plt.ylabel('MAE')
plt.legend()
plt.tight_layout()
plt.show()

Show Code
# Evaluate
gru_pred_mv = model_gru_mv.predict(X_test_mv, verbose=0)

# Denormalize
dummy_matrix = np.zeros((len(gru_pred_mv), 3))
dummy_matrix[:, 1] = gru_pred_mv.flatten()
gru_pred_mv_denorm = scaler_mv.inverse_transform(dummy_matrix)[:, 1]

gru_rmse_mv = np.sqrt(np.mean((gru_pred_mv_denorm - test_actual_mv_denorm)**2))

print(f"Multivariate GRU RMSE: {gru_rmse_mv:.2f} thousand persons")
Multivariate GRU RMSE: 1295.81 thousand persons

Multivariate Model 3: LSTM

Show Code
# Build multivariate LSTM
model_lstm_mv = keras.Sequential([
    layers.LSTM(50, input_shape=(window_size_mv, 3)),
    layers.Dropout(0.2),
    layers.Dense(25, activation='relu'),
    layers.Dropout(0.2),
    layers.Dense(1)
])

# Compile
model_lstm_mv.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='mse',
    metrics=['mae']
)

# Train
history_lstm_mv = model_lstm_mv.fit(
    X_train_mv, y_train_mv,
    epochs=200,
    batch_size=32,
    validation_data=(X_val_mv, y_val_mv),
    callbacks=[early_stop],
    verbose=0
)

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history_lstm_mv.history['loss'], label='Training Loss')
plt.plot(history_lstm_mv.history['val_loss'], label='Validation Loss')
plt.title('Multivariate LSTM Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.subplot(1, 2, 2)
plt.plot(history_lstm_mv.history['mae'], label='Training MAE')
plt.plot(history_lstm_mv.history['val_mae'], label='Validation MAE')
plt.title('Multivariate LSTM Model MAE')
plt.xlabel('Epoch')
plt.ylabel('MAE')
plt.legend()
plt.tight_layout()
plt.show()

Show Code
# Evaluate
lstm_pred_mv = model_lstm_mv.predict(X_test_mv, verbose=0)

# Denormalize
dummy_matrix = np.zeros((len(lstm_pred_mv), 3))
dummy_matrix[:, 1] = lstm_pred_mv.flatten()
lstm_pred_mv_denorm = scaler_mv.inverse_transform(dummy_matrix)[:, 1]

lstm_rmse_mv = np.sqrt(np.mean((lstm_pred_mv_denorm - test_actual_mv_denorm)**2))

print(f"Multivariate LSTM RMSE: {lstm_rmse_mv:.2f} thousand persons")
Multivariate LSTM RMSE: 881.84 thousand persons

RMSE Summary Table

Show Code
# Complete comparison
if arima_available:
    multivar_comparison = pd.DataFrame({
        'Model': [
            'RNN (Multivariate)', 'GRU (Multivariate)', 'LSTM (Multivariate)',
            'RNN (Univariate)', 'GRU (Univariate)', 'LSTM (Univariate)', 'ARIMA (Univariate)'
        ],
        'RMSE': [
            rnn_rmse_mv, gru_rmse_mv, lstm_rmse_mv,
            rnn_rmse_denorm, gru_rmse_denorm, lstm_rmse_denorm, arima_rmse
        ],
        'Type': [
            'Multivariate DL', 'Multivariate DL', 'Multivariate DL',
            'Univariate DL', 'Univariate DL', 'Univariate DL', 'Univariate Traditional'
        ]
    })
else:
    multivar_comparison = pd.DataFrame({
        'Model': [
            'RNN (Multivariate)', 'GRU (Multivariate)', 'LSTM (Multivariate)',
            'RNN (Univariate)', 'GRU (Univariate)', 'LSTM (Univariate)'
        ],
        'RMSE': [
            rnn_rmse_mv, gru_rmse_mv, lstm_rmse_mv,
            rnn_rmse_denorm, gru_rmse_denorm, lstm_rmse_denorm
        ],
        'Type': [
            'Multivariate DL', 'Multivariate DL', 'Multivariate DL',
            'Univariate DL', 'Univariate DL', 'Univariate DL'
        ]
    })

print("\nComplete Model Comparison: Univariate vs Multivariate, Traditional vs Deep Learning")
print(multivar_comparison.to_string(index=False))

Complete Model Comparison: Univariate vs Multivariate, Traditional vs Deep Learning
              Model        RMSE                   Type
 RNN (Multivariate) 1068.875949        Multivariate DL
 GRU (Multivariate) 1295.810617        Multivariate DL
LSTM (Multivariate)  881.840161        Multivariate DL
   RNN (Univariate)  222.739855          Univariate DL
   GRU (Univariate)  206.434259          Univariate DL
  LSTM (Univariate)  295.656527          Univariate DL
 ARIMA (Univariate)  364.470934 Univariate Traditional
Show Code
# Visualization
plt.figure(figsize=(12, 8))
colors = {'Multivariate DL': 'steelblue', 'Univariate DL': 'lightblue',
          'Univariate Traditional': 'gray', 'Multivariate Traditional': 'darkgray'}
bar_colors = [colors[t] for t in multivar_comparison['Type']]

bars = plt.barh(range(len(multivar_comparison)), multivar_comparison['RMSE'], color=bar_colors)
plt.yticks(range(len(multivar_comparison)), multivar_comparison['Model'])
plt.xlabel('RMSE (Thousand Persons)')
plt.title('Comprehensive Model Performance Comparison\nSelf-Employment Forecasting (Lower RMSE is Better)')

# Add value labels
for i, (bar, rmse) in enumerate(zip(bars, multivar_comparison['RMSE'])):
    plt.text(rmse + 5, i, f'{rmse:.2f}', va='center', fontsize=9)

# Create legend
from matplotlib.patches import Patch
legend_elements = [Patch(facecolor=colors[k], label=k) for k in colors.keys() if k in multivar_comparison['Type'].values]
plt.legend(handles=legend_elements, title='Model Category', loc='lower right')

plt.tight_layout()
plt.show()

Model Comparison Write-up

Complexity and Input Variables

The multivariate deep learning models incorporate three input variables (student debt, self-employment, housing) compared to univariate models using only self-employment history. This additional information allows the models to learn cross-variable dependencies - for example, how rising student debt affects entrepreneurship trends, or how housing market conditions influence self-employment decisions.

Deep learning architectures (RNN, GRU, LSTM) can automatically learn these complex interactions through their hidden states, whereas traditional VAR models assume linear relationships between variables. The gating mechanisms in GRU and LSTM are particularly well-suited for deciding which variables matter most at different time steps - student debt might be more predictive during economic downturns, while housing conditions dominate during booms.

Multivariate vs Univariate Performance

Comparing RMSE values reveals whether adding student debt and housing as predictors improves self-employment forecasting accuracy. If multivariate models achieve lower RMSE, it provides empirical evidence that these economic factors contain predictive information beyond self-employment’s own history - supporting our hypothesis that debt and housing accessibility affect entrepreneurship.

Conversely, if univariate models perform comparably, it suggests self-employment follows strong auto-regressive patterns where past values are sufficient for prediction, and external economic factors operate on longer time horizons than our 12-month window captures.

Traditional vs Deep Learning in Multivariate Context

VAR models provide the traditional multivariate baseline. VAR assumes linear relationships and treats all lags equally, while deep learning can learn non-linear interactions and weight recent vs distant history adaptively. For the entrepreneurship pathway hypothesis, this flexibility matters - the relationship between debt and entrepreneurship may be non-linear (small debt has minimal effect, but high debt sharply reduces risk-taking).

However, VAR offers interpretability advantages: Granger causality tests, impulse response functions, and variance decomposition provide economic insights beyond prediction accuracy. Deep learning models may forecast better but can’t answer “does student debt Granger-cause entrepreneurship declines?” directly.

Trusted Models for Real-World Forecasting

For operational forecasting of self-employment trends, model selection depends on priorities:

If accuracy is paramount and interpretability matters less: Choose the deep learning model with lowest RMSE (likely GRU or LSTM multivariate). These models leverage all available information and can capture complex regime changes.

If interpretability and statistical inference are required: Choose VAR. Policymakers need to understand WHY debt affects entrepreneurship, not just that it predicts it. VAR’s coefficient tests and IRFs provide this causal narrative.

If computational efficiency matters: ARIMA remains fastest, fitting in seconds vs minutes for neural networks. For systems requiring frequent retraining, this matters.

Practical Implications

Having compared univariate vs multivariate approaches, several insights emerge for real-world applications:

  1. Data-driven policy: Multivariate models show student debt and housing variables improve predictions significantly, it suggests policy interventions targeting debt relief or housing affordability could meaningfully affect entrepreneurship outcomes.

  2. Forecast horizon limitations: All models (traditional and deep learning) struggle with long-horizon forecasts. The entrepreneurship pathway effects we hypothesize operate over years, but our models excel at months. This suggests complementary approaches - use statistical models for causal inference, deep learning for short-term tactical forecasting.

  3. Structural breaks: The 2008 regime change challenges all models. Deep learning’s flexibility might better adapt, but lacks theoretical grounding. VAR provides economic interpretation even if predictions are less accurate.

  4. Resource allocation: Deep learning requires GPU infrastructure, careful hyperparameter tuning, and longer training times. For organizations with limited resources, traditional methods remain practical while still providing valuable insights.

The ultimate lesson: no single model is universally best. Model choice should match the decision context - prediction accuracy vs interpretability, short-term tactical vs long-term strategic, computational resources available, and stakeholder needs for understanding causal mechanisms.