Introduction to Time Series Forecasting with GRUs using PyTorch
Introduction to Time Series Forecasting with GRUs
Welcome to the first lesson of the course on Time Series Forecasting with GRUs. In this lesson, we will explore the concept of time series forecasting, which involves predicting future values based on previously observed data points. This is particularly useful in fields like finance, weather prediction, and air quality monitoring. We will focus on using Gated Recurrent Units (GRUs), a type of recurrent neural network, to handle univariate time series data. GRUs are well-suited for this task because they can capture temporal dependencies in data efficiently.
To make our learning concrete, we will use the Airline Passenger Traffic dataset. This dataset contains monthly totals of international airline passengers from 1949 to 1960, and we will use it to predict future passenger numbers. By the end of this lesson, you will understand how to preprocess data, build a GRU model, and train it for forecasting.
Data Processing Recap
Before we dive into building our GRU model, let's quickly recap the data processing steps. We will use the Airline Passenger Traffic dataset, which is a well-known example of time series data. This dataset is valuable for illustrating trends, seasonality, and other time series characteristics. We have stored the dataset in a CSV file named data.csv, which includes a 'Month' column representing the time intervals and a 'Passengers' column representing the number of passengers.
You can load and preprocess the dataset using the following code:
Python
import numpy as npimport pandas as pdfrom sklearn.preprocessing import MinMaxScalerdef preprocess_data(filepath, seq_length=10): """Preprocess time series data for RNN training.""" data = pd.read_csv(filepath) data['Month'] = pd.to_datetime(data['Month']) data.set_index('Month', inplace=True) scaler = MinMaxScaler() normalized_data = scaler.fit_transform(data) X, y = [], [] for i in range(len(normalized_data) - seq_length): X.append(normalized_data[i:i+seq_length]) y.append(normalized_data[i+seq_length]) X, y = np.array(X), np.array(y) # Reshape X to ensure it has the correct shape for LSTM input X = X.reshape((X.shape[0], X.shape[1], 1)) return X, y, scalerseq_length = 10X, y, scaler = preprocess_data('data.csv', seq_length)
In this code, we load the dataset, convert the 'Month' column to a datetime format, and set it as the index. We then normalize the data using MinMaxScaler, which scales the data to a range between 0 and 1. This preprocessing is crucial for ensuring that our model can learn effectively from the data.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Understanding the GRU Structure
Before we build our GRU model, it’s important to understand how a Gated Recurrent Unit (GRU) works. GRUs are a type of recurrent neural network (RNN) designed to handle sequential data — such as time series or natural language — by learning patterns over time.
What makes GRUs effective is their ability to control how information flows through the network using two gates: the update gate and the reset gate. These gates allow the model to selectively keep or discard information from the past, helping it focus on what matters most in each step of the sequence.
GRUs are often preferred over LSTMs for their simpler architecture, which leads to faster training while still delivering strong performance.
Here is a diagram of a GRU cell. In the next section, we’ll explore the different parts of this diagram to understand how the GRU processes information.
Inside the GRU Cell
Why Use Tanh?
Building a GRU Model for Univariate Forecasting
Now that we have our data and understand how GRUs work, let's build a GRU model using PyTorch. GRUs are a type of recurrent neural network that can capture temporal dependencies in sequential data. Here's how you can define a GRU model:
import torchimport torch.nn as nnclass GRUModel(nn.Module): def __init__(self, input_size, hidden_size, output_size): super(GRUModel, self).__init__() self.gru1 = nn.GRU(input_size, hidden_size, batch_first=True) self.gru2 = nn.GRU(hidden_size, hidden_size // 2, batch_first=True) self.fc = nn.Linear(hidden_size // 2, output_size) def forward(self, x): x, _ = self.gru1(x) x, _ = self.gru2(x) x = x[:, -1, :] # Take the last time step x = self.fc(x) return x# Define model parametersinput_size = 1 # Univariate inputhidden_size = 32output_size = 1# Instantiate the modelmodel = GRUModel(input_size, hidden_size, output_size)
In this model, we use two GRU layers. The first GRU layer has 32 units and returns sequences, which means it outputs a sequence of data points for each input sequence. The second GRU layer has 16 units and does not return sequences, as it outputs a single data point. Finally, we use a Linear layer with one unit to predict the number of passengers. Our target is the number of passengers, which we aim to forecast.
Training the GRU Model
With our model defined, we can now compile and train it. Compiling the model involves specifying the optimizer and loss function. We will use the Adam optimizer and mean squared error (MSE) as the loss function. Here's how you can train the model:
import torch.optim as optim# Define loss function and optimizercriterion = nn.MSELoss()optimizer = optim.Adam(model.parameters(), lr=0.001)# Convert data to PyTorch tensorsX_tensor = torch.tensor(X, dtype=torch.float32)y_tensor = torch.tensor(y, dtype=torch.float32)# Training loopepochs = 10batch_size = 16for epoch in range(epochs): model.train() for i in range(0, len(X_tensor), batch_size): X_batch = X_tensor[i:i+batch_size] y_batch = y_tensor[i:i+batch_size] # Forward pass outputs = model(X_batch) loss = criterion(outputs.squeeze(), y_batch.squeeze()) # Ensure both are of shape [batch_size] # Backward pass and optimization optimizer.zero_grad() loss.backward() optimizer.step() print(f'Epoch [{epoch+1}/{epochs}], Loss: {loss.item():.4f}')
In this code, we define the loss function and optimizer:
Loss Function: We use mean squared error (MSE) as the loss function, which measures how well the model's predictions match the actual data. It quantifies the difference between predicted and true values, guiding the optimizer in adjusting the model to improve accuracy. MSE is commonly used for regression tasks as it penalizes larger errors more heavily.
Optimizer: We use the Adam optimizer, which is an algorithm that adjusts the model's weights to minimize the loss function. It determines how the model learns from the data by updating weights during training. Adam is popular due to its efficiency and ability to handle sparse gradients.
We then train the model using a training loop, specifying the number of epochs (the number of times the model will iterate over the entire dataset) and the batch size (the number of samples processed before the model is updated). Training the model allows it to learn patterns in the data and make accurate predictions.
Comparing RNN, LSTM, and GRU
To better understand the differences between basic RNNs, LSTMs, and GRUs, let's compare their key features in the table below:
Feature
Basic RNNs
LSTMs
GRUs
Architecture
Simple recurrent structure
Memory cells with input, output, and forget gates
Simplified version of LSTM with update and reset gates
Handling Long-Term Dependencies
Limited due to vanishing gradient problem
Effective with memory cells and gates
Effective with fewer gates and simpler structure
Complexity
Low
High due to multiple gates
Moderate, fewer gates than LSTM
Training Time
Fast
Slower due to complexity
Faster than LSTM, slower than basic RNN
Performance
Limited for long sequences
High for capturing long-term dependencies
Comparable to LSTM, often preferred for efficiency
Use Cases
Short sequences, simple tasks
Tasks requiring long-term context
Efficient for a wide range of tasks, especially with limited resources
In summary, while basic RNNs are suitable for simple tasks with short sequences, LSTMs and GRUs are better equipped to handle long-term dependencies. GRUs offer a good balance between complexity and performance, making them a popular choice for many time series forecasting applications.
Summary and Next Steps
In this lesson, we introduced the concept of time series forecasting with GRUs and walked through the process of building and training a GRU model for univariate forecasting. We covered data preprocessing, model construction, and training, providing you with a solid foundation for using GRUs in time series forecasting.
As you move on to the practice exercises, you'll have the opportunity to apply what you've learned and experiment with different configurations. In future lessons, we will explore how to evaluate the performance of our GRU model and delve into advanced GRU techniques to enhance forecasting accuracy. Keep up the great work, and let's continue to build on this foundation!
ht−1: the hidden state from the previous time step (which carries memory)
Update Gate
The update gate determines how much of the previous hidden state to retain. It is computed as:
zt=σ(Wzxt+Uzht−1)
If zt is close to 1, the model keeps most of the past. If it’s close to 0, it focuses more on the current input.
Reset Gate
The reset gate controls how much of the past to forget when computing new information:
rt=σ(Wrxt+Urht−1)
A low value of rt allows the model to “reset” and ignore the past, which is useful for capturing short-term patterns.
Candidate Hidden State
Using the reset gate, the model calculates a candidate hidden state:
h~t=tanh(Wxt+U(rt⊙ht−1))
Here, ⊙ denotes element-wise multiplication (not a dot product). This candidate state mixes new input with a gated version of past memory.
Final Hidden State
The final output of the GRU cell is a blend of the previous hidden state and the candidate:
ht=zt⊙ht−1+(1−zt)⊙h~t
This allows the GRU to choose between keeping the old information or updating it with new content.
The GRU uses the tanh activation function to create the candidate hidden state. The tanh function outputs values in the range between -1 and 1:
tanh(x)=ex+e−xex−e−x
This helps the GRU model:
Center outputs around zero, making learning more stable
Capture both positive and negative signals
Avoid saturation issues that occur in activations like sigmoid (which only outputs 0 to 1)
The shape of the tanh curve is S-like (similar to sigmoid), but centered at 0. This allows the GRU to generate a more expressive intermediate representation.
Python
import torchimport torch.nn as nnclass GRUModel(nn.Module): def __init__(self, input_size, hidden_size, output_size): super(GRUModel, self).__init__() self.gru1 = nn.GRU(input_size, hidden_size, batch_first=True) self.gru2 = nn.GRU(hidden_size, hidden_size // 2, batch_first=True) self.fc = nn.Linear(hidden_size // 2, output_size) def forward(self, x): x, _ = self.gru1(x) x, _ = self.gru2(x) x = x[:, -1, :] # Take the last time step x = self.fc(x) return x# Define model parametersinput_size = 1 # Univariate inputhidden_size = 32output_size = 1# Instantiate the modelmodel = GRUModel(input_size, hidden_size, output_size)
Python
import torch.optim as optim# Define loss function and optimizercriterion = nn.MSELoss()optimizer = optim.Adam(model.parameters(), lr=0.001)# Convert data to PyTorch tensorsX_tensor = torch.tensor(X, dtype=torch.float32)y_tensor = torch.tensor(y, dtype=torch.float32)# Training loopepochs = 10batch_size = 16for epoch in range(epochs): model.train() for i in range(0, len(X_tensor), batch_size): X_batch = X_tensor[i:i+batch_size] y_batch = y_tensor[i:i+batch_size] # Forward pass outputs = model(X_batch) loss = criterion(outputs.squeeze(), y_batch.squeeze()) # Ensure both are of shape [batch_size] # Backward pass and optimization optimizer.zero_grad() loss.backward() optimizer.step() print(f'Epoch [{epoch+1}/{epochs}], Loss: {loss.item():.4f}')