Revisiting Sequence Models: RNNs, LSTMs, and Their Limits

Introduction

Hello there! Welcome to the first lesson of our course " Sequence Models & The Dawn of Attention ". This initial lesson marks the beginning of an exciting journey that will take us from traditional sequence modeling approaches all the way to the cutting-edge Transformer architecture that powers today's most advanced AI systems. Before we dive into our exploration of sequence models, let's establish what we expect you to bring to this course path. We assume you have solid foundations in Python programming, basic machine learning concepts, and PyTorch fundamentals, including tensor operations and neural network construction. Familiarity with basic linear algebra and calculus will also serve you well as we work through mathematical concepts. Our learning path consists of four comprehensive courses designed to give you deep expertise in Transformer architectures. In this first Course, we'll explore why traditional RNNs and LSTMs struggle with long sequences and build attention mechanisms from scratch, mastering the query-key-value paradigm. Next, in Course 2 we'll systematically construct the complete Transformer architecture, creating reusable components such as multi-head attention modules, feed-forward networks, and positional encodings. By Course 3 we'll be able to combine everything into a complete Transformer model, implementing autoregressive training with teacher forcing and exploring different decoding and inference strategies. Finally, in the last Course we'll harness the powerful Hugging Face ecosystem to work with pre-trained models and fine-tune them for real-world applications. By the end of this path, you'll have gained a profound understanding of how Transformers work under the hood and the practical skills to apply them effectively. Today, in our first lesson, we begin by revisiting sequence models to understand their limitations and why the AI community needed something better. Let's get started!

The Evolution of Sequence Modeling

Sequential data surrounds us everywhere: the words in this sentence follow a specific order, stock prices change over time, and DNA sequences encode genetic information through their arrangement. Traditional feedforward neural networks process each input independently, but sequence modeling requires understanding how current elements relate to previous ones. The challenge lies in temporal dependencies : understanding that "the cat sat on the" should be followed by something like "mat" requires remembering context from earlier in the sequence. Early approaches like n-grams could only look back a fixed number of steps, severely limiting their ability to capture long-range relationships. This limitation sparked the development of recurrent architectures that could theoretically maintain information across arbitrary sequence lengths. However, as we'll discover today, even sophisticated recurrent models face fundamental challenges when sequences grow long. This limitation will motivate the development of attention mechanisms and, later, of the Transformer architecture.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Building Our LSTM Model

Our LSTM implementation follows PyTorch conventions while keeping the architecture simple enough to clearly observe performance patterns across different sequence lengths:

Python

import torch.nn as nn

class SimpleLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(SimpleLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
        self.linear = nn.Linear(hidden_size, output_size)
        
    def forward(self, x):
        # Initialize hidden state and cell state
        h0 = torch.zeros(1, x.size(0), self.hidden_size)
        c0 = torch.zeros(1, x.size(0), self.hidden_size)
        
        # Forward propagate LSTM
        out, (hidden, cell) = self.lstm(x, (h0, c0))
        
        # Decode the hidden state of the last time step
        out = self.linear(out[:, -1, :])
        return out, hidden

import torch.nn as nn

class SimpleLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(SimpleLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
        self.linear = nn.Linear(hidden_size, output_size)
        
    def forward(self, x):
        # Initialize hidden state and cell state
        h0 = torch.zeros(1, x.size(0), self.hidden_size)
        c0 = torch.zeros(1, x.size(0), self.hidden_size)
        
        # Forward propagate LSTM
        out, (hidden, cell) = self.lstm(x, (h0, c0))
        
        # Decode the hidden state of the last time step
        out = self.linear(out[:, -1, :])
        return out, hidden

Our model architecture consists of a single LSTM layer followed by a linear output layer. The batch_first=True parameter ensures our input tensors follow the (batch, sequence, features) convention. We initialize hidden and cell states to zeros for each forward pass, giving each sequence a fresh start.

The key decision here is using out[:, -1, :], that is, we only use the final time step's hidden state for prediction. This forces the LSTM to compress all relevant information from the entire sequence into this final hidden state, creating the exact challenge we want to study.

Systematic Performance Analysis

Now we can systematically test how LSTM performance degrades as sequences grow longer, revealing the fundamental limitations we'll address with attention mechanisms.

Python

def main():
    # Test different sequence lengths
    sequence_lengths = [5, 10, 20, 30, 50, 100]
    results = {}
    
    print("Testing LSTM performance on different sequence lengths...")
    
    for seq_len in sequence_lengths:
        # Generate data and create model
        sequences, targets = generate_sequences(seq_len)
        model = SimpleLSTM(input_size=1, hidden_size=16, output_size=1)
        
        # Train and evaluate
        final_loss = train_and_evaluate(model, sequences, targets)
        results[seq_len] = final_loss
        
        print(f"Sequence length {seq_len}: Final loss = {final_loss:.4f}")

def main():
    # Test different sequence lengths
    sequence_lengths = [5, 10, 20, 30, 50, 100]
    results = {}
    
    print("Testing LSTM performance on different sequence lengths...")
    
    for seq_len in sequence_lengths:
        # Generate data and create model
        sequences, targets = generate_sequences(seq_len)
        model = SimpleLSTM(input_size=1, hidden_size=16, output_size=1)
        
        # Train and evaluate
        final_loss = train_and_evaluate(model, sequences, targets)
        results[seq_len] = final_loss
        
        print(f"Sequence length {seq_len}: Final loss = {final_loss:.4f}")

Our experiment systematically tests sequence lengths from 5 to 100 elements. For each length, we create fresh data according to the current seq_len, as well as a new model to ensure a fair comparison. The results reveal a clear pattern:

text

Testing LSTM performance on different sequence lengths...
Sequence length 5: Final loss = 0.0854
Sequence length 10: Final loss = 0.0651
Sequence length 20: Final loss = 0.6047
Sequence length 30: Final loss = 0.7171
Sequence length 50: Final loss = 0.5680
Sequence length 100: Final loss = 0.9084

Testing LSTM performance on different sequence lengths...
Sequence length 5: Final loss = 0.0854
Sequence length 10: Final loss = 0.0651
Sequence length 20: Final loss = 0.6047
Sequence length 30: Final loss = 0.7171
Sequence length 50: Final loss = 0.5680
Sequence length 100: Final loss = 0.9084

The pattern is striking: performance remains strong for very short sequences but degrades substantially as sequences grow longer, with significant deterioration evident beyond length 10. We can also plot these values to make this trend even clearer: Simple plot showing LSTM Performance Degradation with increasing sequence length.

This performance degradation demonstrates the long-range dependency problem that motivated the development of attention mechanisms: despite LSTMs' theoretical ability to maintain long-term memory, practical limitations emerge when sequences grow beyond certain lengths.

Introduction

The Evolution of Sequence Modeling

Join the 1M+ learners on CodeSignal

Brief Recap: RNNs and LSTMs

Understanding Our Long-Range Dependency Task

Building Our LSTM Model

Training and Evaluation Framework

Systematic Performance Analysis

Conclusion and Next Steps