Using Q-Tables for Decision Making and Encapsulation in Q-Learning

Introduction

Welcome to the third and final lesson of our "Q-Learning Unleashed: Building Intelligent Agents" course! In our previous lessons, we've explored the fundamentals of Q-learning, including how agents learn values for state-action pairs and update their knowledge through interactions with an environment. So far, we've primarily focused on how Q-learning works and the mathematical foundation behind it. Today, we'll focus on mastering two more crucial aspects: How to encapsulate our Q-learning algorithm in a well-structured class; How to use a trained Q-table to make intelligent decisions . By the end of this lesson, we'll have a complete, reusable Q-learning agent that can navigate environments based on learned knowledge. Let's dive in!

Creating an Object-Oriented Q-Learning Agent

One of the best practices in Reinforcement Learning (and in Machine Learning in general) is to encapsulate our algorithms into classes. This approach provides several benefits: Organization : Keeps related data and functionality together. Reusability : Makes it easy to use the agent in different environments. Extensibility : Allows for straightforward modifications or enhancements. Let's create a QLearningAgent class that will house our Q-learning algorithm. This class will feature a__init__constructor method, a learn method that can be used to update the Q-table based on experience, and an act method that is used for decision-making. Starting with the constructor: Python from collections import defaultdict import numpy as np class QLearningAgent: def __init__(self, actions, alpha=0.1, gamma=0.99): # List of possible actions self.actions = actions # Learning rate: controls how quickly the agent updates its Q-values self.alpha = alpha # Discount factor: determines how much future rewards are valued self.gamma = gamma # Q-table: maps state-action pairs to estimated values self.Q = defaultdict(lambda: np.zeros(len(actions))) Using a defaultdict is particularly convenient here because we don't need to explicitly check if a state exists in our Q-table before accessing it. If we try to access a state that doesn't exist yet, it will automatically be created with all Q-values initialized to zero. This approach simplifies our code and allows us to focus on the core learning algorithm rather than data structure management.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

    def learn(self, state, action_idx, reward, next_state, done):
        """Update Q-table using Q-learning."""
        if done: 
            # Terminal states have Q-value of 0
            best_next_q = 0.0
        else:
            # Find the highest Q-value for any action from the next state
            best_next_q = np.max(self.Q[next_state])
        
        # Calculate the TD target: immediate reward + discounted future reward
        td_target = reward + self.gamma * best_next_q
        
        # Calculate the TD error: difference between target and current estimate
        td_error = td_target - self.Q[state][action_idx]
        
        # Update the Q-value using the learning rate
        self.Q[state][action_idx] += self.alpha * td_error

Python

    def act(self, state):
        """Choose best action based on learned Q-values."""
        # Find the action with the highest Q-value for this state
        action_idx = np.argmax(self.Q[state])
        # Return both the action and its index (for learning updates)
        return self.actions[action_idx], action_idx

Python

def print_policy(agent, size=7):
    """Print the policy extracted from the Q-table."""
    goal = size // 2
    
    # Display the policy with directional arrows
    print("Line-world policy:")
    for position in range(size):
        if position == goal:
            print(f"{'G':^3}", end="")  # Mark the goal position
        else:
            best_action, _ = agent.act(position)
            direction = "←" if best_action == -1 else "→"
            print(f"{direction:^3}", end="")
    print()
    
    # Display position numbers for reference
    for position in range(size):
        print(f"{position:^3}", end="")
    print("\n")
    
    # Display the actual Q-values
    print("Q-values:")
    for position in range(size):
        left_q = agent.Q[position][0]  # Q-value for moving left
        right_q = agent.Q[position][1]  # Q-value for moving right
        print(f"Position {position:2}: [left: {left_q:.2f}, right: {right_q:.2f}]")

Python

def main():
    # Define our environment parameters
    size = 10
    goal = size // 2  # Position 5
    actions = [-1, 1]  # -1=left, 1=right
    
    # Create our Q-learning agent
    agent = QLearningAgent(actions)
    
    # Initialize Q-values to demonstrate a learned policy
    for position in range(size):
        if position < goal:
            # For positions left of goal, going right (index 1) is better
            agent.Q[position] = np.array([0.0, 0.8])  # [left, right]
        elif position > goal:
            # For positions right of goal, going left (index 0) is better
            agent.Q[position] = np.array([0.8, 0.0])  # [left, right]
    
    # Display the policy
    print_policy(agent, size)
    # Output:
    #  →  →  →  →  →  G  ←  ←  ←  ← 
    #  0  1  2  3  4  5  6  7  8  9

Introduction

Creating an Object-Oriented Q-Learning Agent

Join the 1M+ learners on CodeSignal

Learning from Experience

Making Decisions with Q-Tables

Visualizing the Agent's Strategy

Implementing Policy Visualization

Conclusion and Next Steps