Training a Q-Learning Agent in a Line-World Environment

Introduction

Welcome back to our " Q-Learning Unleashed: Building Intelligent Agents " course! In the first lesson, we explored the fundamental concepts of Q-Learning, including the Bellman equation, Q-tables, and the Q-Learning update rule. Now, in this second lesson we're ready to take the next step: training a Q-learning agent through interaction with an environment. In this lesson, we'll build a complete Q-Learning agent that learns to navigate a simple environment through trial and error. We'll implement a training loop that allows our agent to explore its environment, collect experiences, and update its Q-table to discover optimal behavior. By the end of this lesson, you'll see how a reinforcement learning agent evolves from random actions to strategic decision-making as it learns from experience. Let's dive in and bring our Q-learning theory to life with practical implementation!

The Line-World Environment

Before we start training our agent, let's understand the environment it will interact with. While our previous course focused on a grid-world environment, for the sake of clarity in this course we'll consider an even simpler environment, called a line-world . In this one-dimensional world, the agent exists on a line with positions numbered from 0 to size-1 . There's a goal state in the middle of the line, and the agent can move left or right. The agent receives a reward of 1 when it reaches the goal and 0 otherwise. The agent starts on a random cell and each episode ends when the agent reaches the goal. This simplified environment allows us to focus on the core aspects of agent training rather than complex environment dynamics. Here's how we can visualize a line-world of size 7: text +-----+-----+-----+-----+-----+-----+-----+ | 0 | 1 | 2 | G | 4 | 5 | 6 | +-----+-----+-----+-----+-----+-----+-----+ Where [G] represents the goal state at position 3 (size // 2).

Implementing Environment: Reset

Let's implement two core functions for our line-world environment: reset() and step() . These functions will allow our agent to interact with the environment. Note we won't be using a class, like we did in Course 1, for simplicty. First, the reset() function initializes the environment at the beginning of each episode: Python def reset(size=7): """ Reset the line-world environment. Returns the initial state (position). """ # Start at a random position (0 to size-1), excluding goal state goal = size // 2 state = np.random.choice([x for x in range(size) if x != goal]) return state This function sets the goal position at the middle of the line (size // 2), randomly places the agent at any position except the goal, and returns the initial state (position) of the agent. This ensures our agent has to learn to navigate to reach the goal from different starting points.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

The Learning Process: Episode by Episode

With our setup complete, we can now implement the core training loop that iterates through episodes:

Python

    for episode in range(n_episodes):
        state = reset(size)
        initial_state = state  # Store initial state for reporting
        done = False
        steps = 0
        
        while not done:
            # Choose random action
            action_idx = np.random.randint(len(actions))
            action = actions[action_idx]
            
            # Take the action
            next_state, reward, done = step(state, action, size)
            
            # Update Q-value
            q_learning_update(Q, state, action_idx, reward, next_state, alpha, gamma, done)
            
            # Move to next state
            state = next_state
            steps += 1
        
        print(f"Episode {episode+1}: reached goal in {steps} steps, starting from position {initial_state}")

    for episode in range(n_episodes):
        state = reset(size)
        initial_state = state  # Store initial state for reporting
        done = False
        steps = 0
        
        while not done:
            # Choose random action
            action_idx = np.random.randint(len(actions))
            action = actions[action_idx]
            
            # Take the action
            next_state, reward, done = step(state, action, size)
            
            # Update Q-value
            q_learning_update(Q, state, action_idx, reward, next_state, alpha, gamma, done)
            
            # Move to next state
            state = next_state
            steps += 1
        
        print(f"Episode {episode+1}: reached goal in {steps} steps, starting from position {initial_state}")

For each episode, we reset the environment, choose an action, update Q-values based on rewards received, and continue until the agent reaches the goal. This exploration process allows the agent to discover which actions lead to better outcomes in each state. Initially, the agent may take many steps to reach the goal, but as it learns, it typically becomes more efficient. The loop represents the core of reinforcement learning: taking actions, receiving feedback, and updating knowledge to improve future decisions. Note we're still taking actions randomly here - this is only done for learning, and in the next lesson we'll discuss how to extract the best action in each state using the trained Q-table!

def q_learning_update(Q, state, action, reward, next_state, alpha, gamma, done): """Update Q-value using the Q-learning update rule""" if done: best_next_q = 0.0 else: best_next_q = np.max(Q[next_state]) Q[state][action] += alpha * (reward + gamma * best_next_q - Q[state][action])

Introduction

The Line-World Environment

Implementing Environment: Reset

Join the 1M+ learners on CodeSignal

Implementing Environment: Step

Q-Learning Update Implementation

Setting Up the Training Environment

The Learning Process: Episode by Episode

Examining the Learning Results

Conclusion and Next Steps