Integrating Agents with Environments in Reinforcement Learning

Introduction

Welcome to the first lesson of " Game On: Integrating RL Agents with Environments "! This is the third course in our " Playing Games with Reinforcement Learning " path, where we finally connect the pieces we've been building. In the previous courses, we developed a grid world environment and a Q-learning agent separately. Now comes the exciting part — bringing them together! This initial lesson serves as a crucial bridge, recapping and integrating the knowledge from our previous two courses: we'll connect the Grid World environment from Course 1 with the Q-learning agent developed in Course 2. Then, in the upcoming lessons we'll explore topics such as the exploration-exploitation tradeoff, plotting learning statistics, and visualizing policy and value functions. By the end of this lesson, you'll be equipped with the knowledge to construct a fully integrated reinforcement learning system. You'll be able to fully train your RL agent to progressively enhance its ability to navigate the grid world, efficiently reaching its objectives. Let's get started!

Recap: Agent-Environment Interaction Loop

To begin, let's quickly recap the fundamental interaction loop between RL agents and environments that we already encountered previously. This cycle is the engine that powers all RL systems: The agent observes the current state from the environment Based on this state, the agent selects an action The environment processes the action and returns: The next state the agent finds itself in. A reward signal indicating how good/bad the action was. A done flag showing if the episode has ended. Additional info that might be helpful (optional). The agent uses this experience tuple (state, action, reward, next_state, done) to learn and improve its policy. This cycle repeats until the episode ends (when done=True), at which point we reset the environment and start a new episode. Through many repetitions of this process, our agent gradually learns the optimal policy!

Designing the Training Function

Let's start by designing a training function to orchestrate the interaction between our agent and environment: Python def train_agent(env, agent, num_episodes=200, eval_interval=20): """ Train the agent for a specified number of episodes. Args: env: The environment instance our agent will interact with agent: The agent that will learn from experiences num_episodes: How many episodes to train for (default 200) eval_interval: How often to display progress metrics (default 20) """ # Initialize tracking variables for performance monitoring rewards_per_episode = [] steps_per_episode = [] success_rate = [] window_size = 10 # For calculating moving averages of recent performance This function is the command center of our learning system. It takes our environment and agent as inputs, along with parameters that control the training process. The tracking variables we've initialized will help us monitor how well our agent is learning over time. Notice how we're using a window_size of 10 episodes for calculating moving averages. This gives us a more stable view of the agent's progress by smoothing out the natural fluctuations that occur while training any RL agent.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing the Core Learning Loop

Now let's implement the heart of our training function — the loop that drives the learning process:

Python

    for episode in range(num_episodes):
        # Reset environment at the start of each episode
        state = env.reset()
        done = False
        total_reward = 0
        
        while not done:  # This inner loop represents a single episode
            # Agent selects an action based on current state
            action = agent.act(state)
            
            # Environment processes the action
            next_state, reward, done, info = env.step(action)
            
            # Agent learns from the experience
            agent.learn(state, action, reward, next_state, done)
            
            # Update for next iteration
            state = next_state
            total_reward += reward

    for episode in range(num_episodes):
        # Reset environment at the start of each episode
        state = env.reset()
        done = False
        total_reward = 0
        
        while not done:  # This inner loop represents a single episode
            # Agent selects an action based on current state
            action = agent.act(state)
            
            # Environment processes the action
            next_state, reward, done, info = env.step(action)
            
            # Agent learns from the experience
            agent.learn(state, action, reward, next_state, done)
            
            # Update for next iteration
            state = next_state
            total_reward += reward

This code implements the agent-environment interaction loop we discussed earlier. For each episode, we:

Reset the environment to get a fresh starting state.
Run the episode until completion (when done becomes True).
In each step, the agent chooses an action, the environment responds, and the agent learns.
We keep track of the accumulated reward throughout the episode.

This elegant loop is where the magic of learning happens — with each iteration, our agent is gathering experiences and refining its understanding of how to navigate the environment effectively.

Monitoring Learning Progress

To understand if our agent is actually improving, we need to track and visualize its performance:

Python

        # Record this episode's outcomes
        success = not info.get("timeout", False)  # Was the goal reached?
        success_rate.append(float(success))
        rewards_per_episode.append(total_reward)
        steps_per_episode.append(env.steps_count)
        
        # Display progress at regular intervals
        if (episode + 1) % eval_interval == 0:
            # Calculate recent performance metrics
            avg_reward = np.mean(rewards_per_episode[-window_size:])
            avg_steps = np.mean(steps_per_episode[-window_size:])
            avg_success = np.mean(success_rate[-window_size:]) * 100
            
            print(f"Episode {episode+1}/{num_episodes} - "
                  f"Avg Reward: {avg_reward:.2f}, "
                  f"Avg Steps: {avg_steps:.2f}, "
                  f"Success Rate: {avg_success:.1f}%")

        # Record this episode's outcomes
        success = not info.get("timeout", False)  # Was the goal reached?
        success_rate.append(float(success))
        rewards_per_episode.append(total_reward)
        steps_per_episode.append(env.steps_count)
        
        # Display progress at regular intervals
        if (episode + 1) % eval_interval == 0:
            # Calculate recent performance metrics
            avg_reward = np.mean(rewards_per_episode[-window_size:])
            avg_steps = np.mean(steps_per_episode[-window_size:])
            avg_success = np.mean(success_rate[-window_size:]) * 100
            
            print(f"Episode {episode+1}/{num_episodes} - "
                  f"Avg Reward: {avg_reward:.2f}, "
                  f"Avg Steps: {avg_steps:.2f}, "
                  f"Success Rate: {avg_success:.1f}%")

After each episode, we record:

Whether the agent successfully reached the goal or timed out. The success variable is determined by checking if the episode ended without a timeout, indicating that the agent reached the goal within the allowed steps.
The total reward accumulated during the episode.
How many steps the episode took.

Then, at regular intervals, we calculate and display moving averages of these metrics. This gives us valuable insights into how our agent's performance is evolving over time. You'll likely see these metrics improve as training progresses — the reward increasing, steps decreasing, and success rate climbing!

Introduction

Recap: Agent-Environment Interaction Loop

Designing the Training Function

Join the 1M+ learners on CodeSignal

Implementing the Core Learning Loop

Monitoring Learning Progress

Finalizing the Training Function

Creating the Main Execution Flow

Conclusion and Next Steps