Integrating Agents with Environments in Reinforcement Learning

Introduction

Welcome to the first lesson of " Game On: Integrating RL Agents with Environments "! This is the third course in our " Playing Games with Reinforcement Learning " path, where we finally connect the pieces we've been building. In the previous courses, we developed a grid world environment and a Q-learning agent separately. Now comes the exciting part — bringing them together! This initial lesson serves as a crucial bridge, recapping and integrating the knowledge from our previous two courses: we'll connect the Grid World environment from Course 1 with the Q-learning agent developed in Course 2. Then, in the upcoming lessons we'll explore topics such as the exploration-exploitation tradeoff, plotting learning statistics, and visualizing policy and value functions. By the end of this lesson, you'll be equipped with the knowledge to construct a fully integrated reinforcement learning system. You'll be able to fully train your RL agent to progressively enhance its ability to navigate the grid world, efficiently reaching its objectives. Let's get started!

Recap: Agent-Environment Interaction Loop

To begin, let's quickly recap the fundamental interaction loop between RL agents and environments that we already encountered previously. This cycle is the engine that powers all RL systems: The agent observes the current state from the environment Based on this state, the agent selects an action The environment processes the action and returns: The next state the agent finds itself in. A reward signal indicating how good/bad the action was. A done flag showing if the episode has ended. Additional info that might be helpful (optional). The agent uses this experience tuple (state, action, reward, next_state, done) to learn and improve its policy. This cycle repeats until the episode ends (when done=True), at which point we reset the environment and start a new episode. Through many repetitions of this process, our agent gradually learns the optimal policy!

Designing the Training Function

Let's start by designing a training function to orchestrate the interaction between our agent and environment: Python def train_agent(env, agent, num_episodes=200, eval_interval=20): """ Train the agent for a specified number of episodes. Args: env: The environment instance our agent will interact with agent: The agent that will learn from experiences num_episodes: How many episodes to train for (default 200) eval_interval: How often to display progress metrics (default 20) """ # Initialize tracking variables for performance monitoring rewards_per_episode = [] steps_per_episode = [] success_rate = [] window_size = 10 # For calculating moving averages of recent performance This function is the command center of our learning system. It takes our environment and agent as inputs, along with parameters that control the training process. The tracking variables we've initialized will help us monitor how well our agent is learning over time. Notice how we're using a window_size of 10 episodes for calculating moving averages. This gives us a more stable view of the agent's progress by smoothing out the natural fluctuations that occur while training any RL agent.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

    for episode in range(num_episodes):
        # Reset environment at the start of each episode
        state = env.reset()
        done = False
        total_reward = 0
        
        while not done:  # This inner loop represents a single episode
            # Agent selects an action based on current state
            action = agent.act(state)
            
            # Environment processes the action
            next_state, reward, done, info = env.step(action)
            
            # Agent learns from the experience
            agent.learn(state, action, reward, next_state, done)
            
            # Update for next iteration
            state = next_state
            total_reward += reward

    for episode in range(num_episodes):
        # Reset environment at the start of each episode
        state = env.reset()
        done = False
        total_reward = 0
        
        while not done:  # This inner loop represents a single episode
            # Agent selects an action based on current state
            action = agent.act(state)
            
            # Environment processes the action
            next_state, reward, done, info = env.step(action)
            
            # Agent learns from the experience
            agent.learn(state, action, reward, next_state, done)
            
            # Update for next iteration
            state = next_state
            total_reward += reward

Python

        # Record this episode's outcomes
        success = not info.get("timeout", False)  # Was the goal reached?
        success_rate.append(float(success))
        rewards_per_episode.append(total_reward)
        steps_per_episode.append(env.steps_count)
        
        # Display progress at regular intervals
        if (episode + 1) % eval_interval == 0:
            # Calculate recent performance metrics
            avg_reward = np.mean(rewards_per_episode[-window_size:])
            avg_steps = np.mean(steps_per_episode[-window_size:])
            avg_success = np.mean(success_rate[-window_size:]) * 100
            
            print(f"Episode {episode+1}/{num_episodes} - "
                  f"Avg Reward: {avg_reward:.2f}, "
                  f"Avg Steps: {avg_steps:.2f}, "
                  f"Success Rate: {avg_success:.1f}%")

        # Record this episode's outcomes
        success = not info.get("timeout", False)  # Was the goal reached?
        success_rate.append(float(success))
        rewards_per_episode.append(total_reward)
        steps_per_episode.append(env.steps_count)
        
        # Display progress at regular intervals
        if (episode + 1) % eval_interval == 0:
            # Calculate recent performance metrics
            avg_reward = np.mean(rewards_per_episode[-window_size:])
            avg_steps = np.mean(steps_per_episode[-window_size:])
            avg_success = np.mean(success_rate[-window_size:]) * 100
            
            print(f"Episode {episode+1}/{num_episodes} - "
                  f"Avg Reward: {avg_reward:.2f}, "
                  f"Avg Steps: {avg_steps:.2f}, "
                  f"Success Rate: {avg_success:.1f}%")

Python

    # Package all training statistics for analysis
    stats = {
        'rewards': rewards_per_episode,
        'steps': steps_per_episode,
        'success_rate': success_rate
    }
    
    return stats

    # Package all training statistics for analysis
    stats = {
        'rewards': rewards_per_episode,
        'steps': steps_per_episode,
        'success_rate': success_rate
    }
    
    return stats

Python

def main():
    """Train and evaluate a Q-learning agent in our grid world."""
    # Create our environment and agent
    env = GridWorldEnv(size=5)  # 5x5 grid world
    actions = [0, 1, 2, 3]      # Up, Down, Left, Right
    agent = QLearningAgent(actions)
    
    # Train the agent and collect performance stats
    print("Training agent...")
    train_stats = train_agent(env, agent, num_episodes=200)
    
    # Display final performance metrics
    print("\nTraining complete!")
    print("Final performance (last 10 episodes):")
    print(f"Average Reward: {np.mean(train_stats['rewards'][-10:]):.2f}")
    print(f"Average Steps: {np.mean(train_stats['steps'][-10:]):.2f}")
    print(f"Success Rate: {np.mean(train_stats['success_rate'][-10:]) * 100:.1f}%")

def main():
    """Train and evaluate a Q-learning agent in our grid world."""
    # Create our environment and agent
    env = GridWorldEnv(size=5)  # 5x5 grid world
    actions = [0, 1, 2, 3]      # Up, Down, Left, Right
    agent = QLearningAgent(actions)
    
    # Train the agent and collect performance stats
    print("Training agent...")
    train_stats = train_agent(env, agent, num_episodes=200)
    
    # Display final performance metrics
    print("\nTraining complete!")
    print("Final performance (last 10 episodes):")
    print(f"Average Reward: {np.mean(train_stats['rewards'][-10:]):.2f}")
    print(f"Average Steps: {np.mean(train_stats['steps'][-10:]):.2f}")
    print(f"Success Rate: {np.mean(train_stats['success_rate'][-10:]) * 100:.1f}%")

Python

    for episode in range(num_episodes):
        # Reset environment at the start of each episode
        state = env.reset()
        done = False
        total_reward = 0
        
        while not done:  # This inner loop represents a single episode
            # Agent selects an action based on current state
            action = agent.act(state)
            
            # Environment processes the action
            next_state, reward, done, info = env.step(action)
            
            # Agent learns from the experience
            agent.learn(state, action, reward, next_state, done)
            
            # Update for next iteration
            state = next_state
            total_reward += reward

Python

        # Record this episode's outcomes
        success = not info.get("timeout", False)  # Was the goal reached?
        success_rate.append(float(success))
        rewards_per_episode.append(total_reward)
        steps_per_episode.append(env.steps_count)
        
        # Display progress at regular intervals
        if (episode + 1) % eval_interval == 0:
            # Calculate recent performance metrics
            avg_reward = np.mean(rewards_per_episode[-window_size:])
            avg_steps = np.mean(steps_per_episode[-window_size:])
            avg_success = np.mean(success_rate[-window_size:]) * 100
            
            print(f"Episode {episode+1}/{num_episodes} - "
                  f"Avg Reward: {avg_reward:.2f}, "
                  f"Avg Steps: {avg_steps:.2f}, "
                  f"Success Rate: {avg_success:.1f}%")

Python

    # Package all training statistics for analysis
    stats = {
        'rewards': rewards_per_episode,
        'steps': steps_per_episode,
        'success_rate': success_rate
    }
    
    return stats

Python

def main():
    """Train and evaluate a Q-learning agent in our grid world."""
    # Create our environment and agent
    env = GridWorldEnv(size=5)  # 5x5 grid world
    actions = [0, 1, 2, 3]      # Up, Down, Left, Right
    agent = QLearningAgent(actions)
    
    # Train the agent and collect performance stats
    print("Training agent...")
    train_stats = train_agent(env, agent, num_episodes=200)
    
    # Display final performance metrics
    print("\nTraining complete!")
    print("Final performance (last 10 episodes):")
    print(f"Average Reward: {np.mean(train_stats['rewards'][-10:]):.2f}")
    print(f"Average Steps: {np.mean(train_stats['steps'][-10:]):.2f}")
    print(f"Success Rate: {np.mean(train_stats['success_rate'][-10:]) * 100:.1f}%")

Introduction

Recap: Agent-Environment Interaction Loop

Designing the Training Function

Join the 1M+ learners on CodeSignal

Implementing the Core Learning Loop

Monitoring Learning Progress

Finalizing the Training Function

Creating the Main Execution Flow

Conclusion and Next Steps