Welcome to the first lesson of "Game On: Integrating RL Agents with Environments"! This is the third course in our "Playing Games with Reinforcement Learning" path, where we finally connect the pieces we've been building.

In the previous courses, we developed a grid world environment and a Q-learning agent separately. Now comes the exciting part — bringing them together! This initial lesson serves as a crucial bridge, recapping and integrating the knowledge from our previous two courses: we'll connect the Grid World environment from Course 1 with the Q-learning agent developed in Course 2. Then, in the upcoming lessons we'll explore topics such as the exploration-exploitation tradeoff, plotting learning statistics, and visualizing policy and value functions.

By the end of this lesson, you'll be equipped with the knowledge to construct a fully integrated reinforcement learning system. You'll be able to fully train your RL agent to progressively enhance its ability to navigate the grid world, efficiently reaching its objectives. Let's get started!

To begin, let's quickly recap the fundamental interaction loop between RL agents and environments that we already encountered previously. This cycle is the engine that powers all RL systems:

1. The agent observes the current state from the environment
2. Based on this state, the agent selects an action
3. The environment processes the action and returns:
   • The next state the agent finds itself in.
   • A reward signal indicating how good/bad the action was.
   • A done flag showing if the episode has ended.
   • Additional info that might be helpful (optional).
4. The agent uses this experience tuple (state, action, reward, next_state, done) to learn and improve its policy.

This cycle repeats until the episode ends (when done=True), at which point we reset the environment and start a new episode. Through many repetitions of this process, our agent gradually learns the optimal policy!

Let's start by designing a training function to orchestrate the interaction between our agent and environment:

def train_agent(env, agent, num_episodes=200, eval_interval=20):
    """
    Train the agent for a specified number of episodes.
    
    Args:
        env: The environment instance our agent will interact with
        agent: The agent that will learn from experiences
        num_episodes: How many episodes to train for (default 200)
        eval_interval: How often to display progress metrics (default 20)
    """
    # Initialize tracking variables for performance monitoring
    rewards_per_episode = []
    steps_per_episode = []
    success_rate = []
    window_size = 10  # For calculating moving averages of recent performance

This function is the command center of our learning system. It takes our environment and agent as inputs, along with parameters that control the training process. The tracking variables we've initialized will help us monitor how well our agent is learning over time.

Notice how we're using a window_size of 10 episodes for calculating moving averages. This gives us a more stable view of the agent's progress by smoothing out the natural fluctuations that occur while training any RL agent.

Now let's implement the heart of our training function — the loop that drives the learning process:

    for episode in range(num_episodes):
        # Reset environment at the start of each episode
        state = env.reset()
        done = False
        total_reward = 0
        
        while not done:  # This inner loop represents a single episode
            # Agent selects an action based on current state
            action = agent.act(state)
            
            # Environment processes the action
            next_state, reward, done, info = env.step(action)
            
            # Agent learns from the experience
            agent.learn(state, action, reward, next_state, done)
            
            # Update for next iteration
            state = next_state
            total_reward += reward

This code implements the agent-environment interaction loop we discussed earlier. For each episode, we:

1. Reset the environment to get a fresh starting state.
2. Run the episode until completion (when done becomes True).
3. In each step, the agent chooses an action, the environment responds, and the agent learns.
4. We keep track of the accumulated reward throughout the episode.

This elegant loop is where the magic of learning happens — with each iteration, our agent is gathering experiences and refining its understanding of how to navigate the environment effectively.

To understand if our agent is actually improving, we need to track and visualize its performance:

        # Record this episode's outcomes
        success = not info.get("timeout", False)  # Was the goal reached?
        success_rate.append(float(success))
        rewards_per_episode.append(total_reward)
        steps_per_episode.append(env.steps_count)
        
        # Display progress at regular intervals
        if (episode + 1) % eval_interval == 0:
            # Calculate recent performance metrics
            avg_reward = np.mean(rewards_per_episode[-window_size:])
            avg_steps = np.mean(steps_per_episode[-window_size:])
            avg_success = np.mean(success_rate[-window_size:]) * 100
            
            print(f"Episode {episode+1}/{num_episodes} - "
                  f"Avg Reward: {avg_reward:.2f}, "
                  f"Avg Steps: {avg_steps:.2f}, "
                  f"Success Rate: {avg_success:.1f}%")

After each episode, we record:

• Whether the agent successfully reached the goal or timed out. The success variable is determined by checking if the episode ended without a timeout, indicating that the agent reached the goal within the allowed steps.
• The total reward accumulated during the episode.
• How many steps the episode took.

Then, at regular intervals, we calculate and display moving averages of these metrics. This gives us valuable insights into how our agent's performance is evolving over time. You'll likely see these metrics improve as training progresses — the reward increasing, steps decreasing, and success rate climbing!

Let's complete our training function by returning the collected statistics:

    # Package all training statistics for analysis
    stats = {
        'rewards': rewards_per_episode,
        'steps': steps_per_episode,
        'success_rate': success_rate
    }
    
    return stats

By returning these statistics, we enable further analysis or visualization after training completes. This is particularly useful if you want to plot learning curves or compare different training runs.

Finally, now that we have our training function, let's create the main function that ties everything together:

def main():
    """Train and evaluate a Q-learning agent in our grid world."""
    # Create our environment and agent
    env = GridWorldEnv(size=5)  # 5x5 grid world
    actions = [0, 1, 2, 3]      # Up, Down, Left, Right
    agent = QLearningAgent(actions)
    
    # Train the agent and collect performance stats
    print("Training agent...")
    train_stats = train_agent(env, agent, num_episodes=200)
    
    # Display final performance metrics
    print("\nTraining complete!")
    print("Final performance (last 10 episodes):")
    print(f"Average Reward: {np.mean(train_stats['rewards'][-10:]):.2f}")
    print(f"Average Steps: {np.mean(train_stats['steps'][-10:]):.2f}")
    print(f"Success Rate: {np.mean(train_stats['success_rate'][-10:]) * 100:.1f}%")

The main() function:

1. Creates a 5×5 grid world environment.
2. Defines the available actions (0=up, 1=down, 2=left, 3=right).
3. Initializes a Q-learning agent.
4. Trains the agent for 200 episodes.
5. Reports the final performance metrics.

Congratulations! You've successfully built the crucial integration code that connects your reinforcement learning system, enabling the learning process through experience collection and policy improvement. In this lesson, we explored the fundamental agent-environment interaction pattern, constructed a comprehensive training function, implemented progress tracking, and created a main execution flow to set up and run the complete system.

Your Q-learning agent is now equipped to navigate the grid world with increasing efficiency, gradually discovering the optimal path to reach the goal. As you run the training, you'll observe the performance metrics improve, showcasing the essence of reinforcement learning in action. Up next, you'll have the opportunity to apply what you've learned in a practice section, reinforcing your understanding and skills. Happy coding!