Balancing Exploration and Exploitation with Epsilon-Greedy Strategy

Introduction

Welcome back to the " Game On: Integrating RL Agents with Environments " course! In our previous lesson, we successfully connected our Q-learning agent with the grid world environment and implemented the core training loop. Now we're ready to explore one of the most fundamental challenges in reinforcement learning: the exploration-exploitation tradeoff . As we continue our journey, we'll discover how intelligent agents balance the need to explore their environment with the desire to exploit what they already know. This balance is crucial for effective learning and optimal performance. In this lesson, we'll implement the epsilon-greedy strategy — a simple yet powerful approach to managing this tradeoff — and see how different exploration rates affect our agent's ability to learn. By the end, you'll understand why proper exploration is essential and how to implement it in your own reinforcement learning agents.

Understanding the Exploration-Exploitation Tradeoff

Before diving into code, let's understand the core dilemma that every reinforcement learning agent faces: Should I explore or should I exploit? Exploitation means using the knowledge the agent already has to select what it believes is the best action. If an agent always exploits, it will consistently choose actions it thinks are optimal based on its current knowledge. Exploration means trying actions the agent is uncertain about to gather more information. This might involve selecting seemingly sub-optimal actions to see if they actually lead to better outcomes. This creates a fundamental tension: If an agent only exploits, it might get stuck in sub-optimal behavior patterns, never discovering better strategies that require initial exploration. In other words, always choosing the greedy action can cause the agent to become trapped in a local optimum, where it settles for a suboptimal reward instead of discovering potentially better rewards If an agent only explores, it wastes time trying random actions without leveraging what it has already learned. Imagine you're in a new city looking for a restaurant. You could either go to the first decent-looking place you find (exploitation) or spend time checking several options (exploration). Go to the first place you see every night, and you might miss better restaurants; spend all your time exploring, and you might never enjoy a good meal! In reinforcement learning, this tradeoff is not just important — it's essential for effective learning. The best agents dynamically balance exploration and exploitation, typically exploring more initially and gradually shifting towards exploitation as they gain confidence in their knowledge.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

def act(self, state, deterministic=False):
    """
    Choose an action using epsilon-greedy policy.
    
    With probability epsilon, choose a random action.
    Otherwise, choose the best action according to Q-values.
    """
    if not deterministic and self.rng.random() < self.epsilon:
        action_idx = self.rng.integers(0, len(self.actions))
    else:
        action_idx = np.argmax(self.Q[state])
    return self.actions[action_idx]

def act(self, state, deterministic=False):
    """
    Choose an action using epsilon-greedy policy.
    
    With probability epsilon, choose a random action.
    Otherwise, choose the best action according to Q-values.
    """
    if not deterministic and self.rng.random() < self.epsilon:
        action_idx = self.rng.integers(0, len(self.actions))
    else:
        action_idx = np.argmax(self.Q[state])
    return self.actions[action_idx]

Python

def main():
    """
    Compare different exploration strategies using the GridWorldEnv.
    We test two strategies by varying epsilon:
      - Greedy (ε=0.0)
      - Epsilon-greedy (ε=0.8)
    """
    # Initialize the environment.
    env = GridWorldEnv(size=5)
    actions = [0, 1, 2, 3]  # Actions: 0=up, 1=down, 2=left, 3=right
    n_episodes = 200
    epsilons = [0.0, 0.8]
    strategy_names = ["Greedy (ε=0.0)", "Epsilon-greedy (ε=0.8)"]

def main():
    """
    Compare different exploration strategies using the GridWorldEnv.
    We test two strategies by varying epsilon:
      - Greedy (ε=0.0)
      - Epsilon-greedy (ε=0.8)
    """
    # Initialize the environment.
    env = GridWorldEnv(size=5)
    actions = [0, 1, 2, 3]  # Actions: 0=up, 1=down, 2=left, 3=right
    n_episodes = 200
    epsilons = [0.0, 0.8]
    strategy_names = ["Greedy (ε=0.0)", "Epsilon-greedy (ε=0.8)"]

Python

    for epsilon, name in zip(epsilons, strategy_names):
        print(f"Strategy: {name}")
        # Initialize a new agent with the specified epsilon.
        agent = QLearningAgent(actions, epsilon=epsilon)

        for episode in range(n_episodes):
            reward, succeeded = run_episode(env, agent)

        rewards = []
        successes = []
        for episode in range(10):
            reward, succeeded = run_episode(env, agent, deterministic=True)
            rewards.append(reward)
            successes.append(1 if succeeded else 0)
        avg_reward = np.mean(rewards)
        success_rate = np.mean(successes)
        print(f"Average Reward: {avg_reward:.2f}, Success Rate: {success_rate:.2f}\n")

    for epsilon, name in zip(epsilons, strategy_names):
        print(f"Strategy: {name}")
        # Initialize a new agent with the specified epsilon.
        agent = QLearningAgent(actions, epsilon=epsilon)

        for episode in range(n_episodes):
            reward, succeeded = run_episode(env, agent)

        rewards = []
        successes = []
        for episode in range(10):
            reward, succeeded = run_episode(env, agent, deterministic=True)
            rewards.append(reward)
            successes.append(1 if succeeded else 0)
        avg_reward = np.mean(rewards)
        success_rate = np.mean(successes)
        print(f"Average Reward: {avg_reward:.2f}, Success Rate: {success_rate:.2f}\n")

text

Strategy: Greedy (ε=0.0)
Average Reward: 0.00, Success Rate: 0.00

Strategy: Epsilon-greedy (ε=0.8)
Average Reward: 1.00, Success Rate: 1.00

Strategy: Greedy (ε=0.0)
Average Reward: 0.00, Success Rate: 0.00

Strategy: Epsilon-greedy (ε=0.8)
Average Reward: 1.00, Success Rate: 1.00

Python

def act(self, state, deterministic=False):
    """
    Choose an action using epsilon-greedy policy.
    
    With probability epsilon, choose a random action.
    Otherwise, choose the best action according to Q-values.
    """
    if not deterministic and self.rng.random() < self.epsilon:
        action_idx = self.rng.integers(0, len(self.actions))
    else:
        action_idx = np.argmax(self.Q[state])
    return self.actions[action_idx]

Python

def main():
    """
    Compare different exploration strategies using the GridWorldEnv.
    We test two strategies by varying epsilon:
      - Greedy (ε=0.0)
      - Epsilon-greedy (ε=0.8)
    """
    # Initialize the environment.
    env = GridWorldEnv(size=5)
    actions = [0, 1, 2, 3]  # Actions: 0=up, 1=down, 2=left, 3=right
    n_episodes = 200
    epsilons = [0.0, 0.8]
    strategy_names = ["Greedy (ε=0.0)", "Epsilon-greedy (ε=0.8)"]

Python

    for epsilon, name in zip(epsilons, strategy_names):
        print(f"Strategy: {name}")
        # Initialize a new agent with the specified epsilon.
        agent = QLearningAgent(actions, epsilon=epsilon)

        for episode in range(n_episodes):
            reward, succeeded = run_episode(env, agent)

        rewards = []
        successes = []
        for episode in range(10):
            reward, succeeded = run_episode(env, agent, deterministic=True)
            rewards.append(reward)
            successes.append(1 if succeeded else 0)
        avg_reward = np.mean(rewards)
        success_rate = np.mean(successes)
        print(f"Average Reward: {avg_reward:.2f}, Success Rate: {success_rate:.2f}\n")

text

Strategy: Greedy (ε=0.0)
Average Reward: 0.00, Success Rate: 0.00

Strategy: Epsilon-greedy (ε=0.8)
Average Reward: 1.00, Success Rate: 1.00

Introduction

Understanding the Exploration-Exploitation Tradeoff

Join the 1M+ learners on CodeSignal

The Epsilon-Greedy Strategy

Implementing Epsilon-Greedy in the Agent

Setting Up the Experiment

Running the Experiment and Episode Function

Analyzing the Results

Conclusion and Next Steps