Implementing Random Goals in Grid World Environments

Introduction

Welcome to the first lesson of " Navigating RL Challenges: Strategies and Future Directions "! This is the fourth and final course in our " Playing Games with Reinforcement Learning " path, great job getting to this point! By now, you've built a solid foundation in Reinforcement Learning concepts through our previous courses, where we covered environment building, Q-learning fundamentals, and agent-environment integration. In this course, we'll take your skills to the next level by exploring more advanced techniques that make Reinforcement Learning agents more flexible, robust, and capable of handling complex scenarios. Today, we'll focus on extending our grid world environment to support random goal states — a seemingly simple change that introduces important new challenges and learning opportunities.

Random Goals: Why They Matter

Before diving into implementation details, let's understand why having random goals in our Grid World is an important advancement in our Reinforcement Learning journey. In our previous courses, the Grid World environment featured fixed goals — the agent always had to reach the same, predetermined location. While this provided a good starting point, real-world problems rarely have such predictable objectives . Consider these examples: A delivery robot that must navigate to different drop-off locations each trip; A game character that must find randomly spawning items; A stock trading agent that must adapt to changing market conditions. By introducing random goals, we're making our agent more versatile and preparing it for real-world scenarios where objectives change. This forces the agent to learn general navigation strategies rather than memorizing a single path, significantly increasing its adaptability and usefulness.

State vs. Observation: A Critical Distinction

When working with random goals, we need to address a fundamental concept in Reinforcement Learning: the distinction between state and observation . State : The complete internal representation of the environment's condition. Observation : What the agent actually perceives and uses for decision-making, Note that, while in some domains states and observations may coincide (think about a chess game), in general the states are not available to the agent: only the observations are. This distinction becomes critical with random goals. Consider this scenario: if our environment places a goal randomly but doesn't tell the agent where it is, then this becomes an impossible task! The agent would need to randomly explore until stumbling upon the goal by chance. For the problem to be solvable, the agent needs to know where the goal is. This means our observation must include not just the agent's position (as we did up to this point) but, crucially, also the goal's position . This additional information in the observation enables the agent to learn meaningful strategies to reach goals regardless of where they appear.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

import numpy as np

class GridWorldEnv:
    def __init__(self, size=5):
        """
        Environment with a randomly selected goal each episode.
        By default, the agent starts at (0,0).
        """
        self.size = size
        self.state = None
        self.goal_state = None
        self.action_space = 4
        self.max_steps = size * 3  # Prevent endless episodes
        self.steps_count = 0
        self.rng = np.random.default_rng(42)  # For reproducibility

import numpy as np

class GridWorldEnv:
    def __init__(self, size=5):
        """
        Environment with a randomly selected goal each episode.
        By default, the agent starts at (0,0).
        """
        self.size = size
        self.state = None
        self.goal_state = None
        self.action_space = 4
        self.max_steps = size * 3  # Prevent endless episodes
        self.steps_count = 0
        self.rng = np.random.default_rng(42)  # For reproducibility

Python

def _sample_random_goal(self):
    """
    Randomly sample a goal cell that's not (0, 0).
    Note: If the environment includes obstacles like walls or mines,
    ensure to exclude those positions from possible goal locations.
    """
    possible_positions = [(r, c) for r in range(self.size)
                          for c in range(self.size)
                          if (r, c) != (0, 0)]  # Exclude starting position
    return possible_positions[self.rng.integers(len(possible_positions))]

def _sample_random_goal(self):
    """
    Randomly sample a goal cell that's not (0, 0).
    Note: If the environment includes obstacles like walls or mines,
    ensure to exclude those positions from possible goal locations.
    """
    possible_positions = [(r, c) for r in range(self.size)
                          for c in range(self.size)
                          if (r, c) != (0, 0)]  # Exclude starting position
    return possible_positions[self.rng.integers(len(possible_positions))]

Python

def reset(self):
    """
    Reset the environment to an initial agent state (0,0)
    and randomly sample a new goal position.

    Returns:
        tuple: ((row, col), (goal_row, goal_col))
    """
    self.state = (0, 0)
    self.goal_state = self._sample_random_goal()
    self.steps_count = 0
    # Note: we must return the goal_state in the observation
    # to make the problem actually solvable
    return (self.state, self.goal_state)

def reset(self):
    """
    Reset the environment to an initial agent state (0,0)
    and randomly sample a new goal position.

    Returns:
        tuple: ((row, col), (goal_row, goal_col))
    """
    self.state = (0, 0)
    self.goal_state = self._sample_random_goal()
    self.steps_count = 0
    # Note: we must return the goal_state in the observation
    # to make the problem actually solvable
    return (self.state, self.goal_state)

Python

def step(self, action):
    """
    Take an action in the environment.

    Args:
        action (int): 0=up, 1=down, 2=left, 3=right

    Returns:
        tuple: (next_state, reward, done, info)
          next_state = ((row, col), (goal_row, goal_col))
          reward = 1.0 if goal reached, otherwise 0.0
          done = True if goal reached or max steps exceeded
          info = {"timeout": True if max steps exceeded}
    """
    self.steps_count += 1
    r, c = self.state

    # Update position based on action
    if action == 0:  # up
        r = max(r - 1, 0)
    elif action == 1:  # down
        r = min(r + 1, self.size - 1)
    elif action == 2:  # left
        c = max(c - 1, 0)
    elif action == 3:  # right
        c = min(c + 1, self.size - 1)

    self.state = (r, c)
    reward = 1.0 if self.state == self.goal_state else 0.0
    done = (self.state == self.goal_state) or (self.steps_count >= self.max_steps)
    info = {"timeout": self.steps_count >= self.max_steps}

    return (self.state, self.goal_state), reward, done, info

def step(self, action):
    """
    Take an action in the environment.

    Args:
        action (int): 0=up, 1=down, 2=left, 3=right

    Returns:
        tuple: (next_state, reward, done, info)
          next_state = ((row, col), (goal_row, goal_col))
          reward = 1.0 if goal reached, otherwise 0.0
          done = True if goal reached or max steps exceeded
          info = {"timeout": True if max steps exceeded}
    """
    self.steps_count += 1
    r, c = self.state

    # Update position based on action
    if action == 0:  # up
        r = max(r - 1, 0)
    elif action == 1:  # down
        r = min(r + 1, self.size - 1)
    elif action == 2:  # left
        c = max(c - 1, 0)
    elif action == 3:  # right
        c = min(c + 1, self.size - 1)

    self.state = (r, c)
    reward = 1.0 if self.state == self.goal_state else 0.0
    done = (self.state == self.goal_state) or (self.steps_count >= self.max_steps)
    info = {"timeout": self.steps_count >= self.max_steps}

    return (self.state, self.goal_state), reward, done, info

Python

def train_agent(env, agent, episodes=1000):
    """Train a Q-learning agent in our random goal environment"""
    rewards = []
    successes = []
    
    # Run one episode with visualization
    print("Testing with a visualized episode:")
    result = run_episode(env, agent, render=True)
    print(f"Initial episode: reward={result[0]}, success={result[1]}\n")
    
    # Train for multiple episodes
    print(f"Training for {episodes} episodes...")
    for ep in range(episodes):
        reward, success = run_episode(env, agent)
        rewards.append(reward)
        successes.append(int(success))
        
    # Report performance
    print(f"Success rate (last 100 episodes): {np.mean(successes[-100:]):.2f}")
    return rewards, successes

def train_agent(env, agent, episodes=1000):
    """Train a Q-learning agent in our random goal environment"""
    rewards = []
    successes = []
    
    # Run one episode with visualization
    print("Testing with a visualized episode:")
    result = run_episode(env, agent, render=True)
    print(f"Initial episode: reward={result[0]}, success={result[1]}\n")
    
    # Train for multiple episodes
    print(f"Training for {episodes} episodes...")
    for ep in range(episodes):
        reward, success = run_episode(env, agent)
        rewards.append(reward)
        successes.append(int(success))
        
    # Report performance
    print(f"Success rate (last 100 episodes): {np.mean(successes[-100:]):.2f}")
    return rewards, successes

Python

import numpy as np

class GridWorldEnv:
    def __init__(self, size=5):
        """
        Environment with a randomly selected goal each episode.
        By default, the agent starts at (0,0).
        """
        self.size = size
        self.state = None
        self.goal_state = None
        self.action_space = 4
        self.max_steps = size * 3  # Prevent endless episodes
        self.steps_count = 0
        self.rng = np.random.default_rng(42)  # For reproducibility

Python

def _sample_random_goal(self):
    """
    Randomly sample a goal cell that's not (0, 0).
    Note: If the environment includes obstacles like walls or mines,
    ensure to exclude those positions from possible goal locations.
    """
    possible_positions = [(r, c) for r in range(self.size)
                          for c in range(self.size)
                          if (r, c) != (0, 0)]  # Exclude starting position
    return possible_positions[self.rng.integers(len(possible_positions))]

Python

def reset(self):
    """
    Reset the environment to an initial agent state (0,0)
    and randomly sample a new goal position.

    Returns:
        tuple: ((row, col), (goal_row, goal_col))
    """
    self.state = (0, 0)
    self.goal_state = self._sample_random_goal()
    self.steps_count = 0
    # Note: we must return the goal_state in the observation
    # to make the problem actually solvable
    return (self.state, self.goal_state)

Python

def step(self, action):
    """
    Take an action in the environment.

    Args:
        action (int): 0=up, 1=down, 2=left, 3=right

    Returns:
        tuple: (next_state, reward, done, info)
          next_state = ((row, col), (goal_row, goal_col))
          reward = 1.0 if goal reached, otherwise 0.0
          done = True if goal reached or max steps exceeded
          info = {"timeout": True if max steps exceeded}
    """
    self.steps_count += 1
    r, c = self.state

    # Update position based on action
    if action == 0:  # up
        r = max(r - 1, 0)
    elif action == 1:  # down
        r = min(r + 1, self.size - 1)
    elif action == 2:  # left
        c = max(c - 1, 0)
    elif action == 3:  # right
        c = min(c + 1, self.size - 1)

    self.state = (r, c)
    reward = 1.0 if self.state == self.goal_state else 0.0
    done = (self.state == self.goal_state) or (self.steps_count >= self.max_steps)
    info = {"timeout": self.steps_count >= self.max_steps}

    return (self.state, self.goal_state), reward, done, info

Python

def train_agent(env, agent, episodes=1000):
    """Train a Q-learning agent in our random goal environment"""
    rewards = []
    successes = []
    
    # Run one episode with visualization
    print("Testing with a visualized episode:")
    result = run_episode(env, agent, render=True)
    print(f"Initial episode: reward={result[0]}, success={result[1]}\n")
    
    # Train for multiple episodes
    print(f"Training for {episodes} episodes...")
    for ep in range(episodes):
        reward, success = run_episode(env, agent)
        rewards.append(reward)
        successes.append(int(success))
        
    # Report performance
    print(f"Success rate (last 100 episodes): {np.mean(successes[-100:]):.2f}")
    return rewards, successes

Introduction

Random Goals: Why They Matter

State vs. Observation: A Critical Distinction

Join the 1M+ learners on CodeSignal

Designing the Random Goal Environment

Sampling Random Goals

Managing Information in Observations

Processing Actions and Rewards

Learning with Random Objectives

Conclusion and Next Steps