Implementing Random Goals in Grid World Environments

Introduction

Welcome to the first lesson of " Navigating RL Challenges: Strategies and Future Directions "! This is the fourth and final course in our " Playing Games with Reinforcement Learning " path, great job getting to this point! By now, you've built a solid foundation in Reinforcement Learning concepts through our previous courses, where we covered environment building, Q-learning fundamentals, and agent-environment integration. In this course, we'll take your skills to the next level by exploring more advanced techniques that make Reinforcement Learning agents more flexible, robust, and capable of handling complex scenarios. Today, we'll focus on extending our grid world environment to support random goal states — a seemingly simple change that introduces important new challenges and learning opportunities.

Random Goals: Why They Matter

Before diving into implementation details, let's understand why having random goals in our Grid World is an important advancement in our Reinforcement Learning journey. In our previous courses, the Grid World environment featured fixed goals — the agent always had to reach the same, predetermined location. While this provided a good starting point, real-world problems rarely have such predictable objectives . Consider these examples: A delivery robot that must navigate to different drop-off locations each trip; A game character that must find randomly spawning items; A stock trading agent that must adapt to changing market conditions. By introducing random goals, we're making our agent more versatile and preparing it for real-world scenarios where objectives change. This forces the agent to learn general navigation strategies rather than memorizing a single path, significantly increasing its adaptability and usefulness.

State vs. Observation: A Critical Distinction

When working with random goals, we need to address a fundamental concept in Reinforcement Learning: the distinction between state and observation . State : The complete internal representation of the environment's condition. Observation : What the agent actually perceives and uses for decision-making, Note that, while in some domains states and observations may coincide (think about a chess game), in general the states are not available to the agent: only the observations are. This distinction becomes critical with random goals. Consider this scenario: if our environment places a goal randomly but doesn't tell the agent where it is, then this becomes an impossible task! The agent would need to randomly explore until stumbling upon the goal by chance. For the problem to be solvable, the agent needs to know where the goal is. This means our observation must include not just the agent's position (as we did up to this point) but, crucially, also the goal's position . This additional information in the observation enables the agent to learn meaningful strategies to reach goals regardless of where they appear.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Let's kick off the implementation by quickly recappping the __init__ constructor method of our GridWorldEnv:

Python

import numpy as np

class GridWorldEnv:
    def __init__(self, size=5):
        """
        Environment with a randomly selected goal each episode.
        By default, the agent starts at (0,0).
        """
        self.size = size
        self.state = None
        self.goal_state = None
        self.action_space = 4
        self.max_steps = size * 3  # Prevent endless episodes
        self.steps_count = 0
        self.rng = np.random.default_rng(42)  # For reproducibility

import numpy as np

class GridWorldEnv:
    def __init__(self, size=5):
        """
        Environment with a randomly selected goal each episode.
        By default, the agent starts at (0,0).
        """
        self.size = size
        self.state = None
        self.goal_state = None
        self.action_space = 4
        self.max_steps = size * 3  # Prevent endless episodes
        self.steps_count = 0
        self.rng = np.random.default_rng(42)  # For reproducibility

This initialization sets up a grid world with configurable size. The key enhancements include:

A goal_state attribute for storing the random goal location;
A max_steps limit proportional to grid size to prevent endless episodes;
A steps counter to track episode progression;
A random number generator obtained via np.random.default_rng with a fixed seed for reproducible results.

Next, let's create the method that generates random goal positions:

Python

def _sample_random_goal(self):
    """
    Randomly sample a goal cell that's not (0, 0).
    Note: If the environment includes obstacles like walls or mines,
    ensure to exclude those positions from possible goal locations.
    """
    possible_positions = [(r, c) for r in range(self.size)
                          for c in range(self.size)
                          if (r, c) != (0, 0)]  # Exclude starting position
    return possible_positions[self.rng.integers(len(possible_positions))]

def _sample_random_goal(self):
    """
    Randomly sample a goal cell that's not (0, 0).
    Note: If the environment includes obstacles like walls or mines,
    ensure to exclude those positions from possible goal locations.
    """
    possible_positions = [(r, c) for r in range(self.size)
                          for c in range(self.size)
                          if (r, c) != (0, 0)]  # Exclude starting position
    return possible_positions[self.rng.integers(len(possible_positions))]

This helper method:

Creates a list of all valid grid positions except the starting point (0,0);
Randomly selects one position using our random number generator;
Returns the chosen position as a (row, column) tuple.

The underscore prefix indicates this is an internal method not intended to be called directly from outside the class — a common Python convention for implementation details.

Now let's implement the environment's reset method, which initializes a new episode:

Python

def reset(self):
    """
    Reset the environment to an initial agent state (0,0)
    and randomly sample a new goal position.

    Returns:
        tuple: ((row, col), (goal_row, goal_col))
    """
    self.state = (0, 0)
    self.goal_state = self._sample_random_goal()
    self.steps_count = 0
    # Note: we must return the goal_state in the observation
    # to make the problem actually solvable
    return (self.state, self.goal_state)

def reset(self):
    """
    Reset the environment to an initial agent state (0,0)
    and randomly sample a new goal position.

    Returns:
        tuple: ((row, col), (goal_row, goal_col))
    """
    self.state = (0, 0)
    self.goal_state = self._sample_random_goal()
    self.steps_count = 0
    # Note: we must return the goal_state in the observation
    # to make the problem actually solvable
    return (self.state, self.goal_state)

Notice the crucial design decision in our reset() method: instead of just returning the agent's position, we return a tuple containing both the agent's position and the goal position. This expanded observation provides the agent with the necessary information to solve the task.

This is a perfect practical example of the state vs. observation distinction. Without the goal position in the observation, our agent would face an unsolvable problem — like trying to find a treasure without a map. By including this vital piece of information, we enable the agent to develop meaningful strategies.

Processing Actions and Rewards

Let's now focus on the step method:

Python

def step(self, action):
    """
    Take an action in the environment.

    Args:
        action (int): 0=up, 1=down, 2=left, 3=right

    Returns:
        tuple: (next_state, reward, done, info)
          next_state = ((row, col), (goal_row, goal_col))
          reward = 1.0 if goal reached, otherwise 0.0
          done = True if goal reached or max steps exceeded
          info = {"timeout": True if max steps exceeded}
    """
    self.steps_count += 1
    r, c = self.state

    # Update position based on action
    if action == 0:  # up
        r = max(r - 1, 0)
    elif action == 1:  # down
        r = min(r + 1, self.size - 1)
    elif action == 2:  # left
        c = max(c - 1, 0)
    elif action == 3:  # right
        c = min(c + 1, self.size - 1)

    self.state = (r, c)
    reward = 1.0 if self.state == self.goal_state else 0.0
    done = (self.state == self.goal_state) or (self.steps_count >= self.max_steps)
    info = {"timeout": self.steps_count >= self.max_steps}

    return (self.state, self.goal_state), reward, done, info

def step(self, action):
    """
    Take an action in the environment.

    Args:
        action (int): 0=up, 1=down, 2=left, 3=right

    Returns:
        tuple: (next_state, reward, done, info)
          next_state = ((row, col), (goal_row, goal_col))
          reward = 1.0 if goal reached, otherwise 0.0
          done = True if goal reached or max steps exceeded
          info = {"timeout": True if max steps exceeded}
    """
    self.steps_count += 1
    r, c = self.state

    # Update position based on action
    if action == 0:  # up
        r = max(r - 1, 0)
    elif action == 1:  # down
        r = min(r + 1, self.size - 1)
    elif action == 2:  # left
        c = max(c - 1, 0)
    elif action == 3:  # right
        c = min(c + 1, self.size - 1)

    self.state = (r, c)
    reward = 1.0 if self.state == self.goal_state else 0.0
    done = (self.state == self.goal_state) or (self.steps_count >= self.max_steps)
    info = {"timeout": self.steps_count >= self.max_steps}

    return (self.state, self.goal_state), reward, done, info

This step method:

Updates the agent's position based on the chosen action;
Provides the binary reward (1.0 for reaching the goal, 0.0 otherwise);
Determines if the episode should end (goal reached or max steps exceeded);
Returns the complete observation, reward, done flag, and additional info.

Consistent with our reset method, we return both the agent position and goal position in each observation. This consistent format ensures the agent always has the complete information needed to make informed decisions.

Learning with Random Objectives

The remarkable thing about our implementation is that the standard Q-learning algorithm can handle random goals without any special modifications, once we include the goal state coordinates in the observation. Let's quickly look at a basic training loop, similar to what we encountered in previous courses:

Python

def train_agent(env, agent, episodes=1000):
    """Train a Q-learning agent in our random goal environment"""
    rewards = []
    successes = []
    
    # Run one episode with visualization
    print("Testing with a visualized episode:")
    result = run_episode(env, agent, render=True)
    print(f"Initial episode: reward={result[0]}, success={result[1]}\n")
    
    # Train for multiple episodes
    print(f"Training for {episodes} episodes...")
    for ep in range(episodes):
        reward, success = run_episode(env, agent)
        rewards.append(reward)
        successes.append(int(success))
        
    # Report performance
    print(f"Success rate (last 100 episodes): {np.mean(successes[-100:]):.2f}")
    return rewards, successes

def train_agent(env, agent, episodes=1000):
    """Train a Q-learning agent in our random goal environment"""
    rewards = []
    successes = []
    
    # Run one episode with visualization
    print("Testing with a visualized episode:")
    result = run_episode(env, agent, render=True)
    print(f"Initial episode: reward={result[0]}, success={result[1]}\n")
    
    # Train for multiple episodes
    print(f"Training for {episodes} episodes...")
    for ep in range(episodes):
        reward, success = run_episode(env, agent)
        rewards.append(reward)
        successes.append(int(success))
        
    # Report performance
    print(f"Success rate (last 100 episodes): {np.mean(successes[-100:]):.2f}")
    return rewards, successes

The key insight here is that our agent learns a general policy that maps combinations of (agent position, goal position) to appropriate actions. Rather than memorizing a single path, it develops a flexible strategy that works for any goal placement.

This is a powerful capability — our agent effectively learns to generate appropriate behavior on the fly based on the current objective. In Reinforcement Learning terms, it has developed a goal-conditioned policy.

In this lesson, we've enhanced our Reinforcement Learning toolkit by implementing an environment with random goal states. We've explored the critical distinction between state and observation, seeing firsthand why providing the agent with goal information is essential for successful learning. By structuring our environment to include goal positions in the observations, we've enabled our agent to develop flexible navigation strategies that adapt to changing objectives.

This advancement represents an important step toward more realistic and practical Reinforcement Learning applications. Rather than solving a single, fixed problem, our agent now learns generalized behaviors that can handle variability. Now, time to practice this newly acquired knowledge!

Introduction

Random Goals: Why They Matter

State vs. Observation: A Critical Distinction

Join the 1M+ learners on CodeSignal

Designing the Random Goal Environment

Sampling Random Goals

Managing Information in Observations

Processing Actions and Rewards

Learning with Random Objectives

Conclusion and Next Steps