Navigating Environmental Hazards in Reinforcement Learning

Introduction

Welcome back to our course on " Navigating RL Challenges: Strategies and Future Directions "! This is the third lesson in our journey, and I'm excited to continue building on what we've learned so far. So far, we implemented random goals to create more dynamic grid world environments, followed by an exploration of reward shaping to provide more informative feedback to our agents, helping them learn more efficiently. Today, we'll tackle another critical challenge in Reinforcement Learning: environmental hazards . In real-world applications, agents often need to navigate environments with dangers or obstacles that must be avoided. Think of a robot that needs to avoid stairs while cleaning a house or an autonomous vehicle that must recognize and avoid hazards on the road. We'll implement mines in our grid world that the agent must learn to avoid, adding risk assessment and danger avoidance to our agent's learning objectives. This brings us one step closer to training agents that can handle the complexities and dangers of real-world environments.

Implementing Mines in the Environment

Let's start by modifying our grid world environment class to include mines. We'll focus first on how to initialize our environment with mines: Python def __init__(self, size=5, use_reward_shaping=True, n_mines=3): """ An extended GridWorld with: - Optional reward shaping - Optional mines that cause immediate failure """ self.size = size self.use_reward_shaping = use_reward_shaping self.n_mines = n_mines # Number of mines to place self.state = None self.goal_state = None self.mines = set() # Will store mine positions self.prev_distance = None self.action_space = 4 self.max_steps = size * 3 self.steps_count = 0 self.rng = np.random.default_rng(42) This constructor introduces a new parameter n_mines that determines how many hazards will be placed in the environment. We also initialize an empty set self.mines that will store the locations of these mines. To visualize how mines appear in our environment, here's how a 5x5 grid would render with the agent A at (1, 1), the goal G at (4, 4), and mines * at positions (1, 2), (2, 3), and (3, 1) : text . . . . . --> safe for A to go up . A * . . --> --> safe for A to go left, but if it goes right, game over! . . . * . --> safe for A to go down . * . . . . . . . G

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Now we should update the reset method to place mines when the environment is reset:

Python

def reset(self):
    """
    Reset environment with agent at (0,0), random goal, and random mines
    """
    self.state = (0, 0)
    self.goal_state = self._sample_random_goal()
    self.mines = self._sample_mines() if self.n_mines > 0 else set()
    self.prev_distance = self._manhattan_distance(self.state, self.goal_state)
    self.steps_count = 0
    mines_near = self._check_nearby_mines()
    return (self.state, self.goal_state, mines_near)

def reset(self):
    """
    Reset environment with agent at (0,0), random goal, and random mines
    """
    self.state = (0, 0)
    self.goal_state = self._sample_random_goal()
    self.mines = self._sample_mines() if self.n_mines > 0 else set()
    self.prev_distance = self._manhattan_distance(self.state, self.goal_state)
    self.steps_count = 0
    mines_near = self._check_nearby_mines()
    return (self.state, self.goal_state, mines_near)

When you call this updated reset(), the environment now places mines randomly with self._sample_mines(), and returns information about nearby mines as part of the state via self._check_nearby_mines(). This gives your agent sensory data about potential dangers in its vicinity.

Let's implement the self._sample_mines() method to place mines randomly:

Python

def _sample_mines(self):
    """Randomly place mines, avoiding (0, 0) and the goal."""
    available = [(r, c) for r in range(self.size)
                for c in range(self.size)
                if (r, c) not in [(0, 0), self.goal_state]]
    
    n_placements = min(self.n_mines, len(available))
    chosen = self.rng.choice(len(available), n_placements, replace=False)
    mines = {available[i] for i in chosen}
    return mines

def _sample_mines(self):
    """Randomly place mines, avoiding (0, 0) and the goal."""
    available = [(r, c) for r in range(self.size)
                for c in range(self.size)
                if (r, c) not in [(0, 0), self.goal_state]]
    
    n_placements = min(self.n_mines, len(available))
    chosen = self.rng.choice(len(available), n_placements, replace=False)
    mines = {available[i] for i in chosen}
    return mines

This method carefully places mines by:

Creating a list of all valid positions that aren't the start or goal.
Randomly selecting positions based on your specified number of mines.
Returning these positions as a set for efficient lookups.

Now let's add the danger detection functionality with the self._check_nearby_mines() method:

Python

def _check_nearby_mines(self):
    """
    Check if there are mines in adjacent cells.
    Returns a binary tuple (up, down, left, right) where 1 indicates danger.
    """
    r, c = self.state
    nearby = [
        (max(r - 1, 0), c),             # up
        (min(r + 1, self.size - 1), c),   # down
        (r, max(c - 1, 0)),             # left
        (r, min(c + 1, self.size - 1))    # right
    ]
    return tuple(1 if pos in self.mines else 0 for pos in nearby)

def _check_nearby_mines(self):
    """
    Check if there are mines in adjacent cells.
    Returns a binary tuple (up, down, left, right) where 1 indicates danger.
    """
    r, c = self.state
    nearby = [
        (max(r - 1, 0), c),             # up
        (min(r + 1, self.size - 1), c),   # down
        (r, max(c - 1, 0)),             # left
        (r, min(c + 1, self.size - 1))    # right
    ]
    return tuple(1 if pos in self.mines else 0 for pos in nearby)

This method simulates a simple danger detection "sensor" for your agent. It examines the four adjacent cells and returns a tuple indicating which directions contain mines. This is crucial for your agent's learning since it provides predictive information about dangers before encountering them directly — much like how real robots use sensors to detect obstacles.

At this point, we need to modify the step method to handle what happens when the agent moves onto a mine:

Python

# Part of the step method that handles mine collisions
if self.state in self.mines:
    mines_near = self._check_nearby_mines()
    return (self.state, self.goal_state, mines_near), -1.0, True, {"hit_mine": True}

# Part of the step method that handles mine collisions
if self.state in self.mines:
    mines_near = self._check_nearby_mines()
    return (self.state, self.goal_state, mines_near), -1.0, True, {"hit_mine": True}

When your agent steps on a mine, three important things happen:

It receives a significant negative reward (-1.0).
The episode immediately ends (done = True).
The information dictionary includes a flag indicating a mine was hit.

This creates a clear negative consequence that your agent will learn to avoid. The negative reward is strategically set to cancel out the positive reward for reaching the goal, creating a balanced risk-reward structure.

Training with Risk Assessment

Now let's implement the usual code to train and evaluate our agent in these hazardous environments. First, we need to update our episode function:

Python

def run_episode(env, agent, render=False, deterministic=False):
    """Execute a single episode and return results"""
    state = env.reset()
    total_reward = 0.0
    done = False

    if render:
        print("Initial State:")
        env.render()

    while not done:
        # Agent selects action (explore or exploit)
        action = agent.act(state, deterministic=deterministic)
        
        # Environment responds with next state and reward
        next_state, reward, done, info = env.step(action)
        
        # Agent learns from the experience
        agent.learn(state, action, reward, next_state, done)
        
        state = next_state
        total_reward += reward

        if render:
            env.render()

    success = (not info.get("timeout", False)) and (not info.get("hit_mine", False))
    hit_mine = info.get("hit_mine", False)
    return total_reward, success, hit_mine

def run_episode(env, agent, render=False, deterministic=False):
    """Execute a single episode and return results"""
    state = env.reset()
    total_reward = 0.0
    done = False

    if render:
        print("Initial State:")
        env.render()

    while not done:
        # Agent selects action (explore or exploit)
        action = agent.act(state, deterministic=deterministic)
        
        # Environment responds with next state and reward
        next_state, reward, done, info = env.step(action)
        
        # Agent learns from the experience
        agent.learn(state, action, reward, next_state, done)
        
        state = next_state
        total_reward += reward

        if render:
            env.render()

    success = (not info.get("timeout", False)) and (not info.get("hit_mine", False))
    hit_mine = info.get("hit_mine", False)
    return total_reward, success, hit_mine

Notice how we've updated the success condition — your agent only succeeds if it reaches the goal without hitting a mine or timing out. We also track whether a mine was hit, which will help you evaluate safety performance.

Finally, let's implement code that lets you compare performance across environments with different hazard levels:

Python

def main():
    """Compare performance with different hazard densities"""
    env_size = 5
    actions = [0, 1, 2, 3]
    n_episodes = 1000

    # Try environments with increasing danger
    mine_counts = [0, 2, 5]

    for n_mines in mine_counts:
        print(f"\nTraining with {n_mines} mines...")
        env = GridWorldEnv(size=env_size, use_reward_shaping=True, n_mines=n_mines)
        agent = QLearningAgent(actions)
        train_agent(env, agent, n_episodes)

def main():
    """Compare performance with different hazard densities"""
    env_size = 5
    actions = [0, 1, 2, 3]
    n_episodes = 1000

    # Try environments with increasing danger
    mine_counts = [0, 2, 5]

    for n_mines in mine_counts:
        print(f"\nTraining with {n_mines} mines...")
        env = GridWorldEnv(size=env_size, use_reward_shaping=True, n_mines=n_mines)
        agent = QLearningAgent(actions)
        train_agent(env, agent, n_episodes)

This experiment trains agents in environments with varying numbers of mines, allowing you to directly observe how increasing environmental hazards affects learning performance. When you run this code, you'll typically observe:

Environments with no mines are easiest to learn.
As you add mines, success rates decrease and training takes longer.
With many mines, finding safe paths becomes significantly challenging.

This progression mirrors real-world RL applications, where increased environmental complexity often requires more sophisticated learning strategies or additional training time.

Below is a plot showing the success rate of three agents trained in environments with different numbers of mines (0, 2, and 5). The x-axis represents the training episodes (from 0 to 1000), and the y-axis represents the success rate — i.e., the fraction of episodes in which the agent reached the goal without hitting a mine or timing out.

Plot showing the success rate of three agents over time. Each agent differs in the mine count of the environment.

0 mines (blue line): As expected, this scenario shows the highest success rate overall. Without mines, the agent faces a straightforward navigation problem, and it quickly learns to reach the goal consistently. We see the success rate rapidly climb above 80% and hover near 90% or higher as training progresses.
2 mines (green line): Introducing a few mines increases the difficulty, causing the success rate to dip lower than in the 0-mine scenario. Early on, the agent often hits mines while exploring, resulting in lower success rates. Over time, however, the agent learns to avoid the hazards and its performance steadily improves, ultimately stabilizing around 70–80% success.
5 mines (red line): With more mines, the environment becomes substantially riskier. The success rate starts low and remains more volatile, reflecting the agent’s greater challenge in finding safe paths. Although performance does improve with training, the success rate plateaus at a lower level compared to the less hazardous environments, highlighting the increased complexity of learning safe navigation.

Overall, these results illustrate how adding mines — even a relatively small number — significantly impacts the agent’s learning curve and final performance. More mines mean the agent must devote more exploration and learning capacity to avoiding hazards, slowing down progress toward high success rates.

In this lesson, you've enhanced your grid world to include environmental hazards, transforming a simple navigation task into a risk-aware planning problem. You've implemented random mine placement, a danger detection system, negative rewards for collisions, and metrics to track safety performance. These additions create a more realistic learning challenge where your agent must balance reaching its goal with avoiding hazards — a fundamental concern in real-world applications.

As you continue your Reinforcement Learning journey, the techniques explored today — negative rewards for hazards, sensory information about dangers, and safety metrics — will serve as building blocks for developing safer and more capable learning systems. The Q-learning algorithm has shown remarkable adaptability, autonomously discovering safer paths without explicit programming about what to avoid — demonstrating the power of Reinforcement Learning to handle complex, risk-aware decision-making.

Introduction

Implementing Mines in the Environment

Join the 1M+ learners on CodeSignal

Updating Resetting Logic

Placing Mines

Detecting Nearby Danger

Handling Mine Collisions

Training with Risk Assessment

Comparing Risk Levels

Visualizing Different Mine Counts

Conclusion and Next Steps