Understanding the Foundations of Reinforcement Learning

Introduction

Welcome to the very first lesson of " Environment Engineering: The Foundation of RL Systems "! This course is part of our " Playing Games with Reinforcement Learning " path, where you'll embark on an exciting journey to master Reinforcement Learning by building intelligent agents that can autonomously navigate and learn in complex environments. Throughout this learning journey, you'll build your skills step by step - first mastering how to create environments where agents can learn, then discovering how Q-Learning enables these agents to make intelligent decisions. You'll then bring everything together by connecting your smart agents with the environments you've built, before finally exploring techniques to optimize their performance. By the time you complete this path, you'll be equipped to build complete reinforcement learning systems that can tackle challenging problems and even learn to master games on their own! Today, we'll begin with the very foundations of Reinforcement Learning (RL) — understanding what it is and the core components that make RL systems work. Let's dive in together!

What is Reinforcement Learning?

Reinforcement Learning is a subset of Machine Learning where an agent learns to make decisions by interacting with an environment. Unlike Supervised Learning (where we train with labeled examples) or Unsupervised Learning (which finds patterns in unlabeled data), Reinforcement Learning is about learning through trial and error and receiving feedback . Think about how you learned to ride a bicycle. Nobody explicitly told you the exact angle to turn the handlebar or how much pressure to apply to the pedals. Instead, you tried different approaches, fell a few times, and gradually improved by getting feedback (staying upright felt good, falling hurt!). This process of learning from experience is the essence of Reinforcement Learning. The RL setting involves several components: An agent that makes decisions; An environment the agent interacts with; States that represent the situation of the environment; Actions the agent can take in each state; Rewards that provide feedback on how good the actions were. Here's a common visual representation of the so-called "RL loop" that comprises the above components:

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Transitions: How Environments Evolve

Transitions are the rules that govern how an environment changes when an agent takes an action in a given state. They form the backbone of the environment's dynamics, determining what happens next after each decision. Think of transitions like the laws of physics in your virtual world — they define what's possible and how things change.

Let's code a simple transition function for our grid world:

Python

# Determines the next state given the current state and action.
def step(state, action, grid_size=4):
    row, col = state
    
    # Calculate the next position based on the action
    if action == 0:  # up
        next_row = max(0, row - 1)
        next_col = col
    elif action == 1:  # down
        next_row = min(grid_size - 1, row + 1)
        next_col = col
    elif action == 2:  # left
        next_row = row
        next_col = max(0, col - 1)
    elif action == 3:  # right
        next_row = row
        next_col = min(grid_size - 1, col + 1)
    else:
        raise ValueError("Invalid action")
    
    return (next_row, next_col)

# Example usage
current_state = (2, 1)
action = 3  # right
next_state = step(current_state, action) # (2, 2)

# Determines the next state given the current state and action.
def step(state, action, grid_size=4):
    row, col = state
    
    # Calculate the next position based on the action
    if action == 0:  # up
        next_row = max(0, row - 1)
        next_col = col
    elif action == 1:  # down
        next_row = min(grid_size - 1, row + 1)
        next_col = col
    elif action == 2:  # left
        next_row = row
        next_col = max(0, col - 1)
    elif action == 3:  # right
        next_row = row
        next_col = min(grid_size - 1, col + 1)
    else:
        raise ValueError("Invalid action")
    
    return (next_row, next_col)

# Example usage
current_state = (2, 1)
action = 3  # right
next_state = step(current_state, action) # (2, 2)

This function implements a deterministic transition model for our grid world. Notice how we use max and min to prevent the agent from moving outside the grid boundaries — these constraints are part of our environment's rules. If the agent attempts to move into a wall or off the grid, it simply stays in place along that dimension.

Let's visualize this, again in the following we're using A to represent the agent's position, G for the goal, and numbers represent the possible actions:

plaintext

+---+---+---+---+
|   | 0 |   |   |
+---+---+---+---+
| 2 | A | 3 |   |
+---+---+---+---+
|   | 1 |   |   |
+---+---+---+---+
|   |   |   | G |
+---+---+---+---+

+---+---+---+---+
|   | 0 |   |   |
+---+---+---+---+
| 2 | A | 3 |   |
+---+---+---+---+
|   | 1 |   |   |
+---+---+---+---+
|   |   |   | G |
+---+---+---+---+

Have you ever noticed in video games how characters can't walk through walls? That's a transition rule! In more complex environments, transitions might involve sophisticated physics simulations or intricate rule systems, but the fundamental concept remains the same: they define how the world responds to an agent's actions.

Reward Functions: Learning Signals

The reward function is the heart and soul of Reinforcement Learning. It provides the feedback that guides your agent's learning process, signaling what outcomes are desirable or undesirable. Think about how we train pets — we give them treats when they perform the desired behavior. The reward function works the same way, but instead of treats, we're giving numerical values that guide the learning process.

As the famous saying in RL goes: "You get what you reward, not what you want." This highlights how critical it is to design your reward function carefully. A poorly designed reward can lead to surprising and unintended agent behaviors!

Let's explore some reward function examples:

In our grid world, we might give a positive reward for reaching the goal and zero elsewhere.
Similarly, in a game like chess, rewards might come only at the end (1 for winning, -1 for losing, and 0 for draw).
For a self-driving car, rewards could penalize dangerous maneuvers and reward smooth, efficient driving.
For a data center cooling system, rewards could be tied to maintaining optimal temperatures while penalizing excessive energy consumption.

Now, let's define a simple reward function for our grid world:

Python

# Define a simple reward function for our grid world
def reward_fn(state, goal_state=(3, 3)):
    return 1.0 if state == goal_state else 0.0

state = (1, 2)    # Agent is somewhere in the grid
reward = get_reward(state)
print(f"Moving to {state}, Reward: {reward}")

state = (3, 3)       # Goal state
reward = get_reward(state)
print(f"Moving to goal {state}, Reward: {reward}")

# Define a simple reward function for our grid world
def reward_fn(state, goal_state=(3, 3)):
    return 1.0 if state == goal_state else 0.0

state = (1, 2)    # Agent is somewhere in the grid
reward = get_reward(state)
print(f"Moving to {state}, Reward: {reward}")

state = (3, 3)       # Goal state
reward = get_reward(state)
print(f"Moving to goal {state}, Reward: {reward}")

This reward function gives your agent a reward of 1.0 when it reaches the goal state (3, 3) and 0.0 in all other states. Each time the agent takes an action and moves to a new state, the environment would call this function to determine what reward to provide, forming the crucial feedback signal that drives learning.

More sophisticated reward functions might include additional components, such as small positive rewards for moving closer to the goal (to incentivize good behaviour that leads to the goal), but also penalties. A penalty can be thought of as a negative reward. For example, we might give a small penalty for every step taken, to encourage learning a behaviour that gets the agent to the goal as fast as possible, or we might give a large penalty for visiting certain "dangerous" states we don't want the agent to visit.

Designing effective reward functions is both an art and a science — one you'll get better at with practice!

Introduction

What is Reinforcement Learning?

Join the 1M+ learners on CodeSignal

The Grid World Environment

States: Representing the World

Actions: Making Decisions

Transitions: How Environments Evolve

Reward Functions: Learning Signals

Conclusion and Next Steps