LRL-2: Formalizing the Problem - The RL Framework

Okay, let's move on to Chapter 2: Formalizing the Problem - The RL Framework. In the first chapter, we got a taste of Reinforcement Learning using the puppy training analogy. Now, it's time to get a bit more precise and build a solid framework for understanding RL problems.

Chapter 2: Formalizing the Problem - The RL Framework

In this chapter, we'll dissect the core components of an RL system in more detail and establish a common language to describe RL problems. Think of it as learning the grammar of the RL language.

1. The Environment: The World the Agent Interacts With

We've already mentioned the "environment" as the world the agent lives in and interacts with. Let's elaborate on this. In RL, the environment is more than just the physical space. It encompasses everything that the agent can perceive and interact with.

• Broad Definition: The environment is everything outside the agent that it can interact with and that influences the consequences of its actions.
• Abstraction: We often think of the environment as a black box that takes the agent's action as input and produces two outputs: the next state and a reward.

(Diagram: Agent interacting with Environment Black Box)

      Action (at)
    ------------>
+-------+       +-------------+        Next State (st+1), Reward (rt+1)
| Agent | ------> | Environment | ---------->
+-------+       +-------------+

Examples of Environments:

• Game Environment (e.g., Chess, Atari Game):

  • The game rules, the game board, the opponent (if any), and everything within the game's simulation constitute the environment.
  • The environment dictates how the game state changes when the agent makes a move and determines the reward (e.g., points, win/loss).

• Robotics Environment (e.g., Robot Arm in a Lab):

  • The physical space, objects the robot can manipulate, physics of the world, sensors, and actuators of the robot are all part of the environment.
  • The environment determines how the robot's actions (motor commands) affect its position and the objects around it, and it provides sensory information (camera images, joint angles) as the "state". Rewards are given based on task completion (e.g., reaching a target, grasping an object).

• Financial Market Environment (e.g., Stock Trading):

  • The stock market, other traders, market dynamics, economic factors are all part of the environment.
  • The environment dictates how stock prices change based on the agent's trading actions (buy, sell, hold) and market forces. Rewards are related to profits and losses.

• Web Recommender System Environment (e.g., Movie Recommendation Website):

  • Users, their preferences, the catalog of movies, website interface, and user interaction tracking systems form the environment.
  • The environment determines how users react to recommended movies (clicks, watch time, ratings) and provides feedback in the form of user engagement metrics, which are used as rewards.

Key Aspects of the Environment:

• Dynamics: How the environment changes over time, especially in response to the agent's actions. This includes the transition from one state to the next.
• Observability: How much information about the environment is available to the agent. Is the agent able to fully observe the current state, or is it only getting partial observations? (We'll touch upon this concept of "partial observability" later).
• Stochasticity vs. Determinism: Is the environment's response to actions predictable (deterministic) or random (stochastic)? In a deterministic environment, the same action in the same state always leads to the same next state and reward. In a stochastic environment, there can be randomness in the transitions and rewards. Real-world environments are often stochastic.

2. States and State Space: What Does the Agent Observe?

The state is a representation of the environment at a particular moment in time. It's the information the agent uses to make decisions. Think of it as a snapshot of the environment.

• Definition: A state, often denoted as s or s_t (at time t), is a description of the relevant aspects of the environment that the agent can observe. It should contain enough information for the agent to make informed decisions.
• State Space: The set of all possible states is called the state space, often denoted as S.

Examples of States:

• Chess Game State: The state can be represented by the position of all pieces on the chessboard, whose turn it is to play, and possibly castling rights, etc. Each unique arrangement of pieces is a different state.

  (Image: Chessboard configuration as a state)

• Atari Game State (e.g., Breakout): The raw pixels of the game screen can be considered the state. Alternatively, we might use processed features from the screen, like the position of the ball, paddle, and bricks.

  (Image: Screenshot of Breakout game as a state representation)

• Robot Navigation State: The robot's current position (x, y coordinates), its orientation, and readings from its sensors (e.g., distance to obstacles from lidar, camera image) could be part of the state.

  (Diagram: Robot with sensors and its state represented by position and sensor readings)

• Stock Trading State: The current prices of stocks, trading volume, historical price trends, and other market indicators can form the state.

Important Considerations about States:

• Information Content: The state should capture all the relevant information needed to make optimal decisions going forward. It doesn't need to contain irrelevant historical information unless that history is crucial for predicting the future. (This relates to the Markov Property, which we'll discuss in Chapter 4).
• Representation: How we represent the state is crucial. It could be a vector of numbers, an image, a set of variables, or a more complex data structure. The chosen representation impacts the learning process.
• State Space Size: The number of possible states can be finite or infinite, and it can be very large. A larger state space generally makes learning more challenging.

3. Actions and Action Space: What Can the Agent Do?

Actions are the choices the agent can make to interact with the environment. They are the agent's way of influencing the environment and trying to achieve its goals.

• Definition: An action, denoted as a or a_t (at time t), is a choice made by the agent that can affect the environment.
• Action Space: The set of all possible actions the agent can take in a given state is called the action space, often denoted as A.

Types of Action Spaces:

• Discrete Action Space: The agent can choose from a finite, countable set of actions. Examples:

  • In Chess: Moving a piece to a valid square.
  • In Atari Games: "Move Left," "Move Right," "Jump," "Fire," "No-op."
  • In Robotics: "Move forward," "Turn left," "Turn right," "Grasp."

  (Diagram: List of discrete actions like "Left", "Right", "Up", "Down")

• Continuous Action Space: The agent can choose actions from a continuous range of values. Examples:

  • In Robotics: Setting the joint angles of a robot arm directly.
  • In Autonomous Driving: Steering angle, acceleration rate, braking force.
  • In Stock Trading: Choosing the exact amount of stock to buy or sell.

  (Diagram: Slider representing a continuous action value range from -1 to 1)

Examples of Actions:

• Chess Game Actions: Moving a piece from one square to another (e.g., "Move Knight from B1 to C3").
• Atari Game Actions (Breakout): "Move paddle left," "Move paddle right," "No action" (in some games).
• Robot Arm Actions: Setting joint angles for each joint of the arm to move the end-effector to a desired position.
• Autonomous Driving Actions: Steering wheel angle (e.g., -30 degrees to +30 degrees), acceleration (e.g., 0 to 10 m/s²), braking (e.g., 0 to 5 m/s²).

4. Rewards: The Feedback Mechanism

Rewards are scalar signals from the environment that tell the agent how "good" or "bad" its actions are. They are the primary way the agent learns what to do in the environment.

• Definition: A reward, denoted as r or r_t+1 (received after taking action a_t in state s_t and transitioning to state s_t+1), is a numerical value that quantifies the immediate consequence of the agent's action.
• Goal of the Agent: The agent's objective in RL is to learn a policy that maximizes the total cumulative reward it receives over time.

Designing Effective Reward Functions:

Designing good reward functions is crucial in RL and can be challenging. Here are some key considerations:

• Clarity and Alignment: Rewards should clearly signal what we want the agent to achieve. The reward function should be aligned with the true objective. Be careful of unintended consequences! For example, if you reward a robot for "reaching a destination quickly," it might learn to drive dangerously fast if safety is not explicitly rewarded.
• Sparsity vs. Density:
  • Sparse Rewards: Rewards are given infrequently, often only when a goal is achieved (e.g., +1 for winning a game, 0 otherwise). Sparse rewards can make learning very difficult, especially in complex environments, because the agent may not get enough feedback to guide its learning.
  • Dense Rewards: Rewards are given more frequently, even for intermediate steps that move the agent closer to the goal. For example, in a navigation task, you might give a small positive reward for each step taken towards the target and a larger reward upon reaching the target. Dense rewards can speed up learning but can also shape the agent's behavior in unintended ways if not carefully designed.
• Shaping Rewards: Sometimes, we use "reward shaping" to provide more informative rewards to guide the agent towards better solutions, especially in the early stages of learning. However, reward shaping should be done cautiously to avoid creating "shortcuts" or unintended behaviors.
• Negative Rewards (Punishments): Negative rewards (penalties) can be used to discourage undesirable actions (e.g., -1 for crashing in a driving simulation, -0.1 for taking too much time). Punishments can sometimes speed up learning by quickly steering the agent away from bad actions.

Examples of Rewards:

• Game Playing Rewards:

  • +1 for winning the game.
  • -1 for losing the game.
  • 0 for a draw or for each time step (if we want to minimize game duration).
  • Intermediate scores or points in the game can also be used as rewards.

• Robotics Rewards (Navigation Task):

  • +10 for reaching the goal location.
  • -0.1 for each time step (to encourage faster navigation).
  • -1 for colliding with an obstacle.

• Recommender System Rewards:

  • +1 for a user clicking on a recommended item.
  • +5 for a user purchasing a recommended item.
  • 0 for no interaction.
  • -0.5 for showing a recommendation that the user explicitly dislikes.

5. Episodes and Time Steps: Structuring the Learning Process

To structure the interaction and learning process, we often divide it into episodes.

• Episode: An episode is a sequence of interactions between the agent and the environment that starts from an initial state and ends when a terminal state is reached or a predefined time limit is exceeded. Think of it as one "game" or one "trial."
• Time Step: Each interaction within an episode, where the agent takes an action and the environment responds with a next state and reward, is called a time step. We can index time steps as t = 0, 1, 2, 3, ... within an episode.

(Diagram: Episode as a sequence of states, actions, and rewards)

Episode 1:  s0, a0, r1, s1, a1, r2, s2, a2, r3, s3, ... , st, at, rt+1, st+1 (Terminal State)
Episode 2:  s'0, a'0, r'1, s'1, a'1, r'2, s'2, ...
...

Types of Episodes:

• Episodic Tasks (Tasks with Episodes): Tasks that naturally break down into episodes with a clear starting and ending point. Examples:

  • Game playing (each game is an episode).
  • Robot navigation from a starting point to a goal.
  • Completing a single assembly task with a robot arm.
  • These are often called "finite-horizon" tasks if episodes are guaranteed to terminate.

• Continuing Tasks (Tasks without Natural Episodes): Tasks that go on indefinitely without a clear end. Examples:

  • Controlling a power plant.
  • Managing a stock portfolio.
  • An autonomous driving system that is always operating.
  • These are often called "infinite-horizon" tasks. We still break them into time steps, but there are no natural episode boundaries.

6. Goals in Reinforcement Learning: Maximizing Cumulative Reward

The ultimate goal of an RL agent is to learn a policy that maximizes the cumulative reward it receives over time. But what exactly do we mean by "cumulative reward"?

• Return (Cumulative Reward): The total reward accumulated over an episode (or over a sequence of time steps in a continuing task) is called the return, often denoted as G. For an episode of length T, the return can be calculated as:

  G = r₁ + r₂ + r₃ + ... + r_T

• Discounted Return: In many RL problems, especially continuing tasks, we use a discount factor, denoted as γ (gamma), where 0 ≤ γ ≤ 1. The discount factor determines how much we value future rewards compared to immediate rewards. A discount factor of 0 means we only care about immediate rewards. A discount factor closer to 1 means we value future rewards more. The discounted return is calculated as:

  G_t = r_t+1 + γr_t+2 + γ²r_t+3 + ... = ∑_k=0^∞ γ^k r_t+k+1

  (Equation: Discounted Return Formula)

  • γ is typically set to a value like 0.9, 0.99, or even 0.999.
  • The discount factor serves several purposes:
    • Mathematical Convenience: It makes the sum of future rewards finite in infinite-horizon tasks (if rewards are bounded).
    • Preference for Immediate Rewards: In many real-world scenarios, getting a reward sooner is better than getting it later. Discounting reflects this preference.
    • Handling Uncertainty: Future rewards are often more uncertain than immediate rewards. Discounting can implicitly account for this uncertainty.

• Objective: The agent's goal is to find a policy that maximizes the expected return. Since the environment can be stochastic, we often talk about maximizing the average return over many episodes or time steps.

In Summary (Chapter 2):

We've now formalized the key components of the Reinforcement Learning framework:

• Environment: The world the agent interacts with.
• State: The agent's perception of the environment.
• Action: The agent's choice of interaction.
• Reward: Feedback from the environment, indicating the value of actions.
• Episode: A sequence of interactions.
• Time Step: Each step within an episode.
• Goal: Maximize cumulative (discounted) reward.

Understanding these concepts is fundamental to grasping how Reinforcement Learning works. In the next chapter, we'll introduce Policies and Value Functions, which are crucial tools that agents use to learn and make decisions in RL environments. We'll see how agents can use policies to choose actions and value functions to estimate the "goodness" of states and actions, guiding them towards maximizing their cumulative reward.