Reinforcement Learning Complete Guide: From Theory to the Latest Algorithms and Real Implementations

• 1. An Overview of Reinforcement Learning
  • 1.1 The Core Components
  • 1.2 The Interaction Loop of Reinforcement Learning
  • 1.3 A Taxonomy of Reinforcement Learning Algorithms
• 2. Mathematical Foundations
  • 2.1 Markov Decision Process (MDP)
  • 2.2 Return and the Definition of the Value Functions
  • 2.3 Deriving the Bellman Equation
  • 2.4 Bellman Optimality Equation
  • 2.5 Advantage Function
• 3. The Classical Algorithms
  • 3.1 Dynamic Programming (DP)
  • 3.2 Monte Carlo (MC) Methods
  • 3.3 Temporal-Difference (TD) Learning
  • 3.4 SARSA (On-Policy TD Control)
  • 3.5 Q-Learning (Off-Policy TD Control)
• 4. The Deep RL Revolution
  • 4.1 DQN: Deep Q-Network (2013/2015)
  • 4.2 Improved DQN Variants
  • 4.3 Policy Gradient: REINFORCE
  • 4.4 Actor-Critic: A2C, A3C
  • 4.5 TRPO and PPO
  • 4.6 SAC: Soft Actor-Critic
  • 4.7 DDPG and TD3: Continuous Action Spaces
• 5. Model-based RL
  • 5.1 The Advantages and Challenges of Model-based RL
  • 5.2 World Models (Ha & Schmidhuber, 2018)
  • 5.3 MuZero (Schrittwieser et al., 2020)
  • 5.4 Dreamer (Hafner et al., 2020, 2021, 2023)
  • 5.5 Autonomous Algorithm Discovery: DiscoRL (2025)
• 6. RLHF: Reinforcement Learning from Human Feedback
  • 6.1 The RLHF Pipeline
  • 6.2 DPO: Direct Preference Optimization
  • 6.3 GRPO: Group Relative Policy Optimization
  • 6.4 Recent Trends in LLM Alignment (2025~2026)
• 7. Major Applications
  • 7.1 AlphaGo / AlphaZero / AlphaFold
  • 7.2 Robotics
  • 7.3 Game AI
  • 7.4 Recommender Systems and Other Applications
• 8. PyTorch Code Examples
  • 8.1 A DQN Implementation (CartPole)
  • 8.2 A PPO Implementation (CartPole)
• 9. Key Paper References
• 10. The Current Limits of RL and Its Future
  • 10.1 Current Limits
  • 10.2 The Future
• Conclusion

1. An Overview of Reinforcement Learning

Reinforcement Learning (RL) is one of the three pillars of machine learning, and it is a fundamentally different paradigm from supervised learning or unsupervised learning. Where supervised learning learns patterns from labelled data and unsupervised learning works out the structure of data without labels, reinforcement learning learns the optimal strategy of action through trial and error, by interacting with an environment.

Think of how a person learns to ride a bicycle. Nobody tells you to "turn the handlebars 37 degrees to the left and pedal 1.2 times per second". Instead you take the punishment of pain when you fall and the reward of moving forward when you keep your balance, and you work out the optimal balancing strategy for yourself. That is exactly what reinforcement learning is.

1.1 The Core Components

A reinforcement learning system is made up of the following core components.

Agent: the learning subject that acts inside the environment. A player in a game, a robot, or a self-driving car are all agents.

Environment: the outside world the agent interacts with. A game board, a physics simulation space, or a real road environment. The environment returns the next state and a reward in response to the agent's action.

State ($s \in \mathcal{S}$): the information that describes the current state of the environment. The board layout in chess, a robot's joint angles, or the current pattern of a stock chart all serve as states. The State Space $\mathcal{S}$ is the set of all possible states.

Action ($a \in \mathcal{A}$): the actions the agent can choose in each state. A joystick direction in a game, or the torque value of a robot arm. The Action Space $\mathcal{A}$ may be discrete or continuous.

Reward ($r \in \mathbb{R}$): the scalar feedback signal the agent receives from the environment when it takes a particular action in a particular state. The goal of reinforcement learning is to maximize the long-run cumulative reward, not to maximize the immediate reward.

Policy ($\pi$): the agent's strategy of action. It is a function that takes a state as input and outputs an action (or a probability distribution over actions), and it is the ultimate object of learning in reinforcement learning.

• Deterministic Policy: $a = \pi(s)$
• Stochastic Policy: $a \sim \pi(a|s)$

Value Function ($V^\pi(s)$): the cumulative reward expected from state $s$ when following policy $\pi$. It puts a number on "how good this state is".

Action-Value Function ($Q^\pi(s, a)$): the cumulative reward expected when taking action $a$ in state $s$ and following policy $\pi$ afterwards. It puts a number on "how good this action is in this state".

Discount Factor ($\gamma \in [0, 1]$): the discount rate that decides the present value of future rewards. At $\gamma = 0$ the agent considers only the immediate reward; the closer it gets to $\gamma = 1$, the more nearly it values distant future rewards as much as present ones.

1.2 The Interaction Loop of Reinforcement Learning

The core structure of reinforcement learning is the following repeating interaction loop.

At time t:
  1. The agent observes the current state s_t
  2. It selects action a_t according to policy π
  3. The environment performs the action and returns reward r_{t+1} and next state s_{t+1}
  4. The agent updates its policy from the experience (s_t, a_t, r_{t+1}, s_{t+1})
  5. t ← t + 1, repeat

The sequence of experience this loop produces, $s_0, a_0, r_1, s_1, a_1, r_2, s_2, \ldots$, is called a Trajectory or an Episode, and it is the training data of a reinforcement learning agent.

1.3 A Taxonomy of Reinforcement Learning Algorithms

Reinforcement learning algorithms can be classified along several axes.

2. Mathematical Foundations

2.1 Markov Decision Process (MDP)

The mathematical framework of reinforcement learning is the Markov Decision Process (MDP). An MDP is a mathematical model that formalizes sequential decision problems, and it is defined by the following 5-tuple.

• $\mathcal{S}$: the State Space
• $\mathcal{A}$: the Action Space
• $P(s'|s, a)$: the Transition Probability — the probability of moving to state $s'$ when taking action $a$ in state $s$
• $R(s, a, s')$: the Reward Function — the immediate reward for the transition $(s, a, s')$
• $\gamma \in [0, 1)$: the Discount Factor

The heart of an MDP is the Markov Property. The future state depends only on the current state and the current action, and is independent of the whole history of past states and actions.

Thanks to this property we may assume the current state $s_t$ alone carries all the information needed for a decision, and that is the fundamental basis on which reinforcement learning becomes mathematically tractable.

2.2 Return and the Definition of the Value Functions

The agent's goal is to maximize the Return. The Return is defined as the discounted cumulative reward from time $t$ onwards.

The important recursive property of the Return is as follows.

This recursion is what the Bellman equations are later built on.

State-Value Function: the Return you can expect starting from state $s$ while following policy $\pi$.

Action-Value Function: the Return you can expect when following policy $\pi$ after first taking action $a$ in state $s$.

The relationship between the two functions is as follows.

2.3 Deriving the Bellman Equation

The Bellman Equation is the single most central mathematical tool in reinforcement learning. It uses the recursive property of the Return to express the value function recursively.

Deriving the Bellman Equation for the State-Value Function:

Substituting the recursion $G_t = r_{t+1} + \gamma G_{t+1}$ for the Return:

Expanding by the linearity of expectation:

Here the action $a$ is decided by the policy $\pi(a|s)$ and the next state $s'$ by the transition probability $P(s'|s,a)$, so unrolling the whole expectation into a double sum gives:

This is the Bellman Expectation Equation for $V^\pi$. The value of the current state is expressed as the expectation of the immediate reward plus the discounted value of the next state.

The Bellman Equation for the Action-Value Function:

The same approach derives the equation for $Q^\pi$.

2.4 Bellman Optimality Equation

The optimal policy $\pi^*$ is the policy that achieves the highest value in every state.

The optimal value functions $V^*$ and $Q^*$ are defined as follows.

Deriving the Bellman Optimality Equation for $V^*$:

Under the optimal policy, the agent selects the optimal action in every state. So instead of an expectation over the policy we use a maximization (max).

Substituting the Bellman equation for $Q^*$:

Bellman Optimality Equation for $Q^*$:

The essential meaning of this equation is the following. If you know the optimal action-value function $Q^*$, you can obtain the optimal policy simply by choosing, in each state, the action that maximizes $Q^*$.

2.5 Advantage Function

Here we define the Advantage Function $A^\pi(s, a)$, which will be central later in the Policy Gradient family.

The Advantage Function expresses "how much better it is to take action $a$ in state $s$ than to act on average". If $A > 0$ the action is above average, if $A < 0$ it is below average. The expectation of the Advantage over all actions is 0.

3. The Classical Algorithms

3.1 Dynamic Programming (DP)

Dynamic Programming is the method available when you know a complete model of the environment (the transition probability $P$ and the reward function $R$). In practice it is rare to know the environment model exactly, but DP is the theoretical foundation of every other RL algorithm.

Policy Evaluation: computes the value function $V^\pi$ of a given policy $\pi$. You apply the Bellman Expectation Equation repeatedly until it converges.

Policy Improvement: obtains the greedy policy with respect to the current value function.

Policy Iteration: alternates Policy Evaluation and Policy Improvement until it converges on the optimal policy.

Value Iteration: applies the Bellman Optimality Equation repeatedly and directly.

Once it converges you extract the optimal policy. Value Iteration can be seen as Policy Iteration in which Policy Evaluation performs only a single sweep.

3.2 Monte Carlo (MC) Methods

In the model-free situation where the environment model is unknown, Monte Carlo methods estimate the value function from actual experience (episodes). The core idea is simple: run many episodes and use the average of the Returns observed at each state (or state-action pair) as the estimate of the value function.

Here $N(s)$ is the number of times state $s$ was visited and $G_t^{(i)}$ is the actual Return on the $i$-th visit.

The advantage of MC: it needs no environment model, and it gives an estimate with no bias.

The disadvantage of MC: learning is only possible once an episode has ended (so it applies only to episodic tasks), and the variance of the Return is large.

3.3 Temporal-Difference (TD) Learning

TD Learning combines the advantages of DP and MC. It can learn without an environment model (like MC), while being able to update at every step before an episode ends (like DP).

TD(0) — the most basic TD method:

Here $\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$ is called the TD Error. You update by the difference between the current estimated value and the TD Target ($r_{t+1} + \gamma V(s_{t+1})$), which is the actual reward $r_{t+1}$ plus the estimated value of the next state. This is called Bootstrapping — updating an estimate using another estimate.

MC vs. TD:

TD($\lambda$): a method that interpolates between MC and TD(0). At $\lambda = 0$ it is TD(0), at $\lambda = 1$ it is equivalent to MC. The $\lambda$-Return is defined as follows.

Here $G_t^{(n)}$ is the $n$-step Return.

3.4 SARSA (On-Policy TD Control)

SARSA is an on-policy TD control algorithm, and its name comes from the experience tuple used in the update, $(S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1})$.

SARSA updates using the Q-value of the next action it actually took, $a_{t+1}$. Because that reflects the value of the action the current policy (usually $\varepsilon$-greedy) will really take, it also takes account of risky actions taken during exploration.

3.5 Q-Learning (Off-Policy TD Control)

Q-Learning is the off-policy TD control algorithm proposed by Watkins (1989), and one of the most important algorithms in the history of reinforcement learning.

The decisive difference from SARSA is that it uses the action giving the maximum Q-value (the greedy action) in the next state rather than the action actually taken. That is what makes Q-Learning off-policy — whatever action the agent takes in order to explore, it always converges towards the optimal Q-values.

# Q-Learning pseudocode
import numpy as np

def q_learning(env, num_episodes, alpha=0.1, gamma=0.99, epsilon=0.1):
    Q = np.zeros((env.observation_space.n, env.action_space.n))

    for episode in range(num_episodes):
        state = env.reset()
        done = False

        while not done:
            # ε-greedy action selection
            if np.random.random() < epsilon:
                action = env.action_space.sample()  # exploration
            else:
                action = np.argmax(Q[state])         # exploitation

            next_state, reward, done, _ = env.step(action)

            # Q-Learning update: max over next actions
            td_target = reward + gamma * np.max(Q[next_state]) * (1 - done)
            td_error = td_target - Q[state, action]
            Q[state, action] += alpha * td_error

            state = next_state

    return Q

Convergence conditions for Q-Learning: if every state-action pair is visited infinitely often and the learning rate satisfies the Robbins-Monro conditions ($\sum \alpha_t = \infty$, $\sum \alpha_t^2 < \infty$), Q-Learning is proven to converge to $Q^*$.

4. The Deep RL Revolution

Classical RL algorithms use a table-shaped value function, so they cannot be applied to problems with a large state space. Atari games alone make the screen pixels the state, so the state space is an astronomical $256^{210 \times 160 \times 3}$. Deep RL, which uses a deep neural network as a function approximator, appeared to solve this problem.

4.1 DQN: Deep Q-Network (2013/2015)

Papers: "Playing Atari with Deep Reinforcement Learning" (Mnih et al., 2013), "Human-level control through deep reinforcement learning" (Mnih et al., 2015, Nature)

DQN is the landmark paper that opened the Deep RL era. It replaces the Q-table of Q-Learning with a CNN (Convolutional Neural Network), estimating action values directly from raw pixel input.

Here $\theta$ is the network parameters. The training goal is to minimize the following loss function.

But simply approximating the Q-function with a neural network made training extremely unstable. DQN has two key innovations that solve this.

Innovation 1: Experience Replay

Consecutive experiences $(s_t, a_t, r_{t+1}, s_{t+1})$ are strongly correlated in time. That violates the i.i.d. (independent and identically distributed) condition that stochastic gradient descent (SGD) assumes. Experience Replay stores experiences in a Replay Buffer $\mathcal{D}$ and then draws a random mini-batch at training time, breaking the correlation between data points.

from collections import deque
import random

class ReplayBuffer:
    def __init__(self, capacity):
        self.buffer = deque(maxlen=capacity)

    def push(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))

    def sample(self, batch_size):
        batch = random.sample(self.buffer, batch_size)
        states, actions, rewards, next_states, dones = zip(*batch)
        return states, actions, rewards, next_states, dones

    def __len__(self):
        return len(self.buffer)

Innovation 2: Target Network

When you update the Q-values, if the target ($r + \gamma \max_{a'} Q(s', a'; \theta)$) and the prediction ($Q(s, a; \theta)$) both depend on the parameters $\theta$ of the same network, the target keeps moving and training can diverge. The Target Network creates a duplicate network with its own parameters $\theta^-$ and copies the main network's parameters over periodically (for example every 10,000 steps), which stabilizes the target.

A soft update is also used: $\theta^- \leftarrow \tau \theta + (1 - \tau) \theta^-$

Atari results: DQN achieved above-human performance on 29 of 49 Atari 2600 games. In games such as Breakout and Pong in particular, it learned remarkable strategies on its own.

4.2 Improved DQN Variants

Countless improvements followed DQN. Let us look at the main variants.

Double DQN (van Hasselt et al., 2016): standard DQN tends to overestimate Q-values because of the max operator. Double DQN solves this by separating action selection from action evaluation.

Action selection is done with the online network ($\theta$) and action evaluation with the target network ($\theta^-$).

Dueling DQN (Wang et al., 2016): changes the network architecture to decompose the Q-function into a State-Value $V(s)$ and an Advantage $A(s, a)$.

Thanks to this decomposition, in states where the choice of action does not matter it is enough to learn $V(s)$ accurately, which is more efficient.

Prioritized Experience Replay (Schaul et al., 2016): instead of sampling every experience with equal probability, it samples experiences with a large TD Error more often. The point is to prioritize the experiences that are more useful for learning.

An importance sampling weight corrects the resulting bias: $w_i = \left( \frac{1}{N \cdot P(i)} \right)^\beta$

Noisy DQN (Fortunato et al., 2018): explores by adding learnable noise to the network weights instead of using $\varepsilon$-greedy.

Categorical DQN / C51 (Bellemare et al., 2017): a Distributional RL approach that learns the whole Return distribution instead of the expectation of the Q-value.

Rainbow (Hessel et al., 2018): an agent that combines all 6 of the improvements above. It performed far better than any of the individual techniques.

4.3 Policy Gradient: REINFORCE

Value-based methods (DQN and the rest) learn a Q-function and then select actions greedily. But that approach has limits.

1. In a continuous action space it is hard to compute $\max_a Q(s, a)$.
2. It cannot express a stochastic policy directly.
3. A small change in the Q-function can cause an abrupt change in the policy.

Policy Gradient methods parameterize the policy $\pi_\theta$ directly and optimize the policy parameters $\theta$ directly.

Deriving the Policy Gradient Theorem:

Define the objective function as follows.

Here $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \ldots)$ is a trajectory and $R(\tau)$ is the total return of the trajectory.

To take the gradient of $J(\theta)$, express the probability of a trajectory as follows.

Taking the gradient:

Applying the Log-Derivative Trick $\nabla_\theta p_\theta(\tau) = p_\theta(\tau) \nabla_\theta \log p_\theta(\tau)$ here:

Expanding $\log p_\theta(\tau)$:

The environment dynamics $\rho_0$ and $P$ do not depend on $\theta$, so they vanish when the gradient is taken.

The Policy Gradient Theorem is therefore as follows.

The essential meaning of this result is that you can estimate the gradient of the policy without knowing the environment model (the transition probability $P$).

The REINFORCE algorithm (Williams, 1992):

REINFORCE is the most direct implementation of the Policy Gradient Theorem. It collects trajectories in Monte Carlo fashion and estimates the gradient from the observed Returns.

Variance reduction with a baseline:

The biggest problem with REINFORCE is its high variance. Subtracting a baseline $b(s)$ from the Return $R(\tau)$ leaves the expectation of the gradient (the bias) unchanged, but can reduce the variance considerably.

The most commonly used baseline is the state-value function $b(s_t) = V(s_t)$, in which case $G_t - V(s_t)$ becomes an estimate of the Advantage Function.

4.4 Actor-Critic: A2C, A3C

Actor-Critic methods learn a Policy Gradient (the Actor) and a Value Function (the Critic) at the same time.

• Actor: learns the policy $\pi_\theta(a|s)$ — it decides the action
• Critic: learns the value function $V_\phi(s)$ — it evaluates the action

REINFORCE could only learn once an episode had ended, and suffered from the high variance of the Return. Actor-Critic uses the value estimate the Critic provides as a baseline, so it updates at every step while still reducing variance.

Actor update:

Here the Advantage is estimated with the TD Error: $\hat{A}(s_t, a_t) = r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$

Critic update:

A3C: Asynchronous Advantage Actor-Critic (Mnih et al., 2016)

A3C greatly improved both training stability and speed through parallelization. The core ideas are:

1. Many Workers independently collect experience, each in its own copy of the environment.
2. Each Worker computes gradients from its own experience and updates the global parameters asynchronously.
3. The asynchronous updates supply a natural diversity of exploration, which eases the data-correlation problem without Experience Replay.

A2C: Advantage Actor-Critic

A2C is the synchronous version of A3C. All Workers collect experience at the same time, then the gradients are summed and applied in a single update. Because there is no stale-gradient problem from asynchronous updates, in practice A2C often performs on a par with A3C or better.

4.5 TRPO and PPO

The core problem of Policy Gradient is that the step size is hard to set. Too large and the policy changes abruptly and performance collapses; too small and learning is slow. TRPO and PPO are the key algorithms for solving this.

TRPO: Trust Region Policy Optimization (Schulman et al., 2015)

TRPO explicitly limits the size of a policy update. Using KL Divergence, it performs the update only within a Trust Region that bounds the distance between the previous policy and the new one.

TRPO theoretically guarantees monotonic performance improvement, but it needs Conjugate Gradient and Line Search to solve the constrained optimization problem, which makes it very complex to implement.

PPO: Proximal Policy Optimization (Schulman et al., 2017)

PPO keeps the stability of TRPO while being far simpler to implement. It is the most widely used Policy Gradient algorithm today, and it was used to train OpenAI's ChatGPT as well.

Deriving the PPO-Clip objective:

Define the policy ratio.

TRPO's surrogate objective is $L^{CPI}(\theta) = \mathbb{E}_t[r_t(\theta) \hat{A}_t]$. Maximize this objective without a constraint and the ratio $r_t(\theta)$ can grow excessively, changing the policy abruptly.

PPO's core idea is to limit the policy change by clipping the ratio into the range $[1 - \varepsilon, 1 + \varepsilon]$. ($\varepsilon$ is usually 0.1~0.2)

Analyzing case by case how this objective works:

Case 1: $\hat{A}_t > 0$ (a good action):

• If $r_t(\theta) > 1 + \varepsilon$ (the new policy picks this action far more often): it gets clipped and limited to $(1 + \varepsilon) \hat{A}_t$. The incentive for any further policy movement is removed.
• If $r_t(\theta) < 1 + \varepsilon$: $r_t(\theta) \hat{A}_t$ is used with no clipping.

Case 2: $\hat{A}_t < 0$ (a bad action):

• If $r_t(\theta) < 1 - \varepsilon$ (the new policy picks this action far less often): it gets clipped and limited to $(1 - \varepsilon) \hat{A}_t$.
• If $r_t(\theta) > 1 - \varepsilon$: $r_t(\theta) \hat{A}_t$ is used with no clipping.

In both cases, once the policy has already moved far enough in the right direction, any further incentive is cut off. That is the heart of PPO's stability.

The full PPO objective:

Real implementations combine a Policy Loss, a Value Loss and an Entropy Bonus.

• $L_t^{VF}$: the MSE Loss of the Value Function
• $S[\pi_\theta]$: the Entropy of the policy (encourages exploration)
• $c_1, c_2$: weighting coefficients

GAE (Generalized Advantage Estimation):

PPO normally estimates the Advantage using GAE (Schulman et al., 2016). GAE is the Advantage version of TD($\lambda$).

Here $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ is the TD Error. At $\lambda = 0$ it becomes the 1-step TD estimate, at $\lambda = 1$ the MC estimate.

4.6 SAC: Soft Actor-Critic

Paper: "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" (Haarnoja et al., 2018)

SAC is an off-policy actor-critic algorithm built on the Maximum Entropy RL framework. It is one of the most successful algorithms in continuous action spaces, and performs especially well in robotics.

The Maximum Entropy objective:

Unlike standard RL, which maximizes only the Return, SAC maximizes the Return and the Entropy at the same time.

Here $\mathcal{H}(\pi(\cdot|s_t)) = -\mathbb{E}_{a \sim \pi}[\log \pi(a|s_t)]$ is the Entropy of the policy and $\alpha > 0$ is the Temperature parameter that controls how much the Entropy counts.

The benefits of entropy regularization are as follows.

1. Encourages exploration: the policy learns to maximize reward while behaving as randomly as it can, which prevents it from falling into a local optimum.
2. A robust policy: it keeps several near-optimal actions available, so it stands up to changes in the environment.
3. Faster learning: broad exploration early on discovers useful actions sooner.

Soft Bellman Equation:

The three networks of SAC:

1. Soft Q-Function $Q_\phi(s, a)$: uses two Q-networks to prevent overestimation (Clipped Double Q)
2. Policy $\pi_\theta(a|s)$: a Gaussian policy (it outputs a mean and a variance)
3. Temperature $\alpha$: tuned automatically (via an entropy constraint)

Q-function update:

Here $\tilde{a}' \sim \pi_\theta(\cdot|s')$.

Policy update:

The Reparameterization Trick is used to make it differentiable: $a = f_\theta(\varepsilon; s) = \mu_\theta(s) + \sigma_\theta(s) \odot \varepsilon$, where $\varepsilon \sim \mathcal{N}(0, I)$.

Automatic temperature tuning:

Here $\bar{\mathcal{H}}$ is the target Entropy (usually $-\dim(\mathcal{A})$).

4.7 DDPG and TD3: Continuous Action Spaces

DDPG: Deep Deterministic Policy Gradient (Lillicrap et al., 2015)

DQN applies only to discrete action spaces. DDPG is an off-policy actor-critic algorithm that extends the ideas of DQN to continuous action spaces. You can think of it as "continuous control with DQN".

• Deterministic Actor: $\mu_\theta(s)$ outputs a continuous action directly
• Critic: $Q_\phi(s, a)$ evaluates the action value
• Uses DQN's Experience Replay and Target Network as they are
• Adds Ornstein-Uhlenbeck noise to the action for exploration

Actor update (Deterministic Policy Gradient):

TD3: Twin Delayed DDPG (Fujimoto et al., 2018)

DDPG had problems with Q-value overestimation and training instability. TD3 solves them with three techniques.

1. Twin Q-networks: uses the smaller of two Q-networks as the target → eases overestimation

2. Delayed Policy Updates: updates the Actor less often than the Critic (for example, 1 Actor update per 2 Critic updates) → the policy is updated on the basis of more accurate Q-values

3. Target Policy Smoothing: adds clipped noise to the target action → smooths the target Q-value

5. Model-based RL

Every algorithm covered so far is a model-free method — it learns directly from experience without knowing the environment model. Model-based RL learns a dynamics model of the environment and uses it to learn more efficiently.

5.1 The Advantages and Challenges of Model-based RL

Advantages:

• Sample efficiency: it can sharply reduce the number of real interactions with the environment, because it can generate a large volume of virtual experience through simulation ("imagination") inside the learned model.
• Planning: it can "simulate" the future before acting and search for the optimal sequence of actions.
• Transfer: the learned model can be reused across a variety of tasks.

Challenges:

• If the model is inaccurate (model error), that error accumulates during planning and performance degrades badly.
• Learning the dynamics of a complex environment accurately is itself hard.

5.2 World Models (Ha & Schmidhuber, 2018)

World Models is the pioneering work that models the environment in a latent space compressed by a VAE (Variational Autoencoder).

• Vision Model (V): encodes high-dimensional observations into a low-dimensional latent vector $z$ with a VAE
• Memory Model (M): predicts the dynamics in latent space with an RNN (MDN-RNN): $P(z_{t+1}|z_t, a_t, h_t)$
• Controller (C): a simple linear model that takes $z_t$ and $h_t$ and decides the action

The key innovation is that the Controller can be trained inside a "dream". Virtual episodes are generated inside the world model that V and M have learned, and the Controller learns from that virtual experience.

5.3 MuZero (Schrittwieser et al., 2020)

MuZero is a model-based RL algorithm that inherits from DeepMind's AlphaZero and extends it so that it can learn without knowing the rules of the environment.

AlphaZero had to know the rules of the game (a perfect simulator). MuZero learns the following three functions.

1. Representation Function $h_\theta$: maps observations $o$ to a latent state $s$: $s^0 = h_\theta(o_1, \ldots, o_t)$
2. Dynamics Function $g_\theta$: predicts transitions in latent state: $r^k, s^k = g_\theta(s^{k-1}, a^k)$
3. Prediction Function $f_\theta$: predicts the policy and value from a latent state: $p^k, v^k = f_\theta(s^k)$

Using these three functions it runs Monte Carlo Tree Search (MCTS) in latent space. Even without knowing the real transition rules of the environment, it can simulate the future with the learned dynamics function.

MuZero matched AlphaZero on Go, Chess and Shogi while additionally reaching SOTA performance on Atari games.

5.4 Dreamer (Hafner et al., 2020, 2021, 2023)

The Dreamer series learns a World Model in latent space and learns a policy through imagination inside that model.

Dreamer v3 (Hafner et al., 2023) showed the generality of learning more than 150 varied tasks (Atari, DMC, Minecraft and others) successfully with a single algorithm and one set of hyperparameters.

Its core components:

• RSSM (Recurrent State Space Model): a world model that combines a deterministic state with a stochastic state
• Actor-Critic in Imagination: "imagines" trajectories inside the learned world model and runs actor-critic learning inside that imagination
• Symlog Predictions: a normalization technique for learning independently of the scale of the reward

5.5 Autonomous Algorithm Discovery: DiscoRL (2025)

DeepMind research published in Nature in 2025 opened a new paradigm for RL. It showed that a machine can discover a SOTA RL algorithm by itself. The discovered rule, named DiscoRL (Discovered RL), is expressed as a neural network; it is more flexible than an algorithm in the form of a conventional mathematical equation, and it outperformed hand-designed algorithms on a range of benchmarks.

6. RLHF: Reinforcement Learning from Human Feedback

RLHF is the most impactful recent application of reinforcement learning. It is the core training pipeline of modern LLMs such as ChatGPT, Claude and Gemini, and it aligns models by using human preference as the reward signal.

6.1 The RLHF Pipeline

The RLHF pipeline consists of three stages.

Stage 1: Supervised Fine-Tuning (SFT)

Fine-tune a pre-trained LLM on high-quality, human-written conversation data. At this stage the model acquires the basic ability to follow instructions.

Stage 2: Reward Model (RM) Training

The SFT model generates several responses to the same prompt, and human labellers compare them and rank them by preference. The reward model is trained on this preference data.

For a preference pair $(y_w, y_l)$ (where $y_w$ is preferred over $y_l$), a loss function based on the Bradley-Terry model is used.

Here $r_\psi$ is the reward model and $\sigma$ the sigmoid function.

Stage 3: RL Optimization (PPO Fine-tuning)

Using the trained reward model as the reward function, the LLM policy is optimized with the PPO algorithm. A KL Penalty is added so the policy does not drift too far from the SFT model.

Here $\beta$ is the coefficient controlling the strength of the KL penalty. Without this KL term the model can "hack" the reward model and produce unnatural output (Reward Hacking).

6.2 DPO: Direct Preference Optimization

Paper: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (Rafailov et al., 2023)

DPO is an innovative approach that cuts the complexity of RLHF drastically. The core insight is that the two stages of training a separate reward model and then performing RL can be merged into one.

Deriving the optimal policy of the KL-constrained optimization problem:

Solving this backwards for $r$:

Substituting this relation into the Bradley-Terry model (the partition constant $Z(x)$ cancels):

The advantages of DPO:

• No need to train a separate reward model
• No need for a complicated PPO training loop
• Simple to implement and stable to train
• Memory and compute costs are cut sharply

6.3 GRPO: Group Relative Policy Optimization

Paper: GRPO, first used in DeepSeek-R1, is the most talked-about RL alignment technique of 2025~2026.

GRPO removes the Value Network (the Critic), the core inefficiency of PPO. Instead it generates several responses in a group for a single prompt and estimates the Advantage from the relative reward statistics within that group.

Here $G$ is the group size and $r_i$ the reward of the $i$-th response. The advantages of this approach are:

• It removes the memory and compute overhead of the Value Network entirely
• Relative comparison within the group makes learning independent of the reward scale
• It is especially effective on tasks with verifiable answers, such as mathematics and coding

RLVR (Reinforcement Learning with Verifiable Rewards):

Since the success of DeepSeek-R1, the RLVR paradigm has risen fast. Instead of subjective human preference it uses objectively verifiable rewards — whether a maths answer is correct, whether code passes its tests. This approach has a very high capability-to-cost ratio, and it is being extended beyond mathematics and coding into other domains (chemistry, biology and so on).

6.4 Recent Trends in LLM Alignment (2025~2026)

The main advances in LLM alignment in 2025~2026 are as follows.

• OpenAI GPT-5 (August 2025): refined RLHF greatly reduced hallucination and improved factual accuracy
• Anthropic Claude Opus 4.5 (November 2025): trained by combining Constitutional AI with RLHF, and released a detailed 80-page constitution
• RLAIF (RL from AI Feedback): an AI rather than a human supplies the preference feedback, which improves scalability greatly
• Hybrid approaches: a trend of selectively combining PPO + DPO + GRPO depending on the task

7. Major Applications

7.1 AlphaGo / AlphaZero / AlphaFold

AlphaGo (Silver et al., 2016):

AlphaGo, which beat the human world champion Lee Sedol 4:1 at Go, announced the potential of reinforcement learning to the world. The state space of Go is about $10^{170}$, far more than the number of atoms in the universe ($10^{80}$).

AlphaGo's training pipeline:

1. SL Policy Network: supervised learning on 160,000 human game records (57% → professional level)
2. RL Policy Network: reinforcement learning through self-play against itself
3. Value Network: evaluates positions by predicting the game result
4. Monte Carlo Tree Search (MCTS): searches by combining the Policy Network and the Value Network

AlphaGo Zero (Silver et al., 2017):

Using no human game records at all, purely through self-play RL, it beat AlphaGo 100:0 in 3 days. That result showed the potential of RL dramatically.

AlphaZero (Silver et al., 2018):

The general-purpose version of AlphaGo Zero: the same algorithm surpassed the previous best programs at Go, chess and shogi alike. Its core consists of just two things: a Deep Neural Network + MCTS.

AlphaFold (Jumper et al., 2021) / AlphaFold 3 (Abramson et al., 2024):

Not RL directly, but a protein 3D structure prediction system inspired by the search strategies of RL, which solved a 50-year grand challenge in biology. AlphaFold 3 predicts the interaction structures of all biomolecules — not only proteins but DNA, RNA, ligands and more.

7.2 Robotics

Reinforcement learning is producing breakthrough results in robotics.

Sim-to-Real Transfer: a technique for transferring a policy learned by RL in a simulated environment onto a real robot. Domain Randomization varies the physics parameters of the simulation at random, so the policy learned is robust enough to work in the real environment too.

Major results:

• OpenAI Rubik's Cube (2019): succeeded in solving a Rubik's cube with a robot hand
• Quadruped Locomotion: 4-legged walking robots moving stably over varied terrain
• Dexterous Manipulation: learning object manipulation skills at a human level
• Google DeepMind Gemini Robotics (2025): released an RL-based robot manipulation model that greatly improves the ability to interact with the physical world

7.3 Game AI

Reinforcement learning has shown its most dramatic results in games.

• Atari Games (DQN, 2015): human-level performance on 49 games
• StarCraft II (AlphaStar, 2019): better results than more than 99.8% of human players
• Dota 2 (OpenAI Five, 2019): beat the world champion team 2:0
• Gran Turismo (GT Sophy, 2022): professional-driver-level performance in a racing game
• Minecraft (Dreamer v3, 2023): the first success at mining diamonds

7.4 Recommender Systems and Other Applications

Recommender systems: learn sequential recommendations that maximize a user's long-run satisfaction. Used at Netflix, YouTube, TikTok and elsewhere, they optimize long-run user engagement rather than short-run click-through rate.

Finance: RL is used for portfolio optimization, high-frequency trading, optimal order execution and more.

Autonomous driving: Waymo, Tesla and others are exploring RL-based approaches in their decision modules.

Healthcare: RL is being researched for dynamic treatment regimes in chronic disease, accelerating drug discovery, optimizing resource allocation and more.

Network/system optimization: it is producing real results in data-centre cooling optimization (Google), network routing, and chip design (Google TPU placement optimization).

8. PyTorch Code Examples

8.1 A DQN Implementation (CartPole)

The complete code for implementing DQN in the CartPole environment. It includes Experience Replay, the Target Network and $\varepsilon$-greedy exploration.

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import gymnasium as gym
from collections import deque
import random

# ============================================================
# Q-Network definition
# ============================================================
class QNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super(QNetwork, self).__init__()
        self.network = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )

    def forward(self, x):
        return self.network(x)

# ============================================================
# Replay Buffer
# ============================================================
class ReplayBuffer:
    def __init__(self, capacity=10000):
        self.buffer = deque(maxlen=capacity)

    def push(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))

    def sample(self, batch_size):
        batch = random.sample(self.buffer, batch_size)
        states, actions, rewards, next_states, dones = zip(*batch)
        return (
            np.array(states),
            np.array(actions),
            np.array(rewards, dtype=np.float32),
            np.array(next_states),
            np.array(dones, dtype=np.float32)
        )

    def __len__(self):
        return len(self.buffer)

# ============================================================
# DQN Agent
# ============================================================
class DQNAgent:
    def __init__(self, state_dim, action_dim, lr=1e-3, gamma=0.99,
                 epsilon_start=1.0, epsilon_end=0.01, epsilon_decay=500,
                 target_update=10, buffer_size=10000, batch_size=64):
        self.action_dim = action_dim
        self.gamma = gamma
        self.epsilon_start = epsilon_start
        self.epsilon_end = epsilon_end
        self.epsilon_decay = epsilon_decay
        self.target_update = target_update
        self.batch_size = batch_size
        self.steps_done = 0

        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        # Main network and target network
        self.q_network = QNetwork(state_dim, action_dim).to(self.device)
        self.target_network = QNetwork(state_dim, action_dim).to(self.device)
        self.target_network.load_state_dict(self.q_network.state_dict())
        self.target_network.eval()

        self.optimizer = optim.Adam(self.q_network.parameters(), lr=lr)
        self.buffer = ReplayBuffer(buffer_size)

    def get_epsilon(self):
        """Decay epsilon by exponential decay"""
        eps = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \
              np.exp(-1.0 * self.steps_done / self.epsilon_decay)
        return eps

    def select_action(self, state):
        """ε-greedy action selection"""
        self.steps_done += 1
        epsilon = self.get_epsilon()

        if random.random() < epsilon:
            return random.randrange(self.action_dim)  # exploration
        else:
            state_t = torch.FloatTensor(state).unsqueeze(0).to(self.device)
            with torch.no_grad():
                q_values = self.q_network(state_t)
            return q_values.argmax(dim=1).item()      # exploitation

    def update(self):
        """Sample a mini-batch from Experience Replay and update the Q-Network"""
        if len(self.buffer) < self.batch_size:
            return

        states, actions, rewards, next_states, dones = self.buffer.sample(self.batch_size)

        states_t = torch.FloatTensor(states).to(self.device)
        actions_t = torch.LongTensor(actions).to(self.device)
        rewards_t = torch.FloatTensor(rewards).to(self.device)
        next_states_t = torch.FloatTensor(next_states).to(self.device)
        dones_t = torch.FloatTensor(dones).to(self.device)

        # Current Q-value: Q(s, a; θ)
        current_q = self.q_network(states_t).gather(1, actions_t.unsqueeze(1)).squeeze(1)

        # Target Q-value: r + γ * max_a' Q(s', a'; θ⁻)
        with torch.no_grad():
            next_q = self.target_network(next_states_t).max(dim=1)[0]
            target_q = rewards_t + self.gamma * next_q * (1 - dones_t)

        # Compute the loss and backpropagate
        loss = nn.MSELoss()(current_q, target_q)
        self.optimizer.zero_grad()
        loss.backward()
        # Gradient Clipping improves stability
        nn.utils.clip_grad_norm_(self.q_network.parameters(), max_norm=1.0)
        self.optimizer.step()

        return loss.item()

    def update_target_network(self):
        """Sync the target network with the main network parameters"""
        self.target_network.load_state_dict(self.q_network.state_dict())

# ============================================================
# Training loop
# ============================================================
def train_dqn(num_episodes=500, render=False):
    env = gym.make("CartPole-v1")
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.n

    agent = DQNAgent(state_dim, action_dim)
    episode_rewards = []

    for episode in range(num_episodes):
        state, _ = env.reset()
        total_reward = 0
        done = False

        while not done:
            action = agent.select_action(state)
            next_state, reward, terminated, truncated, _ = env.step(action)
            done = terminated or truncated

            agent.buffer.push(state, action, reward, next_state, done)
            agent.update()

            state = next_state
            total_reward += reward

        # Periodic target network update
        if episode % agent.target_update == 0:
            agent.update_target_network()

        episode_rewards.append(total_reward)

        if (episode + 1) % 50 == 0:
            avg_reward = np.mean(episode_rewards[-50:])
            print(f"Episode {episode+1}, Avg Reward (last 50): {avg_reward:.1f}, "
                  f"Epsilon: {agent.get_epsilon():.3f}")

    env.close()
    return agent, episode_rewards

if __name__ == "__main__":
    agent, rewards = train_dqn()

8.2 A PPO Implementation (CartPole)

The complete implementation of the PPO-Clip algorithm. It includes the Actor-Critic network, GAE and the Clipped Surrogate Objective.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Categorical
import numpy as np
import gymnasium as gym

# ============================================================
# Actor-Critic network
# ============================================================
class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=64):
        super(ActorCritic, self).__init__()

        # Actor (policy network)
        self.actor = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, action_dim),
            nn.Softmax(dim=-1)
        )

        # Critic (value network)
        self.critic = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, 1)
        )

    def forward(self, state):
        action_probs = self.actor(state)
        state_value = self.critic(state)
        return action_probs, state_value

    def act(self, state):
        """Action selection (sampling)"""
        action_probs, state_value = self.forward(state)
        dist = Categorical(action_probs)
        action = dist.sample()
        return action.item(), dist.log_prob(action), state_value

    def evaluate(self, states, actions):
        """Evaluate a stored batch of experience"""
        action_probs, state_values = self.forward(states)
        dist = Categorical(action_probs)
        log_probs = dist.log_prob(actions)
        entropy = dist.entropy()
        return log_probs, state_values.squeeze(-1), entropy

# ============================================================
# PPO Agent
# ============================================================
class PPOAgent:
    def __init__(self, state_dim, action_dim, lr=3e-4, gamma=0.99,
                 lam=0.95, clip_epsilon=0.2, epochs=10, batch_size=64,
                 entropy_coef=0.01, value_coef=0.5):
        self.gamma = gamma
        self.lam = lam
        self.clip_epsilon = clip_epsilon
        self.epochs = epochs
        self.batch_size = batch_size
        self.entropy_coef = entropy_coef
        self.value_coef = value_coef

        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.policy = ActorCritic(state_dim, action_dim).to(self.device)
        self.optimizer = optim.Adam(self.policy.parameters(), lr=lr)

        # Experience storage
        self.states = []
        self.actions = []
        self.log_probs = []
        self.rewards = []
        self.dones = []
        self.values = []

    def select_action(self, state):
        state_t = torch.FloatTensor(state).to(self.device)
        with torch.no_grad():
            action, log_prob, value = self.policy.act(state_t)
        return action, log_prob.item(), value.item()

    def store_transition(self, state, action, log_prob, reward, done, value):
        self.states.append(state)
        self.actions.append(action)
        self.log_probs.append(log_prob)
        self.rewards.append(reward)
        self.dones.append(done)
        self.values.append(value)

    def compute_gae(self, next_value):
        """Compute the Generalized Advantage Estimation (GAE)"""
        advantages = []
        gae = 0
        values = self.values + [next_value]

        for t in reversed(range(len(self.rewards))):
            delta = self.rewards[t] + self.gamma * values[t + 1] * (1 - self.dones[t]) - values[t]
            gae = delta + self.gamma * self.lam * (1 - self.dones[t]) * gae
            advantages.insert(0, gae)

        advantages = torch.FloatTensor(advantages).to(self.device)
        returns = advantages + torch.FloatTensor(self.values).to(self.device)
        return advantages, returns

    def update(self, next_value):
        """Perform the PPO-Clip update"""
        advantages, returns = self.compute_gae(next_value)

        # Tensor conversion
        states_t = torch.FloatTensor(np.array(self.states)).to(self.device)
        actions_t = torch.LongTensor(self.actions).to(self.device)
        old_log_probs_t = torch.FloatTensor(self.log_probs).to(self.device)

        # Advantage normalization (variance reduction)
        advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)

        # Iterate over the PPO epochs
        for _ in range(self.epochs):
            # Mini-batch training over the full data
            indices = np.arange(len(self.states))
            np.random.shuffle(indices)

            for start in range(0, len(self.states), self.batch_size):
                end = start + self.batch_size
                batch_idx = indices[start:end]

                # Re-evaluate under the current policy
                new_log_probs, values, entropy = self.policy.evaluate(
                    states_t[batch_idx], actions_t[batch_idx]
                )

                # Policy ratio: r_t(θ) = π_θ(a|s) / π_θ_old(a|s)
                ratio = torch.exp(new_log_probs - old_log_probs_t[batch_idx])

                # PPO-Clip objective
                surr1 = ratio * advantages[batch_idx]
                surr2 = torch.clamp(ratio, 1 - self.clip_epsilon,
                                    1 + self.clip_epsilon) * advantages[batch_idx]
                policy_loss = -torch.min(surr1, surr2).mean()

                # Value Loss (MSE)
                value_loss = nn.MSELoss()(values, returns[batch_idx])

                # Entropy Bonus (encourages exploration)
                entropy_loss = -entropy.mean()

                # Total loss
                loss = (policy_loss
                        + self.value_coef * value_loss
                        + self.entropy_coef * entropy_loss)

                self.optimizer.zero_grad()
                loss.backward()
                nn.utils.clip_grad_norm_(self.policy.parameters(), max_norm=0.5)
                self.optimizer.step()

        # Reset the experience buffer
        self.states.clear()
        self.actions.clear()
        self.log_probs.clear()
        self.rewards.clear()
        self.dones.clear()
        self.values.clear()

# ============================================================
# Training loop
# ============================================================
def train_ppo(num_episodes=500, update_interval=2048):
    env = gym.make("CartPole-v1")
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.n

    agent = PPOAgent(state_dim, action_dim)
    episode_rewards = []
    total_steps = 0

    state, _ = env.reset()
    current_episode_reward = 0

    for step in range(1, 200001):
        action, log_prob, value = agent.select_action(state)
        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated

        agent.store_transition(state, action, log_prob, reward, done, value)
        current_episode_reward += reward
        total_steps += 1

        if done:
            episode_rewards.append(current_episode_reward)
            current_episode_reward = 0
            state, _ = env.reset()

            if len(episode_rewards) % 50 == 0:
                avg = np.mean(episode_rewards[-50:])
                print(f"Episode {len(episode_rewards)}, "
                      f"Avg Reward (last 50): {avg:.1f}, "
                      f"Total Steps: {total_steps}")
        else:
            state = next_state

        # Run the PPO update every so many steps
        if step % update_interval == 0:
            with torch.no_grad():
                state_t = torch.FloatTensor(state).to(agent.device)
                _, next_value = agent.policy(state_t)
                next_value = next_value.item()
            agent.update(next_value)

    env.close()
    return agent, episode_rewards

if __name__ == "__main__":
    agent, rewards = train_ppo()