LabHub

Blog

Modern Reinforcement Learning Ecosystems 2026 Deep Dive - RLlib (Anyscale) · Stable-Baselines3 · Tianshou · CleanRL · OpenSpiel (DeepMind) · Gymnasium (Farama) · Acme · PufferLib · Pearl (Meta) · TorchRL

2026-05-16 · 21 min read · #reinforcement-learning#rllib#stable-baselines-3#tianshou#cleanrl#openspiel#gymnasium#acme#pufferlib#pearl-meta#torchrl#english

한국어English日本語

Intro — May 2026, RL Has Entered Its Second Golden Age

If 2018-2020 (AlphaStar, OpenAI Five, AlphaFold 1) was RL's first golden age, 2024-2026 is the second. Three triggers stand out. First, DeepSeek-R1's GRPO and OpenAI's o1/o3 line of Test-Time Compute put RL back at the center of LLM reasoning. Second, NVIDIA GR00T, Isaac Lab, and Cosmos (covered in iter69) pushed robotics simulation and sim-to-real into commercial deployment. Third, Wayve GAIA-2 (iter97) and Tesla FSD v13 brought RL-based evaluation and policy learning back into autonomous driving.

This article is not a marketing catalog. We catalog the libraries, environments, algorithms, and benchmarks that are actually used in RL production and research as of May 2026. All code snippets follow current APIs. We compare RLlib, Stable-Baselines3, Tianshou, CleanRL, TorchRL, OpenSpiel, Acme, Pearl, PufferLib, Gymnasium, PettingZoo, MuJoCo, Brax, and Isaac Lab in one place.

The 2026 RL Landscape — Four Axes

Start with the big picture. The 2026 RL ecosystem decomposes along four axes.

  1. Algorithm library: policy/value networks, trainers, replay buffers.
  2. Environment API: the abstraction that exposes state, action, reward to libraries.
  3. Simulator: physics, games, robots, cities, autonomous driving — the domain simulation.
  4. Distributed runtime: actor-learner topology, actor pools, replay sharding.

In 2018, a single library tried to do all four (own baselines, own Gym envs, own Atari wrappers, own distributed runtime). In 2026 these axes are cleanly separated. The typical stack is Gymnasium (environment API) + PufferLib (env compatibility shim) + RLlib/SB3/Tianshou (algorithms) + Ray/Slurm (distributed).

General-Purpose RL Library Market — Two Leaders Plus Four Challengers

The algorithm library market in May 2026 has two clear leaders.

The challenger pack has matured.

Each has its place. Production → RLlib. Paper baselines → SB3. Fast experimentation / paper reproduction → CleanRL. PyTorch-native multi-agent → TorchRL. Training stability + performance → Tianshou. TPU and JAX → Acme.

RLlib — Industrial RL on Ray

RLlib is the RL submodule of Ray, maintained by Anyscale. Its biggest strength in May 2026 is that distributed training actually works. The same API scales from a lightweight single machine to 1000+ actors.

A typical RLlib script:

import gymnasium as gym
import ray
from ray.rllib.algorithms.ppo import PPOConfig

ray.init()

config = (
    PPOConfig()
    .environment(env="CartPole-v1")
    .framework("torch")
    .training(gamma=0.99, lr=3e-4, train_batch_size=4000)
    .rollouts(num_rollout_workers=8)
    .resources(num_gpus=1)
)

algo = config.build()
for i in range(100):
    result = algo.train()
    print(f"iter={i} reward={result['episode_reward_mean']:.2f}")

algo.save(checkpoint_dir="/tmp/ppo_cartpole")
ray.shutdown()

RLlib has the deepest algorithm catalog: PPO, IMPALA, APPO, DQN, Rainbow, SAC, DDPG, TD3, MARWIL, BC, CQL, MARL via PettingZoo wrappers, and RLlib Offline for RLHF. The downside is deep abstractions — the initial learning curve is steep.

Stable-Baselines3 — The De Facto Research Baseline

SB3 is a PyTorch-based RL library maintained by the DLR-RM team (originally from the German Aerospace Center). It leads on usability and readability. More than half of baseline numbers in new RL papers are produced with SB3.

A complete SB3 PPO training run is four lines:

import gymnasium as gym
from stable_baselines3 import PPO

env = gym.make("LunarLander-v2")
model = PPO("MlpPolicy", env, verbose=1, learning_rate=3e-4, n_steps=2048)
model.learn(total_timesteps=1_000_000)
model.save("ppo_lunar")

SB3 officially ships:

SB3 is the standard recommendation for single-machine, medium-scale training. For distributed training, RLlib or TorchRL is the better fit.

Tianshou — Fast, Modular, Known for Training Stability

Tianshou is a PyTorch-based RL library started by the RL group at Tsinghua University. It has grown rapidly since 2020, and citations as a baseline at NeurIPS/ICLR have climbed sharply over 2024-2026. Its strengths are fast convergence and stable hyperparameters.

Tianshou's core abstractions split into Collector, Policy, and Trainer.

import gymnasium as gym
import tianshou as ts
import torch
from tianshou.utils.net.common import Net
from tianshou.utils.net.discrete import Actor, Critic

env = gym.make("CartPole-v1")
state_shape = env.observation_space.shape or env.observation_space.n
action_shape = env.action_space.n

net = Net(state_shape, hidden_sizes=[64, 64])
actor = Actor(net, action_shape)
critic = Critic(net)
optim = torch.optim.Adam(set(actor.parameters()) | set(critic.parameters()), lr=3e-4)

policy = ts.policy.PPOPolicy(actor, critic, optim, dist_fn=torch.distributions.Categorical)
train_envs = ts.env.DummyVectorEnv([lambda: gym.make("CartPole-v1") for _ in range(8)])
buf = ts.data.VectorReplayBuffer(20000, 8)
collector = ts.data.Collector(policy, train_envs, buf)
collector.collect(n_step=4000)

Tianshou is often cited first for algorithmic correctness. The downside compared to SB3 is shorter documentation.

CleanRL — One File Per Algorithm, Peak Research Reproducibility

CleanRL was started by Costa Huang (Vector Institute, now at Hugging Face) and finishes each algorithm in a single file. The PPO implementation lives entirely in ppo.py with almost no abstraction layer. Readability and reproducibility are unmatched.

As of May 2026, CleanRL provides these algorithms as single-file scripts:

W&B integration is built in, so running a script automatically logs metrics to the cloud. It is one of the first repos researchers clone when reproducing a paper or building a baseline.

TorchRL — Modern RL with PyTorch as a First-Class Citizen

TorchRL is an RL library built directly by Meta's PyTorch team. It stabilized in 2023 and accelerated through 2024-2026. PyTorch tensors and the TensorDict abstraction are first-class citizens, which makes it natural for PyTorch developers.

The core abstractions:

A short TorchRL snippet:

import torch
from torchrl.envs import GymEnv, TransformedEnv, ObservationNorm
from torchrl.modules import MLP, ProbabilisticActor
from torchrl.objectives import ClipPPOLoss

env = TransformedEnv(GymEnv("CartPole-v1"), ObservationNorm(in_keys=["observation"]))
actor_net = MLP(in_features=env.observation_spec["observation"].shape[-1], out_features=env.action_spec.shape[-1], num_cells=[64, 64])
actor = ProbabilisticActor(module=actor_net, in_keys=["observation"], out_keys=["action"])

loss_module = ClipPPOLoss(actor, critic_network=None, entropy_bonus=True)
optim = torch.optim.Adam(loss_module.parameters(), lr=3e-4)

The strength of TorchRL is multi-agent, offline, and meta-RL under one API. The downsides are deep abstractions and a still-evolving surface.

PFRL — Preferred Networks' Japanese PyTorch RL

PFRL is the PyTorch RL library from Preferred Networks (PFN, Tokyo). Its predecessor was the Chainer-based ChainerRL. As of May 2026 it backs many baseline runs at ICML and NeurIPS by Japanese teams.

PFRL's strengths are algorithmic breadth and reliable reproductions. DQN-family variants like Rainbow, IQN, R2D2, and NoisyNet are well covered, and Atari 50M training is validated end to end. The trade-off is that English-language documentation is shallower than SB3's.

TF-Agents — Google's TensorFlow RL Library

TF-Agents is a TensorFlow RL library built by Google. It remains active in May 2026 but has lost share to the PyTorch ecosystem. Inside Google and on TPUs, however, it is still the first choice. Several follow-up projects to AlphaGo / AlphaStar, plus the RL components inside Vertex AI Pipelines, sit on TF-Agents.

OpenSpiel — DeepMind's Game-Theoretic Multi-Agent Bundle

OpenSpiel is DeepMind's standard toolkit for game theory and multi-agent RL. It bundles 60+ games (chess, Go, poker, hex, Liar's Dice, Goofspiel, Hanabi, Catch the Cat, and more) with equilibrium-learning algorithms like PSRO, CFR, NFSP, and MMD.

import pyspiel

game = pyspiel.load_game("tic_tac_toe")
state = game.new_initial_state()
while not state.is_terminal():
    legal_actions = state.legal_actions()
    action = legal_actions[0]
    state.apply_action(action)
print(state.returns())

OpenSpiel is one of the canonical environments for multi-agent RL research. It covers card games (Hanabi), strategy games (Catan), auctions (Sealed-Bid Auction), and Liar's Dice in a single API.

Acme — DeepMind's Modular Researcher-Facing Framework

Acme is DeepMind's internal research framework, released to the public. The core abstraction is a clean separation of Actor + Learner + Replay. It supports both JAX and TF.

Acme's strength is expressing distributed topologies cleanly (R2D2, IMPALA, Ape-X). Combined with DeepMind's Reverb replay service, scaling out to thousands of actors becomes a normal API call.

Sister libraries in the same DeepMind stack:

Pearl — Meta's Production Decision-System RL Library

Pearl is an RL library released by Meta's Applied Research / Production RL team in late 2023. Its full name is PEarl (Production-Ready Reinforcement Learning AI Library). As of May 2026 it is squarely focused on online decision systems: ad bidding, content recommendation, notification timing, and similar.

Pearl emphasizes:

Inside Meta, parts of the ads, recommendations, and notifications stacks run on Pearl. The OSS release exposes the same abstractions to outside users.

Gymnasium — Farama Foundation's API Standard

OpenAI Gym went unmaintained after 2021, and the Farama Foundation forked it as Gymnasium, which is now the de facto standard. Every major RL library in May 2026 (SB3, RLlib, Tianshou, CleanRL, TorchRL) targets the Gymnasium API first.

The differences between Gym and Gymnasium are small but consequential.

import gymnasium as gym

env = gym.make("CartPole-v1", render_mode="rgb_array")
obs, info = env.reset(seed=42)
for _ in range(1000):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()
env.close()

Under the same Farama umbrella sit PettingZoo (multi-agent), MiniGrid (grid-world), MiniWorld (3D mini), Procgen (procedurally generated), and Highway-env (mini autonomous driving).

PettingZoo + MARL — The Multi-Agent RL API Standard

PettingZoo is Gymnasium's multi-agent sibling. It supports both the AEC (Agent Environment Cycle) API and the Parallel API. RLlib, Tianshou, and TorchRL all consume PettingZoo environments first-class.

Categories shipped with PettingZoo:

Sample code:

from pettingzoo.classic import chess_v6

env = chess_v6.env(render_mode="human")
env.reset(seed=42)
for agent in env.agent_iter():
    obs, reward, term, trunc, info = env.last()
    action = None if term or trunc else env.action_space(agent).sample()
    env.step(action)
env.close()

Multi-agent RL algorithms include MAPPO, IPPO, QMIX, MADDPG, and COMA, and libraries like RLlib MARL, MARLlib, and EPyMARL run on top of PettingZoo.

Atari, MuJoCo, DeepMind Control Suite — Classic Benchmarks Today

The standard RL benchmarks remain strong.

MuJoCo 3 has shipped a MJX JAX backend since 2024. The same model can run thousands of parallel rollouts on a single GPU, which is heavily cited in sim-to-real work.

import gymnasium as gym

env = gym.make("HalfCheetah-v5")
obs, info = env.reset(seed=0)
for _ in range(200):
    obs, r, term, trunc, info = env.step(env.action_space.sample())

In the same category: PyBullet (open-source physics engine, an alternative to MuJoCo) and Gazebo (ROS-integrated robotics simulator).

Brax — JAX-Based Differentiable Physics Simulator

Brax is a JAX-based physics simulator from Google. It is differentiable physics + GPU parallelism + JIT compilation, so RL throughput can be 100-1000x higher than CPU-based MuJoCo in some setups.

Brax environments are written to be MuJoCo-compatible: Ant, HalfCheetah, Humanoid, Walker2d, Hopper, Pusher, Reacher are all available under the same names.

import jax
import brax.envs
from brax.training.agents.ppo import train as ppo

env = brax.envs.create(env_name="ant", backend="positional")
make_inference_fn, params, _ = ppo.train(
    environment=env,
    num_timesteps=50_000_000,
    num_evals=10,
    reward_scaling=10,
    episode_length=1000,
    normalize_observations=True,
    action_repeat=1,
    unroll_length=5,
    num_minibatches=32,
    num_updates_per_batch=4,
    discounting=0.97,
    learning_rate=3e-4,
    entropy_cost=1e-2,
    num_envs=4096,
    batch_size=2048,
    seed=0,
)

The trick is running thousands of envs on the GPU at once — note num_envs=4096. Brax + Acme + RLax can finish 1B+ training steps in under a day on a single GPU.

NVIDIA Isaac Lab + Cosmos — The Industry Standard for Robotics Sim-to-Real

In 2023-2024 NVIDIA moved from IsaacGym to Isaac Lab. As of May 2026, Isaac Lab (absorbing the older OmniIsaac / IsaacGymEnvs) is the industry standard for robotics RL simulation. Cosmos (covered in iter69) provides a generative world model layer for sim-to-real on top.

Isaac Lab features:

Many humanoid companies — Boston Dynamics, Agility Robotics, Figure AI, 1X — have publicly stated that Isaac Lab is their standard for RL policy training. Common pairings are Isaac Lab + RSL-RL (ETH Zurich) and Isaac Lab + Stable-Baselines3 or in-house PPO.

VizDoom, MineRL, MineDojo, Crafter, NetHack — The Procedural-Env Comeback

Procedural and open-world RL benchmarks regained attention in 2024-2026.

import minerl

env = minerl.make("MineRLObtainDiamond-v0")
obs = env.reset()
done = False
while not done:
    action = env.action_space.sample()
    obs, r, done, info = env.step(action)

These environments are also heavily used for LLM-based agent evaluation. Voyager (NVIDIA, a GPT-4 Minecraft agent), DEPS, and JARVIS-1 all live on top of MineDojo.

MetaWorld, RoboCasa, LIBERO — Robotics Task Suites

Robotics RL is evaluated not against a single simulator but against task suites paired with simulators.

MetaWorld usage:

import metaworld
import random

mt10 = metaworld.MT10()
training_envs = []
for name, env_cls in mt10.train_classes.items():
    env = env_cls()
    task = random.choice([t for t in mt10.train_tasks if t.env_name == name])
    env.set_task(task)
    training_envs.append(env)

PufferLib — Env Compatibility and a Throughput Boost

PufferLib is a library that unifies environment APIs and throws in GPU vectorization. It has grown fast since 2024. The core claim: keep your CPU envs, then run 100-1000 of them at max throughput from a single GPU host.

PufferLib delivers all of these in one place:

import pufferlib
import pufferlib.emulation
import pufferlib.vectorization

env_creator = lambda: pufferlib.emulation.GymnasiumPufferEnv("CartPole-v1")
vecenv = pufferlib.vectorization.Multiprocessing(env_creator, num_envs=64)
obs = vecenv.reset()

PufferLib + CleanRL, or PufferLib + RLlib, commonly delivers a 2-10x speedup in practice.

LLM + RL — From PPO to GRPO, plus TRL and the Training Infrastructure

In late 2024, DeepSeek-R1 was trained with a PPO variant called GRPO (Group Relative Policy Optimization), and GRPO variants have since become the standard for LLM reasoning model training.

A short summary of PPO vs. GRPO:

AspectPPOGRPO
critic networkrequirednot used
advantage estimationGAE + value baselinegroup-normalized rewards
memoryactor + criticactor only
stabilitywell-known stable regionsgroup size g is the key knob
LLM fitstandard but critic is heavyde facto standard for reasoning RL

TRL (Hugging Face Transformers Reinforcement Learning) is the LLM-side RL library (covered in iter62 LLM fine-tuning). As of May 2026 it supports PPO, DPO, GRPO, KTO, ORPO, and reward-model training.

from trl import GRPOTrainer, GRPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

config = GRPOConfig(
    learning_rate=1e-6,
    num_generations=8,
    max_prompt_length=512,
    max_completion_length=1024,
    beta=0.04,
)

trainer = GRPOTrainer(
    model=model,
    args=config,
    reward_funcs=[lambda completions, **kw: [len(c) for c in completions]],
    train_dataset=...,
    processing_class=tokenizer,
)
trainer.train()

The LLM RL stack has a different shape than classic RL. Rollouts run on vLLM or SGLang, training runs on DeepSpeed / FSDP, and algorithms are implemented in TRL, OpenRLHF, or verl. A training cluster of 16-128 GPUs is typical.

The 2026 Algorithm Landscape — PPO Still Rules, with New Companions

As of May 2026, here is the standing per algorithm.

Benchmark Landscape — MuJoCo / Atari / DM Control / NetHack / Crafter

Standard benchmarks for algorithm comparison:

Korean RL Research — KAIST, SNU, NCSOFT, Krafton, NAVER

Korean RL research has exploded since the 2020s. Key groups:

Korean RL is strong on game RL and industrial decision systems, and academically active on theory and meta-RL.

Japanese RL Research — Preferred Networks, DeepMind Tokyo, Sony AI, NTT

Japanese RL research is strong in games, robotics, and industrial applications.

Gran Turismo Sophy is one of the few cases that combines a high-fidelity simulator with a comparable human baseline — one of the most visible demonstrations of what industrial RL can do.

Real-World Deployments — From AlphaGo to AlphaChip and Boston Dynamics

Notable RL deployments:

OpenAI Spinning Up and Educational Resources

Core resources for getting started:

A natural path is read Spinning Up code → walk through CleanRL's ppo.py line by line → experiment with SB3 / Tianshou → scale up with RLlib / TorchRL to distributed and multi-agent.

Tool Selection Guide — Recommendations by Scenario

A short cheat sheet:

The most important question when picking a tool is "where does environment throughput bottleneck?" CPU envs are fine with SB3 / Tianshou. GPU sims fit Isaac Lab / Brax. Distributed jobs go to RLlib / Acme. LLM training uses TRL.

Wrap-Up — May 2026, RL Is a Stack, Not a Single Tool

The conclusion of this article is simple. The single-tool era of RL is over. You assemble a stack of environment + library + simulator + distributed runtime that fits your domain.

Two big trends to close on. First, the split between LLM-RL and classic RL. LLM-RL has become its own area — TRL + vLLM + GRPO/DPO. Classic RL continues with SB3 / RLlib / Tianshou + Gymnasium + simulators. Second, robotics RL has crossed into commercial deployment. Isaac Lab + Cosmos + Diffusion Policy + Q-Transformer is now the default stack for several humanoid and manipulator companies.

Do not overspend time picking tools. As long as you nail a consistent environment interface, a stable distributed runtime, and good experiment tracking, 90% of the problem is solved. The remainder is domain-specific tuning.

References

Read the original: https://www.youngju.dev/blog/culture/2026-05-16-modern-reinforcement-learning-ecosystems-2026-rllib-stable-baselines-3-tianshou-cleanrl-openspiel-gymnasium-acme-pufferlib-deep-dive.en

Comments

No comments yet.

Sign in to leave a comment