LabHub

Blog

How Reasoning Models Are Built — From Chain-of-Thought to GRPO, in Twelve Papers

2026-09-06 · 12 min read · #ai#llm#reasoning#reinforcement-learning#grpo#paper-review#deepseek

한국어English日本語

Introduction — What "thinking longer" actually means

Since late 2024, model names started carrying the word reasoning: o1, R1, QwQ, Qwen3's thinking mode. The announcements usually say the same thing. "It thinks before it answers."

What is rarely explained is what that means at the level of implementation. No parameters were added. No new architecture appeared. The transformer is unchanged. The only thing that changed is what the model was trained to emit.

This article traces that change through twelve papers. There is a single thread running from the prompting tricks of 2022 to the reinforcement learning pipelines of 2025. At the end, we work out how far you can reproduce this on a single 24GB consumer GPU.

By the end you should be able to answer three questions precisely:

Part 1 — When we still prompted our way to reasoning (2022)

Chain-of-Thought: eight examples that changed the field

It starts with Chain-of-Thought Prompting (Wei et al., 2022) from Google Brain. The idea is almost disappointingly simple. When you give few-shot examples, don't just show the answer. Show the working as well.

Q: Roger has 5 tennis balls. He buys 2 cans of 3 balls each. How many now?
A: Roger started with 5. Two cans of 3 balls is 6 balls. 5 + 6 = 11. The answer is 11.

PaLM 540B went from 17.9% to 58.1% on GSM8K (grade-school word problems). Nothing about the model was touched — only the prompt.

One observation from that paper matters more than the numbers. The effect only appeared in sufficiently large models. Below roughly 10B parameters, chain-of-thought made things worse. If a model cannot produce a coherent intermediate step, asking it for intermediate steps only multiplies the nonsense.

Self-Consistency: forty samples instead of one

The follow-up, Self-Consistency (Wang et al., 2022), proposes something even simpler. Raise the temperature, sample forty answers, and take the most common one.

GSM8K went from 58% to 74%. This is the first clear appearance of the trade that would later be called test-time scaling: spend more compute at inference, get more accuracy.

But every method of this era shared a ceiling. A prompt can only elicit an ability the model already has; it cannot create one. The next step was to train the ability itself.

Part 2 — Turning thoughts into training data (2022–2023)

STaR: teaching yourself with your own solutions

STaR: Self-Taught Reasoner (Zelikman et al., 2022) reads, in hindsight, as a direct ancestor of R1. The loop is four lines:

  1. Let the model solve problems, producing a rationale.
  2. Keep only the rationales that reached the correct answer.
  3. Fine-tune on what survived.
  4. Repeat.

For problems it got wrong, feed the answer as a hint, let it produce a rationale backwards (rationalisation), and keep that too. No human-written solution is needed anywhere. You only need the answers.

That property — answers are enough — becomes, three years later, the thing that turns the field over, under the name verifiable rewards.

Let's Verify Step by Step: scoring the process

OpenAI's Let's Verify Step by Step (Lightman et al., 2023) asks a different question. If you are picking one solution out of many candidates, what should you look at?

Two graders are compared:

GraderTraining signalLabels required
ORM (outcome reward model)Was the final answer right?One answer
PRM (process reward model)Is each step right?A human label per step

They paid for 800K step-level human labels (PRM800K), and PRM clearly beat ORM on MATH: 78.2% against 72.4%.

The reason is intuitive. Outcome rewards cannot filter out solutions that were right by accident. A derivation that flips a sign in step 3, flips it back in step 5, and lands on the right answer is a perfect score to an ORM. Train on those and the model learns the wrong habits.

The paper's conclusion was clear: supervise the process. Two years later DeepSeek reported the opposite. We will come back to that in Part 5.

Part 3 — Reinforcement learning turns the table (2024–2025)

o1: buy accuracy with inference time

OpenAI's o1 (September 2024) disclosed no method, but it published one chart. Training compute on one axis, test-time compute on another, AIME accuracy on the vertical. Both were straight lines on a log scale.

The message was this: there is a second axis besides model size. Let the model think longer before answering. And that "thinking longer" is learned through reinforcement learning.

DeepSeek-R1-Zero: pure RL, no supervised data at all

DeepSeek-R1 (DeepSeek-AI, January 2025) published the method. The striking part is the R1-Zero ablation: they took the base model (DeepSeek-V3-Base) and ran reinforcement learning without a single line of human reasoning data.

There were only two rewards:

That is all. And as training progressed, the model lengthened its own responses. Nobody asked it to. Thinking longer simply produced more correct answers, so the optimiser went that way. AIME 2024 accuracy rose from 15.6% to 71.0%.

The paper's "aha moment" appears here. A mid-training checkpoint began emitting sentences like:

Wait, wait. Wait. That's an aha moment I can flag here.
Let's reevaluate this step-by-step to identify if the correct sum can be...

Reviewing its own work and backtracking emerged without being taught. It was a strategy that reinforcement learning found.

GRPO: why the value network had to go

R1 uses GRPO (Group Relative Policy Optimization). Understanding why they avoided PPO tells you a lot about the practical constraints of this field.

PPO trains a separate value network to estimate per-token advantage. That network is roughly the size of the policy. Training a 671B model while also training a 671B critic is not a reasonable proposition.

GRPO removes the critic. Instead, it samples a group of G answers to the same question and scores them relative to each other.

Sample 16 answers to one question → rewards r_1 … r_16
A_i = (r_i - mean(r)) / std(r)          ← this is the advantage

If an answer beats its group's average, push the policy toward it; if it is worse, push away. The learned baseline is replaced by a sample statistic.

The objective keeps PPO's clipping shape and adds a KL penalty:

J(θ) = E[ min( ρ_i · A_i,  clip(ρ_i, 1-ε, 1+ε) · A_i ) ] − β · D_KL(π_θ ‖ π_ref)
       ρ_i = π_θ(o_i|q) / π_θ_old(o_i|q)

Memory drops by more than half, and in domains with verifiable rewards — maths, code — it performs as well as PPO. This is a case where resource constraints, not theoretical elegance, decided the algorithm.

R1's four-stage pipeline

R1-Zero scored well but was unusable. It was hard to read and mixed Chinese and English inside a single answer. So the final R1 goes through four stages:

StageWhat happensWhy
1. Cold-start SFTFine-tune on a few thousand long CoT samplesInstall a readable output format first
2. Reasoning RLGRPO with accuracy, format and language-consistency rewardsPush reasoning ability up
3. Rejection-sampling SFTGenerate 600K samples with the stage-2 model, keep the good ones, retrainRecover general ability
4. All-domain RLRL again, now including helpfulness and harmlessnessMake it usable by people

Stage 3 deserves attention. Training only on reasoning breaks everything else. You end up with a model that solves olympiad geometry but cannot write an email. So 600K reasoning samples are mixed with 200K general samples to pull the distribution back.

Kimi k1.5: putting a price on length

Kimi k1.5 (Moonshot AI, 2025), released the same month, attacks the same problem from another angle. Longer responses raise accuracy, but they also raise cost. So they put a length penalty directly in the reward.

len_reward = short and correct → positive,  long and wrong → negative

They also propose long2short, transferring a long-thinking model's ability into a short-thinking one. In production this axis matters. Optimise for accuracy alone and you cannot pay the token bill.

Part 4 — Doing it cheaply (2025)

s1: a thousand samples and the word "Wait"

Stanford's s1: Simple test-time scaling (Muennighoff et al., 2025) is the most satisfying paper in this area. It uses no reinforcement learning at all.

That budget forcing alone makes the model re-examine its answer and correct errors. It beat o1-preview on AIME24.

The implication is heavy. Long-form reasoning may already be acquired during pre-training, and post-training may be little more than the key that unlocks it. Shanghai AI Lab's LIMO (Ye et al., 2025) reached the same conclusion with 817 samples.

Distillation: moving a large model's thinking into a small one

The most practically important section of the R1 paper is the distillation study. They fine-tuned Qwen and Llama on 800K R1-generated samples. A 1.5B model beat GPT-4o on AIME (28.9% vs 9.3%).

The controlled comparison is more interesting still. Running RL directly on Qwen-32B was worse than distilling from R1's outputs.

Distilling the reasoning patterns a large model discovered is cheaper and stronger than training a small model with reinforcement learning directly.

A small model rarely produces a good solution for RL to reinforce in the first place. What exploration cannot find, learning cannot capture.

RLVR: removing the reward model entirely

Allen AI's Tülu 3 (Lambert et al., 2024) crystallised the term RLVR — Reinforcement Learning with Verifiable Rewards. Rather than estimating reward with a neural network, verify it with a program:

With no reward model there is less surface for reward hacking, and grading is deterministic, so results reproduce. This is why reasoning training succeeded first in maths and code — those are the domains where grading is already automatic.

Part 5 — The walls you hit in practice

Reward hacking

A model optimises the reward function, not your intention. Reported cases:

That is precisely why R1 avoided neural reward models: at scale, reward hacking made the whole pipeline more complicated to defend than it was worth.

Why PRM lost in practice

Part 2 said PRM beats ORM. Yet R1 did not use PRM, and gave three reasons:

  1. Defining a "step" in general reasoning is hard. A proof splits into lines; an open-ended problem does not.
  2. Judging an intermediate step automatically is hard. Human labels do not scale, and a model labeller injects its own errors.
  3. The PRM itself becomes a target. The policy learns the phrasings that score well step by step.

The summary is this: PRM is correct but expensive. A verifiable outcome reward is crude, but cheap and hard to hack. At scale, crude and cheap won.

Overthinking

Do NOT Think That Much for 2+3=? (Chen et al., 2025) points at the other failure mode. Reasoning models spend thousands of tokens on trivial questions, exploring several approaches to compute 2+3.

There is a regime where cost rises and accuracy does not. That is why current models ship a switch for the thinking budget — Qwen3's thinking mode, the enable_thinking flag. LabHub turns that flag off when it calls its local model: script generation does not need long deliberation, and leaving it on triples the token count.

Part 6 — How far does a single 24GB card get you?

The homelab behind this blog has one RTX 5090 Laptop with 24GB. Let us work out what fits.

Inference is comfortable

The card currently runs an NVFP4 quantisation of Qwen3.8-27B under vLLM. Measured:

ItemValue
Weights16.2GB (NVFP4)
Context length16K
Concurrent requests4
Single-request throughput39 tok/s
Aggregate at concurrency 4138 tok/s

A 27B-class model fits in 24GB because of 4-bit quantisation. For inference there is room to spare.

Training is a different calculation

Stack up the memory a GRPO run needs, assuming full bf16 fine-tuning of a 7B model:

ItemSize
Policy weights (bf16)14GB
Optimizer state (AdamW, two fp32 moments)56GB
Gradients14GB
Reference model (frozen, for KL)14GB
Activations + KV cache (16 samples × 4K tokens)10GB+
Total~108GB

24GB does not start. So a homelab makes three adjustments:

  1. Switch to LoRA. Optimizer state attaches only to the adapters, dropping 56GB below 1GB, and the reference model becomes the same weights with adapters disabled, dropping another 14GB to zero.
  2. Shrink the model. Use the 1.5B–4B class. As R1's distillation study showed, small models are better distilled than RL-trained — but if the goal is to understand the pipeline, small is enough.
  3. Separate generation. GRPO spends most of its wall-clock time generating. Run vLLM as a separate process to serve rollouts and train elsewhere. Both verl and TRL support this split.

With that, Qwen3-1.7B + LoRA rank 16 + group size 8 + 1K generation length fits in 24GB.

Write the reward function first

If you actually try this, write the grader before the training loop. For GSM8K it looks like this:

import re

ANSWER = re.compile(r"<answer>\s*(-?[\d,]+(?:\.\d+)?)\s*</answer>")

def reward(completion: str, gold: str) -> float:
    """Verifiable reward: 0.2 for format, 1.0 for the answer. No neural network."""
    score = 0.0
    if "<think>" in completion and "</think>" in completion:
        score += 0.1
    m = ANSWER.search(completion)
    if not m:
        return score                      # cannot even hold the format
    score += 0.1
    got = m.group(1).replace(",", "")
    return score + (1.0 if got == gold.replace(",", "") else 0.0)

Those twenty lines are the heart of the pipeline. If they are sloppy, the model will find exactly where. Note that this function only reads the first <answer> tag — once the policy discovers that, it learns to list many candidates and put the most plausible one first. This is the kind of thing you actually run into.

Conclusion — what was the essence?

Retrace the lineage and it converges on one sentence. A reasoning model is not a new architecture; it is long output optimised toward a gradeable objective.

One constraint runs through all three periods. The method works only where grading can be automated. It is no accident that maths and code fell first. And the frontier today is exactly this question: how do we make the ungradeable domains gradeable?

🧠 Comprehension check

1. Why could GRPO drop PPO's value network?

Because it samples a group of answers to the same question and normalises the advantage by that group's mean and standard deviation. The learned baseline is replaced by a sample statistic, so there is no need to train a critic the size of the policy.

2. Give at least two reasons PRM wins on benchmarks but is not used in large-scale RL.

Step boundaries are ill-defined in general reasoning; judging intermediate steps automatically is hard and human labels do not scale; and the PRM itself becomes a reward-hacking target, which complicates the pipeline.

3. What happens if you remove stage 3 (rejection-sampling SFT) from R1's pipeline?

Reasoning holds up, but general abilities — writing, world knowledge, role-play — degrade. Training only on reasoning narrows the distribution, so general data must be mixed back in.

4. What is the single biggest reason full GRPO fine-tuning of a 7B model does not fit in 24GB?

The optimizer state. AdamW keeps two fp32 moments per parameter, about 56GB for 7B — far larger than the weights themselves (14GB). LoRA solves it by shrinking that term to adapter size.

References

Comments

No comments yet.

Sign in to leave a comment