Frontier Reasoning & GRPO

Compare answers to the same prompt

GRPO samples a group of responses for one prompt and uses relative rewards to construct advantages, avoiding a learned value critic in this formulation. “No critic” does not mean no baseline, no reference model, or free training. Generation and scoring still cost compute.

Use this lesson after the RLHF policy-ratio explanation. The lab implements advantage normalization and a masked surrogate; it does not train a reasoning language model or demonstrate emergent reasoning.

Initializing Interactive Playground...

Trace reward → advantage → contribution

For rewards [0,0,1,1], mean=0.5 and population standard deviation=0.5. Advantages are [−1,−1,1,1]. Normalize within each prompt, not across unrelated prompts. If all rewards are equal, this implementation returns zero advantages.

For epsilon=0.2, ratio=1.4 and advantage=1, the clipped contribution is min(1.4,1.2)=1.2. For advantage=−1 it is min(−1.4,−1.2)=−1.4. Applying a clipped ratio unconditionally would discard the wrong-direction penalty.

The downloaded lab averages valid completion tokens within each response, then responses within a group. Prompt and padding tokens are excluded, and an empty completion is rejected. It uses a fixed old-policy probability ratio, no value network, and beta=0 to isolate the reward surrogate. Reference covers the separate KL term and its sampling assumptions.

Test the reward before trusting the update

A formatted wrong answer can earn positive relative advantage when the verifier rewards formatting too heavily. In our fixture, increasing the format bonus from 0 to 3 promotes a wrong-but-formatted completion above the group mean. Correctness remains unchanged.

Run adversarial checks: identical rewards, padded tokens, a negative advantage, and a wrong answer with the expected format. Keep a separate held-out correctness metric. Use strict parsers or sandboxed verifiers; never execute arbitrary generated code in your application process.

Experiment notes

Record your experiment

Set the new/old probability ratio to 1.4 and switch the advantage sign. Predict both clipped contributions, then test what changes when all group rewards are equal.

PredictionWhat changed, and why?

Implementation

Find a reward bug before updating a policy

A format bonus makes a wrong completion attractive; padding also distorts a naive token average.

Repair the browser normalization function.
Implement masked clipping in starter.py and compare positive/negative advantages.
Sweep format reward while reporting independent correctness and prompt-level advantages.

Check: Submit a reproducible reward audit, mask invariance tests and a failure example; do not label it a trained reasoning model.

Failing starter lab

Normalize one prompt group

These CPU labs isolate mechanisms. Passing the drill is not completion of the project, LLM training, or independent evidence of mastery.

Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.

Dataset: Authored completion rewards v1

Console

Run the code to inspect its output. Use Check solution only when you want to run the exercise checks.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

ALL TESTS PASSED

Prerequisites: Policy probability ratios · Mean and standard deviation · PPO clipping and token masks

Build the complete CPU experiment

Download all six files into one folder. Python 3.10+; standard library only. No API key, GPU, account, model download or paid service. The browser exercise above is one component of this project.

Run the reference tests, then the reference experiment. Inspect the configuration, fixture hash, numerical results and limitations.
Implement the functions in starter.py, then run the learner tests and experiment. The unfinished starter intentionally fails; it never falls back to the answer.
Change one setting using the experiments below. Save a report with a new filename and add your prediction, result, explanation and limitation.

python3 test_labs.py --reference
python3 run_lab.py grpo --implementation reference
python3 test_labs.py --topic grpo
python3 run_lab.py grpo --output my-grpo-report.json

Experiment plan & debugging checks

Replace [0,0,1,1] by [1,1,1,1].

Predict: All advantages become zero; no relative reward signal remains.

Inspect: Population std, advantages and zero reward-surrogate contribution.

Add masked padding and test ratios 0.6 and 1.4 for both signs.

Predict: Padding has no effect; clipping flattens opposite sides for opposite signs.

Inspect: At ratio 1.4: +1 gives 1.2, −1 gives −1.4.

Use --format-weight 0 and --format-weight 3.

Predict: A wrong but formatted answer may acquire positive advantage.

Inspect: Raw correctness, format scores, mixed rewards and per-group advantages.

Next: port these invariants to a real model only after you can reproduce the small case. Freeze the checkpoint, tokenizer, split, objective, masks and evaluation before comparing runs. GPU training and independent learner validation remain separate work.

Where this fits in the GPT build path →

Back to Topic Library

Feedback & sharing

Frontier Reasoning & GRPO

Compare answers to the same prompt

Trace reward → advantage → contribution

Test the reward before trusting the update

Record your experiment

Implementation

Normalize one prompt group

Build the complete CPU experiment

Replace [0,0,1,1] by [1,1,1,1].

Add masked padding and test ratios 0.6 and 1.4 for both signs.

Use --format-weight 0 and --format-weight 3.

Knowledge check

8 questions

Prove it on a new case

Reasoning, search, and a fixed compute budget

Compare like with like

Further reading

DeepSeekMath: original GRPO formulation

DeepSeek-R1

References

DeepSeekMath: original GRPO formulation

DeepSeek-R1

Approximating KL Divergence

TRL GRPO trainer

Helpful?