DeepSeek-R1 style Group Relative Policy Optimization, rule-based RL, and test-time compute.
For a small example, four solutions to one arithmetic problem receive different rewards. Center or normalize rewards within the group and see which sampled trajectories receive positive learning signal.
A model solves coding tasks with executable tests. Sample groups, use outcome and format rewards, monitor pass rate and response length, and inspect reward exploitation.
Initializing Interactive Playground...
Predict how one control changes the Frontier Reasoning & GRPO result.
The starter deliberately follows a tempting but flawed assumption: Assuming a verifiable reward is ungameable
Check: Your saved hypothesis predicts a direction, the experiment tests it, and the result changes or supports your explanation.
Explanations after each answer · 80% to pass · Frontier Reasoning & GRPO
Chapter 1 of 5 · Core
Reasoning systems can spend extra inference compute on a longer response, multiple independent candidates, verification, or an explicit search procedure. These are different mechanisms. A single autoregressive trace does not automatically implement tree search, and visible reasoning text is not a complete account of a model's internal computation. Tags such as <think> are formatting conventions used by some models, not a universal reasoning architecture.
Separate training-time improvement from inference-time allocation. A trained policy determines what candidates are available. A selection rule determines which candidate becomes the answer. A verifier can help only when its errors are sufficiently controlled; otherwise additional candidates can increase the chance of selecting an answer that exploits the verifier.
Fix a task set and a total compute or latency budget. Compare one longer sample, several short samples with majority voting, and several samples ranked by a verifier. Record accuracy, generated tokens, wall time, and verifier cost. Keep prompt selection and verifier tuning off the test set. Report both average performance and hard subgroups.
For independent trials with identical success probability p, the probability that at least one of k candidates succeeds is 1−(1−p)^k. This is an oracle availability calculation, not deployed answer accuracy: an actual system still has to identify the successful candidate. Correlated samples and imperfect selection break the simple operational interpretation. More budget can yield diminishing returns, saturation, or worse selected answers.
Key equations
Two free references selected for Frontier Reasoning & GRPO.
Primary sources and free technical reading for Frontier Reasoning & GRPO.
The original group-relative objective and mathematical reasoning experiments.
Reported reasoning experiments and distinctions between R1-Zero and R1.
John Schulman derives the sampled KL approximation and its assumptions.
Check current implementation choices against the original objective.