Ongoing work

GRPO-Unhinged: The Hidden Advantage of Classification in RL Post-Training

RL for LLMs is classification-based policy iteration. With binary rewards, the clipped GRPO update is a weighted hinge loss on outcome-labeled tokens.

Builds on · RL in Name Only? (NeurIPS 2025 LAW workshop)

Hinge losswhat the clipped GRPO update becomes with binary rewards

4 componentsclass weights, margins, clipping and the score function, the only ways GRPO variants differ

+10%plan accuracy over GRPO on 30×30 mazes with per-class loss normalization

The view

Each token's log-ratio score acts as an estimate of its Q-value, the probability of success after that token. With a binary reward and no discounting, a token's advantage is how much that token raises the probability of success. The clipped GRPO update then classifies tokens against the outcome labels as a weighted hinge loss, with the score function as the decision boundary.

This places RLVR in classification-based policy iteration, and it shows that GRPO variants such as DAPO, GSPO and CISPO differ only in how they set four components: the class weights, the margins, the clipping and the score function.

Hinge loss plot: correct tokens have a loss that falls to zero past the upper margin 1 plus epsilon, incorrect tokens have a loss that rises past the lower margin 1 minus epsilon. — The clipped GRPO update as a weighted hinge loss. Correct tokens push their score up until the upper margin, incorrect tokens push it down past the lower margin, and the class weights set how hard each side pushes.

Per-class loss normalization

Existing remedies for length growth change the normalization but treat correct and incorrect responses alike. A per-class loss normalization follows from the hinge loss perspective and normalizes the loss within each class. On maze navigation, with transformers trained from scratch to produce an A* search trace and a plan, it keeps incorrect responses at their original length, where GRPO doubles them, and reaches the highest accuracy.

30×30 maze navigation after 70 steps (greedy decoding, 1,000 mazes per set, one seed). Lengths in tokens on Wilson mazes. Plan accuracy in %.
Method	Correct length	Incorrect length	Wilson	Kruskal	DFS-41f	DFS-42
Base model	1,336	4,412	49.6	49.7	65.0	30.3
GRPO	1,283	9,130	65.1	67.2	85.2	39.5
DAPO (token-level aggregation)	1,381	9,404	44.5	42.7	79.7	37.1
Per-class normalization (ours)	1,405	4,199	72.9	74.4	86.7	42.0

Next directions

Multi-turn RL for LLM agents. Extending the classification view to credit assignment across turns, using turn-level advantages to decide which turns mattered while the tokens within each turn still receive the right signal.
Off-policy RL post-training. Deriving updates from the hinge loss perspective that remain valid when rollouts come from earlier policies, so stale or asynchronous rollouts can be reused instead of discarded.

This is ongoing work and the paper is in preparation. Interested in collaborating? Write to ssamine4@asu.edu.