NeurIPS 2025 · LAW workshop

RL in Name Only? Analyzing the Structural Assumptions in RL Post-Training for LLMs

Under the assumptions that RLVR makes about the LLM as an MDP, on-policy GRPO reduces to filtered iterative supervised fine-tuning. The growth in response length traces back to the same assumptions, not to emergent reasoning.

S. R. Samineni*, D. Kalwar*, K. Valmeekam, K. Stechly, S. Kambhampati (*equal contribution)

arXiv Follow-up · GRPO-Unhinged

Two structural assumptions

RL with verifiable rewards treats generation as an MDP in which the state is the question plus the tokens so far, the action is the next token, and a verifier returns one binary reward at the end. Two assumptions sit inside this formulation. The next state is the current state with one token appended, with no observation from the environment, so the MDP is degenerate. And GRPO copies the group-relative advantage of a response to every one of its tokens, so no credit is assigned within a response.

Turn level: prompt, response and outcome reward from the verifier. Token level: each token is appended to the state with no environment observation (Assumption 1), and the outcome reward is distributed equally to every token (Assumption 2). — The two levels of RLVR. The environment acts only at the start and end of a turn. Inside the response, each token is appended to the state (Assumption 1), and every token receives the label of its response (Assumption 2).

GRPO reduces to filtered iterative SFT

With no transition, each token is a (state, action) pair, and with the outcome copied to every token, each pair carries a label. The responses sampled for a question become an outcome-labeled dataset, and the GRPO update becomes a step of supervised fine-tuning on it, filtered by the verifier. We confirm this empirically by post-training Qwen2.5, Llama 3.2 and DeepSeek 7B with GRPO and with filtered iterative SFT on GSM8K, and the two perform comparably.

Diagram. On the RL side, a degenerate MDP, a group-based advantage and uniform credit assignment lead to the RLVR update. On the SFT side, generating on-policy samples, filtering and labeling with the verifier, and updating iteratively lead to the FISFT update. The RLVR update reduces to the FISFT update. — The RLVR update reduces to the filtered iterative SFT (FISFT) update. The verifier supplies the outcome reward to RLVR and the labels to FISFT.

Where the length growth comes from

Because every token of a response gets the same advantage, a token's credit is divided by the length of its response, which favors short correct responses and long incorrect ones. The growth in length often read as the model learning to think is confined to the incorrect responses, and it appears only when the model is trained on them.

Line chart of average response length over training steps on Countdown. Incorrect responses under GRPO and under FISFT trained on both classes grow from about 60 to over 800 tokens; correct responses stay near 200; FISFT trained only on correct responses stays flat near 60. — Countdown, Qwen-2.5-1.5B. Left, average response length. Right, number of correct and incorrect responses. Dashed lines are incorrect responses. Length grows only when the model trains on incorrect responses (GRPO and FISFT+−), not when it trains on correct ones alone (FISFT+).

Line chart of the number of correct and incorrect responses over training steps on Countdown for GRPO and the FISFT variants. — Countdown, Qwen-2.5-1.5B. Left, average response length. Right, number of correct and incorrect responses. Dashed lines are incorrect responses. Length grows only when the model trains on incorrect responses (GRPO and FISFT+−), not when it trains on correct ones alone (FISFT+).

What it means

The gains of RLVR in post-training come from a supervised loss applied at each update, not from solving the credit-assignment problem that sets RL apart from supervised learning. GRPO-Unhinged takes this further and shows that the clipped GRPO update is a weighted hinge loss.