What RL post-training actually changes in a model

RLVR as classification-based policy iteration

My findings show that RLVR acts as a classification-based policy iteration method for learning to search. In my ongoing work, GRPO-Unhinged, I show that with binary rewards the clipped GRPO update is a weighted hinge loss on outcome-labeled tokens. Each token's log-ratio score acts as an estimate of its Q-value, the probability of success after that token, and GRPO variants such as DAPO, GSPO and CISPO differ only in how they set four components, the class weights, the margins, the clipping and the score function. A per-class loss normalization that follows from the hinge loss perspective raised plan accuracy by 10% over GRPO on 30×30 maze navigation and kept incorrect responses at their original length, where GRPO doubled them.

This builds on RL in Name Only? (NeurIPS 2025 LAW workshop), where I showed that under the LLM-as-MDP assumptions used in RLVR, on-policy GRPO reduces to filtered iterative SFT, confirmed by post-training Qwen2.5, Llama 3.2 and DeepSeek 7B on GSM8K, and traced the growth in response length to these structural assumptions rather than to emergent reasoning.

Skeptical evaluation of reasoning

In RLVR as Prompt Augmentation (under review at ICLR 2027), I derive the optimal policy of the RLVR objective as the base policy conditioned on success, and prove that a chain of thought is up-weighted only by how much it raises the solve probability, regardless of its validity or coherence. On math reasoning benchmarks, GRPO raises accuracy while the rate of correct answers reached through incoherent reasoning also rises in every setting, contradicting the claim that RLVR implicitly incentivizes correct reasoning.

In Local Coherence or Global Validity? (NeurIPS 2025 MATH-AI workshop), I built an LLM-as-judge pipeline that translates reasoning traces into first-order logic and tags errors, validated against human annotations, and showed that RLVR improves local step coherence without making solutions valid. I am also a co-author on an ICML 2026 position paper arguing against anthropomorphizing intermediate tokens as reasoning traces.

From prototypes to working systems

Before the PhD I spent more than two years turning RL into working systems. At Quantiphi I built an RL scheduler for flexible job-shop manufacturing with graph attention networks, and I am a co-inventor on a US patent application. At Microsoft Research India I built the RL framework for a renewable-energy market, where the agent handled carbon arbitrage and profit maximization. My M.Tech thesis at the Indian Institute of Science applied RL to robotics, and the resulting approach, which combines model-based lookahead with model-free RL, improved sample efficiency by 30% in simulation and doubled it on a real robot arm and quadruped (ICRA 2022). Before moving into computer science, I earned a B.Tech in Civil Engineering from NIT Warangal and worked as an Assistant Executive Engineer in the Roads and Buildings Department of the Government of Telangana.

What I want to work on next

• Multi-turn RL for LLM agents, where credit has to be assigned across turns as well as tokens, while the tokens within each turn still receive the right signal.
• Off-policy RL post-training, with updates derived from the hinge loss perspective that remain valid when rollouts come from earlier policies, so stale or asynchronous rollouts can be reused instead of discarded.
• Evaluation of reasoning in non-ergodic worlds, where averages over many runs differ from what one agent experiences along a single trajectory.

I am looking for a research internship for summer 2027, and I would welcome the chance to collaborate on these questions. Write to ssamine4@asu.edu.