NeurIPS 2025 · MATH-AI workshop

Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains

S. R. Samineni, D. Kalwar, V. Gangal, S. Bhambri, S. Kambhampati

arXiv Follow-up · RLVR as Prompt Augmentation
Footsteps, each marked with a check, climb a small hill to local coherence, while the taller peak, global validity, is never reached.
The analogy. Every step of the trace checks out, so the climb reaches local coherence. Global validity, the valid solution, is a different and higher peak, and an outcome reward does not ask the model to climb it.
Poster · NeurIPS 2025 Workshop on Mathematical Reasoning and AI

RLVR makes reasoning traces locally coherent, not necessarily valid

RLVR rewards only the final answer, yet claims are made about better reasoning traces. We measure what happens to the intermediate tokens that are never directly rewarded.

Question

Does RL post-training improve the reasoning, or only the answer?

GRPO on Qwen-2.5-0.5B with GSM8K. Each test problem falls into one of four patterns by whether the base model (first digit) and the RL model (second digit) answer it correctly.

00 both wrong01 only RL right10 only base right11 both right
Method

Trace coherence, measured with first-order logic

  • Translate each trace into a first-order-logic representation and tag errors with a taxonomy (false premise, false rule, calculator error, format error).
  • A trace is coherent if it has no errors. Validity, logical soundness, is the stronger property.
  • Pass@K trace coherence, checked against human annotations.
Finding 0% → ~85%

trace coherence on problems only the RL model solves (pattern 01), across Pass@K.

up to 96%

trace coherence for problems both models solve (pattern 11) after RL.

RL improves local coherence without necessarily producing valid or correct solutions. Claims of improved reasoning via RL need to measure validity, not coherence.
Confusion matrices of trace validity against answer correctness for patterns 00 and 01, base model and RL model, at Pass@1. Confusion matrices of trace validity against answer correctness for patterns 10 and 11, base model and RL model, at Pass@1.
Confusion matrices for patterns 00 and 01 (left) and 10 and 11 (right) at Pass@1, before (step 0) and after (step 100) RL.

Why it matters

Improved local coherence in reasoning steps does not guarantee a valid proof or a correct answer. A model can learn to write steps that look consistent while the solution as a whole is still wrong. RLVR as Prompt Augmentation follows up with a theory of why the objective behaves this way.