Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Hugging Face Daily Papers Papers
reasoning-models self-evolution reinforcement-learning question-generation math-reasoning benchmark-evaluation

Summary

The paper identifies why self-evolving reasoning models suffer performance collapse — invalid questions and mathematically-equivalent question repetition — and proposes R-Quest, which uses validity and novelty feedback to sustain self-evolution, outperforming R-Zero by 17.32 points across 12 benchmarks over ten rounds.

Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.
Original Article
View Cached Full Text

Cached at: 10/08/26, 12:31 PM

Paper page - Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Source: https://huggingface.co/papers/2610.04299

Abstract

Self-evolvingreasoningmodelslearnfromtheirowngeneratedquestions,yetrepeatedself-trainingcanleadtoperformancecollapse.Inthispaper,weinvestigatewhyperformancedeterioratesoversuccessiveroundsandhowtosustainself-evolution.Ouranalysisidentifiestworecurringqualityproblemsinself-generatedquestions:invalidquestionsandrepeatedvariantsofthesamemathematicalquestions.First,invalidquestionsbecomemoreprevalentacrossrounds,andanswer-consistencyfilteringfurtherincreasestheirproportionintrainingdata.Second,existingquestiondiversitycontrolsbasedonlexicalsimilaritycanmissmathematicallyequivalentquestionsexpressedindifferentways,whichleadstoquestiondiversitycollapseinlatertrainingrounds.Buildingonthesefindings,weintroduceR-Quest,whichusesquestionvalidityandnoveltyfeedbacktoguideself-evolution.Wefirsttrainthesolvertorecognizeandrejectinvalidquestions,thenuseitsjudgmentstoguidequestionerrewardsandfiltersolvertrainingdata.Toavoidquestionrepetition,weuseafrozenbasemodeltocomparesampledquestionpairsandprovidenoveltyfeedback.Empirically,ourmethodconsistentlyachievesthehighestaverageperformanceon12benchmarksinmathematicalreasoning,general-domainreasoning,andcodegenerationacrosstwomodelfamilies.Additionally,R-Questmaintainsstableperformancegainsovertenroundsofself-evolution,peakinginthefinalroundandoutperformingR-Zeroby17.32points.

View arXiv pageView PDFProject pageAdd to collection

Models citing this paper1

#### jinyuan222/R-Quest-Qwen3-4B-Validity-RL Text Generation• 4B• Updated2 days ago • 418

Datasets citing this paper1

#### jinyuan222/R-Quest-Validity-RL Viewer• Updated2 days ago • 2.23k • 93

Spaces citing this paper1

Collections including this paper1

Similar Articles

What reasoning model are you actually running in production?

Reddit r/AI_Agents

A practitioner seeks real-world feedback on reasoning models like o3, Claude extended thinking, Gemini 2.5 Pro, and Ring 2.6 1T for production agent tasks, questioning the practical performance of Ring's dual-reasoning-effort modes versus benchmarks.

Flow Reasoning Models: Scaling Reasoning Through Iterative Self-Refinement

arXiv cs.AI

Flow Reasoning Models (FRMs) introduce a training and test-time-scaling framework for discrete flow models on structured reasoning tasks. By using self-verification and self-conditioning, FRMs achieve nearly 100% solve rates on Sudoku and Zebra puzzles with far fewer passes than previous baselines.