Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds
Summary
This paper introduces Code-Guided Reasoning (CGR), an evaluation protocol for measuring how executable reasoning scaffolds improve small language model performance on multiple-choice question answering tasks, showing a significant accuracy improvement over direct answering.
View Cached Full Text
Cached at: 05/20/26, 06:36 AM
Paper page - Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds
Source: https://huggingface.co/papers/2605.18827
Abstract
Code-Guided Reasoning (CGR) evaluates how executable reasoning scaffolds enhance small language model performance on multiple-choice question answering tasks through standardized components and measured improvements.
Multiple-choice QAbenchmarks usually evaluatesmall language models(SLMs) as direct answerers, but deployed language-model systems increasingly rely on external scaffolds such as tools, code, and repeated model calls. We introduceCode-Guided Reasoning(CGR), an evaluation protocol and generated-program resource for measuring whenexecutable reasoningscaffolds improve SLM performance on MCQA tasks. CGR standardizes six components: a normalized item interface, adirect solver prompt, agenerator prompt, aPython scaffold, solver-call and extraction helpers, and a three-channel result record. On 20,498 retained result rows from a locally prepared MCQA bundle and six metadata-registered solver models, the observed non-zero-baseline partition shows 66.21% macro assisted accuracy versus 38.11% direct accuracy, a +28.10 percentage-point difference with a pair-bootstrap intervalof [20.32, 36.43]. Under a stricter Ab > 30%direct-signal gate, the macro difference is +14.11 points. These estimates are descriptive. Assisted inference uses a larger solver-call budget,answer extractionis brittle, Time-MQA contains the observed regressions, and somegenerated programsviolate the no-hard-coding instruction. CGR provides thetrace packageneeded to interpret these results, including direct, assisted, and generator-side answers, partition definitions,generated programs,response metadata, and audits.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.18827
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.18827 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.18827 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.18827 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Reasoning, Code, or Both? How Large Language Models Handle Variations in Math Questions
This paper evaluates three approaches (pure chain-of-thought reasoning, single-shot code execution, and iterative code execution) on 1,000 GSM-Symbolic problems using Claude Haiku 4.5, finding that chain-of-thought is the most robust to perturbation, while code execution does not improve reasoning robustness on grade-school math problems.
Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games
This paper introduces a multi-turn interactive framework for reasoning evaluation where LLMs must query a hidden environment and integrate partial observations, instantiated as a benchmark of 474 executable games across five difficulty levels, showing discriminative power and exposing differences in reasoning.
MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning
MedGuideX transforms clinical practice guidelines into executable decision logic to generate factual and counterfactual QA data for training medical LLMs, achieving a 10.28% relative improvement in average accuracy across clinical reasoning benchmarks.
Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models
This paper investigates whether structured synthetic reasoning data can improve arithmetic reasoning in small language models under consumer-hardware constraints. Fine-tuning Qwen3-0.6B and Qwen3-1.7B with LoRA on a synthetic corpus derived from GSM8K improved exact-match accuracy by 12-13 percentage points, with strong transfer to related benchmarks.
Efficient Reasoning Distillation: Small Video-Language Models via Synthetic CoT and Difficulty-Aware Fine-Tuning
The paper presents a method to distill reasoning into compact video-language models using synthetic chain-of-thought rationales and difficulty-aware fine-tuning, enabling smaller models to outperform larger ones with minimal compute.