self-consistency

A practitioner experiment found that when a single model writes both the code and the tests, the generated test suites falsely rejected a known-correct reference solution 77% of the time, even with high mutation scores, and the full agent pipeline was outperformed by simple pass@k sampling at similar token cost.

本文将多路径 LLM 推理形式化为无线通信中的分集合并问题，分析路径正确性相关性对多数投票有效样本量的上限，并在 5 个模型和 12 个基准上验证提示模板分集可降低路径相关性，同时提出基于四路径试点的 Adaptive-K 规则来选择最优采样数。

The paper introduces an algorithm allowing unfaithful LLM providers to artificially inflate reasoning paths for overcharging while evading detection, validated through experiments with models like Llama and Qwen.

This paper introduces an LLM-as-a-judge method to measure perturbation strength for assessing self-consistency in LLM explanations, showing that input perturbations generally affect LLMs more strongly than CoT perturbations.

This article covers inference scaling techniques for AI text generation, such as temperature scaling, top-p filtering, and self-consistency, aiming to improve answer accuracy by over 2x through diverse sampling and majority voting.

The paper evaluates local open-weight LLM judges against human ratings, finding high self-consistency but limited agreement with human judgments, highlighting the need for dual assessment.

This paper presents a novel approach to simulating coupled dynamical systems by using neural surrogates and self-consistency to solve fixed-point problems over full trajectories, replacing traditional timestep-by-timestep integration. It demonstrates the method on van der Pol oscillators and Hodgkin–Huxley neuron networks, highlighting efficiency gains and unique gradient properties.

This paper examines the trade-off between model size and inference compute for grammar-constrained text-to-SQL tasks using small language models, finding that beam search outperforms sample+vote and that scaling model size generally yields better results than scaling inference compute.

This paper investigates the self-consistency of cardinal LLM preference judgments across multiple models and finds large persistent inconsistencies, suggesting they cannot be faithfully summarized by a single utility function.

This paper shows that majority-vote self-consistency backfires on a majority of hard science problems in GPQA Diamond for small instruction-tuned LLMs, and that verifier-free gates like plurality agreement or token entropy cannot prevent this. Pre-registered confirmatory experiments quantify the accuracy drop and suggest confidence does not track correctness.

This position paper argues that many LLM failures stem from evaluating outputs independently and proposes a self-consistency framework that treats diverse techniques as special cases of consistency optimization.

This paper introduces CALVER, a training-free symbolic verifier that scores structured causal reasoning traces against Pearl's criteria to select the best candidate, outperforming plurality voting and other selection methods on causal reasoning benchmarks where multiple valid answers exist.

This paper investigates whether stochastic sampling (self-consistency) in LLMs can capture cross-question structure similar to diverse ensembles. Using a Marchenko–Pastur test, the authors find that within a single model, stochastic variation yields at most one significant dimension, while an ensemble of 24 models yields four, revealing a dimensionality gap that limits self-consistency as an ensemble substitute.

This paper introduces SSC-GRPO, a step-level reward optimization method that uses self-consistency scores across multiple rollouts to reduce context-sensitive factual hallucinations in LLM reasoning, achieving state-of-the-art performance on math reasoning and hallucination benchmarks.

This paper investigates test-time scaling techniques for small open vision-language models (≤7B parameters) on the multilingual visual MCQ benchmark EXAMS-V, finding that inference budget and parseability matter more than complex search or verification methods. The best configuration achieves 84.1% on the ImageCLEF 2026 test split, ranking first on the leaderboard.

This paper introduces GraphEVAL, a graph-based framework for quantifying uncertainty in LLM reasoning, and proposes a new metric, Graph Reasoning Coherence Score (GRCS), that captures semantic-structural consensus and detects confident hallucinations. The authors also present Graph Self-Consistency (GSC), a decoding strategy that prioritizes reasoning fidelity over nominal accuracy.

This paper studies which signals best predict correctness in text-to-SQL for selective prediction. It finds that verification-based signals from LLM judges outperform black-box statistical signals like self-consistency, and that a two-provider ensemble achieves 0.82 AUROC with well-calibrated probabilities.

A new paper shows that scaling inference compute via methods like self-consistency improves LLM accuracy in math and code but fails to improve truthfulness in domains without external verifiers, as model errors are too correlated.

Ali distilled Claude Fable 5 reasoning traces into Qwen3-4B, claiming 100% self-consistency and zero hallucination variance, and open-sourced the result.

Introduces SFL-MTSC, a structured aggregation framework for robust multi-intent spoken language understanding using LLM self-consistency at the semantic frame level, showing improved slot F1 and overall accuracy on the MAC-SLU benchmark.