似然排名在LLMs中的扩展与提示不同

# Likelihood Ranking doesn’t Scale Like Prompting in LLMs
Source: [https://arxiv.org/html/2609.29390](https://arxiv.org/html/2609.29390)
Alessandro BondielliAffiliation:CoLingLab, Department of Philology, Literature and Linguistics, University of PisaAffiliation:Department of Computer Science, University of Pisa \*Equal contribution\.Correspondence:[alessandro\.bondielli@unipi\.it](mailto:[email protected]),[lucia\.passaro@unipi\.it](mailto:[email protected])Lucia PassaroAffiliation:CoLingLab, Department of Philology, Literature and Linguistics, University of PisaAffiliation:Department of Computer Science, University of Pisa \*Equal contribution\.Correspondence:[alessandro\.bondielli@unipi\.it](mailto:[email protected]),[lucia\.passaro@unipi\.it](mailto:[email protected])Davide BacciuAffiliation:Department of Computer Science, University of Pisa \*Equal contribution\.Correspondence:[alessandro\.bondielli@unipi\.it](mailto:[email protected]),[lucia\.passaro@unipi\.it](mailto:[email protected])Alessandro Lenci

###### Abstract

LLM evaluation is commonly performed either by prompting models to produce answers or by scoring candidate outputs with likelihood\-based metrics\. In multiple\-choice QA, however, standard likelihood\-based scoring is still conditioned on the question and answer set, and can therefore leverage the same task\-conditioned answer\-selection interface used in prompting\. We study a complementary protocol based on likelihood ranking of declarative statements constructed from the same question–answer pairs\. Across 95 decoder\-only models, ranging from 0\.1B to 104B parameters, and 10 MCQA datasets, we find a systematic divergence between declarative\-statement likelihood ranking and prompted answering\. Statement\-likelihood accuracy remains comparatively stable across scale, whereas prompted answering improves sharply with scale and instruction\-tuning\. These results suggest that likelihood preferences over controlled declarative alternatives and task\-conditioned answer selection probe distinct aspects of model behavior, and should not be treated as interchangeable\.

## 1Introduction

Evaluation protocols play a central role in shaping conclusions about capabilities of LLMs\. Two paradigms that dominate current practice areprompting\-based evaluation, in which models are directly asked to produce answers, often using a multiple\-choice format[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.29390#bib.bib2), andlikelihood\-based evaluation[Hu and Levy \(2023\)](https://arxiv.org/html/2609.29390#bib.bib1), in which candidate outputs are scored using perplexity or related metrics\.

![Refer to caption](https://arxiv.org/html/2609.29390v1/Figure1.png)Figure 1:Overview of the experimental setting and evaluation\. The design enables a comparison between prompting\-based evaluation and declarative\-statement likelihood ranking\.Prompting\-based evaluation requires the model to map its likelihood preferences onto a discrete action conditioned on task framing, instructions, and output conventions\. This mapping constitutes a learned behavioral policy that is shaped by model scale and alignment\. By contrast, Likelihood\-based evaluation probes how probability mass is distributed over alternatives, e\.g\. factual statements, under a fixed linguistic form\. In this work, we instantiate this paradigm through declarative statements derived from multiple\-choice questions[Petroni et al\. \(2019\)](https://arxiv.org/html/2609.29390#bib.bib25)\.

There is mounting evidence in the literature that prompting based evaluations, especially those involving multiple\-choice selections, are inherently flawed and lack robustness[Wei et al\. \(2024\)](https://arxiv.org/html/2609.29390#bib.bib3);[Zheng et al\. \(2024\)](https://arxiv.org/html/2609.29390#bib.bib5);[Molfese et al\. \(2025\)](https://arxiv.org/html/2609.29390#bib.bib4);[Balepur et al\. \(2025\)](https://arxiv.org/html/2609.29390#bib.bib7)\. Likelihood\-based approaches are less “task\-oriented”, and have been shown to provide informative signals about linguistic and semantic plausibility[Hu and Levy \(2023\)](https://arxiv.org/html/2609.29390#bib.bib1);[Kauf et al\. \(2024\)](https://arxiv.org/html/2609.29390#bib.bib6)\. However, likelihood scores are sensitive to formulation and surface form, making it important to distinguish different likelihood\-based protocols\. Standard likelihood\-based MCQA still scores answer options in the original multiple\-choice context; declarative\-statement ranking removes this explicit answer\-selection interface\. This distinction is related to recent work showing that multiple\-choice answer selection may diverge from token\-level likelihoods, and that likelihood estimates are affected by surface\-form competition[Wang et al\. \(2024a\)](https://arxiv.org/html/2609.29390#bib.bib21);[Wang et al\. \(2024b\)](https://arxiv.org/html/2609.29390#bib.bib22);[Holtzman et al\. \(2021\)](https://arxiv.org/html/2609.29390#bib.bib23)\. How these signals diverge with scale and instruction\-tuning is underexplored\.

Here, we present large scale empirical evidence that declarative\-statement likelihood ranking and prompting\-based evaluations systematically diverge as a function of model scale and post\-training\. Our main findings are: i\.\) likelihood\-based accuracy over declarative statements improves slowly and remains relatively bounded across scale, while prompting\-based accuracy exhibits strong scaling behavior and rapidly surpasses it; ii\.\) instruction\-tuning accelerates the divergence, shifting the crossover point to smaller model sizes\.

## 2Evaluation Setup

As illustrated in Figure[1](https://arxiv.org/html/2609.29390#S1.F1), we formulate two closely related tasks on the same data, namelyMultiple\-Choice Question Answering\(MCQA\) andStatement Ranking, to compare prompting\-based evaluation with declarative\-statement likelihood ranking\. Model performance is assessed with accuracy under two complementary metrics\. We define Accuracy on Prompt Execution \(APX\) as the proportion of correct answers produced in response to explicit prompts, and Accuracy on Perplexity Score \(APS\) as the proportion of cases where the model assigns higher likelihood to the correct declarative statement than to distractors\. Note that we distinguish APS from standard likelihood\-based MCQA, where candidate answers are scored in the original question\-\-option context\. We argue that under our testing condition it follows the same APX\-like scaling, with no significant differences\.111We explicitly show this on preliminary experiments done on 51 models in Appendix[B](https://arxiv.org/html/2609.29390#A2)\.This motivates declarative\-statement ranking as a complementary protocol that reduces the task\-conditioned answer\-selection interface, rather than replacing standard likelihood\-based MCQA\. Intermediate option\-conditioned formulations are valuable, but reintroduce part of the selection context that APS is designed to abstract away from\.

We tested models on 10 HuggingFace MCQA datasets\. First, we unified them into a common format: question, choices \(labeled, e\.g\., "A", "B", etc\.\), correct label, and its index in the choices list\. We preserved the original choices order when available, and randomized it otherwise\. Then, to allow for declarative\-statement likelihood comparison under controlled linguistic forms, we automatically constructnnaffirmative declarative statements for each question withnnanswer options, one per option, usinggpt\-oss\-20b[Agarwal et al\. \(2025\)](https://arxiv.org/html/2609.29390#bib.bib8)\. We provided the model with the question, choices, and instructions to generate declarative statements for each of the choices, one per line\. We filtered out cases in which the model failed to provide exactlynnstatements\.222Appendix[A](https://arxiv.org/html/2609.29390#A1)details the dataset creation \(parameters, prompt, error handling, answer distribution, and validation\)\.Table[1](https://arxiv.org/html/2609.29390#S2.T1)summarizes the final dataset composition and statistics\.333We will release the final dataset via HuggingFace upon acceptance of the paper\.Since APS depends on the quality of the automatically constructed statements, we validate the generation pipeline for semantic fidelity and surface\-form consistency\. On a stratified 1% sample of the dataset, we observe a high similarity of generated statements to their source question–answer pair: BLEU mean/median is 0\.69/0\.75; BERTScore F1 mean/median is is 0\.68/0\.73; the median length gap is \-1 token; the meaning preservation under two LLM judges \(raw agreement 0\.98\) is 99% \. Within each item, a LLM judge finds candidate statements sharing a structural template in 95% of cases\. Full details on the validation process are provided in Appendix[A\.2](https://arxiv.org/html/2609.29390#A1.SS2)\.

Table 1:Statistics of multiple\-choice datasets used in our evaluation, same order as the table:[Wang et al\. \(2024c\)](https://arxiv.org/html/2609.29390#bib.bib9);[Clark et al\. \(2018\)](https://arxiv.org/html/2609.29390#bib.bib10);[Mihaylov et al\. \(2018\)](https://arxiv.org/html/2609.29390#bib.bib11);[Khot et al\. \(2020\)](https://arxiv.org/html/2609.29390#bib.bib14);[Welbl et al\. \(2017\)](https://arxiv.org/html/2609.29390#bib.bib16);[Suzgun et al\. \(2023\)](https://arxiv.org/html/2609.29390#bib.bib18);[Talmor et al\. \(2019\)](https://arxiv.org/html/2609.29390#bib.bib12);[Lin et al\. \(2022\)](https://arxiv.org/html/2609.29390#bib.bib13);[Science \(2025\)](https://arxiv.org/html/2609.29390#bib.bib15);[Sakai et al\. \(2024\)](https://arxiv.org/html/2609.29390#bib.bib17)\.We tested a total of 95 open\-weights decoder\-only language models, ranging from around 0\.1B to 104B parameters \(see Table[2](https://arxiv.org/html/2609.29390#A3.T2), Appendix[C](https://arxiv.org/html/2609.29390#A3)\)\. To avoid confounding factors, we excluded Mixture\-of\-Expert and reasoning\-enabled models from the evaluation\. We consider both the instruction\-tuned \(henceforth,instruct\) and pre\-trained only \(henceforth,base\) variants, when available\.444The full model inventory is available in Appendix[C](https://arxiv.org/html/2609.29390#A3)\.

In the prompting\-based setting, models were provided with instructions to solve the MCQA task, the MC question, its options, and prompted to directly and only provide the correct answer\. We adapted the final part of the prompt to address differences between base and instruct models, e\.g\., “the correct answer is: ” vs\. “what is the correct answer?” respectively\. We also followed the chat template of the model, when available\. We set temperature to 0 for greedy generation\. Finally, we parsed the models’ responses via regular expressions to obtain a single, final answer, that we then compared with the ground truth to compute accuracy\. In the declarative\-statement likelihood setting, we passed each declarative statement independently through the model, and computed its perplexity asexp\(−1N∑i=1Nlogp\(xi∣x<i\)\)\\exp\\\!\\left\(\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log p\(x\_\{i\}\\mid x\_\{<i\}\)\\right\)\. Then, we selected the statement with the lowest perplexity and compared it with the ground truth to compute accuracy\. All our experiments were conducted on HuggingFace models using vLLM\.555See Appendix[D](https://arxiv.org/html/2609.29390#A4)for further details\.

## 3Results

Figure[2](https://arxiv.org/html/2609.29390#S3.F2)illustrates the relationship between model scale and performance under APS and APX, across both base and instruct models \(each point corresponds to an individual model variant\)\.

To better characterize the relationship between scale and performance, we fit parametric scaling curves separately for each evaluation metric \(APS, APX\) and model type \(base, instruct\)\. We fit several monotonic scaling functions and select the one minimizing the Akaike Information Criterion \(AIC\)[Akaike \(1974\)](https://arxiv.org/html/2609.29390#bib.bib19), computed from the residual sum of squares\. AIC enables the comparison of non\-nested models while penalizing overparameterization\. The trend lines shown in Figure[2](https://arxiv.org/html/2609.29390#S3.F2)correspond to the best function fit for each subset\. Finally, we estimate uncertainty around each scaling curve via non\-parametric bootstrap resampling[Efron and Tibshirani \(1994\)](https://arxiv.org/html/2609.29390#bib.bib20)\. Shaded regions in the plot indicate 95% confidence intervals\.

![Refer to caption](https://arxiv.org/html/2609.29390v1/prompting_vs_ppl.png)Figure 2:Accuracies under prompting \(APX\) and declarative statement likelihood \(APS\) settings, across model scales and families\. APS shows weak scaling and limited gains from instruction\-tuning; APX improves sharply with scale, and larger models outperform APS\.Across datasets and model families, we observe a consistent pattern\. APS exhibits weak scaling behavior: declarative\-statement likelihood accuracy improves gradually with model size and remains within a relatively narrow range, saturating well below the best prompted results\. Instruction\-tuning has only a limited effect on this metric\. Both base and instruct models follow roughly the same rational function, with minor differences particularly for smaller models\. APX, by contrast, improves sharply with scale\. Smaller models often perform substantially worse when prompted than when evaluated via declarative\-statement likelihood ranking, while larger models exhibit rapid gains and ultimately outperform APS by a wide margin\. We refer to the threshold at which APX surpasses APS as theprompting break\-even point\. The best fit for both base and instruct models is a log\-linear function\. Both are very similar in terms of steepness, but instruction\-tuned models’ APX reaches the break\-even point with APS substantially earlier\. It is also worth noticing the much smaller variance of APS scores than APX ones, with the latter more loosely spread in the accuracy space\. This pattern differs widely from the standard likelihood\-based MCQA control reported in Appendix[B](https://arxiv.org/html/2609.29390#A2), which instead follows similar scaling behavior as APX\. This supports the interpretation that declarative\-statement ranking captures a signal distinct from the task\-conditioned answer\-selection interface\. We observe the same qualitative patterns by breaking down the analysis by dataset\.666See Appendix[E](https://arxiv.org/html/2609.29390#A5)for Figures and further details\.

These trends indicate that prompted answer selection and declarative\-statement likelihood ranking respond to different drivers of improvement\. While likelihood preferences over declarative alternatives slowly improve with scale, the ability to act on these preferences under task prompts is strongly amplified by instruction\-tuning\. APX and APS exhibit similar trends across datasets, but the prompting break\-even point varies:Sciqrequires larger models thanOpenBookQA; moreover APS onSciqis consistently high \(0\.65\-\-0\.85\), suggesting that correct statements receive more stable likelihood support than in other datasets, with lower and more stable PPL\.777See Appendix[E](https://arxiv.org/html/2609.29390#A5)for further details\.

## 4Interpreting the mismatch

Declarative\-statement likelihood ranking probes how probability mass is distributed over the various alternatives under a fixed linguistic form\. This signal is often diffuse, with small differences between correct and incorrect statements, and improves only gradually with scale\. We verify this by computing the average PPL delta between the first and the second preferred answer\. We see that, aside from few outliers, notably in the 4\-\-10B parameter range, all models display near\-zero differences between their first and second ‘‘choice’’ in terms of PPL\.888See Appendix[E](https://arxiv.org/html/2609.29390#A5)for details and visualizations\.

Prompting\-based evaluation, by contrast, requires the model to interpret task instructions, compare alternatives, and commit to a discrete decision\. This does not mean that prompted gains are disconnected from knowledge; rather, they conflate distributional support for the correct answer with the model’s ability to express that support through the requested answer format\. This behavior shows a learned answer\-selection policy that maps diffuse likelihood preferences onto task\-appropriate outputs\. Improvements under this metric reflect not only sharper underlying distributions, but also more effective interfaces between distributional preferences and decision\-making\.

We view instruction\-tuning as primarily optimizing the interface between likelihood preferences and task\-conditioned behavior, rather than substantially reshaping the underlying probability distribution learned in pre\-training\. By training models to produce explicit, task\-appropriate outputs under natural language instructions, instruction\-tuning strengthens the mapping from diffuse distributional preferences to discrete decisions\. The gap widens with scale: prompted accuracy increasingly reflects improved decision behavior and task compliance rather than proportional gains in declarative\-statement likelihood accuracy\. As model size, training data, and post\-training procedures are often opaque in current model releases, we interpret the observed gap as reflecting the combined effect of scale and instruction\-tuning on the answer\-selection interface\. Declarative\-statement likelihood ranking and prompting\-based evaluations target different objectives, with the latter resembling an easier discriminative decision task\. As a result, gains in prompted performance may reflect improved task\-conditioned decision behavior rather than sharper underlying distributions\. Our interpretation is related to work on factuality and calibration, including truth\-evaluation protocols such asP⁡\(True∣s\)P\(\\mathrm\{True\}\\mid s\)[Kadavath et al\. \(2022\)](https://arxiv.org/html/2609.29390#bib.bib24)\. APS, however, is not intended as a calibrated truth estimator, but as a controlled likelihood\-based signal for comparing declarative\-statement ranking with prompted answer selection\. This signal remains sensitive to surface form, paraphrase competition, and candidate wording\.

Our validation mitigates these confounds by preserving the original question wording and enforcing structural similarity across candidate statements\. Further normalizations, such as discounting unconditional option fluency, may help isolate statement\-conditioned preferences more precisely\. Nevertheless, our central finding is comparative: under the same controlled statement format, likelihood ranking and prompted answer selection scale differently across models, datasets, and instruction\-tuning regimes\.

## 5Conclusion

Current evaluation practices risk conflating task accuracy with likelihood\-based signals used to assess model capabilities\. Instead, our findings suggest that these two practices capture different signals of model knowledge\. Improvements seen under prompting, particularly those induced by instruction\-tuning, reflect advances in behavioral alignment and answer selection mechanisms at least as much as gains in likelihood preferences over semantically matched alternatives\.

We argue that prompting\-based evaluation, standard likelihood\-based MCQA, and declarative\-statement likelihood ranking measure related but distinct properties and should not be used interchangeably\. As LLMs continue to scale and are increasingly optimized for interactive use, evaluation protocols must more carefully distinguish between latent likelihood preferences and how effectively models are trained to act on them\.

## Limitations

This study has several limitations\. First, we focus exclusively on MCQA\. While this enables a controlled comparison between prompting\-based and likelihood\-based evaluation, it represents a limited class of tasks\.

Second, model families are not uniformly balanced across scale\. Some families are overrepresented at particular sizes, which may introduce residual family\-specific effects beyond scale\.

Third, we consider only open\-weights models\. Although this supports reproducibility, it limits the generality of our findings to proprietary systems that may employ different alignment strategies\.

Fourth, our analysis focuses on model scale and does not consider the amount of training tokens\. Albeit the two are strongly correlated, exposure to different amounts of tokens could have independent effects and may be a confounding factor in the analysis\. However, this aspect was not possible to analyze due to the lack of information on training data for many of the tested models, especially smaller variants of top\-of\-the\-line models\.

Fifth, APS relies on automatically generated declarative statements\. Although we validate them for semantic fidelity, structural consistency, and length balance, residual surface\-form differences, paraphrase competition, or stylistic preferences may still affect likelihood estimates[Holtzman et al\. \(2021\)](https://arxiv.org/html/2609.29390#bib.bib23)\. Thus, APS should be interpreted as a likelihood\-based signal over controlled declarative alternatives, not as a direct or exhaustive measure of model knowledge\.

Finally, our declarative\-statement protocol is complementary to standard likelihood\-based MCQA, where answer options are scored in the original multiple\-choice context\. We include this formulation as a control, but leave fuller comparisons with calibrated likelihood scores, option\-fluency corrections, open\-ended factual probes, and truth\-evaluation protocols such asP⁡\(True∣s\)P\(\\mathrm\{True\}\\mid s\)[Kadavath et al\. \(2022\)](https://arxiv.org/html/2609.29390#bib.bib24)to future work\. We also leave a systematic qualitative analysis of APX–APS mismatch cases to future work\.

Despite these limitations, the observed divergence between prompting\-based and declarative\-statement likelihood evaluation is robust across datasets and model families and highlights fundamental distinctions between evaluation paradigms\.

## References

- Agarwalet al\.\(2025\)S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§2](https://arxiv.org/html/2609.29390#S2.p2.1)\.
- Akaike \(1974\)H\. AkaikeA new look at the statistical model identification\.IEEE Transactions on Automatic Control19\(6\),pp\. 716–723\.External Links:[Document](https://dx.doi.org/10.1109/TAC.1974.1100705)Cited by:[§3](https://arxiv.org/html/2609.29390#S3.p2.1)\.
- Balepuret al\.\(2025\)N\. Balepur, R\. Rudinger, and J\. L\. Boyd\-GraberWhich of these best describes multiple choice evaluation with LLMs? a\) forced B\) flawed C\) fixable D\) all of the above\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3394–3418\.External Links:[Link](https://aclanthology.org/2025.acl-long.169/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.169),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p3.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Efron and Tibshirani \(1994\)B\. Efron and R\. J\. TibshiraniAn introduction to the bootstrap\.1 edition,Chapman and Hall/CRC\.External Links:[Document](https://dx.doi.org/10.1201/9780429246593)Cited by:[§3](https://arxiv.org/html/2609.29390#S3.p2.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p1.1)\.
- Holtzmanet al\.\(2021\)A\. Holtzman, P\. West, V\. Shwartz, Y\. Choi, and L\. ZettlemoyerSurface form competition: why the highest probability answer isn’t always right\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 7038–7051\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.564/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.564)Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p3.1),[Limitations](https://arxiv.org/html/2609.29390#Sx1.p5.1)\.
- Hu and Levy \(2023\)J\. Hu and R\. LevyPrompting is not a substitute for probability measurements in large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5040–5060\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.306/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.306)Cited by:[Appendix B](https://arxiv.org/html/2609.29390#A2.p1.1),[§1](https://arxiv.org/html/2609.29390#S1.p1.1),[§1](https://arxiv.org/html/2609.29390#S1.p3.1)\.
- Kadavathet al\.\(2022\)S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. KaplanLanguage models \(mostly\) know what they know\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[§4](https://arxiv.org/html/2609.29390#S4.p3.1),[Limitations](https://arxiv.org/html/2609.29390#Sx1.p6.1)\.
- Kaufet al\.\(2024\)C\. Kauf, E\. Chersoni, A\. Lenci, E\. Fedorenko, and A\. A\. IvanovaLog probabilities are a reliable estimate of semantic plausibility in base and instruction\-tuned language models\.InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Y\. Belinkov, N\. Kim, J\. Jumelet, H\. Mohebbi, A\. Mueller, and H\. Chen \(Eds\.\),Miami, Florida, US,pp\. 263–277\.External Links:[Link](https://aclanthology.org/2024.blackboxnlp-1.18/),[Document](https://dx.doi.org/10.18653/v1/2024.blackboxnlp-1.18)Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p3.1)\.
- Khotet al\.\(2020\)T\. Khot, P\. Clark, M\. Guerquin, P\. Jansen, and A\. SabharwalQASC: A dataset for question answering via sentence composition\.InThe Thirty\-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty\-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7\-12, 2020,pp\. 8082–8090\.External Links:[Link](https://doi.org/10.1609/aaai.v34i05.6319),[Document](https://dx.doi.org/10.1609/AAAI.V34I05.6319)Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTruthfulqa: measuring how models mimic human falsehoods\.InProceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 3214–3252\.Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2381–2391\.External Links:[Link](https://aclanthology.org/D18-1260/),[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Molfeseet al\.\(2025\)F\. M\. Molfese, L\. Moroni, L\. Gioffré, A\. Scirè, S\. Conia, and R\. NavigliRight answer, wrong score: uncovering the inconsistencies of LLM evaluation in multiple\-choice question answering\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18477–18494\.External Links:[Link](https://aclanthology.org/2025.findings-acl.950/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.950),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p3.1)\.
- Petroniet al\.\(2019\)F\. Petroni, T\. Rocktäschel, S\. Riedel, P\. Lewis, A\. Bakhtin, Y\. Wu, and A\. MillerLanguage models as knowledge bases?\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 2463–2473\.External Links:[Link](https://aclanthology.org/D19-1250/),[Document](https://dx.doi.org/10.18653/v1/D19-1250)Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p2.1)\.
- Sakaiet al\.\(2024\)Y\. Sakai, H\. Kamigaito, and T\. WatanabeMCSQA: multilingual commonsense reasoning dataset with unified creation strategy by language models and humans\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14182–14214\.External Links:[Link](https://aclanthology.org/2024.findings-acl.844/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.844)Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Science \(2025\)Y\. ScienceLLM health benchmarks dataset\.External Links:[Link](https://huggingface.co/datasets/yesilhealth/Health_Benchmarks)Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Suzgunet al\.\(2023\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. WeiChallenging BIG\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13003–13051\.External Links:[Link](https://aclanthology.org/2023.findings-acl.824/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824)Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Talmoret al\.\(2019\)A\. Talmor, J\. Herzig, N\. Lourie, and J\. BerantCommonsenseqa: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 4149–4158\.Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Wanget al\.\(2024a\)X\. Wang, C\. Hu, B\. Ma, P\. Röttger, and B\. PlankLook at the text: instruction\-tuned language models are more robust multiple choice selectors than you think\.External Links:2404\.08382,[Link](https://arxiv.org/abs/2404.08382)Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p3.1)\.
- Wanget al\.\(2024b\)X\. Wang, B\. Ma, C\. Hu, L\. Weber\-Genzel, P\. Röttger, F\. Kreuter, D\. Hovy, and B\. Plank“My answer is C”: first\-token probabilities do not match text answers in instruction\-tuned language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7407–7416\.External Links:[Link](https://aclanthology.org/2024.findings-acl.441/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.441)Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p3.1)\.
- Wanget al\.\(2024c\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Weiet al\.\(2024\)F\. Wei, X\. Chen, and L\. LuoRethinking generative large language model evaluation for semantic comprehension\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p3.1)\.
- Welblet al\.\(2017\)J\. Welbl, N\. F\. Liu, and M\. GardnerCrowdsourcing multiple choice science questions\.InProceedings of the 3rd Workshop on Noisy User\-generated Text,L\. Derczynski, W\. Xu, A\. Ritter, and T\. Baldwin \(Eds\.\),Copenhagen, Denmark,pp\. 94–106\.External Links:[Link](https://aclanthology.org/W17-4413/),[Document](https://dx.doi.org/10.18653/v1/W17-4413)Cited by:[Table 1](https://arxiv.org/html/2609.29390#S2.T1)\.
- Zhenget al\.\(2024\)C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. HuangLarge language models are not robust multiple choice selectors\.InThe Twelfth International Conference on Learning Representations \(ICLR\)\),External Links:[Link](https://openreview.net/forum?id=shr9PXz7T0)Cited by:[§1](https://arxiv.org/html/2609.29390#S1.p3.1)\.

## Appendix

## Appendix ADatasets Details

### A\.1Declarative Statements Generation

To generate affirmative declarative statements for likelihood\-based evaluation from multiple choice questions, we usegpt\-oss\-20b\. We use the HuggingFace implementation with vLLM\. Below we report the prompt used\.

#### System prompt

> You are a precise text transformation assistant\. Your task is to generate, for each multiple\-choice question, one affirmative statement per answer option, both for correct and incorrect options\. If the question includes blanks \(e\.g\., ‘‘\_\_\_ is the capital of France\.’’\), fill them with each answer choice\. If not, form a natural affirmative statement by appending the answer, e\.g\., Question: ‘‘What type of water formation is formed by clouds?’’ Expected output format: ‘‘The type of water formation formed by clouds is \[answer\]\.’’ Always output only the list of generated statements, one per line, in the same order as the provided choices\. Each statement must: - •Retain as much as possible the original text\. Do not shorten or omit anything\. - •Be rephrased only as needed to form a grammatically correct affirmative sentence\. - •Insert or append the answer option in a grammatically coherent and correct way\. - •Preserve the order of the answer options\. - •Produce one output per choice, in the same order as the provided list\. - •Be affirmative only \(no question marks\)\. - •Contain no commentary, explanations, or metadata\.

#### User prompt

> Question: \{\{ question \}\} Choices: \{% for label, text in choices %\} \{\{ label \}\}\. \{\{ text \}\} \{% endfor %\} Generate one affirmative statement for each choice, maintaining their order and preserving the question wording\.

#### Generation parameters

We set the reasoning effort to “medium”,t​e​m​p​e​r​a​t​u​r​e=0\.7temperature=0\.7, andt​o​p​\_​p=0\.95top\\\_p=0\.95\. We let the model generate a maximum of 1500 tokens, to fit both the reasoning trace and the final answer\.

#### Error Handling

To handle errors in generating declarative statements, we proceed as follows\. We simply parse the model’s response, excluding the reasoning trace, by splitting on new lines\. Then, we remove empty strings and programmatically assert whether the number of declarative statements actually corresponds to the number of possible answers for that question\. We filter cases where there were more or less declarative statements than possible answers\. Figure[3](https://arxiv.org/html/2609.29390#A1.F3)shows the proportion of errors for each dataset\.

We see that the number of errors for each dataset is relatively small on all datasets exceptMMLUPro\. This may be attributable to the fact that it is one of the most complex datasets, with the highest number of average choices \(9\.45±\\pm1\.5\)\.

![Refer to caption](https://arxiv.org/html/2609.29390v1/examples-and-errors-dataset.png)Figure 3:Number of examples and errors in declarative statements generation for each dataset\.

### A\.2Statement Validation

We validate the generated declarative statements for semantic fidelity and surface\-form consistency\.

Semantic fidelity\.We sample 900 items, corresponding to approximately 1% of the dataset, stratified by dataset and length bin to gather a representative sample of the dataset\. We leverage LLM\-as\-a\-judge with two independent judges, namelygpt\-oss\-120b\(implemented locally, via vLLM\) andgpt\-5\.2\(via API\)\. Judges are given the original question, answer option, and generated statement, and asked whether the statement preserves the meaning of the corresponding question–answer pair without omissions or distortions\. The prompt used for both models is the following:

> \[System\]:You are a strict semantic equivalence checker for multiple\-choice QA\. Task: Determine whether the variable "statement" is semantically equivalent to the variable "choice\_text" as an answer to the variable "question"\. Context definition: - •"question" is asked by Speaker A\. - •Speaker B answers using exactly "choice\_text"\. - •"statement" is a candidate full sentence that may express B’s answer\. Equivalence criteria: - •The statement must assert exactly the same answer as choice\_text\. - •Ignore superficial differences such as punctuation, capitalization, or grammatical form\. - •Do not use external knowledge\. Only compare semantic content\. - •If the statement expresses a different answer, adds incompatible meaning, introduces uncertainty, or changes the asserted answer, it is not equivalent\. Output format \(JSON only, no extra text\): \{"equivalent": boolean, "choice\_label": string \} \[User\]:Variables: question: \{\{question\}\} choice\_label: \{\{choice\_label\}\} choice\_text: \{\{choice\_text\}\} statement: \{\{statement\}\} Determine whether "statement" is semantically equivalent to "choice\_text" as an answer to "question"\. Return JSON only\.

Both judges label 99% of the sampled statements as semantically equivalent, with raw agreement of 0\.98\. We do not report Cohen’sκ\\kappadue to the strong class imbalance\.

We further compute automatic similarity measures\. We use BERTScore101010Model:roberta\-largeand BLEU111111sentence\_bleu, NLTK implementationto assess generated statements against their source question–answer pair\. The BERTScore F1 mean/median is 0\.68/0\.73\. Figure[4](https://arxiv.org/html/2609.29390#A1.F4)shows the complete distribution of F1 scores\. The BLEU score is similar as well, with 0\.69/0\.75 mean/median across questions \(Figure[5](https://arxiv.org/html/2609.29390#A1.F5)\)\. Finally, we also assess wether candidate statements from the same item are comparable in length with their respective question–answer pair\. We tokenize with thegpt\-oss\-20btokenizer and obtain an average within\-item \(statement vs question–answer\) median difference of \-1\.01 \(mean \-9\.1\), indicating that the median statement is roughly 1 token shorter than its source question–\-answer pair\.

![Refer to caption](https://arxiv.org/html/2609.29390v1/bertscore-f1.png)Figure 4:Distribution of BERTScore F1 for each statement and question–answer pair\.![Refer to caption](https://arxiv.org/html/2609.29390v1/mean-bleu.png)Figure 5:Distribution of Mean BLEU scores per question\.Surface\-form consistency\.Finally, we perform an additional assessment of the structural consistency of generated statements, to verify whether generated statements for each question follow the same surface\-form template\. On the same 1% sample, we ask a judge \(gpt\-oss\-120b, implemented locally with vLLM\) to determine whether all generated statements followed the same surface\-form template\. We propmt the model as follows:

> \[System\]:You are an expert linguistic analyst evaluating structural and surface\-form consistency across statements derived from the same multiple\-choice question\. Your task is NOT to evaluate correctness\. Your task is NOT to evaluate semantics\. Your task is NOT to determine whether the answer is valid\. Your task is ONLY to determine whether the statements share the same surface\-form template\. Two statements share a template if: - •They have the same clause structure - •Negation appears in the same position - •Predicate framing is identical - •Tense and modality are identical - •Differences are limited to insertion of the answer content You must: 1. 1\.Look for a shared template \(if present\)\. 2. 2\.Compare each statement to that template\. 3. 3\.Decide whether all statements follow the shared template\. Return ONLY valid JSON\. Output format \(JSON\): \{ "shared\_template": true or false \} \[User\]:Question: \{\{ question \}\} Original options: \{\{ answers \}\} Generated statements: \{\{ statements \}\} Instructions: 1. 1\.Infer the surface\-form template used by the statements\. 2. 2\.Replace the option\-specific span with \[OPTION\]\. 3. 3\.Compare each statement to the template\. 4. 4\.Decide whether all statements share the same template\. Return only JSON in exactly this format: \{"shared\_template": true or false \}

The judge declared that 95% of items have structurally consistent statements across answer options\.

Despite limited by the size of the dataset sample, these results highlight that the generated dataset is viable for our evaluation\.

### A\.3MCQA Answer Distribution

Figure[6](https://arxiv.org/html/2609.29390#A1.F6)shows the distribution of answers in each dataset\. We observe that the distribution is uniform in most datasets\. The main exceptions areBigBenchHardandTruthful\_QA\. However, in both cases the distribution of answers is also attributable to a high variability in the number of possible answers for each question\. If we look at Table[1](https://arxiv.org/html/2609.29390#S2.T1)we see in fact that both dataset have a much higher standard deviation than all other datasets\.

![Refer to caption](https://arxiv.org/html/2609.29390v1/answer-distribution.png)Figure 6:Distribution of answers in each dataset\.
### A\.4Dataset Licensing and Release

All datasets are publicly available on HuggingFace and used in accordance with their licenses\. We release our dataset121212Via HuggingFace upon acceptanceincluding original QA pairs and generated declarative statements—under CC\-BY, the most restrictive license among the sources\.

## Appendix BStandard MCQA Likelihood

In this Section, we aim to clarify the relation and highlight the difference between our formulation of APS and standard likelihood\-based MCQA[Hu and Levy \(2023\)](https://arxiv.org/html/2609.29390#bib.bib1)\. We clarify that our current formulation stems from preliminary experiments conducted early on a subset of models \(i\.e\.,5151\)\. These experiments act also as a control experiment in which we follow the common likelihood\-based MCQA protocol\. Specifically, each answer option is scored via perplexity as a continuation conditioned on the question and the full answer set, i\.e\.,log⁡P⁡\(option∣question,options\)\\log P\(\\mathrm\{option\}\\mid\\mathrm\{question\},\\mathrm\{options\}\)\. We note that in these experiments we kept the prompt identical for both the base and instruct models\. The prompt formulation is the one used for base models in our main experiments \(see Appendix[D\.1](https://arxiv.org/html/2609.29390#A4.SS1)\)\. This was done to limit variations and directly study APX vs likelihood\-MCQA regardless of model tuning\.

![Refer to caption](https://arxiv.org/html/2609.29390v1/L-MCQAvsAPX.png)Figure 7:Prompting\-based \(APX\) and standard likelihood\-based MCQA \(Likelihood\-MCQA\) accuracy across model scales and families\. Regardless of the presence of Instruction fine\-tuning, likelihood\-based MCQA and APX move in the same general direction, with comparable steepness\.With this setting we evaluate 51 models in the 0\.135B–67B parameter range\. We provide a visualization analogous to that of our main experiment in Figure[7](https://arxiv.org/html/2609.29390#A2.F7)\. We plot performances against size, and we fit a log linear function to each of the groups\. This standard formulation of the problem produced trends highly similar to prompted answering\. APX and standard option\-likelihood scores are moderately correlated \(ρ=0\.5444\\rho=0\.5444,p=3\.025×10−5p=3\.025\{\\times\}10^\{\-5\}\)\. Moreover, a linear model with interaction terms shows no significant difference between the scaling slopes of APX and standard option\-likelihood MCQA \(p=0\.58p=0\.58\)\. This supports our interpretation that standard likelihood\-based MCQA largely reflects the same task\-conditioned answer\-selection interface as greedy prompted answering, rather than an independent likelihood signal\.

## Appendix CModel Inventory

Table[2](https://arxiv.org/html/2609.29390#A3.T2)provides an overview of the model families evaluated in this work\. Below we report the full list of evaluated models, grouped by family\.

Table 2:Overview of the model families evaluated in this work, reporting the number of base and instruction\-tuned variants and their parameter scale\.#### Phi\-4\.

phi\-4, Phi\-4\-mini\-instruct\.

#### Qwen2\.5\.

Qwen2\.5\-0\.5B, Qwen2\.5\-0\.5B\-Instruct, Qwen2\.5\-1\.5B, Qwen2\.5\-1\.5B\-Instruct, Qwen2\.5\-3B, Qwen2\.5\-3B\-Instruct, Qwen2\.5\-7B, Qwen2\.5\-7B\-Instruct, Qwen2\.5\-14B, Qwen2\.5\-14B\-Instruct, Qwen2\.5\-32B, Qwen2\.5\-32B\-Instruct, Qwen2\.5\-72B, Qwen2\.5\-72B\-Instruct\.

#### Qwen3\.

Qwen3\-0\.6B\-Base, Qwen3\-0\.6B, Qwen3\-1\.7B\-Base, Qwen3\-1\.7B, Qwen3\-4B\-Base, Qwen3\-4B\-Instruct\-2507, Qwen3\-8B\-Base, Qwen3\-8B, Qwen3\-14B\-Base, Qwen3\-14B, Qwen3\-30B\-A3B\-Base, Qwen3\-30B\-A3B\-Instruct\-2507\.

#### Llama\-3\.

Llama\-3\.2\-1B, Llama\-3\.2\-1B\-Instruct, Llama\-3\.2\-3B, Llama\-3\.2\-3B\-Instruct, Llama\-3\.1\-8B, Llama\-3\.1\-8B\-Instruct, Llama\-3\.1\-70B, Llama\-3\.3\-70B\-Instruct, Meta\-Llama\-3\-70B\.

#### Gemma\-2\.

gemma\-2\-2b, gemma\-2\-2b\-it, gemma\-2\-9b, gemma\-2\-9b\-it, gemma\-2\-27b, gemma\-2\-27b\-it\.

#### Gemma\-3\.

gemma\-3\-270m, gemma\-3\-270m\-it, gemma\-3\-1b\-pt, gemma\-3\-1b\-it, gemma\-3\-4b\-pt, gemma\-3\-4b\-it, gemma\-3\-12b\-pt, gemma\-3\-12b\-it, gemma\-3\-27b\-pt, gemma\-3\-27b\-it\.

#### OLMo\-2\.

OLMo\-2\-0425\-1B, OLMo\-2\-0425\-1B\-Instruct, OLMo\-2\-1124\-7B, OLMo\-2\-1124\-7B\-Instruct, OLMo\-2\-1124\-13B, OLMo\-2\-1124\-13B\-Instruct, OLMo\-2\-0325\-32B, OLMo\-2\-0325\-32B\-Instruct\.

#### DeepSeek\.

deepseek\-llm\-7b\-base, deepseek\-llm\-7b\-chat, deepseek\-llm\-67b\-base, deepseek\-llm\-67b\-chat\.

#### Falcon\-3\.

Falcon3\-1B\-Base, Falcon3\-1B\-Instruct, Falcon3\-3B\-Base, Falcon3\-3B\-Instruct, Falcon3\-7B\-Base, Falcon3\-7B\-Instruct, Falcon3\-10B\-Base, Falcon3\-10B\-Instruct\.

#### Falcon\.

falcon\-7b, falcon\-7b\-instruct, falcon\-40b, falcon\-40b\-instruct\.

#### SmolLM2\.

SmolLM2\-135M, SmolLM2\-135M\-Instruct, SmolLM2\-360M, SmolLM2\-360M\-Instruct, SmolLM2\-1\.7B, SmolLM2\-1\.7B\-Instruct\.

#### SmolLM3\.

SmolLM3\-3B, SmolLM3\-3B\-Base\.

#### Apertus\.

Apertus\-8B\-2509, Apertus\-8B\-Instruct\-2509, Apertus\-70B\-2509, Apertus\-70B\-Instruct\-2509\.

#### Command\-R\.

c4ai\-command\-r7b\-12\-2024, c4ai\-command\-r\-08\-2024, c4ai\-command\-r\-plus\-08\-2024\.

#### Mistral\.

Mistral\-7B\-v0\.3, Mistral\-7B\-Instruct\-v0\.3, Ministral\-8B\-Instruct\-2410\.

## Appendix DDetails of Evaluation Setup

Here, we provide further details on the evaluation setup, including prompts used for MCQA for base and instruction\-tuned models, and implementation via vLLM\.

### D\.1Prompts

For the sake of comparison, we chose not to experiment with prompting techniques tailored to each specific model\. We followed general prompting guidelines and kept the prompt short, simple and to the point\. We divided the prompt in two parts, namelySystemandQuestionprompts\. The System prompt is identical for all models\. The main differences, that we report below, are in the way the question is asked to each model, with instruction\-tuned models receiving a direct question, and base models a statement to complete, and in the fact that, when available, the prompt was wrapped into the chat template of each model\. If the chat template was not available for the specific model \(i\.e\., for base models and older instruction\-tuned ones\), we simply concatenated the System and Question prompts and fed them to the model\.

#### System

> You are an expert AI\. Your task is to read a multiple\-choice question and provide the most likely Correct Answer based on the Options given\. Always output only the letter of the correct option\. Do not add the actual answer, commentary, explanations, or metadata — only output the letter corresponding to the correct answer\.

#### Question \- Instruct

> Question: \{\{ question \}\} Options: \{% for option in options \-%\} \{\{ option \}\} \{% endfor %\} Which is the correct answer?

#### Question \- Base

> Question: \{\{ question \}\} Options: \{% for option in options \-%\} \{\{ option \}\} \{% endfor %\} The letter corresponding to the correct answer is:

Note that neither the instruct nor base variant have trailing whitespace after the final character\. During early experimentation we found that while instruction\-tuned variants were resilient to this kind of variation, several base models struggled if a trailing whitespace was added after the colon, often generating the end of sequence token\. Thus, we chose to not include any trailing white space in the prompts\.

#### Chat Templates

For models that had it available, we wrap the prompt into the model’s chat template as follows:

```
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": QUESTION},
]
```

### D\.2Implementation

All models were evaluated locally on a GPU node equipped with A100 80GB GPUs\. Depending on the size of the model being tested, either one, two, or four GPUs were allocated for the experiment\. For example, models smaller than 20B parameters could be fitted on a single GPU, while 70\-100B parameter models required at least four GPUs to run\. All models were evaluated using FP\-16 variants provided by the original authors via HuggingFace\. Models were called using vLLM, specifically wrapping them into theLLMobject\.

For MCQA, we simply let the model generate a maximum of 15 new tokens with temperature set to zero\. To obtain a single, clear choice from model generations, we adopt a simple regex\-based strategy where we search for possible answers in the generated text\. We consider as possible answers either the letter in isolation, or the actual text of the answers\. For cases in which the model provided more than a single answer, e\.g\., by repeating the original list of answers, we mark it as an error\.

For likelihood\-based evaluation, we pass each statement through the same pipeline, without any further prompting\. We generate only one new token with temperature zero, and output log probabilities for the prompt\. Then, we compute log probabilities of the whole sequence, and for each data point we assign a rank to the statements, from most likely \(i\.e\., lowest logprobs\) to the least likely\.

## Appendix EAdditional Results

### E\.1Per\-dataset results

![Refer to caption](https://arxiv.org/html/2609.29390v1/Figure3.png)Figure 8:Prompting\-based and declarative\-statement likelihood accuracies, for instruction\-tuned and base models, as a function of model scale, for each dataset\.Figure[8](https://arxiv.org/html/2609.29390#A5.F8)reports the same analysis as Figure[2](https://arxiv.org/html/2609.29390#S3.F2)broken down by dataset\. We still observe the same qualitative pattern: APS exhibits weak scaling and limited sensitivity to instruction\-tuning, while APX scales sharply with model size and benefits substantially from instruction\-tuning\. Although absolute performance levels vary across datasets, the divergent behavior between APS and APX is generally consistent, indicating that the mismatch is not driven by any single benchmark\.

### E\.2PPL of correct answer

Figure[9](https://arxiv.org/html/2609.29390#A5.F9)displays the average PPL score assigned to the correct answer by each model, divided by dataset\. We observe that average PPL tends to decrease with model size, despite variability especially for middle\-sized models\.

![Refer to caption](https://arxiv.org/html/2609.29390v1/PPL-correct.png)Figure 9:Average PPL score assigned to the correct answer by each model, divided by dataset\.
### E\.3PPL delta

Figure[10](https://arxiv.org/html/2609.29390#A5.F10)shows the delta between second lowest and lowest PPL score for declarative statements vs model size, i\.e\., between the model’s actual choice and its second one\. Aside from some notable excepitions, especially in the 4 to 10B parameter range, the difference between the models’ choices and their second option is mostly near zero\.

![Refer to caption](https://arxiv.org/html/2609.29390v1/PPL-delta.png)Figure 10:Average delta between second lowest and lowest PPL score for declarative statements vs model size\.