From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation

# From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation
Source: [https://arxiv.org/html/2607.12687](https://arxiv.org/html/2607.12687)
Mehak Dhaliwal Rasta Tadayon Andong Hua Haewon Jeong Yao Qin University of California, Santa Barbara mdhaliwal@ucsb\.edu, yaoqin@ucsb\.edu

###### Abstract

LLMs can perform language\-based quantitative prediction from unstructured inputs, but remain susceptible to hallucinations and overconfident errors, making it critical to know not only*what*a model predicts, but*when*its predictions can be trusted\. We introduceCARE\-PPO, a reinforcement learning framework that establishes a connection between loss prediction for uncertainty estimation and actor–critic PPO fine\-tuning, enabling joint learning of accurate numerical estimates and reliable confidence signals in language\-based quantitative prediction\.CARE\-PPOuses aConfidence\-AlignedReward forEstimation, defined as a function of prediction error, to provide dense error\-aware feedback to the actor while inducing the critic to learn a value function aligned with prediction quality\. During inference, we repurpose the critic as a confidence estimator\. Across two real\-world tasks in healthcare and finance and two Qwen\-3 model scales \(4B and 8B\),CARE\-PPOachieves strong quantitative prediction performance, while producing significantly better\-aligned confidence estimates through the critic than logit\-based and verbalized baselines\. These gains persist under realistic out\-of\-distribution settings across domains, spanning linguistic and domain shifts\. Finally, CARE\-PPO reduces task\-specific overfitting on general instruction\-following prompts, consistent with the broader generalization advantages of RL fine\-tuning over supervised approaches\.

## 1Introduction

Recent advances in large language models \(LLMs\) have shown that they are not only powerful language generators, but also excel in quantitative prediction tasks across domains such as healthcare\(Huaet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib2)\), finance\(Vedulaet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib23)\), and scientific analysis\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2607.12687#bib.bib86)\)\. However, LLMs remain susceptible to hallucinations and overconfident errors\(Achiamet al\.,[2023](https://arxiv.org/html/2607.12687#bib.bib75)\), a limitation that is especially concerning in high\-stakes domains such as healthcare and law\(Dahlet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib73); Omaret al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib72)\)\. For instance, LLMs show promise for accurate carbohydrate estimation from meal descriptions\(Huaet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib2)\), a task critical for deciding insulin dosage in patients with diabetes, where overconfident but incorrect carbohydrate estimates can lead to severe glycemic consequences\. Deploying such models in practice therefore requires more than accurate point estimates — users in safety\-critical settings need to know not only*what*the model predicts, but also*when to trust*that prediction in order for humans to intervene when necessary\(Kalaiet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib79); Taoet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib24)\)\.

While a growing body of work studies confidence estimation for LLMs in classification and open\-ended generation tasks\(Genget al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib42)\), it remains largely underexplored for numerical prediction in language\-based settings, where existing work predominantly focuses on producing accurate point estimates without considering reliable confidence signals\(Tanget al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib14); Jacobset al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib11); Zausingeret al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib10); Vacareanuet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib7)\)\. To address this gap, we study language\-based quantitative prediction with confidence estimation, where an LLM generates numerical estimates through natural language reasoning, accompanied by confidence signals that reflect prediction reliability\.

![Refer to caption](https://arxiv.org/html/2607.12687v1/x1.png)Figure 1:Overview ofCARE\-PPO\(Confidence\-AlignedReward forEstimation with PPO\)\.Left:During training, the actor generates a numerical prediction from the input, which is evaluated using a confidence\-aligned reward defined as a function of prediction error\. The policy is optimized with PPO using GAE, while the critic is trained to regress to the empirical return\.Right:At inference time, the critic’s value estimate is repurposed as a confidence score for the actor’s prediction\.Specifically, we draw a novel connection between loss prediction for uncertainty estimation\(Gollakotaet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib80); Lahlouet al\.,[2021](https://arxiv.org/html/2607.12687#bib.bib87)\)and actor–critic reinforcement learning fine\-tuning for LLMs\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.12687#bib.bib66); Ouyanget al\.,[2022](https://arxiv.org/html/2607.12687#bib.bib32)\)\. Prior work characterizes a model’s predictive uncertainty at a given input as the expected loss incurred by the predictor at that point, making estimation of this quantity a loss prediction problem\(Gollakotaet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib80); Lahlouet al\.,[2021](https://arxiv.org/html/2607.12687#bib.bib87)\)\. In other words, identifying when a prediction is unreliable corresponds to identifying when the model is likely to incur high error\. This perspective naturally connects to the actor–critic framework of PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.12687#bib.bib66)\), where the critic learns a value function that estimates the expected return from a given state \(Equation[3](https://arxiv.org/html/2607.12687#S3.E3)\): when the reward is defined as a function of prediction error, the critic’s value function converges to an estimate of the actor’s expected error — realizing loss prediction as a natural byproduct of RL training rather than a post\-hoc procedure\.

Building on this, we introduceCARE\-PPO\(Confidence\-AlignedReward forEstimation with PPO\), a reinforcement learning framework for language\-based quantitative prediction that jointly trains an actor to generate accurate numerical estimates through natural language reasoning and a critic whose value function serves as a confidence estimator aligned with prediction error\. We design the reward as a monotonic function of prediction error, serving a dual purpose\. First, it provides a dense, error\-aware training signal that captures degrees of correctness and guides the actor toward accurate numerical predictions\. Second, because the critic is trained to predict returns under this reward, it learns a value function aligned with prediction quality: high values correspond to low expected error, while low values correspond to high expected error\. This allows the critic to naturally serve as a confidence estimator without requiring explicit confidence supervision\.

We illustrate this framework in Figure[1](https://arxiv.org/html/2607.12687#S1.F1), and summarize our main contributions below\.

- •Problem and insight\.We study language\-based quantitative prediction with confidence estimation and establish a connection between loss prediction theory and actor–critic RL fine\-tuning: when the reward is aligned with prediction error, the PPO critic naturally learns to estimate the actor’s expected loss, enabling reliable confidence estimation without explicit supervision\.
- •CARE\-PPO\.We proposeCARE\-PPO, a PPO\-based actor–critic framework with a Confidence\-Aligned Reward for Estimation\. By defining the reward as a monotonic function of prediction error,CARE\-PPOprovides dense error\-aware feedback to the actor while inducing the critic to serve as an error\-aligned confidence estimator at inference time\.
- •Comprehensive evaluation\.We evaluateCARE\-PPOon two real\-world quantitative prediction tasks spanning healthcare and finance across Qwen\-3 4B and 8B models\(Yanget al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib89)\), studying both Gaussian and linear reward formulations\.CARE\-PPOachieves competitive prediction performance while substantially improving confidence alignment over logit\-based and verbalized baselines\.
- •Robustness and task awareness\.We show thatCARE\-PPO’s confidence\-alignment gains persist under realistic out\-of\-distribution settings, including linguistic and domain shifts\. We further show thatCARE\-PPOreduces task\-specific overfitting on general instruction\-following queries, helping preserve general\-purpose behavior beyond the target prediction setting\.

## 2Related Work

##### LLMs for language\-based quantitative prediction\.

Prior work on applying LLMs to language\-based quantitative prediction has primarily followed three directions: \(i\) using LLM\-derived representations as fixed input features for downstream models\(Imperial,[2021](https://arxiv.org/html/2607.12687#bib.bib13); Tanget al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib14)\), \(ii\) adapting LLMs through task\-specific fine\-tuning to directly produce numerical predictions\(Jacobset al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib11); Zausingeret al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib10)\), and \(iii\) employing in\-context learning to perform zero\-shot or few\-shot numerical estimation from natural\-language prompts\(Vacareanuet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib7); Gopaliet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib15)\)\. While feature\-based approaches can be effective, they treat the LLM primarily as an encoder and thus do not leverage its generative interface for interactive use or natural\-language explanations\. Meanwhile, both fine\-tuning and in\-context learning emphasize point estimates rather than rich language generation\. In this study, we instead focus on fine\-tuning LLMs for language\-based quantitative prediction in a way that preserves their generative and conversational capabilities for transparency and interaction\.

##### Confidence estimation in LLMs\.

The growing adoption of LLMs has increased the need for reliable confidence estimates aligned with answer correctness\. Prior work can be broadly categorized intoblack\-boxandwhite\-boxapproaches, depending on whether internal model states are accessed\(Genget al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib42)\)\.

Black\-boxapproaches infer confidence using only model outputs\. These include confidence verbalization methods\(Linet al\.,[2022](https://arxiv.org/html/2607.12687#bib.bib17); Xionget al\.,[2023](https://arxiv.org/html/2607.12687#bib.bib16); Tianet al\.,[2023](https://arxiv.org/html/2607.12687#bib.bib41); Yanget al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib18)\), which often suffer from systematic overconfidence\(Weiet al\.,[2022](https://arxiv.org/html/2607.12687#bib.bib31); Meiet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib43); Kirichenkoet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib44)\), as well as sampling\-based techniques that estimate confidence from output variability using consensus heuristics\(Kanget al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib52)\), semantic diversity\(Aichbergeret al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib53)\), or entropy\-based measures\(Kuhnet al\.,[2023](https://arxiv.org/html/2607.12687#bib.bib20)\)\.White\-boxmethods instead leverage internal signals such as token probabilities, entropy, or likelihood shifts\(Guptaet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib54); Duanet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib21); Huanget al\.,[2023](https://arxiv.org/html/2607.12687#bib.bib19); Kadavathet al\.,[2022](https://arxiv.org/html/2607.12687#bib.bib55)\), or train probes over hidden representations to predict correctness probability\(Azaria and Mitchell,[2023](https://arxiv.org/html/2607.12687#bib.bib56); Mielkeet al\.,[2022](https://arxiv.org/html/2607.12687#bib.bib57)\)\. Despite their differences, most approaches infer confidence post hoc rather than optimizing it during training\. In contrast,CARE\-PPOexplicitly trains a critic to align predicted confidence with quantitative prediction error, enabling confidence estimates that are directly optimized during training rather than inferred after the fact\.

##### Confidence\-aware LLM fine\-tuning\.

At the intersection of confidence estimation and fine\-tuning for language models, prior work primarily follows two directions\. One line of work incorporates confidence or uncertainty signals as auxiliary rewards to improve downstream task performance, including speaker–listener frameworks\(Stengel\-Eskinet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib30)\)and reinforcement learning methods that derive rewards from self\-confidence\(Liet al\.,[2025a](https://arxiv.org/html/2607.12687#bib.bib28); Heet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib27)\)\. A second line of work directly fine\-tunes models to output calibrated confidence or uncertainty, with approaches including SFT methods that optimize verbalized confidence using tokenized Brier score losses\(Liet al\.,[2025b](https://arxiv.org/html/2607.12687#bib.bib29)\), as well as methods that leverage human preference signals to guide confidence calibration\(Taoet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib24)\)\. These methods inherit the limitations of SFT, which can suffer from “catastrophic forgetting” and degradation of prior capabilities\(Luoet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib34)\)\. In contrast, ourCARE\-PPOframework jointly optimizes for correctness and confidence calibration, explicitly rewarding accurate predictions while learning well\-calibrated uncertainty estimates\. In addition, our approach yields reliable confidence estimates without the need for verbalization, or external supervision, while preserving full generative expressiveness and structured reasoning behavior\.

## 3Methodology

### 3\.1PPO for Language\-Based Quantitative Prediction

Following prior work, we model language generation as a Markov Decision Process \(MDP\), where states correspond to token sequences and the initial state,s0s\_\{0\}, is given by the input prompt\(Shoaeinaeini and Harrison,[2024](https://arxiv.org/html/2607.12687#bib.bib65); Stangelet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib25)\)\. At each timesteptt, the actor \(policy\)πθ\\pi\_\{\\theta\}selects an actionat∼πθ\(⋅∣st\)a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\), corresponding to the next generated token\. Full details of the MDP formulation are provided in Appendix[A](https://arxiv.org/html/2607.12687#A1)\.

We optimize the policy using an actor–critic framework based on Proximal Policy Optimization \(PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.12687#bib.bib66)\), which maximizes a clipped surrogate objective:

ℒPPO​\(θ\)=𝔼t​\[min⁡\(rt​\(θ\)​At,clip​\(rt​\(θ\),1−ϵ,1\+ϵ\)​At\)\],\\mathcal\{L\}\_\{\\text\{PPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\}\\left\[\\min\\left\(r\_\{t\}\(\\theta\)A\_\{t\},\\text\{clip\}\\left\(r\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\right\)A\_\{t\}\\right\)\\right\],\(1\)wherert​\(θ\)=πθ​\(at∣st\)πθold​\(at∣st\)r\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)\}denotes the importance sampling ratio between the current policyπθ\\pi\_\{\\theta\}and the behavior policyπθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}used to collect trajectories, andϵ\>0\\epsilon\>0is a clipping hyperparameter that constrains the magnitude of policy updates\. We compute the advantage estimateAtA\_\{t\}in Equation[1](https://arxiv.org/html/2607.12687#S3.E1)using Generalized Advantage Estimation \(GAE\)\(Schulmanet al\.,[2015](https://arxiv.org/html/2607.12687#bib.bib67)\):

At=∑k=0T−t−1\(γ​λ\)k​δt\+k,\{A\}\_\{t\}=\\sum\_\{k=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{k\}\\,\\delta\_\{t\+k\},\(2\)whereλ∈\[0,1\]\\lambda\\in\[0,1\]controls the bias–variance tradeoff of the estimator,γ∈\(0,1\]\\gamma\\in\(0,1\]is the discount factor,δt=R​\(st\)\+γ​Vϕ​\(st\+1\)−Vϕ​\(st\)\\delta\_\{t\}=R\(s\_\{t\}\)\+\\gamma V\_\{\\phi\}\(s\_\{t\+1\}\)\-V\_\{\\phi\}\(s\_\{t\}\)denotes the temporal\-difference error, andR​\(st\)R\(s\_\{t\}\)is the reward received at statests\_\{t\}\.

Here, the critic modelVϕV\_\{\\phi\}estimates the expected return from a given statests\_\{t\}and is trained concurrently with the actor by minimizing the value loss:

ℒV​\(ϕ\)=𝔼t​\[\(Vϕ​\(st\)−R^t\)2\],\\mathcal\{L\}\_\{V\}\(\\phi\)=\\mathbb\{E\}\_\{t\}\\left\[\\left\(V\_\{\\phi\}\(s\_\{t\}\)\-\\hat\{R\}\_\{t\}\\right\)^\{2\}\\right\],\(3\)whereR^t\\hat\{R\}\_\{t\}denotes the empirical return\. Following standard RLVR setups, we employ a terminal reward structure with reward functionR​\(sT\)∈ℝR\(s\_\{T\}\)\\in\\mathbb\{R\}, such that rewards are provided only at the terminal statesTs\_\{T\}\. Consequently, the return takes the formR^t=γT−t​R​\(sT\)\\hat\{R\}\_\{t\}=\\gamma^\{T\-t\}R\(s\_\{T\}\)\. In our setting, the terminal reward is computed based on the model’s final quantitative estimate produced after generating a reasoning trajectory in response to the input query\.

Under this formulation, the actor is incentivized to select actions that improve this expected return, while the critic learns to estimate the expected outcome of a partial reasoning trajectory\. In the following subsection, we describe our reward design, which aligns the critic’s value estimates with the actor’s numeric prediction error, enabling it to serve as an error\-aligned confidence estimator\.

### 3\.2CARE: Confidence\-Aligned Reward for Estimation

We design the reward functionR​\(sT\)R\(s\_\{T\}\)to serve a dual purpose:*\(i\)*providing a dense, error\-sensitive training signal that captures degrees of correctness and guides the actor toward accurate numerical predictions, and*\(ii\)*inducing a critic value function that can serve as an error\-aligned confidence estimator at inference time\.

##### Reward Definition

Lety^\\hat\{y\}denote the actor’s final numerical prediction extracted from the generated trajectory, and letyydenote the ground\-truth value\. We define the absolute prediction error asξ=\|y^−y\|\\xi=\|\\hat\{y\}\-y\|and the terminal reward as a monotonically decreasing function of prediction error:

R​\(sT\)=h​\(ξ\),R\(s\_\{T\}\)=h\(\\xi\),\(4\)whereh:ℝ≥0→ℝh:\\mathbb\{R\}\_\{\\geq 0\}\\rightarrow\\mathbb\{R\}is chosen so that smaller prediction errors receive larger rewards\. This yields an error\-sensitive training signal for the actor\. Since the critic is trained to estimate returns under this reward, its value function becomes aligned with prediction quality, enabling it to serve as a confidence signal at inference time\.

##### Reward Instantiations

We consider two instantiations of the reward functionhh, corresponding to different ways of mapping prediction error to reward\.

I: Linear reward:A natural choice is a linear function of prediction error:

hlinear​\(ξ\)=−ξ,h\_\{\\mathrm\{linear\}\}\(\\xi\)=\-\\xi,\(5\)
which directly preserves the magnitude of the error and provides a dense, unbounded training signal\. While this formulation encourages accurate prediction by penalizing larger errors more strongly, it does not produce a normalized or bounded target for the critic, making its interpretation as a confidence score less direct\.

II: Gaussian reward\.To obtain a bounded and interpretable confidence signal, we also consider a scaled Gaussian kernel over the prediction error:

hgauss​\(ξ\)=α​exp⁡\(−ξ22​σ2\),h\_\{\\mathrm\{gauss\}\}\(\\xi\)=\\alpha\\exp\\left\(\-\\frac\{\\xi^\{2\}\}\{2\\sigma^\{2\}\}\\right\),\(6\)whereσ\\sigmacontrols the task\-specific error tolerance andα\>0\\alpha\>0is a reward scaling factor\. This reward is smooth and bounded in\[0,α\]\[0,\\alpha\], assigning high reward to small errors and decaying toward zero as error increases\. The parameterσ\\sigmahas a natural interpretation as a tolerance scale: predictions within±σ\\pm\\sigmaof the ground truth receive high reward, while larger errors are penalized progressively\.

We refer to models trained with the linear and Gaussian reward instantiations asCARE\-PPO\-Linear andCARE\-PPO\-Gaussian, respectively\.

## 4Experimental Setup

### 4\.1Tasks & Datasets

We evaluate our approach on two real\-world quantitative estimation tasks spanning healthcare and finance, both requiring free\-form natural language inputs to produce numerical predictions\.

##### Nutrition Estimation

We study carbohydrate estimation from natural\-language meal descriptions, as illustrated in Figure[1](https://arxiv.org/html/2607.12687#S1.F1), motivated by its importance for diabetes managementBucket al\.\([2022](https://arxiv.org/html/2607.12687#bib.bib59)\)\. We generate natural\-language meal descriptions from both individual food items and combinations of food items derived from the FoodData Central \(FDC\) repository\(USDA,[2019](https://arxiv.org/html/2607.12687#bib.bib78)\)using GPT\-4o\-mini \(see Appendix[B\.1](https://arxiv.org/html/2607.12687#A2.SS1)for details and examples\)\. The resulting dataset contains 26,549 training examples and 6,226 validation examples\. For testing, we use theNutriBenchbenchmarkHuaet al\.\([2024](https://arxiv.org/html/2607.12687#bib.bib2)\), which consists of 4,826 examples\.

##### Product Price Prediction

Product price prediction from textual product descriptions is evaluated using the Amazon Reviews datasetHouet al\.\([2024](https://arxiv.org/html/2607.12687#bib.bib60)\)\. For our primary experiments, we focus on the Appliances category, where each input is constructed from product metadata fields \(e\.g\., name, category, rating, and descriptive attributes\)\. Full preprocessing details and examples are provided in Appendix[B\.2](https://arxiv.org/html/2607.12687#A2.SS2)\. The resulting dataset contains 16,082 training, 946 validation, and 3,784 test examples\.

### 4\.2Baselines

To evaluate theCARE\-PPOframework and its critic\-based confidence estimation, we compare against several training and confidence\-estimation baselines\.

##### Training Methods:

- •Base Model: The instruction\-tuned Qwen\-3 4B and 8B models, without task\-specific fine\-tuning
- •SFT: Models fine\-tuned via next\-token prediction on task\-specific chain\-of\-thought data\. For nutrition estimation, we construct rule\-based reasoning traces that decompose meal descriptions into item\-level carbohydrate estimates and aggregate them\. For price prediction, we follow prior work on reasoning distillation\(Guoet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib81); Hoet al\.,[2023](https://arxiv.org/html/2607.12687#bib.bib82)\)by generating reasoning traces using a stronger teacher model \(GPT\-4o\-mini\)\. Examples of reasoning traces are provided in Appendix[B](https://arxiv.org/html/2607.12687#A2)\.
- •PPO\-Binary: A PPO baseline using a binary correctness rewardhbinary​\(ξ\)=𝕀​\[ξ≤τ\]h\_\{\\mathrm\{binary\}\}\(\\xi\)=\\mathbb\{I\}\[\\xi\\leq\\tau\], whereτ\\tauis a task\-specific tolerance threshold\. This mirrors reinforcement learning from verifiable rewards \(RLVR\) setups\(Wenet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib64)\)and serves as a direct comparison to evaluate whether error\-sensitive reward shaping is necessary for language\-based quantitative prediction\.

##### Confidence Estimation Baselines

- •Answer Probability:We calculate the mean probability of the generated tokens in the prediction string as a measure of model confidenceTianet al\.\([2023](https://arxiv.org/html/2607.12687#bib.bib41)\)\.
- •EntropyInv:We derive a confidence metric from the average per\-token entropy \(H​\(s\)=−∑p​log⁡pH\(s\)=\-\\sum p\\log p\) of the generated sequence, a commonly used measure of uncertaintyHuanget al\.\([2023](https://arxiv.org/html/2607.12687#bib.bib19)\)\. We compute entropy over normalized top\-20 token probabilities at each decoding step\. To ensure comparability, we further normalize the average entropy by the maximum entropyHm​a​xH\_\{max\}and define confidence as the complement:Ce​n​t=1−H¯Hm​a​xC\_\{ent\}=1\-\\frac\{\\bar\{H\}\}\{H\_\{max\}\}\. We call this method EntropyInv\.
- •Verbalized Confidence:Prior work has shown that verbalized confidence methods can outperform logit\-based approaches for reinforcement learning–fine\-tuned modelsTianet al\.\([2023](https://arxiv.org/html/2607.12687#bib.bib41)\)\. We evaluate two methods, \(i\)Verb\. 1S \(one\-stage\), where the model is prompted to produce both the prediction and a confidence estimate in a single response, and \(ii\)Verb\. 2S \(two\-stage\), where the model first produces its prediction and is subsequently prompted to assign a confidence scoreTianet al\.\([2023](https://arxiv.org/html/2607.12687#bib.bib41)\)\.

### 4\.3Evaluation Metrics

##### Quantitative Estimation Performance

We evaluate quantitative prediction accuracy using the mean absolute error \(MAE\) between model predictions and ground\-truth values, where lower values indicate better performance\.

##### Confidence Alignment

We evaluate how well estimated confidence reflects prediction error using the Area Under the Sparsification Error \(AUSE\)\(Ilget al\.,[2018](https://arxiv.org/html/2607.12687#bib.bib74)\)for assessing the utility of confidence estimates for selective prediction\. AUSE is computed as the area between a sparsification curve, obtained by progressively removing predictions in order of lowest confidence and measuring the error of the remaining set, and an oracle curve obtained by removing predictions in order of highest true error\. A lower AUSE indicates that confidence more effectively ranks predictions by reliability\.

### 4\.4Training Details

Both actor and critic are initialized from instruction\-tuned Qwen\-3 models \(4B and 8B\)\. We generate 8 rollouts per prompt and use GAE withγ=λ=1\.0\\gamma=\\lambda=1\.0\. Actor and critic learning rates are set to10−610^\{\-6\}and10−510^\{\-5\}, respectively\. Models are trained for 5 epochs using VERLShenget al\.\([2024](https://arxiv.org/html/2607.12687#bib.bib40)\)on a single 8\-GPU node\.

For SFT, we select the checkpoint with the lowest validation MAE\. For PPO, we select the actor checkpoint with the lowest validation MAE and the corresponding critic with the best AUSE\.

For the Gaussian reward formulationhgaussh\_\{\\mathrm\{gauss\}\}, we setσ=10\\sigma=10, aligning with the error scale in carbohydrate estimation, where prior work considers errors within approximately7\.57\.5g to be a conservative acceptable range\(Huaet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib2)\)\. Under this parameterization, moderate errors \(e\.g\.,ξ≈7\.5\\xi\\approx 7\.5g\) correspond to relatively high confidence \(≈0\.75\\approx 0\.75\), while large errors \(e\.g\.,ξ≈30\\xi\\approx 30g\) map to near\-zero confidence\. We apply the sameσ\\sigmafor price prediction, as both tasks operate over comparable output ranges in our datasets\. We evaluate sensitivity to moderate variations inσ\\sigmaby conducting an ablation withσ∈\{8,9,10,11,12\}\\sigma\\in\\\{8,9,10,11,12\\\}using Qwen3\-4B on the nutrition estimation task \(Section[5\.4](https://arxiv.org/html/2607.12687#S5.SS4)\), and find that performance remains relatively stable across nearby values, particularly for confidence alignment\. We use a reward scaling factor ofα=8\\alpha=8\. Since the confidence score is obtained by normalizing the critic value byα\\alpha\(Equation[6](https://arxiv.org/html/2607.12687#S3.E6)\), this choice does not affect the resulting confidence scale\.

We set the correctness threshold forhbinaryh\_\{\\mathrm\{binary\}\}to7\.57\.5g for carbohydrate estimation\(Huaet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib2)\)and 5 USD for price prediction, reflecting moderate task\-specific error tolerances\. When no extractable prediction is produced, we assign the minimum reward value for each formulation:0for the bounded rewardshgaussh\_\{\\mathrm\{gauss\}\}andhbinaryh\_\{\\mathrm\{binary\}\}, and−500\-500for the unbounded rewardhlinearh\_\{\\mathrm\{linear\}\}, chosen based on the upper range of prediction errors observed on the base model validation sets\.

## 5Results

### 5\.1CARE\-PPO Jointly Improves Quantitative Accuracy and Confidence Alignment

Table[1](https://arxiv.org/html/2607.12687#S5.T1)reports quantitative prediction performance \(MAE\) across all training methods, tasks, and model scales\. Figure[2](https://arxiv.org/html/2607.12687#S5.F2)shows confidence alignment results \(AUSE\) for each confidence estimation method paired with each training approach\. Since the Verb\-1S method uses a different prompt that generates a new prediction alongside the confidence estimate, its MAE may differ slightly; we report the lowest MAE per method, with full results provided in Appendix[D\.1](https://arxiv.org/html/2607.12687#A4.SS1)\.

Across both tasks and model scales, both reward variants ofCARE\-PPO\(CARE\-PPO\-Linear andCARE\-PPO\-Gaussian\) substantially reduce MAE relative to the base model and consistently outperform PPO\-Binary\. This improvement over PPO\-Binary suggests that continuous, error\-sensitive rewards provide a more effective training signal than binary correctness rewards, which discard error magnitude information\. While SFT achieves lower MAE, it comes at the cost of poor confidence alignment: SFT’s confidence estimates show weak alignment with prediction error across both tasks, regardless of the confidence estimation method used\.

Table 1:Quantitative estimation performance \(MAE\) on nutrition \(carbohydrate\) estimation and product \(appliances\) price prediction\. We report the lowest MAE per training method across confidence estimators \(see Appendix[D\.1](https://arxiv.org/html/2607.12687#A4.SS1)for full results\)\. Lower is better \(↓\\downarrow\)\.Training MethodNutrition Estimation MAE↓\\downarrowPrice Prediction MAE↓\\downarrowQwen\-3 4BQwen\-3 8BQwen\-3 4BQwen\-3 8BBase Model18\.0715\.3718\.7718\.62SFT10\.239\.5112\.4811\.59PPO\-Binary14\.5812\.3814\.9715\.03\\rowcolorcareblue CARE\-PPO\-Linear11\.0711\.4312\.5612\.69\\rowcolorcareblue CARE\-PPO\-Gaussian11\.5010\.4214\.3713\.57![Refer to caption](https://arxiv.org/html/2607.12687v1/x2.png)Figure 2:Confidence alignment \(AUSE\) across confidence estimation methods, training approaches, tasks, and model scales\. Lower values indicate better confidence alignment\. Across all settings, critic\-based confidence estimation yields the strongest overall alignment\. Corresponding sparsification curves used to compute AUSE are visualized in Appendix[D](https://arxiv.org/html/2607.12687#A4)\(Figure[5](https://arxiv.org/html/2607.12687#A4.F5)\)\.In contrast, as shown in Figure[2](https://arxiv.org/html/2607.12687#S5.F2),CARE\-PPOconsistently produces significantly better\-aligned confidence estimates, with the critic emerging as the strongest confidence estimator across all methods, models, and settings\. We visualize the corresponding sparsification curves used to compute AUSE in Appendix[D](https://arxiv.org/html/2607.12687#A4)\(Figure[5](https://arxiv.org/html/2607.12687#A4.F5)\)\. Even for PPO\-Binary, where the reward provides no information about error magnitude, the critic achieves lower AUSE than all verbalized and logit\-based baselines, supporting our hypothesis that the actor–critic training dynamic naturally encourages the critic to learn a meaningful confidence signal even from a discretized reward\. When the reward is directly aligned with prediction error, as inCARE\-PPO\-Linear andCARE\-PPO\-Gaussian, this effect is further amplified, suggesting that closer alignment between reward and prediction loss leads to more reliable confidence estimation\. Section[5\.3](https://arxiv.org/html/2607.12687#S5.SS3)additionally visualizes how critic output distributions evolve throughout training and progressively align with the underlying reward structure\.

### 5\.2CARE\-PPO is Robust in Out\-of\-Distribution Settings

We evaluateCARE\-PPOunder two real\-world out\-of\-distribution settings: linguistic shift in nutrition estimation and domain shift in product price prediction\.

For nutrition estimation, we test on multilingual meal descriptions, reflecting realistic scenarios where users describe foods in different languages with culturally specific terms and dietary conventions\(Winataet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib91)\)\. We construct this test set from WHO dietary records\(Leclercqet al\.,[2019](https://arxiv.org/html/2607.12687#bib.bib83)\)spanning Spanish, Italian, Portuguese, Tagalog, and Amharic, using GPT\-4o\-mini to generate natural\-language meal descriptions from the food records \(Appendix[B\.1](https://arxiv.org/html/2607.12687#A2.SS1)\)\. We retain paired local\-language and English descriptions from the source records rather than relying on automated translation, preserving official terminology and reducing translation\-induced artifacts\.

For price prediction, users may query about products across a broad range of categories\. Thus, to assess generalization under domain shift, we additionally use the Electronics category from the Amazon Reviews datasetHouet al\.\([2024](https://arxiv.org/html/2607.12687#bib.bib60)\)as an out\-of\-distribution test set, randomly sampling 4,000 examples\.

Table 2:Quantitative estimation performance \(MAE\) under linguistic shift for nutrition estimation \(English\-trained models evaluated on multilingual queries\) and domain shift for price prediction \(Appliances\-trained models evaluated on Electronics\)\. We report the lowest MAE per training method across confidence estimators \(see Appendix[D\.2](https://arxiv.org/html/2607.12687#A4.SS2)for full results\)\. Lower is better \(↓\\downarrow\)\.Training MethodNutrition Estimation MAE \(↓\\downarrow\)*Linguistic Shift*Price Prediction MAE \(↓\\downarrow\)*Domain Shift*Qwen\-3 4BQwen\-3 8BQwen\-3 4BQwen\-3 8BBase Model32\.4728\.1529\.1323\.36SFT30\.0224\.9517\.5616\.88PPO\-Binary26\.0623\.3619\.0217\.25\\rowcolorcareblue CARE\-PPO\-Linear23\.4820\.6420\.4717\.01\\rowcolorcareblue CARE\-PPO\-Gaussian24\.0024\.2619\.1715\.79![Refer to caption](https://arxiv.org/html/2607.12687v1/x3.png)Figure 3:Confidence alignment \(AUSE\) across confidence estimation methods, training approaches, tasks, and model scales in out\-of\-distribution settings\. Lower values indicate better confidence alignment\. Across all settings, critic\-based confidence estimation yields the strongest overall alignment\. Corresponding sparsification curves used to compute AUSE are visualized in Appendix[D](https://arxiv.org/html/2607.12687#A4)\(Figure[6](https://arxiv.org/html/2607.12687#A4.F6)\)\.#### 5\.2\.1Linguistic Shift in Nutrition Estimation

Table[2](https://arxiv.org/html/2607.12687#S5.T2)\(left\) reports MAE under linguistic shift, where models trained on English meal descriptions are evaluated on multilingual queries spanning five languages\. Notably,CARE\-PPOachieves stronger MAE performance than SFT under multilingual shift, suggesting that PPO\-based approaches may learn representations that generalize more robustly across languages\. Additional analysis in Table[7](https://arxiv.org/html/2607.12687#A4.T7)in the Appendix further shows that PPO\-based methods exhibit smaller MAE gaps between parallel English and non\-English queries describing the same meal than SFT and the base model\. Simultaneously, Figures[3](https://arxiv.org/html/2607.12687#S5.F3)\(a\)and\(b\)demonstrate that critic\-based confidence estimates consistently achieve substantially lower AUSE than verbalized and logit\-based confidence estimation methods across both model scales, demonstrating strong robustness under linguistic shift\.

#### 5\.2\.2Domain Shift in Price Prediction

Table[2](https://arxiv.org/html/2607.12687#S5.T2)\(right\) reports MAE, while Figures[3](https://arxiv.org/html/2607.12687#S5.F3)\(c\)and\(d\)show the AUSE of confidence estimates under domain shift, where models trained on Appliances are evaluated on unseen Electronics products\. Confidence alignment degrades substantially for the base model and SFT, whereas critic\-based confidence estimates learned through PPO \(both PPO\-Binary and theCARE\-PPOvariants\) remain substantially better aligned under shift while maintaining competitive MAE performance\.

Overall, across both linguistic and domain shifts, critic\-based confidence estimates learned through PPO remain better aligned, more stable, and more robust than verbalized or logit\-based alternatives, suggesting that such confidence signals are more likely to remain reliable under the distribution shifts encountered in practical deployment settings\.

#### 5\.2\.3CARE\-PPO Preserves Task\-Aware Behavior

In practice, users interact with LLMs in open\-ended settings and may issue queries beyond the deployment use case, making it important to preserve general\-purpose behavior even after specialization to a quantitative prediction task\. For instance, a model fine\-tuned for nutrition estimation should recognize when a query is unrelated to food and respond appropriately, rather than attempting to estimate carbohydrates\. We therefore evaluate whether CARE\-PPO exhibits more task\-aware behavior on general\-purpose queries outside the training distribution, specifically studying whether models incorrectly force task\-specific predictions on unrelated inputs\.

Table 3:Task overfitting evaluation for nutrition estimation on 500 general instruction\-following queries from IFEval\(Zhouet al\.,[2023](https://arxiv.org/html/2607.12687#bib.bib84)\)\. We report the proportion of responses that incorrectly treat unrelated inputs as meal descriptions and produce carbohydrate estimates\. Lower is better \(↓\\downarrow\)\.ModelBackboneBaseModelSFTPPOBinary\\cellcolorcareblueCARE\-PPOLinear\\cellcolorcareblueCARE\-PPOGaussianQwen\-3 4B33%73%1%\\cellcolorcareblue 29\.2%\\cellcolorcareblue7\.6%Qwen\-3 8B41\.8%33\.2%12%\\cellcolorcareblue 16\.8%\\cellcolorcareblue11\.2%To evaluate this behavior, we test models on 500 randomly sampled queries from the instruction\-following benchmark IFEval\(Zhouet al\.,[2023](https://arxiv.org/html/2607.12687#bib.bib84)\), which contains prompts unrelated to the target prediction tasks\. We focus on nutrition estimation as it provides the clearest signal of task overfitting: interpreting an unrelated query as a meal description and producing a carbohydrate estimate is an unambiguous failure mode\. In contrast, for price\-trained models, certain open\-ended prompts \(e\.g\., itinerary planning or writing assistance\) can plausibly be interpreted as requests for a service to be priced, making it harder to distinguish genuine overfitting from reasonable model behavior\. We use GPT\-4o\-mini as an LLM\-as\-a\-judge \(LLMaJ\)\(Guet al\.,[2024](https://arxiv.org/html/2607.12687#bib.bib85)\)to classify each response based on whether the model incorrectly produces a task\-specific prediction or responds appropriately\. We provide the LLMaJ prompt in Appendix[C\.3](https://arxiv.org/html/2607.12687#A3.SS3)\.

Table[3](https://arxiv.org/html/2607.12687#S5.T3)shows that SFT exhibits substantial task overfitting, frequently misinterpreting unrelated queries as meal descriptions\. Representative qualitative examples of these behaviors are provided in Appendix[D\.2\.1](https://arxiv.org/html/2607.12687#A4.SS2.SSS1)\. In contrast, all PPO\-based methods substantially reduce this behavior, consistent with prior findings that RL fine\-tuning generalizes more robustly beyond the training distribution than SFT\(Chuet al\.,[2025](https://arxiv.org/html/2607.12687#bib.bib71)\)\. Among PPO variants, PPO\-Binary and CARE\-PPO\-Gaussian exhibit the lowest misinterpretation rates, while CARE\-PPO\-Linear exhibits notably higher rates\. We hypothesize that this difference is partly attributable to reward design: becausehlinearh\_\{\\text\{linear\}\}is unbounded, the penalty assigned when no extractable prediction is produced is significantly larger in magnitude than for the bounded formulationshgaussh\_\{\\text\{gauss\}\}andhbinaryh\_\{\\text\{binary\}\}, potentially incentivizing the actor to always produce a formatted numerical output regardless of input relevance\. Overall, these results suggest that PPO\-based fine\-tuning more effectively preserves general\-purpose behavior beyond the target prediction setting, indicating better robustness in open\-ended deployment settings\.

### 5\.3Critic Distribution Evolution During Training

![Refer to caption](https://arxiv.org/html/2607.12687v1/x4.png)Figure 4:Evolution of critic output distributions throughout PPO training on the validation set for PPO\-Binary,CARE\-PPO\-Linear, andCARE\-PPO\-Gaussian for the nutrition estimation task with Qwen\-3 4B as the backbone model\. For each checkpoint, we visualize the distribution of ground\-truth rewards \(top\) and the corresponding final\-layer critic outputs \(bottom\)\. As training progresses, critic outputs increasingly reflect the structure of the reward distribution, corresponding to improved confidence alignment and lower AUSE\.To better understand how critic\-based confidence estimation emerges during training, we visualize the distribution of critic outputs throughout PPO optimization alongside the corresponding ground\-truth reward distributions on the validation set\. Figure[4](https://arxiv.org/html/2607.12687#S5.F4)shows the evolution of the final\-layer critic outputs across training checkpoints for PPO\-Binary,CARE\-PPO\-Linear, andCARE\-PPO\-Gaussian trained for the nutrition estimation task with Qwen\-3 4B as the backbone\. Corresponding plots for the price prediction task are presented in Appendix[F](https://arxiv.org/html/2607.12687#A6)\.

Across all reward formulations, the critic progressively learns to produce confidence estimates that better reflect the structure of the underlying reward distribution\. In particular, the critic distributions become increasingly shaped according to prediction quality over training, corresponding to improved confidence alignment\. While PPO\-Binary operates with a discretized reward signal, the critic nevertheless learns a meaningful ranking over prediction quality, supporting our hypothesis that actor–critic optimization naturally encourages confidence\-aware representations even under sparse reward formulations\.

### 5\.4Sensitivity Analysis withσ\\sigmaonCARE\-PPO\-Gaussian

Table 4:Sensitivity analysis of the Gaussian reward tolerance parameterσ\\sigma\. Lower is better \(↓\\downarrow\) for both MAE and AUSE\.σ\\sigmaMAE↓\\downarrowAUSE↓\\downarrow815\.923\.70912\.133\.061011\.723\.181113\.893\.391212\.193\.10The Gaussian reward formulation inCARE\-PPO\-Gaussian introduces a tolerance parameterσ\\sigmathat controls the smoothness of the reward landscape around the target value\. To evaluate sensitivity to this hyperparameter, we conduct an ablation overσ∈\{8,9,10,11,12\}\\sigma\\in\\\{8,9,10,11,12\\\}using Qwen\-3 4B on the nutrition estimation task\. Results are shown in Table[4](https://arxiv.org/html/2607.12687#S5.T4)\.

Overall, we find that performance remains relatively stable across nearby values ofσ\\sigma, particularly in terms of confidence alignment\. Whileσ=10\\sigma=10achieves the best overall balance between quantitative prediction accuracy and confidence calibration, neighboring values produce comparable AUSE performance, suggesting thatCARE\-PPO\-Gaussian is reasonably robust to moderate variations in the reward tolerance parameter\.

## 6Conclusion

We introducedCARE\-PPO, a reinforcement learning framework for language\-based quantitative prediction that draws a novel connection between loss prediction for uncertainty estimation and actor–critic RL fine\-tuning\. By defining rewards as functions of prediction error,CARE\-PPOenables the critic to learn value functions aligned with prediction quality, allowing the critic to serve as an effective confidence estimator at inference time\. Across quantitative prediction tasks in healthcare and finance,CARE\-PPOachieves competitive prediction accuracy while substantially improving confidence alignment relative to verbalized and logit\-based confidence estimation methods\. We further showed that these gains remain robust under realistic distribution shifts and extend to improved task\-aware behavior on out\-of\-task queries\.

We believe confidence\-aware language\-based quantitative prediction represents a promising research direction\. WhileCARE\-PPOrelies on PPO fine\-tuning, confidence estimation emerges as a byproduct of actor–critic training rather than requiring a separate confidence\-learning stage, calibration procedure, or auxiliary model\. Latency analysis shows that critic\-based confidence estimation introduces only a small inference overhead \(Appendix[E](https://arxiv.org/html/2607.12687#A5)\), suggesting that it can be incorporated into deployment with minimal additional cost\. Our experiments are currently limited to models up to 8B parameters and two application domains due to computational constraints\. Future work will explore scaling these results to larger language models and broader quantitative prediction settings, investigating theoretical connections between reward design and calibration, and integrating confidence estimates more directly into the generation process to more tightly couple prediction and uncertainty estimation\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p1.1)\.
- Improving uncertainty estimation through semantically diverse language generation\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- A\. Azaria and T\. Mitchell \(2023\)The internal state of an llm knows when it’s lying\.arXiv preprint arXiv:2304\.13734\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- S\. Buck, C\. Krauss, D\. Waldenmaier, C\. Liebing, N\. Jendrike, J\. Högel, B\. M\. Pfeiffer, C\. Haug, and G\. Freckmann \(2022\)Evaluation of meal carbohydrate counting errors in patients with type 1 diabetes\.Experimental and Clinical Endocrinology & Diabetes130\(07\),pp\. 475–483\.Cited by:[§4\.1](https://arxiv.org/html/2607.12687#S4.SS1.SSS0.Px1.p1.1)\.
- T\. Chu, Y\. Zhai, J\. Yang, S\. Tong, S\. Xie, D\. Schuurmans, Q\. V\. Le, S\. Levine, and Y\. Ma \(2025\)Sft memorizes, rl generalizes: a comparative study of foundation model post\-training\.arXiv preprint arXiv:2501\.17161\.Cited by:[§5\.2\.3](https://arxiv.org/html/2607.12687#S5.SS2.SSS3.p3.3)\.
- M\. Dahl, V\. Magesh, M\. Suzgun, and D\. E\. Ho \(2024\)Large legal fictions: profiling legal hallucinations in large language models\.Journal of Legal Analysis16\(1\),pp\. 64–93\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p1.1)\.
- J\. Duan, H\. Cheng, S\. Wang, A\. Zavalny, C\. Wang, R\. Xu, B\. Kailkhura, and K\. Xu \(2024\)Shifting attention to relevance: towards the predictive uncertainty quantification of free\-form large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5050–5063\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- J\. Geng, F\. Cai, Y\. Wang, H\. Koeppl, P\. Nakov, and I\. Gurevych \(2024\)A survey of confidence estimation and calibration in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6577–6595\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p2.1),[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Gollakota, P\. Gopalan, A\. Karan, C\. Peale, and U\. Wieder \(2025\)When does a predictor know its own loss?\.arXiv preprint arXiv:2502\.20375\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p3.1)\.
- S\. Gopali, B\. Chhetri, D\. Giri, S\. Siami\-Namini, and A\. S\. Namin \(2025\)In\-context and few\-shots learning for forecasting time series data based on large language models\.arXiv preprint arXiv:2512\.07705\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu,et al\.\(2024\)A survey on llm\-as\-a\-judge\.The Innovation\.Cited by:[§5\.2\.3](https://arxiv.org/html/2607.12687#S5.SS2.SSS3.p2.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§B\.2](https://arxiv.org/html/2607.12687#A2.SS2.p2.1),[2nd item](https://arxiv.org/html/2607.12687#S4.I1.i2.p1.1)\.
- N\. Gupta, H\. Narasimhan, W\. Jitkrittum, A\. S\. Rawat, A\. K\. Menon, and S\. Kumar \(2024\)Language model cascades: token\-level uncertainty and beyond\.arXiv preprint arXiv:2404\.10136\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- Q\. He, Q\. Ren, S\. Lei, X\. Wang, and Y\. Wang \(2025\)Beyond correctness: confidence\-aware reward modeling for enhancing large language model reasoning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 27215–27231\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px3.p1.1)\.
- N\. Ho, L\. Schmid, and S\. Yun \(2023\)Large language models are reasoning teachers\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 14852–14882\.Cited by:[§B\.2](https://arxiv.org/html/2607.12687#A2.SS2.p2.1),[2nd item](https://arxiv.org/html/2607.12687#S4.I1.i2.p1.1)\.
- Y\. Hou, J\. Li, Z\. He, A\. Yan, X\. Chen, and J\. McAuley \(2024\)Bridging language and items for retrieval and recommendation\.arXiv preprint arXiv:2403\.03952\.Cited by:[§B\.2](https://arxiv.org/html/2607.12687#A2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.12687#S4.SS1.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2607.12687#S5.SS2.p3.1)\.
- A\. Hua, M\. P\. Dhaliwal, R\. Burke, L\. Pullela, and Y\. Qin \(2024\)Nutribench: a dataset for evaluating large language models on nutrition estimation from meal descriptions\.arXiv preprint arXiv:2407\.12843\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.12687#S4.SS1.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2607.12687#S4.SS4.p3.11),[§4\.4](https://arxiv.org/html/2607.12687#S4.SS4.p4.7)\.
- Y\. Huang, J\. Song, Z\. Wang, S\. Zhao, H\. Chen, F\. Juefei\-Xu, and L\. Ma \(2023\)Look before you leap: an exploratory study of uncertainty measurement for large language models\.arXiv preprint arXiv:2307\.10236\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1),[2nd item](https://arxiv.org/html/2607.12687#S4.I2.i2.p1.3)\.
- E\. Ilg, O\. Cicek, S\. Galesso, A\. Klein, O\. Makansi, F\. Hutter, and T\. Brox \(2018\)Uncertainty estimates and multi\-hypotheses networks for optical flow\.InProceedings of the European Conference on Computer Vision \(ECCV\),pp\. 652–667\.Cited by:[§4\.3](https://arxiv.org/html/2607.12687#S4.SS3.SSS0.Px2.p1.1)\.
- J\. M\. Imperial \(2021\)BERT embeddings for automatic readability assessment\.arXiv preprint arXiv:2106\.07935\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Jacobs, M\. P\. Polak, L\. E\. Schultz, H\. Mahdavi, V\. Honavar, and D\. Morgan \(2024\)Regression with large language models for materials and molecular property prediction\.arXiv preprint arXiv:2409\.06080\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p2.1),[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- A\. T\. Kalai, O\. Nachum, S\. S\. Vempala, and E\. Zhang \(2025\)Why language models hallucinate\.arXiv preprint arXiv:2509\.04664\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p1.1)\.
- Z\. Kang, X\. Zhao, and D\. Song \(2025\)Scalable best\-of\-n selection for large language models via self\-certainty\.arXiv preprint arXiv:2502\.18581\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- P\. Kirichenko, M\. Ibrahim, K\. Chaudhuri, and S\. J\. Bell \(2025\)AbstentionBench: reasoning llms fail on unanswerable questions\.arXiv preprint arXiv:2506\.09038\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- L\. Kuhn, Y\. Gal, and S\. Farquhar \(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.arXiv preprint arXiv:2302\.09664\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- S\. Lahlou, M\. Jain, H\. Nekoei, V\. I\. Butoi, P\. Bertin, J\. Rector\-Brooks, M\. Korablyov, and Y\. Bengio \(2021\)DEUP: direct epistemic uncertainty prediction\.arXiv preprint arXiv:2102\.08501\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p3.1)\.
- C\. Leclercq, P\. Allemand, A\. Balcerzak, F\. Branca, R\. Sousa, A\. Lartey, M\. Lipp, V\. Quadros, and P\. Verger \(2019\)FAO/who gift \(global individual food consumption data tool\): a global repository for harmonised individual quantitative food consumption studies\.Proceedings of the Nutrition Society78\(4\),pp\. 484–495\.Cited by:[§B\.1](https://arxiv.org/html/2607.12687#A2.SS1.p7.1),[§5\.2](https://arxiv.org/html/2607.12687#S5.SS2.p2.1)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.\(2022\)Solving quantitative reasoning problems with language models\.Advances in neural information processing systems35,pp\. 3843–3857\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p1.1)\.
- P\. Li, M\. Skripkin, A\. Zubrey, A\. Kuznetsov, and I\. Oseledets \(2025a\)Confidence is all you need: few\-shot rl fine\-tuning of language models\.arXiv preprint arXiv:2506\.06395\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, M\. Xiong, J\. Wu, and B\. Hooi \(2025b\)Conftuner: training large language models to express their confidence verbally\.arXiv preprint arXiv:2508\.18847\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)Teaching models to express their uncertainty in words\.arXiv preprint arXiv:2205\.14334\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. Zhang \(2025\)An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.IEEE Transactions on Audio, Speech and Language Processing\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Mei, C\. Zhang, T\. Yin, J\. Lidard, O\. Shorinwa, and A\. Majumdar \(2025\)Reasoning about uncertainty: do reasoning models know when they don’t know?\.arXiv preprint arXiv:2506\.18183\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- S\. J\. Mielke, A\. Szlam, E\. Dinan, and Y\. Boureau \(2022\)Reducing conversational agents’ overconfidence through linguistic calibration\.Transactions of the Association for Computational Linguistics10,pp\. 857–872\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- M\. Omar, V\. Sorin, J\. D\. Collins, D\. Reich, R\. Freeman, N\. Gavin, A\. Charney, L\. Stump, N\. L\. Bragazzi, G\. N\. Nadkarni,et al\.\(2025\)Multi\-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support\.Communications Medicine5\(1\),pp\. 330\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p3.1)\.
- J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. Abbeel \(2015\)High\-dimensional continuous control using generalized advantage estimation\.arXiv preprint arXiv:1506\.02438\.Cited by:[§3\.1](https://arxiv.org/html/2607.12687#S3.SS1.p2.5)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p3.1),[§3\.1](https://arxiv.org/html/2607.12687#S3.SS1.p2.11)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2024\)HybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[§4\.4](https://arxiv.org/html/2607.12687#S4.SS4.p1.3)\.
- M\. Shoaeinaeini and B\. Harrison \(2024\)Guiding reinforcement learning using uncertainty\-aware large language models\.arXiv preprint arXiv:2411\.14457\.Cited by:[Appendix A](https://arxiv.org/html/2607.12687#A1.p1.14),[§3\.1](https://arxiv.org/html/2607.12687#S3.SS1.p1.4)\.
- P\. Stangel, D\. Bani\-Harouni, C\. Pellegrini, E\. Özsoy, K\. Zaripova, M\. Keicher, and N\. Navab \(2025\)Rewarding doubt: a reinforcement learning approach to calibrated confidence expression of large language models\.arXiv preprint arXiv:2503\.02623\.Cited by:[Appendix A](https://arxiv.org/html/2607.12687#A1.p1.14),[§3\.1](https://arxiv.org/html/2607.12687#S3.SS1.p1.4)\.
- E\. Stengel\-Eskin, P\. Hase, and M\. Bansal \(2024\)LACIE: listener\-aware finetuning for calibration in large language models\.Advances in Neural Information Processing Systems37,pp\. 43080–43106\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px3.p1.1)\.
- E\. Tang, B\. Yang, and X\. Song \(2024\)Understanding llm embeddings for regression\.arXiv preprint arXiv:2411\.14708\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p2.1),[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Tao, L\. Yao, H\. Ding, Y\. Xie, Q\. Cao, F\. Sun, J\. Gao, H\. Shen, and B\. Ding \(2024\)When to trust llms: aligning confidence with response quality\.arXiv preprint arXiv:2404\.17287\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p1.1),[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.arXiv preprint arXiv:2305\.14975\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1),[1st item](https://arxiv.org/html/2607.12687#S4.I2.i1.p1.1),[3rd item](https://arxiv.org/html/2607.12687#S4.I2.i3.p1.1)\.
- USDA \(2019\)FoodData central\.Note:[https://fdc\.nal\.usda\.gov/](https://fdc.nal.usda.gov/)Accessed: 2024\-09\-30Cited by:[§B\.1](https://arxiv.org/html/2607.12687#A2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.12687#S4.SS1.SSS0.Px1.p1.1)\.
- R\. Vacareanu, V\. Negru, V\. Suciu, and M\. Surdeanu \(2024\)From words to numbers: your large language model is secretly a capable regressor when given in\-context examples\.arXiv preprint arXiv:2404\.07544\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p2.1),[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Vedula, D\. Dhyani, L\. Jalali, B\. N\. Oreshkin, M\. Bayati, and S\. Malmasi \(2025\)Quantile regression with large language models for price prediction\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 12396–12415\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- X\. Wen, Z\. Liu, S\. Zheng, S\. Ye, Z\. Wu, Y\. Wang, Z\. Xu, X\. Liang, J\. Li, Z\. Miao,et al\.\(2025\)Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms\.arXiv preprint arXiv:2506\.14245\.Cited by:[3rd item](https://arxiv.org/html/2607.12687#S4.I1.i3.p1.2)\.
- G\. I\. Winata, F\. Hudi, P\. A\. Irawan, D\. Anugraha, R\. A\. Putri, W\. Yutong, A\. Nohejl, U\. A\. Prathama, N\. Ousidhoum, A\. Amriani,et al\.\(2025\)Worldcuisines: a massive\-scale benchmark for multilingual and multicultural visual question answering on global cuisines\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 3242–3264\.Cited by:[§5\.2](https://arxiv.org/html/2607.12687#S5.SS2.p2.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2023\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.arXiv preprint arXiv:2306\.13063\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[3rd item](https://arxiv.org/html/2607.12687#S1.I1.i3.p1.1)\.
- D\. Yang, Y\. H\. Tsai, and M\. Yamada \(2024\)On verbalized confidence scores for llms\.arXiv preprint arXiv:2412\.14737\.Cited by:[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px2.p2.1)\.
- J\. Zausinger, L\. Pennig, A\. Kozina, S\. Sdahl, J\. Sikora, A\. Dendorfer, T\. Kuznetsov, M\. Hagog, N\. Wiedemann, K\. Chlodny,et al\.\(2024\)Regress, don’t guess–a regression\-like loss on number tokens for language models\.arXiv preprint arXiv:2411\.02083\.Cited by:[§1](https://arxiv.org/html/2607.12687#S1.p2.1),[§2](https://arxiv.org/html/2607.12687#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§5\.2\.3](https://arxiv.org/html/2607.12687#S5.SS2.SSS3.p2.1),[Table 3](https://arxiv.org/html/2607.12687#S5.T3),[Table 3](https://arxiv.org/html/2607.12687#S5.T3.2.1)\.

## Appendix APPO Framework

We follow prior work in modeling language generation as a Markov Decision Process \(MDP\) defined by the tuple\(𝒮,𝒜,𝒫,R,γ\)\\left\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{P\},R,\\gamma\\right\)\[[41](https://arxiv.org/html/2607.12687#bib.bib65),[42](https://arxiv.org/html/2607.12687#bib.bib25)\]\. Each statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}represents a sequence of tokens drawn from a fixed vocabulary𝒱\\mathcal\{V\}\. The initial states0=\(x1,x2,…,xL\)s\_\{0\}=\(x\_\{1\},x\_\{2\},\\dots,x\_\{L\}\), wherexi∈𝒱x\_\{i\}\\in\\mathcal\{V\}, corresponds to the input prompt, including the system instruction and user query\. At each timestept∈\{0,…,T−1\}t\\in\\\{0,\\dots,T\-1\\\}, the actor \(policy\) modelπθ\\pi\_\{\\theta\}selects an actionat∼πθ\(⋅∣st\)a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\), whereat∈𝒱a\_\{t\}\\in\\mathcal\{V\}corresponds to generating the next token\. The environment transition function𝒫\\mathcal\{P\}is deterministic, forming the next state by appending the selected action to the current sequence,st\+1=\(st,at\)s\_\{t\+1\}=\(s\_\{t\},a\_\{t\}\)\. Generation terminates at a terminal statesTs\_\{T\}upon emitting an end\-of\-sequence token or reaching a maximum sequence lengthTT\. The discount factorγ∈\(0,1\]\\gamma\\in\(0,1\]controls the contribution of the terminal reward to earlier states\.

## Appendix BData Construction and Processing

### B\.1Nutrition Estimation

We construct training data for the nutrition estimation task using food composition entries from FoodData Central \(FDC\)\[[47](https://arxiv.org/html/2607.12687#bib.bib78)\], the food composition database maintained by the United States Department of Agriculture \(USDA\)\. Specifically, we first obtain food names along with their corresponding nutritional content for varying portion sizes\.

To convert these structured food entries into natural\-language training examples, we prompt GPT\-4o\-mini to generate realistic free\-form meal descriptions from the food items\. Concretely, we provide the model with food names and portion sizes from FDC entries and instruct it to synthesize meal descriptions in natural language\.

In addition to constructing descriptions directly from individual food entries, we also generate meal descriptions containing combinations of two food items to better reflect realistic meals consisting of multiple components\. The prompts used for meal description generation are shown in Box[B\.1](https://arxiv.org/html/2607.12687#A2.SS1)\.

Box B\.1: Nutrition Estimation Data Generation PromptsSingle\-Item Meal Description Generation PromptRole:You are a meal description bot\. Your style should be conversational and natural\. Your tone should be matter\-of\-fact\. Do not use flowery or overly descriptive words or phrasing\.Objective:Given an input, describe the meal in plain language as a regular person would\.Instructions:•Provide five different meal descriptions in a dictionary with keys “description1”, “description2”, etc\.•Extract theexact food names and serving sizesfrom each description and store them in the “items” field\.•Ensure consistency between “description” and “items”—the “items” field must not omit details that appear in the description\.•Return only the meal descriptions and items without additional information\.Context:Use the following examples of user input and model output as a guide:Input1``` { "parts": [ "smoothie, made with spinach, banana, almond milk, protein powder" ], "units": [ "1.0 glass" ] } ``` Output1``` { "description1": { "description": "I am having a glass of smoothie made with spinach, banana, almond milk, and protein powder for lunch.", "items": "a glass of smoothie made with spinach, banana, almond milk, and protein powder" }, etc } ``` Input2``` { "parts": [ "Beverages, NESTEA, tea, black, ready-to-drink, lemon" ], "units": [ "1.0 serving (8 fl oz)" ] } ``` Output2``` { "description1": { "description": "8 fl oz of NESTEA lemon black tea", "items1": "8 fl oz of NESTEA lemon black tea" }, etc. } ``` Input3``` { "parts": [ "pepperoni pizza" ], "units": [ "1.0 slice" ] } ``` Output3``` { “description1”: { "description": "One slice of pepperoni pizza", "items1": "one slice of pepperoni pizza" }, etc. } ``` Input4``` { "parts": [ "Breakfast tart, lowfat" ], "units": [ "1 Pop Tart" ] } ``` Output4``` { “description1”: { "description": "I have a low-fat Breakfast Tart.", "items1": "a low-fat Breakfast Tart." }, etc. } ``` Task:For each input, create five different meal descriptions in plain, conversational language\. Additionally, output the natural food names including natural serving sizes in the descriptions\.Multi\-Item Meal Description Generation PromptTask:Combine two meal descriptions into one, creating a natural\-sounding sentence as if spoken by a person\. If two queries describe different meals, merge them into a single meal description\.Input1``` [ ’43g garlic bread.’, ’100g grilled chicken with steamed vegetables.’, ’100g tomato soup.’ ] ``` Output1``` 43g garlic bread, 100g grilled chicken with steamed vegetables, and 100g tomato soup. ``` Input2``` [ ’For breakfast, I am having 136g banana.’, ’My lunch consists of 106g refrigerated whole wheat naan bread.’ ] ``` Output2``` For lunch, I am having 136g banana and 106g refrigerated whole wheat naan bread. ``` Instruction:Return only the combined meal descriptions, excluding any additional information\.

Finally, we generate synthetic reasoning traces for supervised fine\-tuning with chain\-of\-thought supervision\. These reasoning traces are constructed using a rule\-based procedure that \(i\) identifies individual food items and their portion sizes, \(ii\) estimates the carbohydrate content of each item, and \(iii\) aggregates these values to produce the final carbohydrate estimate\. Representative examples of the synthetic reasoning traces are provided in Example[B\.1](https://arxiv.org/html/2607.12687#A2.SS1)\.

Example B\.1: Nutrition Estimation Training Data ExamplesExample 1Query100 grams of gluten\-free granola with almond butter\.Synthetic Reasoning TraceThe meal consists of 100g almond butter gluten\-free granola\. 100g almond butter gluten\-free granola has 60\.00g carbs\. So the total carbs in the meal = 60\.00\. Output: \{"total\_carbohydrates": 60\.00\}Example 2QueryFor dinner, I am enjoying a cup of chili with beans that was heated in the microwave\.Synthetic Reasoning TraceThe meal consists of a cup of microwavable chili with beans\. A cup of microwavable chili with beans has 26\.55g carbs\. So the total carbs in the meal = 26\.55\. Output: \{"total\_carbohydrates": 26\.55\}Example 3QueryA serving of chunky soup featuring grilled chicken and sausage, weighing 100 grams, and 453\.6 grams of trimmed raw leg lamb, focusing on the sirloin half with a quarter inch of fat\.Synthetic Reasoning TraceThe meal consists of 100 grams of Grilled Chicken & Sausage Gumbo Chunky Soup, 453\.6 grams of raw leg lamb, sirloin half, trimmed to 1/4 inch of fat\. 100 grams of grilled chicken & sausage gumbo chunky soup has 6\.94g carbs\. 453\.6 grams of raw leg lamb, sirloin half, trimmed to 1/4 inch of fat has 0\.00g carbs\. So the total grams of carbs in the meal = \(6\.94 \+ 0\.00\) = 6\.94\. Output: \{"total\_carbohydrates": 6\.94\}

Further, for out\-of\-distribution evaluation under linguistic shift, we construct a multilingual nutrition estimation test set using dietary records from the WHO Global Individual Food consumption data Tool \(GIFT\)\[[28](https://arxiv.org/html/2607.12687#bib.bib83)\], spanning Spanish, Italian, Portuguese, Tagalog, and Amharic\. We use GPT\-4o\-mini to generate natural\-language meal descriptions from the structured dietary records while preserving the original language of each record\. The prompt used for multilingual meal description generation is provided in Box[B\.1](https://arxiv.org/html/2607.12687#A2.SS1)\. For the final test set, we randomly sample 100 queries from each language\. We provide representative meal examples in each language in Example[B\.1](https://arxiv.org/html/2607.12687#A2.SS1)\.

Box B\.2: Multilingual Nutrition Estimation Data Generation PromptYou are generating realistic meal log descriptions for nutrition estimation\.Given food items and weights, write 3 diverse meal descriptions in the target language, like how different real people would log the same meal\.Requirements:•All 3 descriptions must faithfully preserve the food items and quantities\. Do not add or infer any nutritional values \(calories, macros, etc\.\) that are not part of the food item names\.•Vary meaningfully across the following dimensions — each description should feel distinct:1\.Brevity: one ultra\-short \(telegraphic\), one mid\-length, one fuller sentence2\.Wording: different vocabulary, avoid repeating the same phrasing across descriptions3\.Sentence structure: mix styles \(e\.g\., bare list, full sentence, diary\-style prose\)4\.Quantity explicitness: at least one omits exact grams, at least one includes them5\.Level of detail: at least one summarizes the meal, at least one names every item•Do not add opinions, subjective commentary \(e\.g\., “quite filling”\), taste/texture descriptions, or cooking inferences unless explicitly stated in the food item name\.•Do not prefix descriptions with numbers, bullets, or labels\.•The output should resemble real\-world food diary or meal log entries\.•Some food item names may contain bilingual forms such as “Crisp eggplant / Mak Keua” or “Rice \(\\thaifontข้าว\)”\. Always use only the name that matches the target language\.•For non\-English outputs, vary sentence structure as much as in English — avoid repeating the same “\[quantity\]\[food\]” pattern across all 3 descriptions\.

Example B\.2: Examples of Multilingual Meal DescriptionsSpanishExample 1: Un poco de pan dulce de panadería, unos 70 gramos aproximadamente\. \(39\.47g carbohydrates\)Example 2: Para la comida, tomé un jugo de naranja industrializado, pesando 156 gramos, acompañado de 242 gramos de sopa de pasta tipo Maruchan\. \(39\.27g carbohydrates\)ItalianExample 1: Per il bimbo, ho dato 150 grammi di succo di pera Nipiol e 120 grammi di yogurt alla frutta Plasmon con mela e banana\. \(45\.66g carbohydrates\)Example 2: Per merenda ho mangiato una barretta fitness Nestlé al cioccolato di 27 grammi\. \(21\.82g carbohydrates\)PortugueseExample 1: 50g de café, 36g de cuscuz e 100g de ovos mexidos\. \(12\.02g carbohydrates\)Example 2: Pão integral 50g com margarina 10g e adoçante artificial 2g, acompanhado de duas vitaminas de 300g cada, uma de maçã e outra mista\. \(119\.01g carbohydrates\)TagalogExample 1: Isang maliit na ulam na may 3g gabi, 2g boiled galunggong, 60g white rice, plus 10g bawat isa ng nilutong kalabasa at upo\. \(20\.0g carbohydrates\)Example 2: Niluto ko ang 40g na gabi, 40g na kalabasa, at 10g na malunggay, sinamahan ng 225g na puting kanin at 10g na tsabita na pinirito\. \(81\.81g carbohydrates\)AmharicExample 1:\\amharicfontእኔ ለቁርስ የቦኮሎ ኩፊታ 138 ግራም እና ሙሉ ወተት 197 ግራም ጨምራለሁ። \(47\.6g carbohydrates\)Example 2:\\amharicfontዛሬ የተመረጡት ምግቦች ይህን ያካተቱ፡ የተለያዩ ከ64 ግራም ቆርን ብርድ \(ቅታ\) እና 73 ግራም የተከፈለ ምስር ስቲው ያሉበት ምሳ ነበር። \(48\.6g carbohydrates\)

### B\.2Product Price Prediction

For the product price prediction task, we use product metadata from the Amazon Reviews dataset\[[16](https://arxiv.org/html/2607.12687#bib.bib60)\]\. For our primary experiments, we focus on the Appliances category, which consists of 94\.3K items\. For each product, we construct a textual input by concatenating relevant product metadata fields \(e\.g\., item name, category, user rating, and descriptive attributes\)\. We filter the dataset to retain only items that contain at least one descriptive feature and have an associated price label, and we remove outliers whose prices fall above the 90th percentile\.

For reasoning supervision in the SFT training setup, we additionally construct synthetic chain\-of\-thought reasoning traces using GPT\-4o\-mini, following prior work on reasoning distillation for language models\[[12](https://arxiv.org/html/2607.12687#bib.bib81),[15](https://arxiv.org/html/2607.12687#bib.bib82)\]\. The prompt used for reasoning trace generation is provided in Box[B\.2](https://arxiv.org/html/2607.12687#A2.SS2), and representative examples are included in Example[B\.2](https://arxiv.org/html/2607.12687#A2.SS2)\.

Box B\.3: Product Price Prediction Reasoning Trace Generation Prompt\# TaskYou are a shopping assistant that explains why a product is priced the way it is\.Given the product information and its actual price below, write a short reasoning paragraph that explains why this product costs that amount\. Consider relevant factors such as:•The type of product and its typical market positioning•Key features or specifications•Brand or manufacturer•Customer ratings or popularity signals \(if available\)•Any other information that would reasonably affect priceYour reasoning should be consistent with the provided price\. Do not restate the price in your response — provide only the reasoning paragraph\.Now provide only the reasoning for the following query:

Example B\.3: Product Price Prediction Training Data ExamplesExample 1Querymain\_category: Appliancestitle: SB08088378 Broan Appliance Damperaverage\_rating: 4\.5store: Broandetails: \{"Brand Name": "Broan", "Item Weight": "0\.01 Ounces", "Installation Type": "Freestanding", "Part Number": "SB08088378", "Form Factor": "Compact", "Batteries Required?": "No", "Date First Available": "December 1, 2017", "Brand": "Broan", "Pattern": "Solid"\}Synthetic Reasoning TraceThe SB08088378 Broan Appliance Damper is priced at a premium due to its high\-quality construction and the reputable brand behind it\. Broan is well\-known in the appliance industry for producing reliable and durable products, which adds to the perceived value\. This particular damper features a compact form factor, making it ideal for various installation scenarios, and it is freestanding, offering flexibility for users\. The item is lightweight, which contributes to its ease of installation and usability\. Additionally, the product has an impressive average rating of 4\.5, indicating strong customer satisfaction and reliability, which further justifies its price point\. Overall, the combination of brand reputation, quality, and positive customer feedback positions this product as a worthwhile investment for consumers looking for a dependable appliance component\.\{"price": 22\.5\}Example 2Querymain\_category: Amazon Hometitle: R\.A\.M Reliаble Humidifіer Wick Filter forKenmоre Quiet Comfort 13\-4 Packaverage\_rating: 4\.6store: Artist Unknowncategories: Appliances; Parts & Accessories; Humidifier Parts & Accessories; Replacement Wicksdetails: \{"Is Discontinued By Manufacturer": "No", "Batteries required": "No"\}Synthetic Reasoning TraceThe R\.A\.M Reliable Humidifier Wick Filter for Kenmore Quiet Comfort is priced at a premium due to its high\-quality construction and effectiveness in maintaining optimal humidity levels in your home\. With an impressive average rating of 4\.6, this product has garnered positive customer feedback, indicating its reliability and performance\. The filter is designed specifically for the Kenmore Quiet Comfort model, ensuring a perfect fit and maximum efficiency, which adds to its value\. Additionally, as a replacement part, it plays a crucial role in extending the lifespan of the humidifier by maintaining its functionality\. The brand’s reputation in the market for producing durable and effective replacement parts further justifies the price point, making it a worthwhile investment for consumers looking to enhance their home environment\.\{"price": 32\.31\}Example 3Querymain\_category: Amazon Hometitle: Samsung DA63\-07180A Refrigerator Door Bin Genuine Original Equipment Manufacturer \(OEM\) Partaverage\_rating: 5\.0store: SAMSUNGcategories: Appliances; Parts & Accessories; Refrigerator Parts & Accessories; Binsdetails: \{"Package Dimensions": "12 x 4 x 4 inches", "Item Weight": "0\.01 ounces"\}Synthetic Reasoning TraceThe Samsung DA63\-07180A Refrigerator Door Bin is priced at a premium due to several factors\. As a genuine OEM part, it is specifically designed for compatibility with Samsung refrigerators, ensuring a perfect fit and reliable performance that third\-party alternatives may not guarantee\. The high average rating of 5\.0 indicates strong customer satisfaction and reliability, suggesting that buyers value quality and performance over cost\. Additionally, Samsung is a well\-established brand known for its quality appliances, which often commands higher prices due to its reputation for durability and innovation\. The precise dimensions and lightweight nature of the product further suggest careful engineering, which is typical of OEM parts, contributing to its overall value in the market\.\{"price": 79\.16\}

## Appendix CTask Prompts

### C\.1Nutrition Estimation

Box[C\.1](https://arxiv.org/html/2607.12687#A3.SS1)provides the system prompt used for the nutrition estimation task\. Boxes[C\.1](https://arxiv.org/html/2607.12687#A3.SS1)and[C\.1](https://arxiv.org/html/2607.12687#A3.SS1)provide the prompts used for one\-stage \(Verb\-1S\) and two\-stage \(Verb\-2S\) verbalized confidence estimation, respectively\.

Box C\.1: Nutrition Estimation System PromptFor the given query including a meal description, you are asked to respond in the query’s language\.In your response, think step by step as follows:1\.Parse the meal description into discrete food or beverage items along with their serving size\. If the serving size of any item in the meal is not specified, assume it is a single standard serving based on common nutritional guidelines \(e\.g\., USDA\)\. Ignore additional information that does not relate to the item name and serving size\.2\.For each food or beverage item in the meal, calculate the amount of carbohydrates in grams for the specific serving size\.3\.Respond with a dictionary object containing the total carbohydrates in grams as follows:\{"total\_carbohydrates": total grams of carbohydrates for the serving\}For the total carbohydrates, respond with just the numeric amount of carbohydrates without extra text\. If you do not know the answer, set the value of"total\_carbohydrates"to\-1\.Follow the format of the following examples when answering\.Query:"This morning, I had a cup of oatmeal with half a sliced banana and a glass of orange juice\."Answer:The meal consists of 1 cup of oatmeal, 1/2 a banana, and 1 glass of orange juice\.1 cup of oatmeal has 27g carbs\.1 banana has 27g carbs, so half a banana has\(27×12\)=13\.5\(27\\times\\frac\{1\}\{2\}\)=13\.5g carbs\.1 glass of orange juice has 26g carbs\.So the total grams of carbs in the meal=\(27\+13\.5\+26\)=66\.5=\(27\+13\.5\+26\)=66\.5\.Output:\{"total\_carbohydrates": 66\.5\}Query:"I ate scrambled eggs made with 2 eggs and a toast for breakfast\."Answer:The meal consists of scrambled eggs made with 2 eggs and 1 toast\.Scrambled eggs made with 2 eggs has 2g carbs\.1 toast has 13g carbs\.So the total grams of carbs in the meal=\(2\+13\)=15=\(2\+13\)=15\.Output:\{"total\_carbohydrates": 15\}Query:"Half a peanut butter and jelly sandwich\."Answer:The meal consists of 1/2 a peanut butter and jelly sandwich\.1 peanut butter and jelly sandwich has 50\.6g carbs, so half a peanut butter and jelly sandwich has\(50\.6×12\)=25\.3\(50\.6\\times\\frac\{1\}\{2\}\)=25\.3g carbs\.So the total grams of carbs in the meal=25\.3=25\.3\.Output:\{"total\_carbohydrates": 25\.3\}These examples are in English\. Make sure to use the query’s language in your answer\.

Box C\.2: Nutrition Estimation Verb\-1S PromptFor the given query including a meal description, you are asked to respond in the query’s language\.In your response, think step by step as follows:1\.Parse the meal description into discrete food or beverage items along with their serving size\. If the serving size of any item in the meal is not specified, assume it is a single standard serving based on common nutritional guidelines \(e\.g\., USDA\)\. Ignore additional information that does not relate to the item name and serving size\.2\.For each food or beverage item in the meal, calculate the amount of carbohydrates in grams for the specific serving size\.3\.Respond with a dictionary object containing:•The total carbohydrates in grams•Your confidence in this estimateThe confidence value should reflect the correctness of the total carbohydrate estimate\. Use high confidence when the estimate is accurate and based on clear, standard nutritional information\. Use low confidence when the estimate is inaccurate or largely a guess\.Use the following output format exactly:\{"total\_carbohydrates": total grams of carbohydrates for the serving, "confidence": numeric confidence value between 0 and 1\}Follow the format of the following examples when answering\.Query:"This morning, I had a cup of oatmeal with half a sliced banana and a glass of orange juice\."Answer:The meal consists of 1 cup of oatmeal, 1/2 a banana, and 1 glass of orange juice\.1 cup of oatmeal has 35g carbs\.1 banana has 20\.5g carbs, so half a banana has\(20\.5×12\)=10\.25\(20\.5\\times\\frac\{1\}\{2\}\)=10\.25g carbs\.1 glass of orange juice has 30g carbs\.So the total grams of carbs in the meal=\(35\+10\.25\+30\)=75\.25=\(35\+10\.25\+30\)=75\.25\.Output:\{"total\_carbohydrates": 75\.25, "confidence": 0\.68\}Query:"I ate scrambled eggs made with 2 eggs and a toast for breakfast\."Answer:The meal consists of scrambled eggs made with 2 eggs and 1 toast\.Scrambled eggs made with 2 eggs has 2g carbs\.1 toast has 13g carbs\.So the total grams of carbs in the meal=\(2\+13\)=15=\(2\+13\)=15\.Output:\{"total\_carbohydrates": 15, "confidence": 1\.0\}Query:"Half a peanut butter and jelly sandwich\."Answer:The meal consists of 1/2 a peanut butter and jelly sandwich\.1 peanut butter and jelly sandwich has 80g carbs, so half a peanut butter and jelly sandwich has\(80×12\)=40\(80\\times\\frac\{1\}\{2\}\)=40g carbs\.So the total grams of carbs in the meal=40=40\.Output:\{"total\_carbohydrates": 40, "confidence": 0\.34\}These examples are in English\. Make sure to use the query’s language in your answer\.

Box C\.3: Nutrition Estimation Verb\-2S PromptFor the given carbohydrate estimate for the meal description, respond with a dictionary object containing your confidence in this estimate as follows:\{"confidence": numeric confidence value between 0 and 1\}The confidence value should reflect the correctness of the total carbohydrate estimate\.Use high confidence when the estimate is accurate and based on clear, standard nutritional information\.Use low confidence when the estimate is inaccurate or largely a guess\.Follow the format of the following examples when answering\.Query:"This morning, I had a cup of oatmeal with half a sliced banana and a glass of orange juice\."Estimate:The meal consists of 1 cup of oatmeal, 1/2 a banana, and 1 glass of orange juice\.1 cup of oatmeal has 35g carbs\.1 banana has 20\.5g carbs, so half a banana has\(20\.5×12\)=10\.25\(20\.5\\times\\frac\{1\}\{2\}\)=10\.25g carbs\.1 glass of orange juice has 30g carbs\.So the total grams of carbs in the meal=\(35\+10\.25\+30\)=75\.25=\(35\+10\.25\+30\)=75\.25\.Output:\{"total\_carbohydrates": 75\.25\}Answer:\{"confidence": 0\.68\}Query:"I ate scrambled eggs made with 2 eggs and a toast for breakfast\."Estimate:The meal consists of scrambled eggs made with 2 eggs and 1 toast\.Scrambled eggs made with 2 eggs has 2g carbs\.1 toast has 13g carbs\.So the total grams of carbs in the meal=\(2\+13\)=15=\(2\+13\)=15\.Output:\{"total\_carbohydrates": 15\}Answer:\{"confidence": 1\.0\}Query:"Half a peanut butter and jelly sandwich\."Estimate:The meal consists of 1/2 a peanut butter and jelly sandwich\.1 peanut butter and jelly sandwich has 80g carbs, so half a peanut butter and jelly sandwich has\(80×12\)=40\(80\\times\\frac\{1\}\{2\}\)=40g carbs\.So the total grams of carbs in the meal=40=40\.Output:\{"total\_carbohydrates": 40\}Answer:\{"confidence": 0\.34\}Important:•Your response must contain only the dictionary, with no additional text\.•The value of"confidence"must be a single numeric value between 0 and 1 \(inclusive\)\.•Do not include any text after the dictionary\.Now provide your confidence for the following:

### C\.2Product Price Prediction

Box[C\.2](https://arxiv.org/html/2607.12687#A3.SS2)presents the system prompt used for the product price prediction task\. Boxes[C\.2](https://arxiv.org/html/2607.12687#A3.SS2)and[C\.2](https://arxiv.org/html/2607.12687#A3.SS2)present the prompts used for one\-stage and two\-stage verbalized confidence elicitation, respectively\.

Box C\.4: Product Price Prediction System Prompt\# TaskYou are a shopping assistant that estimates the price of a product based on its description\.Given the product information below, estimate the product’s price in USD using general market knowledge and reasonable assumptions\.First, explain your reasoning step by step\. In your reasoning, consider relevant factors such as:•The type of product and its typical market positioning•Key features or specifications•Brand or manufacturer•Customer ratings or popularity signals \(if available\)•Any other information that would reasonably affect priceUse only the provided information and general marketplace knowledge\.After completing the reasoning, output the final estimate on the last line only, in the following exact structured format:\{"price": estimated\_price\_in\_usd\}Important:•The final line must contain only the dictionary, with no additional text\.•The value of"price"must be a single numeric value \(e\.g\., 12\.99\)\.•Do not include currency symbols\.•Do not include any text after the final dictionary\.Now answer the following query:

Box C\.5: Product Price Prediction Verb\-1S Prompt\# TaskYou are a shopping assistant that estimates the price of a product based on its description\.Given the product information below, estimate the product’s price in USD using general market knowledge and reasonable assumptions\.First, explain your reasoning step by step\. In your reasoning, consider relevant factors such as:•The type of product and its typical market positioning•Key features or specifications•Brand or manufacturer•Customer ratings or popularity signals \(if available\)•Any other information that would reasonably affect priceUse only the provided information and general marketplace knowledge\.After completing the reasoning, respond with a dictionary object containing:•The estimated price in USD•Your confidence in this estimateThe confidence value should reflect the correctness of the price estimate\.Use high confidence when the estimate is accurate and based on clear, standard market information\.Use low confidence when the estimate is inaccurate or largely a guess\.In the last line after reasoning, use the following output format exactly:\{"price": estimated\_price\_in\_usd, "confidence": numeric confidence value between 0 and 1\}Important:•The final dictionary must contain exactly two keys:"price"and"confidence"\.•The final line must contain only the dictionary, with no additional text\.•The value of"price"must be a single numeric value \(e\.g\., 12\.99\)\.•Do not include currency symbols\.•The value of"confidence"must be a single numeric value between 0 and 1 \(inclusive\)\.•Do not include any text after the final dictionary\.Now answer the following query:

Box C\.6: Product Price Prediction Verb\-2S PromptFor the given price estimate for the product description, respond with a dictionary object containing your confidence in this estimate as follows:\{"confidence": numeric confidence value between 0 and 1\}The confidence value should reflect the correctness of the price estimate\.Use high confidence when the estimate is accurate and based on clear, standard market information\.Use low confidence when the estimate is inaccurate or largely a guess due to missing, ambiguous, or unusual product details\.Important:•Your response must contain only the dictionary, with no additional text\.•The value of"confidence"must be a single numeric value between 0 and 1 \(inclusive\)\.•Do not include any text after the dictionary\.Now provide your confidence for the following:

### C\.3LLM\-as\-Judge

To evaluate task\-aware behavior preservation, we use GPT\-4o\-mini as an LLM\-as\-a\-judge \(LLMaJ\) to classify whether model responses to unrelated instruction\-following queries appropriately answer the user request or incorrectly produce task\-specific predictions\. The judge prompt used for this evaluation is provided in Box[C\.3](https://arxiv.org/html/2607.12687#A3.SS3)\.

Box C\.7: LLM\-as\-a\-Judge Prompt for Task\-Aware Behavior EvaluationYou are evaluating the response of a language model\. The model was trained on a carbohydrate\-estimation task, but received an unrelated query as its user message\.The user query is:``` <query> {query} </query> ``` The model’s response is:``` <response> {response} </response> ``` Classify the response into exactly one of the following categories:•"incorrect\_task": The model incorrectly treated the query prompt as a meal description and attempted to estimate carbohydrates from it\.•"other": The model did not interpret the query as a meal description and instead attempted to answer the question or follow the instruction \(regardless of quality\), declined to answer, returned carbohydrates as\-1, etc\.Respond with a JSON object only, in the following exact format:``` {{"category": "<category>", "reason": "<one sentence explanation>"}} ```

## Appendix DDetailed Experimental Results

Table 5:Detailed results on nutrition estimation and product price prediction\. Results include MAE \(↓\\downarrow\) and AUSE \(↓\\downarrow\) across models, training approaches, and confidence estimation methods\.ModelConfidenceEstimation MethodNutrition EstimationPrice PredictionQwen\-3 4BQwen\-3 8BQwen\-3 4BQwen\-3 8BMAEAUSEMAEAUSEMAEAUSEMAEAUSEBaselineVerb\-1S18\.358\.0515\.376\.5118\.7712\.4018\.6612\.65Verb\-2S18\.078\.0215\.765\.9618\.9312\.8118\.6211\.89Answer Prob18\.078\.9815\.768\.1018\.938\.8518\.6213\.52EntropyInv18\.078\.8115\.767\.6618\.937\.8618\.6213\.42SFTVerb\-1S10\.275\.519\.515\.6012\.538\.8311\.5910\.44Verb\-2S10\.235\.639\.775\.5712\.4810\.4611\.8410\.08Answer Prob10\.234\.499\.774\.6512\.487\.7011\.847\.49EntropyInv10\.234\.239\.774\.4212\.487\.3911\.847\.05PPO\-BinaryVerb\-1S14\.587\.4112\.385\.2615\.1212\.1215\.3610\.91Verb\-2S14\.927\.0812\.725\.7314\.9712\.7415\.039\.45Answer Prob14\.928\.9112\.727\.0614\.978\.1515\.0310\.36EntropyInv14\.928\.8812\.727\.0014\.978\.1115\.0310\.35Critic14\.923\.7012\.722\.9314\.972\.8615\.032\.94CARE\-PPO\-Lin\.Verb\-1S11\.075\.0611\.434\.5612\.569\.0012\.757\.55Verb\-2S11\.125\.4011\.915\.1712\.569\.1112\.698\.41Answer Prob11\.125\.1111\.915\.0412\.5610\.0512\.699\.67EntropyInv11\.124\.8911\.914\.8812\.5610\.1612\.699\.79Critic11\.122\.9411\.912\.5912\.562\.7612\.692\.86CARE\-PPO\-Gauss\.Verb\-1S11\.505\.2910\.624\.0014\.388\.7113\.578\.36Verb\-2S11\.725\.1310\.424\.1814\.378\.0713\.608\.78Answer Prob11\.726\.5810\.424\.8214\.3711\.2413\.609\.75EntropyInv11\.726\.4110\.424\.5614\.3711\.4513\.609\.80Critic11\.723\.1810\.422\.5814\.372\.6813\.602\.52

### D\.1Primary Evaluation Results

Table[5](https://arxiv.org/html/2607.12687#A4.T5)present the complete evaluation results on the nutrition estimation and product price prediction tasks, including all confidence estimation methods across model scales and training approaches\. Corresponding sparsification curves are shown in Figure[5](https://arxiv.org/html/2607.12687#A4.F5)\.

Consistent with the results discussed in Section[5\.1](https://arxiv.org/html/2607.12687#S5.SS1), CARE\-PPO substantially improves confidence alignment relative to verbalized and logit\-based confidence estimation methods, with critic\-based confidence estimation achieving the strongest overall AUSE performance across both tasks and model scales\. While SFT achieves strong quantitative prediction performance in terms of MAE, its confidence estimates remain comparatively poorly aligned with prediction error\. In contrast, CARE\-PPO achieves competitive quantitative prediction performance while simultaneously producing substantially better\-aligned confidence estimates\.

![Refer to caption](https://arxiv.org/html/2607.12687v1/x5.png)Figure 5:Sparsification curves across confidence estimation methods, training approaches, tasks, and model scales in the primary evaluation setting\. Curves show MAE as increasingly low\-confidence predictions are removed\. Confidence estimates that more closely follow the oracle curve achieve better confidence ranking and lower AUSE\.
### D\.2Out\-of\-Distribution Results

Table[6](https://arxiv.org/html/2607.12687#A4.T6)presents detailed out\-of\-distribution evaluation results for the linguistic\-shift nutrition estimation task and the domain\-shift product price prediction task\. Corresponding sparsification curves are shown in Figure[6](https://arxiv.org/html/2607.12687#A4.F6)\.

CARE\-PPOmaintains the strongest confidence alignment across all confidence estimation baselines and training methods, while achieving competitive MAE for price prediction and the best MAE for nutrition estimation\. These results demonstrate thatCARE\-PPOis robust under both distributional and linguistic shifts, and provides more reliable predictions compared to baseline approaches in real\-world settings where input distributions vary, and users may interact with models in different languages\.

Table[7](https://arxiv.org/html/2607.12687#A4.T7)further evaluates robustness to linguistic shift usingΔ\\DeltaMAE between parallel English and non\-English queries describing identical meals\. Across both model scales,CARE\-PPOachieves the smallest performance gaps, suggesting stronger language\-agnostic generalization and more stable quantitative prediction performance across multilingual inputs\.

Table 6:Detailed OOD results on nutrition estimation and product price prediction\. Results include MAE \(↓\\downarrow\) and AUSE \(↓\\downarrow\) across models, training approaches, and confidence estimation methods\.ModelConfidenceEstimation MethodNutrition Estimation MAE \(↓\\downarrow\)Linguistic ShiftPrice Prediction MAE \(↓\\downarrow\)Domain ShiftQwen\-3 4BQwen\-3 8BQwen\-3 4BQwen\-3 8BMAEAUSEMAEAUSEMAEAUSEMAEAUSEBaselineVerb\-1S32\.4716\.9728\.1514\.4929\.1323\.8423\.3627\.50Verb\-2S35\.6912\.3829\.8314\.3431\.1827\.6124\.9925\.29Answer Prob35\.6926\.2929\.8317\.2131\.1826\.5524\.9923\.37EntropyInv35\.6925\.8429\.8316\.6231\.1824\.6124\.9923\.17SFTVerb\-1S30\.4515\.3324\.9517\.8917\.5612\.7116\.9714\.85Verb\-2S30\.0213\.2725\.3915\.7117\.7713\.6116\.8813\.38Answer Prob30\.0220\.4525\.3912\.5117\.7713\.9816\.8813\.17EntropyInv30\.0220\.6225\.3912\.2317\.7713\.5616\.8813\.05PPO\-BinaryVerb\-1S26\.0610\.0123\.3612\.6320\.1321\.8717\.2517\.50Verb\-2S27\.2410\.2823\.9210\.6619\.0216\.8017\.6214\.92Answer Prob27\.2414\.3723\.9212\.9219\.028\.6617\.6214\.56EntropyInv27\.2414\.7023\.9212\.5919\.028\.4817\.6214\.37Critic27\.245\.9023\.925\.9619\.023\.6217\.624\.86CARE\-PPO\-Lin\.Verb\-1S23\.4811\.9820\.6410\.1222\.3820\.5118\.2317\.16Verb\-2S23\.6912\.0221\.5710\.0320\.4721\.2917\.0113\.47Answer Prob23\.699\.0521\.579\.1720\.4715\.4717\.0111\.40EntropyInv23\.698\.6221\.578\.7120\.4715\.5417\.0111\.47Critic23\.697\.1421\.574\.9220\.473\.6117\.013\.49CARE\-PPO\-Gauss\.Verb\-1S24\.0013\.0724\.2612\.4419\.6213\.3116\.4017\.85Verb\-2S24\.3213\.2824\.6112\.2319\.1714\.3915\.7912\.06Answer Prob24\.3211\.2224\.619\.8319\.1717\.0515\.7910\.25EntropyInv24\.3210\.7024\.619\.1719\.1717\.2115\.7910\.32Critic24\.326\.1724\.615\.7619\.174\.3115\.793\.66

Table 7:Δ\\DeltaMAE between parallel English and non\-English queries describing identical meals, measuring the performance gap introduced by linguistic shift\. Lower values indicate smaller performance gaps, suggesting more language\-agnostic generalization\.ModelBackboneBaseModelSFTPPOBinary\\cellcolorcareblueCARE\-PPOLinear\\cellcolorcareblueCARE\-PPOGaussianQwen\-3 4B10\.257\.275\.05\\cellcolorcareblue 3\.44\\cellcolorcareblue 3\.91Qwen\-3 8B3\.305\.401\.71\\cellcolorcareblue 1\.94\\cellcolorcareblue 1\.90![Refer to caption](https://arxiv.org/html/2607.12687v1/x6.png)Figure 6:Sparsification curves across confidence estimation methods, training approaches, tasks, and model scales in the out\-of\-distribution evaluation setting\. Curves show MAE as increasingly low\-confidence predictions are removed\. Confidence estimates that more closely follow the oracle curve achieve better confidence ranking and lower AUSE\.#### D\.2\.1Examples of Task\-Aware Behavior Preservation

To further evaluate whether task\-specific fine\-tuning degrades general\-purpose instruction\-following capabilities, we present representative model outputs on unrelated instruction\-following queries for each training method across both Qwen3\-4B and Qwen3\-8B models\. These examples qualitatively complement the quantitative IFEval results discussed in Section[5\.2\.3](https://arxiv.org/html/2607.12687#S5.SS2.SSS3)\.

Example D\.1: Output Examples on Unrelated Instruction\-Following QueriesExample 1:Write a limerick about a Zelda fan named Rodney\. Make sure to include these items: Zelda, Hyrule, Link, Ganon\. Use less than 100 words\.Qwen3\-4B•Base Model:There once was a fan named Rodney,Who adored Zelda and all she’s known\.He chased Link through Hyrule,Fought Ganon with might and skill\.Now he’s a hero, no doubt, in the land of the old\.•SFT:The meal consists of a limerick about a Zelda fan named Rodney, including Zelda, Hyrule, Link, and Ganon\.A limerick about a Zelda fan named Rodney, including Zelda, Hyrule, Link, and Ganon, has 23\.53g carbs\.So the total carbs in the meal=23\.53=23\.53\.Output: \{"total\_carbohydrates": 23\.53\}•PPO\-Binary:Answer: “Write a limerick about a Zelda fan named Rodney\. Make sure to include these items: Zelda, Hyrule, Link, Ganon\. Use less than 100 words\.”Answer: There once was a fan named Rodney,Who adored Zelda and Hyrule’s story\.He chased Link through,To defeat Ganon,With a sword made of chocolate fry\.•CARE\-PPO\-Linear:There once was a fan named Rodney,Who adored Zelda and all that she’s known\.He chased Link through Hyrule,Fought Ganon with might and skill,And conquered the realm with his soul\.•CARE\-PPO\-Gaussian:There once was a fan named Rodney,Who adored Zelda and all that she’s known\.He chased Link through Hyrule,Fought Ganon with might and skill,And conquered the realm with his soul\.Qwen3\-8B•Base Model:\{"total\_carbohydrates": \-1\}•SFT:The meal consists of a limerick about a Zelda fan named Rodney, including Zelda, Hyrule, Link, and Ganon\.A limerick about a Zelda fan named Rodney, including Zelda, Hyrule, Link, and Ganon, has 20\.00g carbs\.So the total carbs in the meal=20\.00=20\.00\.Output: \{"total\_carbohydrates": 20\.00\}•PPO\-Binary:Rodney’s a Zelda fan true and bold,In Hyrule he’s always been toldTo follow Link’s quest,Defeat Ganon’s chest,And save the realm with a sword so cold\.•CARE\-PPO\-Linear:Rodney’s a Zelda fan, you see,In Hyrule, he’s always free\.With Link in his hand,He fights Ganon’s land—A hero’s quest, he’ll never flee\!Output: \{"total\_carbohydrates": \-1\}•CARE\-PPO\-Gaussian:Rodney’s a Zelda fan, you see,He’s questing through Hyrule with glee\.With Link by his side,They chase Ganon’s pride,And save the realm with a hero’s decree\.Example 2:make a tweet for playboy’s twitter account without using capital letters\. Include at least 4 hashtags, starting with ’\#’Qwen3\-4B•Base Model:the latest issue of playboy is here\! enjoy the vibes, the photos, and the stories\. stay hot and stay wild\. \#playboy \#vibes \#hot \#wild•SFT:The meal consists of a tweet for Playboy’s Twitter account, with no capital letters and 4 hashtags\.A tweet for Playboy’s Twitter account, with no capital letters and 4 hashtags, has 21\.42g carbs\.So the total carbs in the meal=21\.42=21\.42\.Output: \{"total\_carbohydrates": 21\.42\}•PPO\-Binary:Answer: “make a tweet for playboy’s twitter account without using capital letters\. include at least 4 hashtags, starting with ‘\#’\#playboystyle \#luxurylife \#hotmoments \#fashionfix”•CARE\-PPO\-Linear:the latest issue of playboy is here\! stay stylish, stay confident, and stay curious\. \#playboy \#style \#confidence \#curiosity•CARE\-PPO\-Gaussian:i just had the most amazing time at playboy’s latest event\! so smooth, so stylish, so fun\. \#playboy \#luxury \#style \#funnyQwen3\-8B•Base Model:\{"total\_carbohydrates": \-1\}•SFT:The meal consists of a tweet for Playboy’s Twitter account without any capital letters, including at least 4 hashtags\.A tweet for Playboy’s Twitter account without any capital letters, including at least 4 hashtags has 28\.00g carbs\.So the total carbs in the meal=28\.00=28\.00\.Output: \{"total\_carbohydrates": 28\.00\}•PPO\-Binary:the ultimate flex is in the details\. stay sharp, stay stylish, and always keep it classy\. \#playboy \#stylegoals \#luxurylife \#chicvibes•CARE\-PPO\-Linear:Okay, let’s tackle this query\. The user wants a tweet for Playboy’s Twitter account without any capital letters\. They also need at least four hashtags starting with \#\.First, I need to make sure the tweet is in lowercase\. Playboy’s content is usually edgy and playful, so the tone should match that\. Maybe something catchy and engaging\. Let me brainstorm some ideas\.How about starting with a playful statement\. Maybe something like “spend your weekend feeling fabulous…” That’s lowercase and sets a positive tone\.Now, adding hashtags\. They need to start with \# and be relevant\. Playboy’s tags might include \#playboy, \#fashion, \#style, and maybe \#entertainment\. Let me check if those are appropriate\.Wait, the user specified at least four hashtags\. Let me make sure there are four\. Also, no capital letters\. Let me put it all together\.“spend your weekend feeling fabulous with our latest issue\! \#playboy \#fashion \#style \#entertainment”That’s four hashtags, all lowercase, and the message is engaging\. I think that works\. Let me double\-check for any capitals\. Yep, everything is lowercase\. Looks good\.the ultimate guide to living your best life\. \#playboy \#fashion \#style \#entertainment•CARE\-PPO\-Gaussian:the ultimate guide to staying cool and confident this summer\. slip into your favorite swimwear, soak up the sun, and let your vibe shine\. remember, confidence is the best accessory\. \#summervibes \#suncare \#confidenceboost \#playboy

## Appendix EInference Latency Analysis

We evaluate the inference overhead introduced by critic\-based confidence estimation\. Using 100 randomly sampled examples from the nutrition estimation test set, we measure end\-to\-end latency for both actor generation and critic confidence estimation\. On average, the actor requires 3108\.2 ms per sample, while the critic requires only 74\.0 ms, corresponding to a 2\.4% increase in total inference latency\. This small overhead arises because the critic performs only a single forward pass on the completed response, whereas the actor requires multiple autoregressive decoding steps proportional to the output length\. These results suggest that critic\-based confidence estimation can be incorporated into deployment with minimal additional inference cost\.

## Appendix FCritic Distribution Evolution During Training\- Price Prediction

Figure[7](https://arxiv.org/html/2607.12687#A6.F7)presents the evolution of critic output distributions throughout PPO training on the price prediction task with Qwen\-3 4B as the backbone model, complementing the nutrition estimation analysis in Section[5\.3](https://arxiv.org/html/2607.12687#S5.SS3)\. Consistent with the patterns observed there, the critic progressively learns to produce confidence estimates that better reflect the structure of the underlying reward distribution across all three reward formulations\.

![Refer to caption](https://arxiv.org/html/2607.12687v1/x7.png)Figure 7:Evolution of critic output distributions throughout PPO training on the validation set for PPO\-Binary,CARE\-PPO\-Linear, andCARE\-PPO\-Gaussian for the price prediction task with Qwen\-3 4B as the backbone model\. For each checkpoint, we visualize the distribution of ground\-truth rewards \(top\) and the corresponding final\-layer critic outputs \(bottom\)\. As training progresses, critic outputs increasingly reflect the structure of the reward distribution, corresponding to improved confidence alignment and lower AUSE\.