Text-to-Distribution Prediction with Quantile Tokens and Neighbor Context

# Text-to-Distribution Prediction with Quantile Tokens and Neighbor Context
Source: [https://arxiv.org/html/2604.20216](https://arxiv.org/html/2604.20216)
Yilun Zhu1Yuan Zhuang1Nikhita Vedula1Dushyanta Dhyani1Shaoyuan Xu1 Moyan Li1Mohsen Bayati2Bryan Wang1Shervin Malmasi1

1Amazon\.com, Inc\.2Stanford University yz565@georgetown\.edubayati@stanford\.edu \{zyone, veduln, dhyanidd, shaoyux, moyanli, brywan, malmasi\}@amazon\.com

###### Abstract

Many applications of LLM\-based text regression require predicting a full conditional distribution rather than a single point value\. We study*distributional regression*under empirical\-quantile supervision, where each input is paired with multiple observed quantile outcomes, and the target distribution is represented by a dense grid of quantiles\. We address two key limitations of current approaches: the lack of local grounding for distribution estimates, and the reliance on shared representations that create an indirect bottleneck between inputs and quantile outputs\. In this paper, we introduce*Quantile Token Regression*, which, to our knowledge, is the first work to insert dedicated quantile tokens into the input sequence, enabling direct input\-output pathways for each quantile through self\-attention\. We further augment these quantile tokens with retrieval, incorporating semantically similar*neighbor*instances and their empirical distributions to ground predictions with local evidence from similar instances\. We also provide the first theoretical analysis of loss functions for quantile regression, clarifying which distributional objectives each optimizes\. Experiments on the Inside Airbnb and StackSample benchmark datasets with LLMs ranging from 1\.7B to 14B parameters show that quantile tokens with neighbors consistently outperform baselines \(∼\\sim4 points lower MAPE and 2×\\timesnarrower prediction intervals\), with especially large gains on smaller and more challenging datasets where quantile tokens produce substantially sharper and more accurate distributions\.111Our code is publicly available at[https://github\.com/yilunzhu/text2distribution/](https://github.com/yilunzhu/text2distribution/)\.

Text\-to\-Distribution Prediction with Quantile Tokens and Neighbor Context

Yilun Zhu1††thanks:Work done while at Amazon\. Currently at Apple\.Yuan Zhuang1Nikhita Vedula1Dushyanta Dhyani1Shaoyuan Xu1Moyan Li1Mohsen Bayati2Bryan Wang1Shervin Malmasi11Amazon\.com, Inc\.2Stanford Universityyz565@georgetown\.edubayati@stanford\.edu\{zyone, veduln, dhyanidd, shaoyux, moyanli, brywan, malmasi\}@amazon\.com

## 1Introduction

![Refer to caption](https://arxiv.org/html/2604.20216v1/x1.png)Figure 1:Overview of our approach\.*Left:*Input includes query text and retrieved neighbors with their full empirical distributions \(as quantiles\)\.*Center:*The baseline\(Vedulaet al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib9)\)uses only query text without neighbors and computes all quantiles from a shared hidden state via separate linear heads\. Our Quantile Token approach augments the input with neighbors and inserts dedicated⟨Qτ⟩\\langle Q\_\{\\tau\}\\rangletokens that attend directly to the input, creating direct input\-output pathways for each quantile\.*Right:*Output is a complete predicted distribution viaτ\\tauquantiles\.*Bottom:*We evaluate on two diverse datasets \(Airbnb, Stack Overflow\) with Qwen3 1\.7B–14B models\.Large Language Models \(LLMs\) have shown remarkable capabilities beyond text generation, extending to structured prediction tasks such as time series forecasting\(Gruveret al\.,[2023](https://arxiv.org/html/2604.20216#bib.bib19)\)and regression\(Vacareanuet al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib14); Jacobset al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib20)\)\. Recent work has shown that LLMs can approximate numerical mappings with strong accuracy when fine\-tuned or prompted with in\-context examples, making them attractive for text regression tasks where crucial information lies in unstructured text\(Bitvai and Cohn,[2015](https://arxiv.org/html/2604.20216#bib.bib15); Chen and Si,[2024](https://arxiv.org/html/2604.20216#bib.bib28)\)\.

While most LLM\-based regression work focuses on point estimation, many real\-world use cases require predicting*full probability distributions*rather than single values\. Price prediction, demand forecasting, and risk assessment all benefit from understanding not just central tendencies but also dispersion and tail behavior\(Aroraet al\.,[2023](https://arxiv.org/html/2604.20216#bib.bib21); Kneibet al\.,[2023](https://arxiv.org/html/2604.20216#bib.bib22)\)\. Quantile regression\(Koenker and Bassett,[1978](https://arxiv.org/html/2604.20216#bib.bib10)\)provides a natural framework for distribution prediction by estimating conditional quantiles at different probability levels, offering robustness to outliers and the ability to capture heterogeneous effects across the distribution\.

Recent work byVedulaet al\.\([2025](https://arxiv.org/html/2604.20216#bib.bib9)\)takes an important first step towards LLM\-based distributional prediction by attaching multiple linear regression heads to a shared final hidden state, each predicting a different quantile\. However, this architecture has three key limitations\. First, all quantile predictions derive from the same representation bottleneck, creating only an*indirect*connection between input features and quantile\-specific outputs\. The model must compress everything relevant about the distribution into a single vector, from which separate heads attempt to extract different quantiles\. Second, the method predicts distributions based only on the query text, which may contain limited information about the target distribution\. This contrasts with how humans reason about distributions, which naturally relies on comparison with similar instances\. For example, when estimating the price of a query product, one searches for similar products and builds an understanding by comparing features and observed price ranges\. The model can benefit from a richer local context obtained by finding semantically similar items to inform the distribution estimate, but it does not have explicit access to such relevant reference points\. Third, prior retrieval\-augmented methods\(Wanget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib8)\)rely on training with single point labels for each instance, which provides limited distributional supervision compared to training with full empirical distributions\.

We address these limitations with a quantile token architecture augmented by neighbor context \(Figure[1](https://arxiv.org/html/2604.20216#S1.F1)\)\. Our approach makes two key contributions:

Quantile Token Regression\.We propose a novel architecture that inserts learnable quantile tokens \(⟨Qτ1⟩,…,⟨QτQ⟩\\langle Q\_\{\\tau\_\{1\}\}\\rangle,\\ldots,\\langle Q\_\{\\tau\_\{Q\}\}\\rangle\) directly into the input sequence, which allows each quantile token to attend to different parts of the input and accumulate quantile\-specific information\. This creates a*direct*input\-output pathway for each quantile level, rather than relying on separate linear heads over a shared representation\. The architecture enables more coherent quantile predictions since all quantile tokens are produced jointly within the same attention computation, and offers interpretability by revealing which input features each quantile attends to\.

Retrieval\-Augmented Distribution Estimation\.We augment quantile regression with retrievedneighborinstances, which are semantically similar examples from a candidate pool\. Crucially, while prior retrieval\-augmented approaches attach only a*single point label*to each neighbor\(Wanget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib8)\)and only predict a single price output, we equip each neighbor with its*full empirical distribution*represented as quantiles\. This provides the model with richer supervision\. By grounding predictions in distributional evidence from neighbors, the model can better estimate not only central tendency but also dispersion and tail behavior of the target distribution\.

To evaluate our approach, we construct two text\-to\-distribution datasets from Inside Airbnb\(Inside Airbnb,[2025](https://arxiv.org/html/2604.20216#bib.bib16)\)and StackSample\(Stack Overflow,[2019](https://arxiv.org/html/2604.20216#bib.bib17),[2025](https://arxiv.org/html/2604.20216#bib.bib18)\)\. Experiments with Qwen3 models\(Yanget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib36)\)spanning 1\.7B–14B parameters show that \(1\) retrieval\-augmented inputs consistently improve quantile regression across model scales \(8% relative reduction in avg MAPE on Airbnb, 63% on StackSample\), \(2\) quantile tokens outperform the shared\-representation baseline \(14% relative reduction in avg MAPE on StackSample, 6×\\timesnarrower intervals\), and \(3\) the combination of both techniques yields the best performance\. We also provide a mathematical analysis of different loss functions for quantile regression, clarifying the distributional objectives each optimizes\. These two tasks capture different scales and uncertainty regimes for text\-to\-distribution prediction\.

## 2Related Work

##### LLM\-Based Regression\.

LLMs have been applied to regression through three main paradigms\. First, in\-context learning performs regression without fine\-tuning by providing numeric examples in the prompt\(Garget al\.,[2022](https://arxiv.org/html/2604.20216#bib.bib13); Vacareanuet al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib14)\)\. Second, LLM embeddings can be used as features for conventional regressors\(Imperial,[2021](https://arxiv.org/html/2604.20216#bib.bib23); Tanget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib24)\)\. Third, fine\-tuning directly optimizes LLMs for numeric prediction, either by treating numbers as text tokens or by adding regression heads\(Yanget al\.,[2020](https://arxiv.org/html/2604.20216#bib.bib41); Jacobset al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib20); Songet al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib25)\)\. Recent work also explores regression\-specific objectives, such as decision\-theoretic fine\-tuning \(RAFT\)\(Lukasiket al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib26)\)and coupling chain\-of\-thought with regression losses \(TRACT\)\(Chianget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib27)\)\. Our work follows the fine\-tuning paradigm but targets*distributional*rather than point prediction\.

##### Quantile Regression and Distributional Prediction\.

Quantile regression\(Koenker and Bassett,[1978](https://arxiv.org/html/2604.20216#bib.bib10)\)estimates conditional quantiles via pinball loss, enabling nonparametric characterization of predictive uncertainty\. Distributional prediction is widely used in applications such as forecasting and risk\-sensitive decision making\(Aroraet al\.,[2023](https://arxiv.org/html/2604.20216#bib.bib21); Gürleket al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib34); Guet al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib33)\), and recent work has begun integrating these ideas with LLMs\(Gruveret al\.,[2023](https://arxiv.org/html/2604.20216#bib.bib19); Gillmanet al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib29)\)\. LLM\-based quantile regression often attaches multiple quantile heads to a shared representation\(Vedulaet al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib9); Dorka,[2024](https://arxiv.org/html/2604.20216#bib.bib30)\), which can create a bottleneck between the input and quantile\-specific outputs\. In contrast, our quantile token architecture inserts dedicated tokens that participate in attention throughout the transformer, yielding more direct input–output pathways for each quantile\.

##### Retrieval\-Augmented Prediction\.

Retrieval augmentation has been effective for grounding LLM outputs in external evidence\(Lewiset al\.,[2020](https://arxiv.org/html/2604.20216#bib.bib31); Asaiet al\.,[2023](https://arxiv.org/html/2604.20216#bib.bib32)\)\. For regression, retrieved neighbors can provide context that improves calibration; for example, retrieval\-augmented pricing leverages similar items to support numeric estimates\(Wanget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib8)\)\. Existing work largely targets point prediction\. We extend retrieval augmentation to*distributional*prediction, leveraging intuition that similar instances exhibit similar outcome distributions, which is informative for estimating dispersion and tail behavior\.

##### Text Regression Applications\.

Text regression maps unstructured language to numeric targets in domains including finance, real estate, product pricing, and content scoring\(Guet al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib33); Chen and Si,[2024](https://arxiv.org/html/2604.20216#bib.bib28); Vedulaet al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib9); Wanget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib8); Chianget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib27)\)\. A common challenge is that key signals are embedded in free\-form text and are difficult to capture with manual features\. We formulate text\-to\-distribution prediction as a general problem and evaluate across multiple domains \(e\.g\., Airbnb listings and community Q&A\) to demonstrate breadth beyond pricing\-centric settings\.

## 3Method

### 3\.1Quantile Distribution Regression Task

LetXXdenote an input instance, where each inputX\(i\)X^\{\(i\)\}is a text sequenceX\(i\)=\(x1\(i\),x2\(i\),…,xni\(i\)\)X^\{\(i\)\}=\(x^\{\(i\)\}\_\{1\},x^\{\(i\)\}\_\{2\},\\ldots,x^\{\(i\)\}\_\{n\_\{i\}\}\), and letY∈ℝY\\in\\mathbb\{R\}be a continuous outcome\. The objective is to learn a model that maps an inputXXto a conditional distributionFY∣X\(⋅∣X\)F\_\{Y\\mid X\}\(\\cdot\\mid X\), represented via its conditional quantiles\. Specifically, for a set of quantile levels𝝉=\(τ1,…,τQ\)∈\(0,1\)Q\\boldsymbol\{\\tau\}=\(\\tau\_\{1\},\\ldots,\\tau\_\{Q\}\)\\in\(0,1\)^\{Q\}, we predict the quantile vectorq𝝉​\(X\)=\(qτ1​\(X\),…,qτQ​\(X\)\)q\_\{\\boldsymbol\{\\tau\}\}\(X\)=\\big\(q\_\{\\tau\_\{1\}\}\(X\),\\ldots,q\_\{\\tau\_\{Q\}\}\(X\)\\big\), whereqτ​\(X\)q\_\{\\tau\}\(X\)is theτ\\tau\-th quantile ofFY∣X\(⋅∣X\)F\_\{Y\\mid X\}\(\\cdot\\mid X\)\. This formulation follows the standard view of quantile regression as distribution learning through conditional quantiles\.

Empirical Quantile Supervision\.Each inputXiX\_\{i\}is paired with multiple observed outcomes𝒴i=\{yi​1,…,yi​Mi\}\\mathcal\{Y\}\_\{i\}=\\\{y\_\{i1\},\\ldots,y\_\{iM\_\{i\}\}\\\}, which we treat as realizations ofY∣XiY\\mid X\_\{i\}\. We construct the empirical CDFF^i​\(t\)=1Mi​∑m=1Mi𝟏​\[yi​m≤t\]\\widehat\{F\}\_\{i\}\(t\)=\\frac\{1\}\{M\_\{i\}\}\\sum\_\{m=1\}^\{M\_\{i\}\}\\mathbf\{1\}\[y\_\{im\}\\leq t\]with empirical quantile functionQ^i​\(τ\)=F^i−1​\(τ\)\\widehat\{Q\}\_\{i\}\(\\tau\)=\\widehat\{F\}\_\{i\}^\{\-1\}\(\\tau\)\. Since each instance has a variable number of outcomesMiM\_\{i\}, we interpolateQ^i\\widehat\{Q\}\_\{i\}to a fixed grid ofQ=99Q=99quantile levels𝝉=\{0\.01,0\.02,…,0\.99\}\\boldsymbol\{\\tau\}=\\\{0\.01,0\.02,\\ldots,0\.99\\\}via linear interpolation \(Appendix[B](https://arxiv.org/html/2604.20216#A2)\), producing target vectors𝐪^i∈ℝ99\\widehat\{\\mathbf\{q\}\}\_\{i\}\\in\\mathbb\{R\}^\{99\}\.

Learning Objective\.Given a dataset𝒟=\{\(Xi,𝒴i\)\}i=1N\\mathcal\{D\}=\\\{\(X\_\{i\},\\mathcal\{Y\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, we train a modelfθf\_\{\\theta\}to predict𝐪^i\\widehat\{\\mathbf\{q\}\}\_\{i\}fromXiX\_\{i\}, i\.e\.,fθ​\(Xi\)≈𝐪^if\_\{\\theta\}\(X\_\{i\}\)\\approx\\widehat\{\\mathbf\{q\}\}\_\{i\}\. This turns text\-to\-distribution prediction into structured regression over quantile levels\.

![Refer to caption](https://arxiv.org/html/2604.20216v1/x2.png)Figure 2:Quantile tokens enable specialized representations and direct input\-output pathways for each quantile level\. For a Stack Overflow question about asyncio exception handling, different features signal different response times: the popular “python” tag suggests fast answers, while the niche “asyncio” topic and code complexity suggest slower responses\.*Center:*Each quantile token \(Q10Q\_\{10\},Q50Q\_\{50\},Q90Q\_\{90\}\) learns to attend to the features most predictive of its target quantile—Q10Q\_\{10\}focuses on popularity signals whileQ90Q\_\{90\}focuses on complexity indicators\.*Right:*The resulting time\-to\-answer distribution captures both the possibility of a quick response \(10% within 15 minutes\) and the long tail \(90% within 18 hours\)\.
### 3\.2Quantile Token Regression

We introduce*Quantile Token Regression*, a simple architectural change that makes each quantile prediction depend on a dedicated representation\. Given an input sequenceX=\(x1,…,xn\)X=\(x\_\{1\},\\ldots,x\_\{n\}\)222Including the query and any retrieved neighbor context\., we appendQQspecial quantile tokens⟨Qτ1⟩,…,⟨QτQ⟩\\langle Q\_\{\\tau\_\{1\}\}\\rangle,\\ldots,\\langle Q\_\{\\tau\_\{Q\}\}\\rangleto the end of the sequence\. The resulting sequence is

X~=\(x1,…,xn,⟨Qτ1⟩,…,⟨QτQ⟩\)\.\\widetilde\{X\}=\(x\_\{1\},\\ldots,x\_\{n\},\\langle Q\_\{\\tau\_\{1\}\}\\rangle,\\ldots,\\langle Q\_\{\\tau\_\{Q\}\}\\rangle\)\.\(1\)We feedX~\\widetilde\{X\}into a pretrained transformergθg\_\{\\theta\}and obtain final\-layer hidden statesH=gθ​\(X~\)H=g\_\{\\theta\}\(\\widetilde\{X\}\)\. Lethτk∈ℝdh\_\{\\tau\_\{k\}\}\\in\\mathbb\{R\}^\{d\}denote the hidden state at the position of token⟨Qτk⟩\\langle Q\_\{\\tau\_\{k\}\}\\rangle\. We then predict thekk\-th quantile using a shared linear regressor applied to the corresponding quantile\-token representation\.

q^τk​\(X\)=w⊤​hτk\+b\.\\hat\{q\}\_\{\\tau\_\{k\}\}\(X\)=w^\{\\top\}h\_\{\\tau\_\{k\}\}\+b\.\(2\)
Quantile token regression architecture has two advantages\. First, it creates a direct input\-output relation for each quantile level by allowing⟨Qτk⟩\\langle Q\_\{\\tau\_\{k\}\}\\rangleto collect information across all transformer layers, rather than relying on separate linear heads over a shared final representation\. This improves the alignment between the conditioning evidence and quantile\-specific predictions, which pays more attention to extreme quantiles\. Figure[2](https://arxiv.org/html/2604.20216#S3.F2)illustrates this mechanism on a Stack Overflow question, where the range of response time to a question is predicted\. The⟨Q10⟩\\langle Q\_\{10\}\\rangletoken learns to attend to popularity signals \(e\.g\., “python” tag\) that predict fast answers, while⟨Q90⟩\\langle Q\_\{90\}\\rangleattends to complexity indicators that predict slower responses\. Second, quantile tokens result in more coherent quantile representations, since all quantile tokens are produced jointly within the same attention computation, and they enable interpretability by inspecting which parts of the input each⟨Qτk⟩\\langle Q\_\{\\tau\_\{k\}\}\\rangleattends to when forming its estimate\.

### 3\.3Retrieval\-based Quantile Regression

Semantically similar inputs tend to exhibit similar outcome distributions\. For example, similar product descriptions yield similar price distributions, and similar questions receive similar response\-time distributions\. When conditioning only on the query text, the model must implicitly learn these distributional patterns from the training data, which can be challenging\. We therefore augment quantile regression with*neighbors*, semantically similar, label\-bearing instances retrieved from a candidate pool, to explicitly provide the model with distributional evidence from similar instances\.

Given an input text sequenceXX, we retrieve the top\-KKsemantically similar neighbors from a candidate pool using dense embedding similarity\. Each retrieved neighbor is provided to the model along with its empirical distribution represented as a small set of quantiles\. Implementation details, including the choice of embedding model, retrieval features, input formatting, and which quantiles are selected for neighbors, are described in Section[4\.1](https://arxiv.org/html/2604.20216#S4.SS1)\.

### 3\.4Loss Functions and Theoretical Analysis

A critical design choice in distribution learning is the objective used to align the predicted quantile vectorq^𝝉​\(X\)=\(q^τ1​\(X\),…,q^τQ​\(X\)\)\\hat\{q\}\_\{\\boldsymbol\{\\tau\}\}\(X\)=\\big\(\\hat\{q\}\_\{\\tau\_\{1\}\}\(X\),\\ldots,\\hat\{q\}\_\{\\tau\_\{Q\}\}\(X\)\\big\)with the supervision derived from the outcome set𝒴i=\{yi​1,…,yi​Mi\}\\mathcal\{Y\}\_\{i\}=\\\{y\_\{i1\},\\ldots,y\_\{iM\_\{i\}\}\\\}\. Standard quantile regression applies the pinball loss to*raw*outcomesyy\. In contrast, our training targetsQ^i​\(τ\)\\widehat\{Q\}\_\{i\}\(\\tau\)are*empirical quantile estimators*computed from a finite sample𝒴i\\mathcal\{Y\}\_\{i\}and then interpolated to a fixed grid\. We therefore analyze losses that are appropriate for*quantile supervision*, and clarify what each objective targets in population\.

##### ℓ1\\ell\_\{1\}andℓ2\\ell\_\{2\}losses on empirical quantiles \(Wasserstein matching\)\.

We treat the interpolated empirical quantilesQ^i​\(τk\)\\widehat\{Q\}\_\{i\}\(\\tau\_\{k\}\)as noisy measurements of the latent population quantilesQ∗​\(Xi,τk\)Q^\{\*\}\(X\_\{i\},\\tau\_\{k\}\)and minimize an element\-wiseℓp\\ell\_\{p\}loss:

ℒℓ1​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\ell\_\{1\}\}\(\\theta\)=1N​Q​∑i=1N∑k=1Q\|Q^i​\(τk\)−q^τk​\(Xi\)\|,\\displaystyle=\\frac\{1\}\{NQ\}\\sum\_\{i=1\}^\{N\}\\sum\_\{k=1\}^\{Q\}\\left\|\\widehat\{Q\}\_\{i\}\(\\tau\_\{k\}\)\-\\hat\{q\}\_\{\\tau\_\{k\}\}\(X\_\{i\}\)\\right\|,\(3\)ℒℓ2​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\ell\_\{2\}\}\(\\theta\)=1N​Q​∑i=1N∑k=1Q\(Q^i​\(τk\)−q^τk​\(Xi\)\)2\.\\displaystyle=\\frac\{1\}\{NQ\}\\sum\_\{i=1\}^\{N\}\\sum\_\{k=1\}^\{Q\}\\left\(\\widehat\{Q\}\_\{i\}\(\\tau\_\{k\}\)\-\\hat\{q\}\_\{\\tau\_\{k\}\}\(X\_\{i\}\)\\right\)^\{2\}\.\(4\)This is called Wasserstein distance since in one dimension, thepp\-Wasserstein distance admits the quantile representationWpp​\(F,G\)=∫01\|QF​\(u\)−QG​\(u\)\|p​𝑑uW\_\{p\}^\{p\}\(F,G\)=\\int\_\{0\}^\{1\}\|Q\_\{F\}\(u\)\-Q\_\{G\}\(u\)\|^\{p\}\\,du\(Villani,[2009](https://arxiv.org/html/2604.20216#bib.bib38)\)\. When𝝉\\boldsymbol\{\\tau\}is a dense, approximately uniform grid,ℒℓ1\\mathcal\{L\}\_\{\\ell\_\{1\}\}andℒℓ2\\mathcal\{L\}\_\{\\ell\_\{2\}\}can be interpreted as discrete approximations toW1\(F^i,F^θ\(⋅∣Xi\)\)W\_\{1\}\(\\widehat\{F\}\_\{i\},\\widehat\{F\}\_\{\\theta\}\(\\cdot\\mid X\_\{i\}\)\)andW22\(F^i,F^θ\(⋅∣Xi\)\)W\_\{2\}^\{2\}\(\\widehat\{F\}\_\{i\},\\widehat\{F\}\_\{\\theta\}\(\\cdot\\mid X\_\{i\}\)\), respectively, whereF^θ\\widehat\{F\}\_\{\\theta\}is the distribution implied by the predicted quantiles\.

##### Mismatched pinball on empirical quantiles \(Pinball\-Q\)\.

An alternative is to apply the standard pinball loss to the empirical quantile targets:

ℒPinball\-Q​\(θ\)=1N​Q​∑i=1N∑k=1Qρτk​\(Q^i​\(τk\)−q^τk​\(Xi\)\),\\mathcal\{L\}\_\{\\text\{Pinball\-Q\}\}\(\\theta\)=\\frac\{1\}\{NQ\}\\sum\_\{i=1\}^\{N\}\\sum\_\{k=1\}^\{Q\}\\rho\_\{\\tau\_\{k\}\}\\\!\\left\(\\widehat\{Q\}\_\{i\}\(\\tau\_\{k\}\)\-\\hat\{q\}\_\{\\tau\_\{k\}\}\(X\_\{i\}\)\\right\),\(5\)whereρτ​\(u\)=u​\(τ−𝕀​\[u<0\]\)\\rho\_\{\\tau\}\(u\)=u\\big\(\\tau\-\\mathbb\{I\}\[u<0\]\\big\)\. While pinball is proper for learningQ∗​\(X,τ\)Q^\{\*\}\(X,\\tau\)from raw outcomes, here the “outcome” fed to pinball is itself a random quantile estimator\. As a result, \([5](https://arxiv.org/html/2604.20216#S3.E5)\) generally targets theτ\\tau\-th quantile of the*estimator distribution*Q^i​\(τ\)∣Xi\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}, not the underlying parameterQ∗​\(Xi,τ\)Q^\{\*\}\(X\_\{i\},\\tau\)\(Appendix[A](https://arxiv.org/html/2604.20216#A1)\)\. This yields a systematic “inflation/deflation” effect away fromτ=0\.5\\tau=0\.5\.

##### Scalarized pinball on a single statistic \(Pinball\-Med\)\.

Prior LLM regression work often associates each input with a single scalar label and trains all quantile heads against that scalar using pinball\. To mirror this setting under distribution supervision, we define a scalar pseudo\-targetyi:=Q^i​\(0\.5\)y\_\{i\}:=\\widehat\{Q\}\_\{i\}\(0\.5\)\(the sample median\)333If an application only provides a subset of quantiles, one can analogously setyiy\_\{i\}to the median of the available reported quantiles\.and optimize

ℒPinball\-Med​\(θ\)=1N​Q​∑i=1N∑k=1Qρτk​\(yi−q^τk​\(Xi\)\)\.\\mathcal\{L\}\_\{\\text\{Pinball\-Med\}\}\(\\theta\)=\\frac\{1\}\{NQ\}\\sum\_\{i=1\}^\{N\}\\sum\_\{k=1\}^\{Q\}\\rho\_\{\\tau\_\{k\}\}\\\!\\left\(y\_\{i\}\-\\hat\{q\}\_\{\\tau\_\{k\}\}\(X\_\{i\}\)\\right\)\.\(6\)Appendix[A](https://arxiv.org/html/2604.20216#A1)shows that \([6](https://arxiv.org/html/2604.20216#S3.E6)\) is generally*inconsistent*for learning the full conditional distribution whenMi\>1M\_\{i\}\>1: it learns the conditional distribution of the statisticyiy\_\{i\}\(which concentrates aroundQ∗​\(Xi,0\.5\)Q^\{\*\}\(X\_\{i\},0\.5\)asMiM\_\{i\}grows\)\.

##### Theoretical implications and empirical ordering\.

Under the latent\-sample model in Appendix[A](https://arxiv.org/html/2604.20216#A1), for each fixed\(X,τ\)\(X,\\tau\)the empirical quantile obeys an asymptotic expansionQ^i​\(τ\)=Q∗​\(Xi,τ\)\+εi,τ\\widehat\{Q\}\_\{i\}\(\\tau\)=Q^\{\*\}\(X\_\{i\},\\tau\)\+\\varepsilon\_\{i,\\tau\}whereεi,τ\\varepsilon\_\{i,\\tau\}is approximately centered and symmetric with variance scaling asVar​\(εi,τ∣Xi\)∝τ​\(1−τ\)/Mi\\mathrm\{Var\}\(\\varepsilon\_\{i,\\tau\}\\mid X\_\{i\}\)\\propto\\tau\(1\-\\tau\)/M\_\{i\}\(up to density factors\)\. Consequently,ℒℓ1\\mathcal\{L\}\_\{\\ell\_\{1\}\}andℒℓ2\\mathcal\{L\}\_\{\\ell\_\{2\}\}yield Fisher\-consistent estimations forQ∗Q^\{\*\}in the large\-MiM\_\{i\}regime \(withℓ1\\ell\_\{1\}offering additional robustness under heavy tails or outliers\)\. In contrast,ℒPinball\-Q\\mathcal\{L\}\_\{\\text\{Pinball\-Q\}\}introduces a bias of orderMi−1/2M\_\{i\}^\{\-1/2\}whose magnitude grows away fromτ=0\.5\\tau=0\.5, whileℒPinball\-Med\\mathcal\{L\}\_\{\\text\{Pinball\-Med\}\}discards distributional shape and concentrates toward the median\. This analysis predicts the empirical ordering observed in our experiments:ℒℓ1\\mathcal\{L\}\_\{\\ell\_\{1\}\}performs best overall,ℒPinball\-Q\\mathcal\{L\}\_\{\\text\{Pinball\-Q\}\}is competitive but biased, andℒPinball\-Med\\mathcal\{L\}\_\{\\text\{Pinball\-Med\}\}underperforms due to systematic loss of tail information\.

The same theory suggests variance\-aware weighting by\(Mi,τk\)\(M\_\{i\},\\tau\_\{k\}\)\(and density factors\) to downweight noisy tail quantiles whenMiM\_\{i\}is small; we leave this extension to future work \(Appendix[A](https://arxiv.org/html/2604.20216#A1)\)\.

DatasetSplit\# SamplesAvg\. \# TokensInside Airbnbtrain768,001385val50,000354test50,000355test\_la \(OOD\)61,551385StackSampletrain46,544320val5,818322test5,819324Table 1:Dataset split statistics\. We report number of samples and average number of tokens per sample\.

## 4Experiments

Table 2:Results on the Airbnb test set with various model sizes\. QR denotes quantile regression, QT denotes quantile token regression, andKKis the number of neighbors\. Best value per metric column is in bold\.Table 3:Results on the StackSample test split\.### 4\.1Datasets

We experiment upon two publicly available text\-to\-distribution datasets from different domains: Inside Airbnb\(Inside Airbnb,[2025](https://arxiv.org/html/2604.20216#bib.bib16)\)and StackSample\(Stack Overflow,[2019](https://arxiv.org/html/2604.20216#bib.bib17),[2025](https://arxiv.org/html/2604.20216#bib.bib18)\)\. For both datasets, we construct ground\-truth distributions from multiple observed outcomes per instance, keeping only instances with at least 4 observations \(an empirical threshold chosen to balance label quality with dataset size\)\. We use stratified sampling for train/val/test splits\. For retrieval, we compute dense embeddings usingQwen/Qwen3\-Embedding\-8Bover the full text of each instance\. The retrieval candidate pool is restricted to the training split only, ensuring that no validation or test instances appear in retrieval\. At inference time, each query retrieves top\-KKneighbors \(withK=8K\{=\}8in the main setting\)\. Each retrieved neighbor contributes its title and nine representative empirical quantiles4441, 5, 10, 25, 50, 75, 90, 95, 99 percentiles\., which are appended to the model input\. For Airbnb, we build one index per city and perform same\-city retrieval\. For StackSample, the index is constructed from training questions only\. For out\-of\-domain evaluation on Los Angeles \(LA\), LA listings are excluded from both training and retrieval candidate pools used in the main experiments\. For LA evaluation, we construct a separate retrieval pool using only LA test instances, ensuring no cross\-city information leakage\. Dataset statistics are shown in Table[1](https://arxiv.org/html/2604.20216#S3.T1); full construction details are in Appendix[C](https://arxiv.org/html/2604.20216#A3)\.

##### Airbnb\.

The task is to predict the price distribution for an Airbnb listing given its textual description and metadata\. We construct the dataset from Inside Airbnb\(Inside Airbnb,[2025](https://arxiv.org/html/2604.20216#bib.bib16)\), collecting all available cities from 2024\-09 to 2025\-08 \(119 cities\) and converting prices to U\.S\. dollars\. Each listing is represented by its title, description, amenities, location, and property type\. We apply log transformation to prices to handle their wide range\. We construct the ground\-truth price distribution from observed monthly prices across time snapshots, yielding∼\\sim840k samples from 55 cities after filtering\. We hold out Los Angeles to form an out\-of\-domain \(OOD\) test set\.

##### StackSample\.

The task is to predict the distribution of response times \(time from question posting to receiving an answer\) for a Stack Overflow question given its text\. We use StackSample\(Stack Overflow,[2019](https://arxiv.org/html/2604.20216#bib.bib17),[2025](https://arxiv.org/html/2604.20216#bib.bib18)\), a Kaggle\-hosted subset of Stack Overflow Q&A\. Each question is represented by its title, body, and tags\. We construct the ground\-truth response\-time distribution from observed answer response times\. We apply log transformation to handle the wide range of response times \(from minutes to hours\)\.

### 4\.2Experimental Setup

We fine\-tune Qwen3 models \(1\.7B–14B parameters\)\(Yanget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib36)\)with LoRA\(Huet al\.,[2022](https://arxiv.org/html/2604.20216#bib.bib35)\), predictingQ=99Q\{=\}99uniformly spaced quantiles\. We compare QRK=0\(baseline quantile regression\(Vedulaet al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib9)\)\), QRK=8\(QR withK=8K\{=\}8retrieved neighbors\), and QTK=8\(quantile tokens withK=8K\{=\}8neighbors\)\. We evaluate using average MAPE \(Mean Absolute Percentage Error\), wMAPE, sMAPE for point accuracy, and CRPSS \(Continuous Ranked Probability Skill Score\) and RCIW \(Relative Coverage Interval Width\) for distributional quality\. Full details on dataset construction, experimental setup, and hyperparameters are in Appendix[D\.1](https://arxiv.org/html/2604.20216#A4.SS1)and[D\.2](https://arxiv.org/html/2604.20216#A4.SS2)\. We focus on LLM\-based quantile regression baselines as the strongest directly comparable approach under the same supervision and decoding setting\. For both datasets, training is performed in log space; at inference, predictions are exponentiated back to the original scale before computing all evaluation metrics\.

### 4\.3Evaluation Metrics

We evaluate quantile predictions on a test set\{\(xi,yi\)\}i=1n\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}, where the model outputsq^τ​\(xi\)\\hat\{q\}\_\{\\tau\}\(x\_\{i\}\)forτ∈\{τk\}k=1Q\\tau\\in\\\{\\tau\_\{k\}\\\}\_\{k=1\}^\{Q\}\. Since each sample has a ground\-truth distribution, we first compute the Mean Absolute Percentage Error \(MAPE\\mathrm\{MAPE\}\) at each quantile in a coarse set of target quantiles𝒯=\{0\.1,0\.2,…,0\.9\}\\mathcal\{T\}=\\\{0\.1,0\.2,\\ldots,0\.9\\\}\. Then we compute the average over them, which we refer to asaverageMAPE\\mathrm\{averageMAPE\}:

MAPE​@​τ\\displaystyle\\mathrm\{MAPE\}@\\tau=100n​∑i=1n\|q^τ​\(xi\)−qτ​\(xi\)qτ​\(xi\)\|,\\displaystyle=\\frac\{100\}\{n\}\\sum\_\{i=1\}^\{n\}\\left\|\\frac\{\\hat\{q\}\_\{\\tau\}\(x\_\{i\}\)\-q\_\{\\tau\}\(x\_\{i\}\)\}\{q\_\{\\tau\}\(x\_\{i\}\)\}\\right\|,\(7\)avgMAPE\\displaystyle\\mathrm\{avgMAPE\}=1\|𝒯\|​∑τ∈𝒯MAPE​@​τ,\\displaystyle=\\frac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{\\tau\\in\\mathcal\{T\}\}\\mathrm\{MAPE\}@\\tau,\(8\)
We additionally report the Weighted Mean Absolute Percentage Error \(wMAPE\\mathrm\{wMAPE\}\) and the Symmetric Mean Absolute Percentage Error \(sMAPE\\mathrm\{sMAPE\}\) using the median predictiony^i=q^0\.5​\(xi\)\\hat\{y\}\_\{i\}=\\hat\{q\}\_\{0\.5\}\(x\_\{i\}\)\. We useyi=q0\.5​\(xi\)y\_\{i\}=q\_\{0\.5\}\(x\_\{i\}\)as the benchmark value, which is the median of the ground\-truth distribution\.

wMAPE\\displaystyle\\mathrm\{wMAPE\}=100⋅∑i=1n\|y^i−yi\|∑i=1n\|yi\|,\\displaystyle=100\\cdot\\frac\{\\sum\_\{i=1\}^\{n\}\|\\hat\{y\}\_\{i\}\-y\_\{i\}\|\}\{\\sum\_\{i=1\}^\{n\}\|y\_\{i\}\|\},\(9\)sMAPE\\displaystyle\\mathrm\{sMAPE\}=200n​∑i=1n\|y^i−yi\|\|y^i\|\+\|yi\|\.\\displaystyle=\\frac\{200\}\{n\}\\sum\_\{i=1\}^\{n\}\\frac\{\|\\hat\{y\}\_\{i\}\-y\_\{i\}\|\}\{\|\\hat\{y\}\_\{i\}\|\+\|y\_\{i\}\|\}\.\(10\)
For the evaluation of distributional quality, we report the Continuous Ranked Probability Skill Score \(CRPSS\) and Relative Coverage Interval Width \(RCIW\)\. CRPSS is

CRPSS=1−CRPSmodelCRPSref,\\displaystyle\\mathrm\{CRPSS\}=1\-\\frac\{\\mathrm\{CRPS\}\_\{\\text\{model\}\}\}\{\\mathrm\{CRPS\}\_\{\\text\{ref\}\}\},\(11\)
whereCRPSref\\mathrm\{CRPS\}\_\{\\text\{ref\}\}is the CRPS of the empirical marginal distribution of training targets, followingVedulaet al\.\([2025](https://arxiv.org/html/2604.20216#bib.bib9)\)\. We reportRCIW​@​c\\mathrm\{RCIW\}@cforc∈\{90,95,99\}c\\in\\\{90,95,99\\\}:

RCIW​@​c=100n​∑i=1nq^τu​\(c\)​\(xi\)−q^τℓ​\(c\)​\(xi\)\|q0\.5​\(xi\)\|,\\displaystyle\\mathrm\{RCIW\}@c=\\frac\{100\}\{n\}\\sum\_\{i=1\}^\{n\}\\frac\{\\hat\{q\}\_\{\\tau\_\{u\}\(c\)\}\(x\_\{i\}\)\-\\hat\{q\}\_\{\\tau\_\{\\ell\}\(c\)\}\(x\_\{i\}\)\}\{\\left\|q\_\{0\.5\}\(x\_\{i\}\)\\right\|\},\(12\)
where\(τℓ,τu\)\(\\tau\_\{\\ell\},\\tau\_\{u\}\)are chosen to give the closest available central coverage on our quantile grid \(forQ=99Q\{=\}99, we use\(0\.05,0\.95\)\(0\.05,0\.95\),\(0\.02,0\.98\)\(0\.02,0\.98\), and\(0\.01,0\.99\)\(0\.01,0\.99\)for nominalc=90,95,99c\{=\}90,95,99, respectively\)\.

Table 4:Results on the Airbnb OOD test set \(Los Angeles\)\.
### 4\.4Experimental Results

Our Quantile Tokens Regression approach \(QT\) consistently outperforms the quantile regression \(QR\) baseline across both datasets and all model sizes \(Tables[2](https://arxiv.org/html/2604.20216#S4.T2)and[3](https://arxiv.org/html/2604.20216#S4.T3)\)\. The method ranking is stable: QTK=8outperforms QRK=8, which in turn outperforms QRK=0on all reported metrics\. Compared to retrieval\-augmented QR, QT improves both accuracy and sharpness, consistently reducing average MAPE and producing markedly narrower prediction intervals, with RCIW reduced by multiple factors in both datasets\.555We note that StackSample exhibits substantially heavier\-tailed distributions than Airbnb, leading to much larger absolute metric values \(e\.g\., RCIW\)\. Therefore, metric magnitudes are not directly comparable across datasets\.The advantage of QT is especially pronounced on StackSample, the smaller and more challenging dataset with response times spanning from 1\.01 mins to 12 hrs\. The QRK=0baseline produces extremely poor distributions with very wide confidence intervals \(RCIW@99 of4\.55×1044\.55\\times 10^\{4\}\), while QTK=8converges more reliably\. Comparing QTK=8to the baseline QRK=0, QTK=8reduces average MAPE from 266\.65 to 84\.30666The absolute MAPE on StackSample remains high due to the limited dataset size and the inherently high uncertainty in response\-time prediction compared to price prediction\.\(68% reduction\) and shrinks RCIW@99 from4\.55×1044\.55\\times 10^\{4\}to 346\.90 \(131×\\timesreduction\)\. Even compared to retrieval\-augmented QRK=8, QTK=8achieves 14% lower average MAPE \(84\.30 vs 98\.56\) and 6×\\timesnarrower intervals\.

Neighbor retrieval provides substantial gains across all configurations, with particularly large impact when training data are limited\. On StackSample, retrieval yields dramatic improvements: for the QR baseline, average MAPE drops from 266\.65 to 98\.56 \(63% reduction\) when moving fromK=0K\{=\}0toK=8K\{=\}8\. On Airbnb, retrieval also consistently improves average MAPE for QR across all model sizes: for Qwen3\-4B, average MAPE drops from 30\.31 to 27\.78 \(8% reduction\), and similar gains hold at other scales\. These substantial improvements empirically validate the hypothesis \(Section[3\.3](https://arxiv.org/html/2604.20216#S3.SS3)\) that semantically similar inputs exhibit similar outcome distributions, as retrieved neighbors’ distributions provide informative context for predictions\. This contrast aligns with dataset scale: Airbnb is much larger, with∼\\sim840k listings across 55 cities, while StackSample contains∼\\sim58k questions\. Therefore, retrieval has greater impact when training data are more limited, offering explicit distributional evidence from similar instances that helps ground predictions\.

Model scaling shows diminishing returns at larger sizes on Airbnb\. Moving from 1\.7B to 4B parameters reduces average MAPE by 7% for QTK=8\(from 27\.18 to 26\.89\), while moving from 8B to 14B yields only 1% improvement \(from 26\.56 to 26\.40\)\. This aligns with empirical scaling laws\(Kaplanet al\.,[2020](https://arxiv.org/html/2604.20216#bib.bib39)\)but shows that gains saturate in our settings\. Notably, distributional metrics do not monotonically improve with size: for example, QRK=8shows wider intervals at 14B than at 8B \(RCIW@99 increases from 11\.27 to 27\.51\)\. Since we tune hyperparameters per model, this suggests that larger backbones can be more sensitive to optimization and regularization choices, and better point accuracy does not necessarily translate into sharper confidence intervals\.

To summarize, these results suggest that retrieval augmentation and quantile tokens are especially critical for harder, higher\-uncertainty text\-to\-distribution tasks rather than providing only incremental gains\. We further evaluate generalization by holding out Los Angeles in the Airbnb dataset during training and OOD testing on its listings, with full results in Appendix[4\.5](https://arxiv.org/html/2604.20216#S4.SS5)\.

### 4\.5Experimental Results on OOD Dataset

As shown in Table[4](https://arxiv.org/html/2604.20216#S4.T4), the LA test set can appear easier than the multi\-city test split for two complementary reasons\. First, Los Angeles is a large, high\-density city in the dataset, so retrieval can find closer and more informative neighbors, leading to stronger grounding for distribution prediction\. Second, U\.S\. cities and listings constitute a substantial portion of our training data, so LA is not far from the dominant training distribution in both language and pricing patterns\. As a result, the holdout primarily reflects a city\-level split rather than a severe domain shift, which explains why OOD performance can be comparable to or even better than the stratified test set\.

### 4\.6Ablation Studies

#### 4\.6\.1Loss Functions

Table[5](https://arxiv.org/html/2604.20216#S4.T5)validates the theoretical comparison in Section[3\.4](https://arxiv.org/html/2604.20216#S3.SS4)and Appendix[A](https://arxiv.org/html/2604.20216#A1)by contrasting Wasserstein losses with pinball variants under empirical\-quantile supervision\.

Table 5:Ablation on loss functions on Airbnb dev set using Qwen3\-4B withK=8K\{=\}8neighbors\.The two Wasserstein objectives, which are Fisher\-consistent for the target quantiles as the number of labels per instance increases, achieve the best practical accuracy and sharpness\. In particular,ℓ1\\ell\_\{1\}Wasserstein yields the lowest average MAPE and the tightest confidence intervals, whileℓ2\\ell\_\{2\}Wasserstein is competitive but slightly worse on both average MAPE and RCIW@95\. In contrast, the pinball\-based objectives perform poorly for distribution learning in our setting:Pinball\-Qapplies pinball loss to the empirical quantile targets, andPinball\-Meduses only the empirical median as supervision\. Both incur much larger average MAPE and extremely wide intervals, consistent with the predicted bias ofPinball\-Qand the loss of distributional information underPinball\-Med\. Although the pinball losses attain higher CRPSS than Wasserstein, this is expected: CRPS is mathematically the integral of pinball losses across quantile levels\(Gneiting and Raftery,[2007](https://arxiv.org/html/2604.20216#bib.bib45)\), so pinball training directly optimizes a discrete approximation of the evaluation criterion\. However, this comes at the cost of massively inflated prediction intervals \(RCIW@95 above 150 versus 3–4 for Wasserstein\), rendering the resulting forecasts impractical despite the CRPSS advantage\. Overall, these results support using Wasserstein objectives for empirical\-quantile supervision, withℓ1\\ell\_\{1\}Wasserstein providing the best accuracy–sharpness tradeoff in our experiments\.

#### 4\.6\.2Number of Neighbors

Table[6](https://arxiv.org/html/2604.20216#S4.T6)studies number of retrieved itemsKKon the Airbnb dev set using Qwen3\-4B with QT andℓ1\\ell\_\{1\}Wasserstein loss, applying postprocessed monotonicity \(described in Section[4\.6\.3](https://arxiv.org/html/2604.20216#S4.SS6.SSS3)\)\.

Table 6:Ablation on the number of retrieved neighbors on Airbnb dev set using Qwen3\-4B\.IncreasingKKconsistently improves average MAPE and CRPSS while tightening intervals\. The improvement is most pronounced when neighbors are first introduced, especially fromK=0K\{=\}0toK=2K\{=\}2, and exhibits diminishing returns asKKincreases\. UsingK=16K\{=\}16performs best overall, reducing average MAPE from 29\.54 to 25\.85 and improving CRPSS from 0\.4509 to 0\.4735, while lowering RCIW@95 from 4\.90 to 3\.47\. However, largerKKincreases the number of input tokens, which raises memory usage and training cost \(e\.g\.,K=16K\{=\}16requires approximately 2×\\timesmemory per sample and 1\.4×\\timestotal training time compared toK=8K\{=\}8\)\.

#### 4\.6\.3Monotonicity

Our proposed Quantile Token regression approach provides no guarantee that the predicted quantiles will satisfy the monotonicity constraint, which can cause issues like the 90th percentile prediction being lower than the 80th percentile\. Table[7](https://arxiv.org/html/2604.20216#S4.T7)therefore compares three approaches to ensure monotonicity on the Airbnb dev set using Qwen3\-4B with QT,ℓ1\\ell\_\{1\}Wasserstein loss, andK=8K\{=\}8neighbors\.

Table 7:Ablation on monotonicity method on Airbnb dev set using Qwen3\-4B withK=8K\{=\}8neighbors\.Baselineapplies no monotonicity constraint and uses the raw predicted quantiles as\-is\.Cumsumenforces monotonicity during both training and inference by predicting non\-negative gaps between adjacent quantiles and cumulatively summing them to form ordered quantiles\.Postprocesskeeps training unchanged and enforces monotonicity only at inference time by sorting the predicted quantiles\. Empirically,Cumsumslightly improves average MAPE and yields the best CRPSS, but substantially widens intervals\. In contrast,Postprocessachieves the lowest average MAPE, while maintaining comparable CRPSS and intervals\.

## 5Conclusion

We studied text\-to\-distribution prediction under empirical\-quantile supervision, where each input has multiple observed outcomes and the target distribution is represented by a dense quantile grid\. We introduced a*retrieval\-augmented*approach that grounds distribution estimates with retrieved neighbor instances and their empirical distributions, and*Quantile Token Regression*, which predicts each quantile from a dedicated token representation formed through self\-attention\. Across Inside Airbnb and StackSample, both methods improve accuracy and yield sharper predictive intervals, with especially large gains on the smaller, more challenging StackSample dataset\. We also analyzed training objectives, showing that Wasserstein matching better fits quantile\-target supervision than pinball variants and offers a stronger accuracy–sharpness tradeoff in practice\. Overall, combining retrieval\-based grounding with quantile\-specific representations is a simple, effective approach for scalable text\-to\-distribution prediction, motivating future work on variance\-aware weighting, calibration under sharpness constraints, and broader applications\.

## Limitations

Our evaluation relies on empirical quantiles constructed from multiple observed outcomes per input and interpolated to a fixed quantile grid; since ground\-truth values at every quantile level are typically unavailable, this interpolation can introduce approximation error, especially when each instance has only a few labels, and the resulting distributions may not fully reflect real\-world conditional outcome distributions\. Our evaluation includes two LLM families, Qwen3 and Phi\-3, but broader validation across additional backbones would further strengthen the generality of the findings\. We report paired statistical significance tests, but do not study additional uncertainty estimates such as bootstrap confidence intervals or repeated data splits, which would further strengthen statistical reliability\.

## References

- M\. Abdin, J\. Aneja, H\. Awadalla, A\. Awadallah, A\. A\. Awan, N\. Bach, A\. Bahree, A\. Bakhtiari, J\. Bao, H\. Behl, A\. Benhaim, M\. Bilenko, J\. Bjorck, S\. Bubeck, M\. Cai, Q\. Cai, V\. Chaudhary, D\. Chen, D\. Chen, W\. Chen, Y\. Chen, Y\. Chen, H\. Cheng, P\. Chopra, X\. Dai, M\. Dixon, R\. Eldan, V\. Fragoso, J\. Gao, M\. Gao, M\. Gao, A\. Garg, A\. D\. Giorno, A\. Goswami, S\. Gunasekar, E\. Haider, J\. Hao, R\. J\. Hewett, W\. Hu, J\. Huynh, D\. Iter, S\. A\. Jacobs, M\. Javaheripi, X\. Jin, N\. Karampatziakis, P\. Kauffmann, M\. Khademi, D\. Kim, Y\. J\. Kim, L\. Kurilenko, J\. R\. Lee, Y\. T\. Lee, Y\. Li, Y\. Li, C\. Liang, L\. Liden, X\. Lin, Z\. Lin, C\. Liu, L\. Liu, M\. Liu, W\. Liu, X\. Liu, C\. Luo, P\. Madan, A\. Mahmoudzadeh, D\. Majercak, M\. Mazzola, C\. C\. T\. Mendes, A\. Mitra, H\. Modi, A\. Nguyen, B\. Norick, B\. Patra, D\. Perez\-Becker, T\. Portet, R\. Pryzant, H\. Qin, M\. Radmilac, L\. Ren, G\. de Rosa, C\. Rosset, S\. Roy, O\. Ruwase, O\. Saarikivi, A\. Saied, A\. Salim, M\. Santacroce, S\. Shah, N\. Shang, H\. Sharma, Y\. Shen, S\. Shukla, X\. Song, M\. Tanaka, A\. Tupini, P\. Vaddamanu, C\. Wang, G\. Wang, L\. Wang, S\. Wang, X\. Wang, Y\. Wang, R\. Ward, W\. Wen, P\. Witte, H\. Wu, X\. Wu, M\. Wyatt, B\. Xiao, C\. Xu, J\. Xu, W\. Xu, J\. Xue, S\. Yadav, F\. Yang, J\. Yang, Y\. Yang, Z\. Yang, D\. Yu, L\. Yuan, C\. Zhang, C\. Zhang, J\. Zhang, L\. L\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, and X\. Zhou \(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.External Links:2404\.14219,[Link](https://arxiv.org/abs/2404.14219)Cited by:[§D\.6](https://arxiv.org/html/2604.20216#A4.SS6.p1.1)\.
- Anthropic \(2025\)Claude sonnet 4\.Note:[https://www\.anthropic\.com](https://www.anthropic.com/)Accessed: 2026Cited by:[§D\.5](https://arxiv.org/html/2604.20216#A4.SS5.p1.1)\.
- S\. Arora, J\. W\. Taylor, and H\. Mak \(2023\)Probabilistic forecasting of patient waiting times in an emergency department\.Vol\.25\.External Links:[Document](https://dx.doi.org/10.1287/msom.2023.1210)Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p2.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2023\)Self\-rag: learning to retrieve, generate, and critique through self\-reflection\.arXiv preprint arXiv:2310\.11511\.Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px3.p1.1)\.
- R\. R\. Bahadur \(1966\)A note on quantiles in large samples\.The Annals of Mathematical Statistics37\(3\),pp\. 577–580\.External Links:ISSN 00034851,[Link](http://www.jstor.org/stable/2238725)Cited by:[§A\.1](https://arxiv.org/html/2604.20216#A1.SS1.SSS0.Px1.p2.4)\.
- Z\. Bitvai and T\. Cohn \(2015\)Non\-linear text regression with a deep convolutional neural network\.InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing \(Volume 2: Short Papers\),C\. Zong and M\. Strube \(Eds\.\),Beijing, China,pp\. 180–185\.External Links:[Link](https://aclanthology.org/P15-2030/),[Document](https://dx.doi.org/10.3115/v1/P15-2030)Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p1.1)\.
- T\. Chen and S\. Si \(2024\)Predicting rental price of lane houses in Shanghai with machine learning methods and large language models\.arXiv preprint arXiv:2405\.17505\.Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p1.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px4.p1.1)\.
- C\. Chiang, H\. Lee, and M\. Lukasik \(2025\)TRACT: regression\-aware fine\-tuning meets chain\-of\-thought reasoning for LLM\-as\-a\-judge\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 2934–2952\.External Links:[Link](https://aclanthology.org/2025.acl-long.147/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.147)Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px4.p1.1)\.
- N\. Dorka \(2024\)Quantile regression for distributional reward models in rlhf\.arXiv preprint arXiv:2409\.10164\.Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Garg, D\. Tsipras, P\. Liang, and G\. Valiant \(2022\)What can transformers learn in\-context? a case study of simple function classes\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Gillman, D\. Aggarwal, M\. Freeman, S\. Singh, and C\. Sun \(2025\)Fourier head: helping large language models learn complex probability distributions\.InProceedings of ICLR,Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Gneiting and A\. E\. Raftery \(2007\)Strictly proper scoring rules, prediction, and estimation\.Journal of the American Statistical Association102\(477\),pp\. 359–378\.External Links:[Document](https://dx.doi.org/10.1198/016214506000001437),[Link](https://doi.org/10.1198/016214506000001437),https://doi\.org/10\.1198/016214506000001437Cited by:[§4\.6\.1](https://arxiv.org/html/2604.20216#S4.SS6.SSS1.p2.3)\.
- N\. Gruver, M\. Finzi, S\. Qiu, and A\. G\. Wilson \(2023\)Large language models are zero\-shot time series forecasters\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p1.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Gu, Y\. Zhong, S\. Li, C\. Wei, L\. Dong, Z\. Wang, and C\. Yan \(2024\)Predicting stock prices with FinBERT\-LSTM: integrating news sentiment analysis\.In2024 8th International Conference on Cloud and Big Data Computing,External Links:[Document](https://dx.doi.org/10.1145/3694860.3694870),2407\.16150Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px4.p1.1)\.
- R\. Gürlek, F\. de Véricourt, and D\. K\.K\. Lee \(2024\)Boosted generalized normal distributions: integrating machine learning with operations knowledge\.Working PaperSSRN\.Note:Available at SSRN, Posted: August 1, 2024External Links:[Link](https://ssrn.com/abstract=4906838),[Document](https://dx.doi.org/10.2139/ssrn.4906838)Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§D\.1](https://arxiv.org/html/2604.20216#A4.SS1.p1.2),[§4\.2](https://arxiv.org/html/2604.20216#S4.SS2.p1.6)\.
- J\. M\. Imperial \(2021\)BERT embeddings for automatic readability assessment\.InProceedings of the International Conference on Recent Advances in Natural Language Processing \(RANLP 2021\),Held Online,pp\. 611–618\.External Links:[Document](https://dx.doi.org/10.26615/978-954-452-072-4%5F069),[Link](https://aclanthology.org/2021.ranlp-1.69/)Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1)\.
- Inside Airbnb \(2025\)Get the data\.Note:[https://insideairbnb\.com/get\-the\-data/](https://insideairbnb.com/get-the-data/)Dataset downloads \(city snapshots\)\. Licensed under CC BY 4\.0\. Accessed 2025\-12\-17Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p7.1),[§4\.1](https://arxiv.org/html/2604.20216#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2604.20216#S4.SS1.p1.2)\.
- R\. Jacobs, M\. P\. Polak, L\. E\. Schultz, H\. Mahdavi, V\. Honavar, and D\. Morgan \(2024\)Regression with large language models for materials and molecular property prediction\.arXiv preprint arXiv:2409\.06080\.Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p1.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§4\.4](https://arxiv.org/html/2604.20216#S4.SS4.p3.2)\.
- T\. Kneib, A\. Silbersdorff, and B\. Säfken \(2023\)Rage against the mean–a review of distributional regression approaches\.Econometrics and Statistics26,pp\. 99–123\.Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p2.1)\.
- R\. W\. Koenker and G\. Bassett \(1978\)Regression quantiles\.Econometrica46\(1\),pp\. 33–50\.External Links:[Link](https://econpapers.repec.org/RePEc:ecm:emetrp:v:46:y:1978:i:1:p:33-50)Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p2.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.InProceedings of the 34th International Conference on Neural Information Processing Systems,NIPS ’20,Red Hook, NY, USA\.External Links:ISBN 9781713829546Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Lukasik, Z\. Meng, H\. Narasimhan, Y\. Chang, A\. K\. Menon, F\. Yu, and S\. Kumar \(2025\)Better autoregressive regression with llms via regression\-aware fine\-tuning\.InProceedings of ICLR,Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Song, O\. Li, C\. Lee, B\. Yang, D\. Peng, S\. Perel, and Y\. Chen \(2024\)OmniPred: language models as universal regressors\.Trans\. Mach\. Learn\. Res\.2024\.External Links:[Link](https://openreview.net/forum?id=t9c3pfrR1X)Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1)\.
- Stack Overflow \(2019\)StackSample: 10% of stack overflow q&a\.Note:Kaggle datasetKaggle\-hosted dataset\. The page shows "Updated 6 years ago" when accessed on 2025\-12\-17, implying a last\-updated year of approximately 2019\.External Links:[Link](https://www.kaggle.com/datasets/stackoverflow/stacksample/data)Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p7.1),[§4\.1](https://arxiv.org/html/2604.20216#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2604.20216#S4.SS1.p1.2)\.
- Stack Overflow \(2025\)What is the license for the content i post?\.Note:Stack Overflow Help CenterStates Stack Overflow user contributions are licensed under CC BY\-SA, with version depending on contribution date \(2\.5/3\.0/4\.0\)\.External Links:[Link](https://stackoverflow.com/help/licensing)Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p7.1),[§4\.1](https://arxiv.org/html/2604.20216#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2604.20216#S4.SS1.p1.2)\.
- E\. Tang, B\. Yang, and X\. Song \(2025\)Understanding LLM embeddings for regression\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=Wt6Iz5XNIO)Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Vacareanu, V\. Negru, V\. Suciu, and M\. Surdeanu \(2024\)From words to numbers: your large language model is secretly a capable regressor when given in\-context examples\.External Links:2404\.07544,[Link](https://arxiv.org/abs/2404.07544)Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p1.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Vedula, D\. Dhyani, L\. Jalali, B\. N\. Oreshkin, M\. Bayati, and S\. Malmasi \(2025\)Quantile regression with large language models for price prediction\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 12396–12415\.External Links:[Link](https://aclanthology.org/2025.findings-acl.641/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.641),ISBN 979\-8\-89176\-256\-5Cited by:[Figure 1](https://arxiv.org/html/2604.20216#S1.F1),[§1](https://arxiv.org/html/2604.20216#S1.p3.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2604.20216#S4.SS2.p1.6),[§4\.3](https://arxiv.org/html/2604.20216#S4.SS3.p7.3)\.
- C\. Villani \(2009\)Optimal transport: old and new\.Grundlehren der mathematischen Wissenschaften, Vol\.338,Springer\.External Links:[Document](https://dx.doi.org/10.1007/978-3-540-71050-9),ISBN 978\-3\-540\-71049\-3,[Link](https://link.springer.com/book/10.1007/978-3-540-71050-9)Cited by:[§3\.4](https://arxiv.org/html/2604.20216#S3.SS4.SSS0.Px1.p2.8)\.
- H\. Wang, S\. You, Q\. Zhang, X\. Xie, S\. Han, Y\. Wu, F\. Huang, and J\. Chen \(2025\)LLP: llm\-based product pricing in e\-commerce\.External Links:2510\.09347,[Link](https://arxiv.org/abs/2510.09347)Cited by:[§1](https://arxiv.org/html/2604.20216#S1.p3.1),[§1](https://arxiv.org/html/2604.20216#S1.p6.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px4.p1.1)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. L\. Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. Rush \(2020\)HuggingFace’s transformers: state\-of\-the\-art natural language processing\.External Links:1910\.03771,[Link](https://arxiv.org/abs/1910.03771)Cited by:[§D\.1](https://arxiv.org/html/2604.20216#A4.SS1.p1.2)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§D\.1](https://arxiv.org/html/2604.20216#A4.SS1.p1.2),[§1](https://arxiv.org/html/2604.20216#S1.p7.1),[§4\.2](https://arxiv.org/html/2604.20216#S4.SS2.p1.6)\.
- R\. Yang, J\. Cao, Z\. Wen, Y\. Wu, and X\. He \(2020\)Enhancing automated essay scoring performance via fine\-tuning pre\-trained language models with combination of regression and ranking\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 1560–1569\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.141/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.141)Cited by:[§2](https://arxiv.org/html/2604.20216#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix

## Appendix ATheoretical Analysis of Quantile Supervision

This appendix formalizes the behavior of different objectives when training labels are*empirical quantiles*computed from finite samples\. We justify the empirical ordering observed in our experiments:ℒℓ1\\mathcal\{L\}\_\{\\ell\_\{1\}\}performs best,ℒPinball\-Q\\mathcal\{L\}\_\{\\text\{Pinball\-Q\}\}is slightly worse but better thanℒPinball\-Med\\mathcal\{L\}\_\{\\text\{Pinball\-Med\}\}, and pinball\-style objectives are appropriate only in the extremely sparse supervision regime\.

### A\.1Latent sample model and quantile\-label noise

Fix an input instanceX=xX=x\. LetF∗\(⋅∣x\)F^\{\*\}\(\\cdot\\mid x\)denote the true conditional distribution ofY∣X=xY\\mid X=xwith \(left\-continuous\) quantile function

Q∗​\(x,τ\):=inf\{t∈ℝ:F∗​\(t∣x\)≥τ\},τ∈\(0,1\)\.Q^\{\*\}\(x,\\tau\):=\\inf\\\{t\\in\\mathbb\{R\}:F^\{\*\}\(t\\mid x\)\\geq\\tau\\\},\\qquad\\tau\\in\(0,1\)\.For simplicity, here we assume allMiM\_\{i\}are equal toMM\. In our data, each inputXiX\_\{i\}is paired with a multiset of outcomes𝒴i=\{yi​1,…,yi​M\}\\mathcal\{Y\}\_\{i\}=\\\{y\_\{i1\},\\ldots,y\_\{iM\}\\\}, which we model as

yi​1,…,yi​M∼i\.i\.d\.F∗\(⋅∣Xi\),y\_\{i1\},\\ldots,y\_\{iM\}\\ \\overset\{\\text\{i\.i\.d\.\}\}\{\\sim\}\\ F^\{\*\}\(\\cdot\\mid X\_\{i\}\),with a variable sample sizeMMacross instances\. From𝒴i\\mathcal\{Y\}\_\{i\}we construct the empirical CDFF^i​\(t\)=1M​∑m=1M𝕀​\[yi​m≤t\]\\widehat\{F\}\_\{i\}\(t\)=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\mathbb\{I\}\[y\_\{im\}\\leq t\], and define the empirical quantile estimator \(with the same interpolation rule as in the main text\)

Q^i​\(τ\):=F^i−1​\(τ\)\.\\widehat\{Q\}\_\{i\}\(\\tau\):=\\widehat\{F\}\_\{i\}^\{\-1\}\(\\tau\)\.For theoretical analysis, we treatQ^i​\(τ\)\\widehat\{Q\}\_\{i\}\(\\tau\)as a standard sample quantile estimator; the additional linear interpolation changes the estimator by at mostO​\(1/M\)O\(1/M\)whenf​\(Q\)\>0f\(Q\)\>0that does not change the asymptotics in the largeMMregime\.

##### Regularity assumption\.

We assume that for the quantile levels of interestτ∈\[τmin,1−τmin\]\\tau\\in\[\\tau\_\{\\min\},1\-\\tau\_\{\\min\}\]: \(i\)F∗\(⋅∣x\)F^\{\*\}\(\\cdot\\mid x\)is continuously differentiable in a neighborhood ofQ∗​\(x,τ\)Q^\{\*\}\(x,\\tau\); \(ii\) the conditional densityf∗​\(t∣x\)=∂tF∗​\(t∣x\)f^\{\*\}\(t\\mid x\)=\\partial\_\{t\}F^\{\*\}\(t\\mid x\)exists and satisfiesf∗​\(Q∗​\(x,τ\)∣x\)\>0f^\{\*\}\(Q^\{\*\}\(x,\\tau\)\\mid x\)\>0\.

Under these conditions, sample quantiles admit a Bahadur\-type expansion and a central limit theorem \(e\.g\., “Bahadur representation”,Bahadur \([1966](https://arxiv.org/html/2604.20216#bib.bib42)\)\):

Q^i​\(τ\)=Q∗​\(Xi,τ\)\+τ−F^i​\(Q∗​\(Xi,τ\)\)f∗​\(Q∗​\(Xi,τ\)∣Xi\)\+ri,τ,ri,τ=op​\(M−1/2\)\.\\widehat\{Q\}\_\{i\}\(\\tau\)=Q^\{\*\}\(X\_\{i\},\\tau\)\+\\frac\{\\tau\-\\widehat\{F\}\_\{i\}\(Q^\{\*\}\(X\_\{i\},\\tau\)\)\}\{f^\{\*\}\(Q^\{\*\}\(X\_\{i\},\\tau\)\\mid X\_\{i\}\)\}\+r\_\{i,\\tau\},\\qquad r\_\{i,\\tau\}=o\_\{p\}\(M^\{\-1/2\}\)\.\(13\)As a consequence, asMMgrows to infinity,

M​\(Q^i​\(τ\)−Q∗​\(Xi,τ\)\)⇒𝒩​\(0,τ​\(1−τ\)f∗​\(Q∗​\(Xi,τ\)∣Xi\)2\)\.\\sqrt\{M\}\\Big\(\\widehat\{Q\}\_\{i\}\(\\tau\)\-Q^\{\*\}\(X\_\{i\},\\tau\)\\Big\)\\ \\Rightarrow\\ \\mathcal\{N\}\\\!\\left\(0,\\ \\frac\{\\tau\(1\-\\tau\)\}\{f^\{\*\}\(Q^\{\*\}\(X\_\{i\},\\tau\)\\mid X\_\{i\}\)^\{2\}\}\\right\)\.\(14\)ThusQ^i​\(τ\)\\widehat\{Q\}\_\{i\}\(\\tau\)can be viewed as a noisy measurement ofQ∗​\(Xi,τ\)Q^\{\*\}\(X\_\{i\},\\tau\)with heteroskedastic noise that is \(asymptotically\) centered and symmetric\.

### A\.2Population minimizers forℓ1\\ell\_\{1\}andℓ2\\ell\_\{2\}losses

Because our loss functions sum overkk, the linearity of expectation allows us to evaluate the population minimizers pointwise inτ\\tau\. Therefore, despite the finite\-sample correlation across the empirical quantile vector, our proofs are pointwise inτ\\tauand rely only on the marginal distribution in \([14](https://arxiv.org/html/2604.20216#A1.E14)\)\.

Fixxxandτ\\tauand define the random labelZ:=Q^i​\(τ\)∣\(Xi=x\)Z:=\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid\(X\_\{i\}=x\)\. Consider a scalar predictiona∈ℝa\\in\\mathbb\{R\}\.

###### Proposition 1\.

The conditional risk minimizers satisfy:

arg⁡mina∈ℝ⁡𝔼​\[\(Z−a\)2∣X=x\]\\displaystyle\\arg\\min\_\{a\\in\\mathbb\{R\}\}\\ \\mathbb\{E\}\\big\[\(Z\-a\)^\{2\}\\mid X=x\\big\]=𝔼​\[Z∣X=x\],\\displaystyle=\\mathbb\{E\}\[Z\\mid X=x\],arg⁡mina∈ℝ⁡𝔼​\[\|Z−a\|∣X=x\]\\displaystyle\\arg\\min\_\{a\\in\\mathbb\{R\}\}\\ \\mathbb\{E\}\\big\[\|Z\-a\|\\mid X=x\\big\]∈Median​\(Z∣X=x\)\.\\displaystyle\\in\\mathrm\{Median\}\(Z\\mid X=x\)\.

###### Proof\.

Both losses are convex inaa\. For squared loss, differentiate the conditional risk and set to zero\. For absolute loss, the subdifferential ofa→𝔼​\|Z−a\|a\\to\\mathbb\{E\}\|Z\-a\|is

∂=\[2​ℙ​\(Z​<a\|​x\)−1,2​ℙ​\(Z≤a\|x\)−1\]\\partial=\\left\[2\\mathbb\{P\}\(Z<a\|x\)\-1,2\\mathbb\{P\}\(Z\\leq a\|x\)\-1\\right\]so0∈∂0\\in\\partialif and only ifℙ​\(Z​<a\|​x\)≤12≤ℙ​\(Z≤a\|x\)\\mathbb\{P\}\(Z<a\|x\)\\leq\\frac\{1\}\{2\}\\leq\\mathbb\{P\}\(Z\\leq a\|x\)\. This yields the median characterization\. ∎

###### Proposition 2\(Asymptotic Fisher consistency ofℒℓ1\\mathcal\{L\}\_\{\\ell\_\{1\}\}andℒℓ2\\mathcal\{L\}\_\{\\ell\_\{2\}\}\)\.

Under the latent sample model and regularity assumptions above, for each fixed\(x,τ\)∈𝒳×\[τmin,1−τmin\]\(x,\\tau\)\\in\\mathcal\{X\}\\times\[\\tau\_\{\\min\},1\-\\tau\_\{\\min\}\], the population targets ofℒℓ1\\mathcal\{L\}\_\{\\ell\_\{1\}\}andℒℓ2\\mathcal\{L\}\_\{\\ell\_\{2\}\}converge to the true quantile:

limM→∞𝔼​\[Q^i​\(τ\)∣Xi=x\]=Q∗​\(x,τ\),limM→∞Median​\(Q^i​\(τ\)∣Xi=x\)=Q∗​\(x,τ\)\.\\lim\_\{M\\to\\infty\}\\mathbb\{E\}\\\!\\left\[\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}=x\\right\]=Q^\{\*\}\(x,\\tau\),\\qquad\\lim\_\{M\\to\\infty\}\\mathrm\{Median\}\\\!\\left\(\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}=x\\right\)=Q^\{\*\}\(x,\\tau\)\\,\.

###### Proof\.

Fix an instanceXi=xX\_\{i\}=xand a quantile levelτ∈\[τmin,1−τmin\]\\tau\\in\[\\tau\_\{\\min\},1\-\\tau\_\{\\min\}\]\.

To establish convergence in mean, we use the Bahadur representation from Equation \([13](https://arxiv.org/html/2604.20216#A1.E13)\):

Q^i​\(τ\)=Q∗​\(x,τ\)\+τ−F^i​\(Q∗​\(x,τ\)\)f∗​\(Q∗​\(x,τ\)∣x\)\+ri,τ\.\\widehat\{Q\}\_\{i\}\(\\tau\)=Q^\{\*\}\(x,\\tau\)\+\\frac\{\\tau\-\\widehat\{F\}\_\{i\}\(Q^\{\*\}\(x,\\tau\)\)\}\{f^\{\*\}\(Q^\{\*\}\(x,\\tau\)\\mid x\)\}\+r\_\{i,\\tau\}\\,\.Taking the conditional expectation of both sides, we note that𝔼​\[F^i​\(t\)\]=F∗​\(t∣x\)\\mathbb\{E\}\[\\widehat\{F\}\_\{i\}\(t\)\]=F^\{\*\}\(t\\mid x\)\. By definition,F∗​\(Q∗​\(x,τ\)∣x\)=τF^\{\*\}\(Q^\{\*\}\(x,\\tau\)\\mid x\)=\\tau, meaning the expectation of the middle term is exactly zero\. Under the stated regularity conditions, the expected value of the remainder termri,τr\_\{i,\\tau\}vanishes asM→∞M\\to\\infty\. This gives the mean consistency:

limM→∞𝔼​\[Q^i​\(τ\)∣Xi=x\]=Q∗​\(x,τ\)\.\\lim\_\{M\\to\\infty\}\\mathbb\{E\}\\\!\\left\[\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}=x\\right\]=Q^\{\*\}\(x,\\tau\)\\,\.
To establish median consistency, we rely on Equation \([14](https://arxiv.org/html/2604.20216#A1.E14)\), which shows thatM​\(Q^i​\(τ\)−Q∗​\(x,τ\)\)\\sqrt\{M\}\(\\widehat\{Q\}\_\{i\}\(\\tau\)\-Q^\{\*\}\(x,\\tau\)\)converges in distribution to a centered Gaussian\. Because the limiting normal distribution is continuous and strictly symmetric about zero, the median of the sequence of estimators converges to the median of the limiting distribution, yielding:

limM→∞Median​\(Q^i​\(τ\)∣Xi=x\)=Q∗​\(x,τ\)\.\\lim\_\{M\\to\\infty\}\\mathrm\{Median\}\\\!\\left\(\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}=x\\right\)=Q^\{\*\}\(x,\\tau\)\\,\.
Finally, we bridge these asymptotic limits to our learning objectives via Proposition[1](https://arxiv.org/html/2604.20216#Thmproposition1)\. Recall that for a given target variableZZ, the population risk minimizers for squared error \(ℒℓ2\\mathcal\{L\}\_\{\\ell\_\{2\}\}\) and absolute error \(ℒℓ1\\mathcal\{L\}\_\{\\ell\_\{1\}\}\) correspond to the conditional expectation𝔼​\[Z\]\\mathbb\{E\}\[Z\]and the conditional medianMedian​\(Z\)\\mathrm\{Median\}\(Z\), respectively\. Setting our target variable to the empirical quantile estimator,Z=Q^i​\(τ\)∣Xi=xZ=\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}=x, and substituting the asymptotic limits established above, we obtain the limiting optimal predictions:

limM→∞\(arg⁡mina∈ℝ⁡𝔼​\[\(Q^i​\(τ\)−a\)2∣Xi=x\]\)\\displaystyle\\lim\_\{M\\to\\infty\}\\left\(\\arg\\min\_\{a\\in\\mathbb\{R\}\}\\ \\mathbb\{E\}\\big\[\(\\widehat\{Q\}\_\{i\}\(\\tau\)\-a\)^\{2\}\\mid X\_\{i\}=x\\big\]\\right\)=limM→∞𝔼​\[Q^i​\(τ\)∣Xi=x\]=Q∗​\(x,τ\),\\displaystyle=\\lim\_\{M\\to\\infty\}\\mathbb\{E\}\[\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}=x\]=Q^\{\*\}\(x,\\tau\)\\,,limM→∞\(arg⁡mina∈ℝ⁡𝔼​\[\|Q^i​\(τ\)−a\|∣Xi=x\]\)\\displaystyle\\lim\_\{M\\to\\infty\}\\left\(\\arg\\min\_\{a\\in\\mathbb\{R\}\}\\ \\mathbb\{E\}\\big\[\|\\widehat\{Q\}\_\{i\}\(\\tau\)\-a\|\\mid X\_\{i\}=x\\big\]\\right\)=limM→∞Median​\(Q^i​\(τ\)∣Xi=x\)=Q∗​\(x,τ\)\.\\displaystyle=\\lim\_\{M\\to\\infty\}\\mathrm\{Median\}\(\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}=x\)=Q^\{\*\}\(x,\\tau\)\\,\.Consequently, as the number of observed outcomesM→∞M\\to\\infty, the population targets for bothℒℓ1\\mathcal\{L\}\_\{\\ell\_\{1\}\}andℒℓ2\\mathcal\{L\}\_\{\\ell\_\{2\}\}correctly converge to the true latent quantileQ∗​\(x,τ\)Q^\{\*\}\(x,\\tau\), completing the proof\. ∎

### A\.3Bias of mismatched pinball on empirical quantiles \(Pinball\-Q\)

###### Proposition 3\.

Letρτ​\(u\)=u​\(τ−𝕀​\[u<0\]\)\\rho\_\{\\tau\}\(u\)=u\(\\tau\-\\mathbb\{I\}\[u<0\]\)\. For any scalar random variableZZ,

arg⁡mina∈ℝ⁡𝔼​\[ρτ​\(Z−a\)\]∈Quantileτ​\(Z\),\\arg\\min\_\{a\\in\\mathbb\{R\}\}\\ \\mathbb\{E\}\\big\[\\rho\_\{\\tau\}\(Z\-a\)\\big\]\\ \\in\\ \\mathrm\{Quantile\}\_\{\\tau\}\(Z\),i\.e\., pinball loss targets theτ\\tau\-th quantile of the label distribution\.

###### Proof\.

Similar to the above proof forℓ1\\ell\_\{1\}, the subdifferential ofa↦𝔼​\[ρτ​\(Z−a\)\]a\\mapsto\\mathbb\{E\}\[\\rho\_\{\\tau\}\(Z\-a\)\]is

∂=\[ℙ​\(Z​<a\|​x\)−τ,ℙ​\(Z≤a\|x\)−τ\]\\partial=\\left\[\\mathbb\{P\}\(Z<a\|x\)\-\\tau,\\mathbb\{P\}\(Z\\leq a\|x\)\-\\tau\\right\]so the optimality condition isℙ​\(Z​<a\|​x\)≤τ≤ℙ​\(Z≤a\|x\)\\mathbb\{P\}\(Z<a\|x\)\\leq\\tau\\leq\\mathbb\{P\}\(Z\\leq a\|x\)\. ∎

###### Proposition 4\(Bias ofℒPinball\-Q\\mathcal\{L\}\_\{\\text\{Pinball\-Q\}\}\)\.

Fix\(x,τ\)\(x,\\tau\)and write the empirical quantile label as

Q^i​\(τ\)=Q∗​\(x,τ\)\+εi,τ,εi,τ:=Q^i​\(τ\)−Q∗​\(x,τ\)\.\\widehat\{Q\}\_\{i\}\(\\tau\)=Q^\{\*\}\(x,\\tau\)\+\\varepsilon\_\{i,\\tau\},\\qquad\\varepsilon\_\{i,\\tau\}:=\\widehat\{Q\}\_\{i\}\(\\tau\)\-Q^\{\*\}\(x,\\tau\)\.The population minimizer ofℒPinball\-Q\\mathcal\{L\}\_\{\\text\{Pinball\-Q\}\}at levelτ\\tausatisfies

qPinball\-Q∗​\(x,τ\)=Q∗​\(x,τ\)\+Quantileτ​\(εi,τ∣Xi=x\)\.q^\{\*\}\_\{\\text\{Pinball\-Q\}\}\(x,\\tau\)=Q^\{\*\}\(x,\\tau\)\+\\mathrm\{Quantile\}\_\{\\tau\}\\\!\\big\(\\varepsilon\_\{i,\\tau\}\\mid X\_\{i\}=x\\big\)\.Under the normal approximation in \([14](https://arxiv.org/html/2604.20216#A1.E14)\), the resulting inflation/deflation bias is approximately

qPinball\-Q∗​\(x,τ\)≈Q∗​\(x,τ\)\+Φ−1​\(τ\)​τ​\(1−τ\)M​1f∗​\(Q∗​\(x,τ\)∣x\)\.q^\{\*\}\_\{\\text\{Pinball\-Q\}\}\(x,\\tau\)\\approx Q^\{\*\}\(x,\\tau\)\+\\Phi^\{\-1\}\(\\tau\)\\,\\sqrt\{\\frac\{\\tau\(1\-\\tau\)\}\{M\}\}\\,\\frac\{1\}\{f^\{\*\}\(Q^\{\*\}\(x,\\tau\)\\mid x\)\}\.\(15\)

###### Proof\.

By Proposition[3](https://arxiv.org/html/2604.20216#Thmproposition3),ℒPinball\-Q\\mathcal\{L\}\_\{\\text\{Pinball\-Q\}\}targets theτ\\tau\-quantile of the random labelQ^i​\(τ\)∣Xi=x\\widehat\{Q\}\_\{i\}\(\\tau\)\\mid X\_\{i\}=x\. Quantiles are translation\-equivariant, giving the first display\. Under \([14](https://arxiv.org/html/2604.20216#A1.E14)\),εi,τ\\varepsilon\_\{i,\\tau\}is approximately normal with standard deviationτ​\(1−τ\)/\(M​f∗​\(Q∗∣x\)\)\\sqrt\{\\tau\(1\-\\tau\)\}/\(\\sqrt\{M\}\\,f^\{\*\}\(Q^\{\*\}\\mid x\)\), and theτ\\tau\-quantile of a centered normal isΦ−1​\(τ\)\\Phi^\{\-1\}\(\\tau\)times its standard deviation\. ∎

Equation \([15](https://arxiv.org/html/2604.20216#A1.E15)\) shows that the leadingO​\(M−1/2\)O\(M^\{\-1/2\}\)inflation term vanishes atτ=0\.5\\tau=0\.5, but Pinball\-Q systematically inflates upper quantiles \(τ\>0\.5\\tau\>0\.5\) and deflates lower quantiles \(τ<0\.5\\tau<0\.5\)\. The magnitude of this mismatch isO​\(M−1/2\)O\(M^\{\-1/2\}\)and grows away fromτ=0\.5\\tau=0\.5over typical interior grids and can be further amplified whenf∗f^\{\*\}is small \(often in tails\)\.

### A\.4Scalarized pinball is inconsistent for distribution learning \(Pinball\-Med\)

Define the scalar pseudo\-target

yi:=Q^i​\(0\.5\),y\_\{i\}:=\\widehat\{Q\}\_\{i\}\(0\.5\),and recall the scalarized objectiveℒPinball\-Med​\(θ\)=1N​Q​∑i,kρτk​\(yi−q^τk​\(Xi\)\)\\mathcal\{L\}\_\{\\text\{Pinball\-Med\}\}\(\\theta\)=\\frac\{1\}\{NQ\}\\sum\_\{i,k\}\\rho\_\{\\tau\_\{k\}\}\(y\_\{i\}\-\\hat\{q\}\_\{\\tau\_\{k\}\}\(X\_\{i\}\)\)\.

###### Proposition 5\(WhatℒPinball\-Med\\mathcal\{L\}\_\{\\text\{Pinball\-Med\}\}learns\)\.

For eachτ∈\(0,1\)\\tau\\in\(0,1\)and fixedxx, the population minimizer ofℒPinball\-Med\\mathcal\{L\}\_\{\\text\{Pinball\-Med\}\}at levelτ\\tausatisfies

qPinball\-Med∗​\(x,τ\)∈Quantileτ​\(yi∣Xi=x\)\.q^\{\*\}\_\{\\text\{Pinball\-Med\}\}\(x,\\tau\)\\in\\mathrm\{Quantile\}\_\{\\tau\}\\big\(y\_\{i\}\\mid X\_\{i\}=x\\big\)\.Consequently,ℒPinball\-Med\\mathcal\{L\}\_\{\\text\{Pinball\-Med\}\}is proper for the conditional distribution of the statisticyi=Q^i​\(0\.5\)y\_\{i\}=\\widehat\{Q\}\_\{i\}\(0\.5\), not forY∣XY\\mid X\.

###### Proof\.

Fix\(x,τ\)\(x,\\tau\)and viewyiy\_\{i\}as the random label\. Applying Proposition[3](https://arxiv.org/html/2604.20216#Thmproposition3)toZ=yi∣Xi=xZ=y\_\{i\}\\mid X\_\{i\}=xyields the claim\. ∎

###### Proposition 6\(Concentration in the large\-MMregime\)\.

AssumeM→∞M\\to\\inftyand the regularity conditions above\. Thenyi=Q^i​\(0\.5\)→Q∗​\(x,0\.5\)y\_\{i\}=\\widehat\{Q\}\_\{i\}\(0\.5\)\\to Q^\{\*\}\(x,0\.5\)in probability, and hence

qPinball\-Med∗​\(x,τ\)→Q∗​\(x,0\.5\)for all​τ∈\(0,1\)\.q^\{\*\}\_\{\\text\{Pinball\-Med\}\}\(x,\\tau\)\\to Q^\{\*\}\(x,0\.5\)\\qquad\\text\{for all \}\\tau\\in\(0,1\)\.IfF∗\(⋅∣x\)F^\{\*\}\(\\cdot\\mid x\)is non\-degenerate, this implies a non\-vanishing distributional error; for example,

W1\(F∗\(⋅∣x\),δQ∗​\(x,0\.5\)\)=∫01\|Q∗\(x,u\)−Q∗\(x,0\.5\)\|du\>0\.W\_\{1\}\\\!\\left\(F^\{\*\}\(\\cdot\\mid x\),\\ \\delta\_\{Q^\{\*\}\(x,0\.5\)\}\\right\)=\\int\_\{0\}^\{1\}\\big\|Q^\{\*\}\(x,u\)\-Q^\{\*\}\(x,0\.5\)\\big\|\\,du\\;\>\\;0\.

###### Proof sketch\.

Consistency of the sample median follows from standard quantile consistency\. Asyiy\_\{i\}concentrates, the conditional distribution ofyi∣Xi=xy\_\{i\}\\mid X\_\{i\}=xconverges to a point mass atQ∗​\(x,0\.5\)Q^\{\*\}\(x,0\.5\), whoseτ\\tau\-quantile equals the same point for everyτ\\tau\. TheW1W\_\{1\}identity is standard in one dimension\. ∎

### A\.5Towards a variance\-aware weighting

Equation \([14](https://arxiv.org/html/2604.20216#A1.E14)\) implies heteroskedastic noise across\(M,τ\)\(M,\\tau\)\. A variance\-aware extension replaces the unweighted losses with weights proportional to the inverse asymptotic variance,

wi,k∝M​f∗​\(Q∗​\(Xi,τk\)∣Xi\)2τk​\(1−τk\),w\_\{i,k\}\\ \\propto\\ \\frac\{M\\,f^\{\*\}\(Q^\{\*\}\(X\_\{i\},\\tau\_\{k\}\)\\mid X\_\{i\}\)^\{2\}\}\{\\tau\_\{k\}\(1\-\\tau\_\{k\}\)\},yielding a quasi\-likelihood weighted least squares objective\. In practice, the unknown density factor can be estimated from the predicted quantile slope viaf∗​\(Q∗​\(x,τ\)∣x\)=1/∂τQ∗​\(x,τ\)f^\{\*\}\(Q^\{\*\}\(x,\\tau\)\\mid x\)=1/\\partial\_\{\\tau\}Q^\{\*\}\(x,\\tau\)\(when the quantile function is differentiable and strictly increasing\), suggesting a fully data\-adaptive weighting scheme\. We leave a systematic study of these weights to future work\.

## Appendix BQuantile Interpolation

Each instance is associated with a variable\-size set of observed outcomes𝒴i=\{yi​1,…,yi​Mi\}\\mathcal\{Y\}\_\{i\}=\\\{y\_\{i1\},\\ldots,y\_\{iM\_\{i\}\}\\\}, whereMiM\_\{i\}can differ across instances\. We sort them to obtain order statisticsyi​\(1\)≤⋯≤yi​\(Mi\)y\_\{i\(1\)\}\\leq\\cdots\\leq y\_\{i\(M\_\{i\}\)\}and treat these samples as defining an empirical quantile function\. To obtain a fixed\-dimensional training target, we interpolate the empirical quantile function to a dense grid ofQ=99Q=99quantile levels𝝉=\{0\.01,0\.02,…,0\.99\}\\boldsymbol\{\\tau\}=\\\{0\.01,0\.02,\\ldots,0\.99\\\}, producing

𝐪^i=\(Q^i​\(0\.01\),…,Q^i​\(0\.99\)\)∈ℝ99\.\\widehat\{\\mathbf\{q\}\}\_\{i\}=\\big\(\\widehat\{Q\}\_\{i\}\(0\.01\),\\ldots,\\widehat\{Q\}\_\{i\}\(0\.99\)\\big\)\\in\\mathbb\{R\}^\{99\}\.\(16\)
For eachτk\\tau\_\{k\}, we compute a fractional rankri​k=1\+\(Mi−1\)​τkr\_\{ik\}=1\+\(M\_\{i\}\-1\)\\tau\_\{k\}and setQ^i​\(τk\)\\widehat\{Q\}\_\{i\}\(\\tau\_\{k\}\)by linear interpolation betweenyi​\(⌊ri​k⌋\)y\_\{i\(\\lfloor r\_\{ik\}\\rfloor\)\}andyi​\(⌈ri​k⌉\)y\_\{i\(\\lceil r\_\{ik\}\\rceil\)\}\.

## Appendix CDataset Construction Details

##### Airbnb\.

We collect all available cities from Inside Airbnb data snapshots between 2024\-09 and 2025\-08, covering 119 cities\. We drop listings with fewer than 4 price observations and remove cities with fewer than 10k samples after filtering, yielding∼\\sim840k samples from 55 cities\. For retrieval, we build one training index per city and restrict retrieved neighbors to the same city\. For each retrieved neighbor, we append only its title to the model input\.

##### StackSample\.

We filter out answers with response time exceeding 12 hours, convert each remaining response time to minutes, then apply log transformation to ensure a manageable range\. We perform quantile interpolation over the log\-transformed response times to create the ground\-truth quantile distribution\. For retrieval, we build an index from training questions only\. For each retrieved neighbor question, we append only its title to the model input\.

## Appendix DExperimental Details

### D\.1Experimental Setup

We run all experiments on a single AWS GPU cluster with NVIDIA H100 and H200 GPUs, setQ=99Q=99with uniformly spaced quantile levels, and tune hyperparameters \(e\.g\., number of epochs, batch sizes, learning rates, and maximum sequence length\) across settings, model sizes, and datasets to report the best configuration\. Unless otherwise specified, we useK=8K=8retrieved neighbors\. We fine\-tune the Qwen3 model family\(Yanget al\.,[2025](https://arxiv.org/html/2604.20216#bib.bib36)\)with LoRA\(Huet al\.,[2022](https://arxiv.org/html/2604.20216#bib.bib35)\)using the HuggingFace Transformers library\(Wolfet al\.,[2020](https://arxiv.org/html/2604.20216#bib.bib37)\)\.

### D\.2Hyperparameters

This section provides the hyperparameter configurations used in our experiments\. All models were trained using LoRA fine\-tuning on the Qwen3 family\. Table[8](https://arxiv.org/html/2604.20216#A4.T8)shows hyperparameters for Inside Airbnb experiments, and Table[9](https://arxiv.org/html/2604.20216#A4.T9)shows hyperparameters for StackSample experiments\.

Table 8:Hyperparameters for Inside Airbnb experiments\. K = number of neighbors, LR = learning rate, BS = batch size per device, Max Len = maximum sequence length\. All models use weight decay = 0\.01, LoRA dropout = 0\.1, and Wasserstein W1 loss\.Table 9:Hyperparameters for StackSample experiments\. K = number of neighbors, LR = learning rate, BS = batch size per device, WD = weight decay, Warmup = warmup ratio, Max Len = maximum sequence length\. QR models use LoRA dropout = 0\.1, QT model uses LoRA dropout = 0\.15\. All use Wasserstein W1 loss\.
### D\.3Retrieval Efficiency and Cost

While our main results and ablations show that retrieval augmentation consistently improves predictive performance, incorporating retrieved context increases input length and computational cost\. Table[10](https://arxiv.org/html/2604.20216#A4.T10)quantifies this tradeoff for Qwen3\-4B on the Airbnb dataset\.

Table 10:Training and inference efficiency as the number of retrieved neighborsKKincreases \(Qwen3\-4B on Airbnb\)\.As the number of retrieved neighborsKKincreases, both training and inference throughput decrease substantially\. IncreasingKKfrom 0 to 8 raises training time from 23\.71 to 64\.95 hours \(2\.7×\\times\) and reduces inference throughput from 208\.52 to 76\.43 samples/sec\. Combined with the accuracy results in Table[6](https://arxiv.org/html/2604.20216#S4.T6), these measurements suggestK=8K\{=\}8as a practical operating point that balances predictive gains against computational cost\.

### D\.4Statistical Significance

Table 11:Paired statistical significance tests on per\-instance avg\-MAPE using Qwen3\-4B\. Comparisons areK=0→K=8K\{=\}0\\rightarrow K\{=\}8under QR, and QR→\\rightarrowQT atK=8K\{=\}8\. NegativeΔ\\Deltaindicates lower error for the second condition\.We assess whether the improvements from retrieval augmentation and quantile tokens are statistically significant using paired two\-sidedtt\-tests on per\-instance average MAPE\. For each instance, we first compute the mean absolute percentage error across the nine evaluation quantiles \(10–90\), and then compute paired differences between model variants\.

Table[11](https://arxiv.org/html/2604.20216#A4.T11)reports results for Qwen3\-4B on both datasets\. Increasing the number of retrieved neighbors fromK=0K\{=\}0toK=8K\{=\}8yields statistically significant improvements on both Airbnb and StackSample \(p<0\.001p<0\.001\)\. Similarly, replacing QR with QT atK=8K\{=\}8also leads to significant gains\. The magnitude of improvement is particularly large on StackSample, where retrieval reduces avg\-MAPE by 168\.09 points, and QT further reduces it by 14\.26 points\.

Overall, all comparisons are statistically significant with large effect sizes, confirming that the observed improvements are not due to random variation\.

### D\.5Autoregressive Baseline

We evaluate an autoregressive baseline using Claude Sonnet 4\(Anthropic,[2025](https://arxiv.org/html/2604.20216#bib.bib43)\)in a few\-shot setting on StackSample\. To ensure a controlled comparison, we match the retrieval setup and useK=8K\{=\}8neighbors\. Each prompt includes the query and retrieved examples, and the model is instructed to directly generate the target quantiles\.

Due to the brittleness of long\-form numeric generation, we restrict the model to output a compact set of 9 quantiles \(10–90\)\. As a result, metrics that require dense quantile grids \(e\.g\., RCIW@95 and RCIW@99\) are not applicable\.

Table 12:Comparison with an autoregressive baseline \(Claude Sonnet 4\) on the StackSample test split\. The autoregressive model outputs 9 quantiles \(10–90\), so only metrics defined on this subset are reported\.Table[12](https://arxiv.org/html/2604.20216#A4.T12)compares the autoregressive baseline with fine\-tuned models under the same retrieval setting\. The autoregressive baseline exhibits substantially higher avg\-MAPE, driven by large errors in the upper quantiles\. While metrics such as sMAPE, wMAPE, and CRPSS may appear competitive, they primarily reflect central tendency and can mask extreme deviations\. In particular, the model tends to underestimate high quantiles, resulting in under\-dispersed predictions and poor coverage\. Overall, these results indicate that fine\-tuned regression models, especially QT, are more reliable for distributional prediction\.

### D\.6Cross\-Family Validation

To evaluate whether the proposed method generalizes beyond the Qwen3 model family, we conduct experiments onPhi\-3\-mini\-4k\-instruct\(Abdinet al\.,[2024](https://arxiv.org/html/2604.20216#bib.bib44)\)and compare QR and QT under the same retrieval setup\.

Table 13:Cross\-family validation onPhi\-3\-mini\-4k\-instructon StackSample\. We compare QR and QT withK∈\{0,8\}K\\in\\\{0,8\\\}\.The results in Table[13](https://arxiv.org/html/2604.20216#A4.T13)show consistent trends: retrieval improves the QR baseline, and QT yields further gains in both accuracy and distributional quality\. This suggests that the effectiveness of quantile tokens and retrieval augmentation is not specific to a single backbone model\.