Reliable Financial Named Entity Recognition under Domain Shift

# Reliable Financial Named Entity Recognition under Domain Shift:Confidence Estimation and Selective Prediction
Source: [https://arxiv.org/html/2608.19558](https://arxiv.org/html/2608.19558)
Zihao Zheng1, Baichuan Li2, Junyi Yao1, and Jiayu Long1Affiliation:1Washington University in St\. Louis St\. Louis, Missouri, USA z\.zihaogary@wustl\.edu; j\.yao@wustl\.edu; jiayujacqueline@wustl\.eduAffiliation:2Southern Methodist University Dallas, Texas, USA baichuanl@smu\.edu

###### Abstract

Financial AI systems often train information extractors on one textual register and deploy them across filings, news, and user\-generated content, and standard F1 scores do not indicate which predictions remain safe to automate when that input distribution changes\. We study confidence estimation and selective prediction for financial named entity recognition \(NER\) on a three\-tier stress test spanning SEC filings, financial news, and general\-topic social media as an extreme out\-of\-domain condition, evaluating a BERT tagger and LoRA\-tuned Qwen2\.5\-0\.5B/1\.5B models with five inference\-time confidence signals, three training seeds, and bootstrap intervals\. Confidence rankings themselves change under shift: whole\-output probability is the strongest in\-domain error detector but deteriorates out of domain, whereas entity\-span probability and self\-consistency are more robust; self\-consistency is also better calibrated without post\-hoc fitting\. Abstention reduces sentence error from 34\.3% to below 2% on the highest\-confidence 40% of in\-domain inputs and remains useful on financial news, but recovers no usefully large clean subset under the extreme social\-media shift\. These results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction\-level confidence gating\.

###### Index Terms:

financial named entity recognition, domain shift, confidence estimation, selective prediction, uncertainty estimation, large language models

## IIntroduction

Named entity recognition \(NER\) is a core information\-extraction component of applied financial AI systems for compliance, risk, and market\-intelligence workflows\. A growing line of work fine\-tunes small open\-weight language models for financial NER with parameter\-efficient methods and reports steadily rising F1 on in\-domain test splits\[[17](https://arxiv.org/html/2608.19558#bib.bib9),[35](https://arxiv.org/html/2608.19558#bib.bib10),[20](https://arxiv.org/html/2608.19558#bib.bib8)\]\. Yet purpose\-built financial LLMs still describe span\-level NER as one of their hardest tasks\[[34](https://arxiv.org/html/2608.19558#bib.bib5),[36](https://arxiv.org/html/2608.19558#bib.bib6),[37](https://arxiv.org/html/2608.19558#bib.bib7)\], and a deployed system faces a question that leaderboard F1 does not answer:*when the input distribution drifts away from the training data, can the model tell us which of its extractions to trust?*

This matters because financial text is not one domain: a model trained on formal filings language \(“Apple Inc\. reported revenue of $94\.9 billion\.”\) will meet news headlines and social\-media commentary \(“AAPL just destroyed earnings”\)\. Prior work shows that neural models become*overconfident*precisely when inputs shift away from training data\[[25](https://arxiv.org/html/2608.19558#bib.bib19),[14](https://arxiv.org/html/2608.19558#bib.bib15)\], while complementary work detects harmful covariate shift at the distribution level before unreliable predictions reach downstream systems\[[8](https://arxiv.org/html/2608.19558#bib.bib38)\]\. NER models in particular lean on memorized entity surface forms rather than context\[[18](https://arxiv.org/html/2608.19558#bib.bib30)\]\. In a financial setting, a hallucinated organization or a mistyped person entity is not benign: such errors propagate into downstream compliance, monitoring, and analytical pipelines\.

We therefore study financial NER through the lens of*calibrated selective prediction*\[[1](https://arxiv.org/html/2608.19558#bib.bib11),[5](https://arxiv.org/html/2608.19558#bib.bib12),[7](https://arxiv.org/html/2608.19558#bib.bib13)\]: allow the model to abstain on low\-confidence predictions and route them to human review, and ask how much reliability this buys at what coverage cost\. We fine\-tune a BERT encoder tagger and instruction\-tuned generative models \(Qwen2\.5\-0\.5B and 1\.5B\-Instruct with LoRA\[[26](https://arxiv.org/html/2608.19558#bib.bib35),[11](https://arxiv.org/html/2608.19558#bib.bib34)\]\) on the FIN corpus of SEC filings\[[27](https://arxiv.org/html/2608.19558#bib.bib1)\]and evaluate them on a three\-tier stress test: in\-domain filings \(FIN test\), financial news \(FiNER\-ORD\[[28](https://arxiv.org/html/2608.19558#bib.bib2)\]\), and general\-topic social media as an extreme out\-of\-domain condition \(TweetNER7\[[30](https://arxiv.org/html/2608.19558#bib.bib3)\]\)\. For the generative models we compare five inference\-time confidence signals that require no additional training, spanning whole\-output probabilities, entity\-restricted probabilities, and self\-consistency vote share over sampled decodes\[[33](https://arxiv.org/html/2608.19558#bib.bib24),[21](https://arxiv.org/html/2608.19558#bib.bib25)\]\. Following\[[14](https://arxiv.org/html/2608.19558#bib.bib15)\], all thresholds and calibration parameters are chosen on in\-domain validation data and frozen before touching any test set; results carry multi\-seed variance and bootstrap confidence intervals\.

This study contributes a deployment\-centered reliability evaluation for financial NER: it demonstrates that confidence rankings are domain\-sensitive \(the strongest in\-domain signal can become inferior after a register or topic shift\), identifies entity\-span probability and self\-consistency as complementary signals \(stronger error ranking vs\. better calibration\), and characterizes the operating boundary of confidence gating, whose severe\-shift failures motivate upstream domain\-level screening as future work\.

## IIRelated Work

Three lines of work underpin our study\.

#### Financial NER

The FIN dataset\[[27](https://arxiv.org/html/2608.19558#bib.bib1)\]framed financial NER as domain adaptation from newswire to SEC filings and remains the standard filings\-domain benchmark, adopted by FLUE\[[29](https://arxiv.org/html/2608.19558#bib.bib4)\], BloombergGPT\[[34](https://arxiv.org/html/2608.19558#bib.bib5)\], PIXIU\[[36](https://arxiv.org/html/2608.19558#bib.bib6)\], and FinBen\[[37](https://arxiv.org/html/2608.19558#bib.bib7)\]; FiNER\-ORD\[[28](https://arxiv.org/html/2608.19558#bib.bib2)\]provides manually annotated financial*news*\. Recent work fine\-tunes small LLMs for financial NER with LoRA\-style adapters\[[17](https://arxiv.org/html/2608.19558#bib.bib9),[35](https://arxiv.org/html/2608.19558#bib.bib10)\]or evaluates prompting\[[20](https://arxiv.org/html/2608.19558#bib.bib8)\], reporting F1 on a single\-domain split without uncertainty analysis\. We hold the training recipe fixed and instead ask whether model confidence survives domain shift\.

#### Selective prediction and calibration

Abstention dates to Chow\[[1](https://arxiv.org/html/2608.19558#bib.bib11)\];\[[5](https://arxiv.org/html/2608.19558#bib.bib12),[7](https://arxiv.org/html/2608.19558#bib.bib13)\]formalized the risk–coverage trade\-off, with maximum softmax probability \(MSP\) as the canonical baseline\[[10](https://arxiv.org/html/2608.19558#bib.bib14)\]\. Modern networks are miscalibrated\[[9](https://arxiv.org/html/2608.19558#bib.bib17)\], and calibration degrades under dataset shift\[[25](https://arxiv.org/html/2608.19558#bib.bib19),[3](https://arxiv.org/html/2608.19558#bib.bib20)\]; complementary work detects harmful covariate shift at the distribution level to identify when deployment conditions may invalidate model generalization\[[8](https://arxiv.org/html/2608.19558#bib.bib38)\]\. Closest to our setting, Kamath et al\.\[[14](https://arxiv.org/html/2608.19558#bib.bib15)\]studied selective*question answering*under domain shift and showed softmax confidence becomes unreliable out of domain;\[[31](https://arxiv.org/html/2608.19558#bib.bib16)\]cautioned that simple probability baselines are hard to beat, which our results confirm in\-domain and refute under shift\.

#### Uncertainty for NER and generative extraction

Span\-level confidence for extraction goes back to CRF\-based estimates\[[2](https://arxiv.org/html/2608.19558#bib.bib28)\]; later work calibrated entity\-level confidences of encoder taggers\[[12](https://arxiv.org/html/2608.19558#bib.bib29)\]\. Casting NER as text generation\[[32](https://arxiv.org/html/2608.19558#bib.bib31),[39](https://arxiv.org/html/2608.19558#bib.bib32),[4](https://arxiv.org/html/2608.19558#bib.bib33)\]makes token probabilities, sampling agreement\[[33](https://arxiv.org/html/2608.19558#bib.bib24),[21](https://arxiv.org/html/2608.19558#bib.bib25)\], and semantic entropy\[[15](https://arxiv.org/html/2608.19558#bib.bib22),[6](https://arxiv.org/html/2608.19558#bib.bib23)\]available as confidence signals; sequence probability requires length correction\[[23](https://arxiv.org/html/2608.19558#bib.bib27)\]\. Verbalized confidence is systematically overconfident for small models\[[38](https://arxiv.org/html/2608.19558#bib.bib26)\], so we restrict our study to logit\- and sampling\-based signals\. To our knowledge, prior financial NER work has not evaluated whether confidence rankings and calibration survive such shifts; this is the gap we target\. On the deployment side, learning\-to\-defer\[[22](https://arxiv.org/html/2608.19558#bib.bib36)\]and uncertainty\-guided human–LLM work allocation\[[16](https://arxiv.org/html/2608.19558#bib.bib37)\]motivate our risk–coverage framing\.

## IIIData and Evaluation Protocol

We build a three\-tier stress test from public datasets, harmonized to a shared \{PER, ORG, LOC\} schema \(Table[I](https://arxiv.org/html/2608.19558#S3.T1)\)\. The first two tiers stay within finance while changing register; the third intentionally leaves the financial domain to probe a boundary condition under severe distribution shift\. Because the corpora also differ in annotation process, topic, time, and entity density, the tiers are neither a controlled causal decomposition of domain shift nor a purely gradual register change\.

#### In\-domain: FIN \(SEC filings\)

The FIN corpus\[[27](https://arxiv.org/html/2608.19558#bib.bib1)\]contains sentences from US SEC financial agreements; its standard train/valid/test splits serve for training, calibration/threshold fitting, and in\-domain testing\. Sentences containing MISC entities are dropped \(6 test sentences\) so that all three tiers share the same schema\.

#### Near shift: FiNER\-ORD \(financial news\)

FiNER\-ORD\[[28](https://arxiv.org/html/2608.19558#bib.bib2)\]provides manually annotated PER/ORG/LOC entities over financial news articles\. The register shift from formal filings boilerplate to journalistic prose constitutes our near\-shift tier\.

#### Far shift: TweetNER7 \(social media\)

TweetNER7\[[30](https://arxiv.org/html/2608.19558#bib.bib3)\]is a*general\-topic*Twitter NER dataset, not a financial\-tweet corpus\. We use it only as an extreme out\-of\-domain stress test for a model trained on financial filings\. It annotates seven entity types includingcorporation,person, andlocation, which we map to ORG/PER/LOC\. To keep gold labels clean, our main far\-shift set keeps only tweets whose entities all fall inside the mapped schema, dropping the rest rather than silently relabeling\. This removes 2,179 of 2,807 tweets in the 2021 test split, leaving 628 eligible tweets \(Table[VI](https://arxiv.org/html/2608.19558#S7.T6)\) — a deliberate precision\-over\-size choice quantified by a sensitivity analysis \(§[VII](https://arxiv.org/html/2608.19558#S7)\) whose*relaxed*variant keeps every tweet and instead*ignores*out\-of\-schema gold entities, so matching predictions receive neither credit nor penalty\. URL placeholders and user\-mention markup are normalized to plain surface forms\. Both out\-of\-domain test sets are capped at 300 sentences with a fixed subsampling seed\.

TABLE I:Dataset statistics after harmonization and filtering\.All filtering decisions are recorded in machine\-readable pipeline manifests\.

### III\-AEvaluation protocol

Encoder predictions are scored as exact BIO span matches\. Generative predictions are generated text, so we score them by*position\-less multiset matching*: each predicted \(surface form, type\) pair is matched against the multiset of gold pairs\. Because the two model families are thus scored under different matching procedures, direct encoder–generative F1 comparisons should be read as indicative rather than exact\. Surface forms are lowercased, whitespace\-collapsed, and stripped of boundary punctuation, so tokenization artifacts do not count as errors; duplicate predictions can match at most as many gold occurrences as exist; wrong boundaries or types count as both a false positive and a false negative\. Invalid JSON is salvaged with a tolerant parser and flagged; unparseable output counts every gold entity as missed\. Sentences without entities are scored as correct exactly when the model predicts the empty set, with the sequence probability as the confidence of that claim\. Hallucinations — predicted surface forms absent from the source sentence — are false positives and additionally tracked as a separate rate\.

## IVConfidence Signals

#### Generative NER

The generative model is instruction\-tuned to map a sentence to a JSON object\{"entities":\[\{"text","type"\},…\]\}\\\{\\texttt\{"entities"\}:\[\\\{\\texttt\{"text"\},\\texttt\{"type"\}\\\},\\dots\]\\\}and scored under the protocol of §[III\-A](https://arxiv.org/html/2608.19558#S3.SS1)\.

#### Confidence signals

Lety1:ny\_\{1:n\}be the generated output with token log\-probabilitieslog⁡p⁡\(yi\)\\log p\(y\_\{i\}\)from the greedy decode\. We compare, per predicted entityee:*sequence probability*\(length\-normalized\)Cseq=exp⁡\(1n​∑ilog⁡p⁡\(yi\)\)C\_\{\\text\{seq\}\}=\\exp\\big\(\\tfrac\{1\}\{n\}\\sum\_\{i\}\\log p\(y\_\{i\}\)\\big\)\[[23](https://arxiv.org/html/2608.19558#bib.bib27)\];*token probability*Ctok=1n​∑ip⁡\(yi\)C\_\{\\text\{tok\}\}=\\tfrac\{1\}\{n\}\\sum\_\{i\}p\(y\_\{i\}\);*span probability*Cspan​\(e\)=exp⁡\(1\|Te\|​∑i∈Telog⁡p⁡\(yi\)\)C\_\{\\text\{span\}\}\(e\)=\\exp\\big\(\\tfrac\{1\}\{\|T\_\{e\}\|\}\\sum\_\{i\\in T\_\{e\}\}\\log p\(y\_\{i\}\)\\big\), whereTeT\_\{e\}are the tokens ofee’s surface form;*type probability*Ctype​\(e\)C\_\{\\text\{type\}\}\(e\), defined likewise over the tokens ofee’s type label; and*self\-consistency*Csc\(e\)=1K∑k=1K𝟏\[e∈E^k\]C\_\{\\text\{sc\}\}\(e\)=\\tfrac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbf\{1\}\[e\\in\\hat\{E\}\_\{k\}\], the vote share ofeeoverKKsampled decodesE^k\\hat\{E\}\_\{k\}\.CseqC\_\{\\text\{seq\}\}andCtokC\_\{\\text\{tok\}\}are sentence\-level signals shared by all entities in a sentence; the rest are entity\-level\. Self\-consistency usesK=5K\{=\}5samples at temperature0\.70\.7\[[33](https://arxiv.org/html/2608.19558#bib.bib24),[21](https://arxiv.org/html/2608.19558#bib.bib25)\]; an entity counts as a vote when a sampled decode contains it \(same normalized surface form and type\)\. A validation\-selected convex combination of span and type confidence selected pure span confidence \(test\-tier effect below±\\pm0\.02 AUROC\), so we report span confidence directly \(§[VII](https://arxiv.org/html/2608.19558#S7)\)\.

#### Encoder baseline confidence

For the BERT tagger, span confidence is the mean first\-subword maximum softmax probability within the predicted span\[[10](https://arxiv.org/html/2608.19558#bib.bib14),[2](https://arxiv.org/html/2608.19558#bib.bib28)\], optionally temperature\-scaled\[[9](https://arxiv.org/html/2608.19558#bib.bib17)\]withTTfit on in\-domain validation token\-level NLL\.

#### Selective prediction

We evaluate abstention at two granularities\.*Sentence level:*the sentence confidence is the minimum entity confidence among its predictions \(sequence probability for empty predictions\)\. Ranking sentences by this score yields a risk–coverage curve\[[5](https://arxiv.org/html/2608.19558#bib.bib12),[7](https://arxiv.org/html/2608.19558#bib.bib13)\]: risk is the fraction of answered sentences containing at least one extraction error, coverage the fraction answered; we report its area \(AURC\)\.*Entity level:*following\[[2](https://arxiv.org/html/2608.19558#bib.bib28)\], we rank all emitted entities by confidence and report*selective precision*at fixed entity coverage\. We additionally report entity\-level error\-detection AUROC and expected calibration error \(ECE, 15 equal\-width bins\)\[[24](https://arxiv.org/html/2608.19558#bib.bib18)\]against entity correctness; because binned ECE can be unstable on test tiers of roughly 300 sentences, we interpret calibration differences qualitatively and complement ECE with reliability diagrams \(§[VII](https://arxiv.org/html/2608.19558#S7)\)\. Uncertainty is quantified as mean±\\pmstd over training seeds and 95% bootstrap confidence intervals \(1,000 resamples over test instances, averaged over seeds\)\.

## VExperimental Setup

#### Models

The encoder baseline isbert\-base\-casedwith a token\-classification head \(7 BIO labels\), trained for 5 epochs \(batch 16, lr3×10−53\\times 10^\{\-5\}, linear warmup 10%\)\. The primary generative model isQwen2\.5\-0\.5B\-Instruct\[[26](https://arxiv.org/html/2608.19558#bib.bib35)\]with LoRA\[[11](https://arxiv.org/html/2608.19558#bib.bib34)\]adapters \(r=16r\{=\}16,α=32\\alpha\{=\}32, dropout 0\.05, on q/k/v/o projections\), trained for 3 epochs \(effective batch 16, lr1×10−41\\times 10^\{\-4\}\) with loss on the JSON completion only, in fp16 with fp32 LoRA parameters\. Both are trained on the harmonized FIN training set with three random seeds \(13/42/2026\)\. To test whether the confidence findings are an artifact of the smallest scale, we additionally trainQwen2\.5\-1\.5B\-Instructwith the identical recipe \(single seed\) as a scale check\. All experiments \(∼\\sim25k generations with log\-probabilities and 5\-sample consistency\) ran on a single consumer\-grade machine \(Apple M1 Pro, 16 GB\) in roughly two days; self\-consistency is the only signal with a multiplicative inference cost, while span and sequence probabilities are free by\-products of decoding\.

#### Protocol

The FIN validation split is the only data used for fitting: the encoder temperatureTTand any abstention thresholds are selected there and frozen before evaluating any test domain\[[14](https://arxiv.org/html/2608.19558#bib.bib15)\]; test sets are never used for model or hyperparameter selection\. Every pipeline stage records the command line, input hashes, seeds, and library versions in machine\-readable manifests, and all figures and tables are generated deterministically from per\-entity and per\-sentence prediction dumps\.

## VIResults

### VI\-AExtraction quality degrades under shift

Table[II](https://arxiv.org/html/2608.19558#S6.T2)gives the headline extraction results; the reported ECE is entity\-level, computed on span confidence for the generative models and on MSP for the encoder\. The encoder tagger drops from 70\.3 micro\-F1 in\-domain to 38\.3 on financial news and 24\.1 on tweets — a two\-thirds relative loss\. The 0\.5B generative model is substantially weaker in\-domain \(37\.4±\\pm1\.3 F1; encoder advantage\+\+33\.6 points, paired bootstrap 95% CI\[22\.7,44\.5\]\[22\.7,44\.5\]\), reflecting its scale and the∼\\sim1k\-sentence training budget, but degrades more gently: on the far\-shift tier the encoder’s advantage vanishes \(24\.1±\\pm2\.5 vs\. 26\.7±\\pm2\.8; difference CI\[−7\.2,\+2\.0\]\[\-7\.2,\+2\.0\], consistent with parity\), in line with reports that instruction\-tuned generative models generalize more gracefully\[[39](https://arxiv.org/html/2608.19558#bib.bib32),[4](https://arxiv.org/html/2608.19558#bib.bib33)\]\. Generation\-specific failure modes stay bounded but non\-negligible: 1\.4–5\.4% invalid JSON and 3\.6–8\.2% hallucinated entities across the three tiers\.

TABLE II:Main results \(%; mean±\\pmstd over 3 seeds; Qwen2\.5\-1\.5B is a single\-seed scale check\)\.
### VI\-BWhich confidence signals survive domain shift?

Table[III](https://arxiv.org/html/2608.19558#S6.T3)compares error\-detection AUROC and calibration of the five signals for the 0\.5B model \(mean±\\pmstd over 3 seeds\)\. Three regularities emerge\.

\(1\) The best in\-domain signal is not the best signal under shift\.In\-domain, whole\-output probabilities are the strongest error detectors \(sequence 0\.839±\\pm0\.014\), replicating the strength of simple probability baselines\[[10](https://arxiv.org/html/2608.19558#bib.bib14),[31](https://arxiv.org/html/2608.19558#bib.bib16)\]\. Under shift their AUROC falls to 0\.661/0\.626 \(news/tweets\), while span\-restricted confidence holds at 0\.690/0\.721 and self\-consistency at 0\.671/0\.682; on the far tier the sequence and span bootstrap intervals separate \(\[0\.594,0\.656\]\[0\.594,0\.656\]vs\.\[0\.695,0\.748\]\[0\.695,0\.748\]\), so the reordering is not sampling noise\.

\(2\) Discrimination and calibration dissociate\.Probability\-based signals are severely overconfident as probabilities \(span\-signal entity\-level ECE 0\.36–0\.41 across test tiers\), while self\-consistency vote share is far better calibrated out of the box \(ECE 0\.10–0\.12; Figure[1](https://arxiv.org/html/2608.19558#S6.F1)\) — its reliability curve tracks the diagonal at low and middle confidence on every tier, but on both shifted tiers the highest\-confidence bin*inverts*: under shift the model can be consistently, confidently wrong, echoing\[[25](https://arxiv.org/html/2608.19558#bib.bib19)\]\.

\(3\) The encoder’s confidence collapses under shift; the generative model’s does not\.The encoder’s MSP is an excellent in\-domain error detector \(AUROC 0\.922±\\pm0\.016\) but drops to 0\.633/0\.615 under shift — the same failure Kamath et al\.\[[14](https://arxiv.org/html/2608.19558#bib.bib15)\]report for selective QA\. The generative span signal loses far less \(0\.801→\\to0\.690/0\.721\), so under shift the*weaker*model provides the more useful uncertainty estimate\.

TABLE III:Error\-detection AUROC and ECE per confidence signal \(Qwen2\.5\-0\.5B; mean±\\pmstd over 3 seeds\), by test domain\.![Refer to caption](https://arxiv.org/html/2608.19558v1/figures/reliability.png)Fig\. 1:Reliability diagrams for self\-consistency confidence \(0\.5B, pooled seeds\)\. Near\-diagonal in\-domain; on both shifted tiers the top\-confidence bin inverts\.
### VI\-CA single\-seed scale check at 1\.5B

Table[IV](https://arxiv.org/html/2608.19558#S6.T4)repeats the signal comparison with Qwen2\.5\-1\.5B trained under the identical recipe but with a single seed, so the comparison is indicative rather than a full replication\. The larger model is substantially stronger and more shift\-robust \(F1 46\.2/52\.1/47\.7 across the three tiers; far\-tier sentence error rate 86% vs\.≈\\approx95% for the 0\.5B model\), and the study’s two central regularities recur qualitatively\. First, whole\-sequence probability again degrades most steeply as shift grows \(0\.764→\\to0\.637\), while span probability matches or exceeds it on every tier and self\-consistency becomes the most shift\-robust detector \(0\.744 on news, 0\.663 on tweets\)\. Second, the calibration dissociation persists: span ECE 0\.23–0\.36 vs\. self\-consistency ECE 0\.11–0\.17\. On the far tier the discrimination margins compress \(all signals 0\.64–0\.66\); as a single\-seed check, small differences here should not be over\-read\.

TABLE IV:Scale check: error\-detection AUROC by signal and domain for Qwen2\.5\-0\.5B \(3\-seed mean\) vs\. Qwen2\.5\-1\.5B \(single seed\)\.
### VI\-DSelective prediction: what abstention buys

Figure[2](https://arxiv.org/html/2608.19558#S6.F2)shows sentence\-level risk–coverage curves and Table[V](https://arxiv.org/html/2608.19558#S6.T5)entity\-level selective precision\. In\-domain, abstention is highly effective: answering the 40% of sentences the span signal trusts most drives sentence\-level risk from 34\.3% to below 2% \(every seed\), and the encoder reaches 98\.7% entity precision at 50% coverage \(from 66\.7% at full coverage\)\. On the near\-shift tier abstention remains useful: risk falls from 58\.8% toward∼\\sim15–25% at low coverage, and span\-ranked selective precision rises\+\+10 points at 60% coverage \(30\.7±\\pm1\.3→\\to40\.8±\\pm2\.6\)\. On the far\-shift tier, however,≈\\approx95% of sentences contain an error and no signal finds a usefully large clean subset \(AURC 0\.92–0\.95\):*abstention complements, but cannot substitute for, domain\-appropriate training data*\(§[VII](https://arxiv.org/html/2608.19558#S7)\)\.

![Refer to caption](https://arxiv.org/html/2608.19558v1/figures/risk_coverage.png)Fig\. 2:Sentence\-level risk–coverage \(Qwen2\.5\-0\.5B; line = 3\-seed mean, band = seed range; lower is better\)\. Abstention sharply reduces risk in\-domain and on news, but cannot rescue the far\-shift tier\.TABLE V:Entity\-level selective precision \(%\) at 100/80/60% coverage \(Qwen2\.5\-0\.5B; mean±\\pmstd over 3 seeds\)\.

## VIIAnalysis and Deployment Implications

#### Why is sequence probability vulnerable under shift while span probability is more robust?

The observed pattern is consistent with dilution by output scaffolding: once the JSON format is learned, most generated tokens are structural and receive probability near 1 regardless of input domain, so under shift this structural floor compresses whole\-output probability \(mean sequence probability stays above 0\.9 on tweets even for wrong predictions\)\. Span\-restricted confidence excludes the scaffolding\. Consistently, the validation\-fit convex combination of span and type confidence \(§[IV](https://arxiv.org/html/2608.19558#S4)\) selected pure span confidence in two of three seeds: type confidence contributed no discriminative value \(AUROC 0\.49–0\.54, barely above chance\)\. For these models, uncertainty lives in*where the entity is*, not*what type it is*\.

#### What do the two best signals disagree about?

Span probability and self\-consistency are complementary rather than redundant: self\-consistency provides substantially better calibration \(ECE≤\\leq0\.12 vs\.≥\\geq0\.36 without any post\-hoc fitting\), while span confidence tends to provide the stronger error ranking under far shift \(AUROC 0\.721 vs\. 0\.682, bootstrap intervals\[0\.695,0\.748\]\[0\.695,0\.748\]vs\.\[0\.653,0\.710\]\[0\.653,0\.710\], overlapping only marginally\) at1×1\\timesrather than6×6\\timesdecoding cost\. A deployment can therefore choose by constraint: span probability for a fixed\-budget ranking gate, self\-consistency when thresholds must be expressed in probability units, or a trained calibrator\[[14](https://arxiv.org/html/2608.19558#bib.bib15)\]to combine both\. We caution that setting a threshold to a target precision \(e\.g\., “auto\-accept above 80% estimated precision”\) requires validating threshold\-specific precision directly on domain\-matched data: binned ECE is estimated on≈\\approx300\-sentence tiers where it can be unstable, and Figure[1](https://arxiv.org/html/2608.19558#S6.F1)shows aggregate calibration can mask failures exactly in the high\-confidence region such thresholds rely on\.

#### Where does abstention stop working?

The general\-topic far\-shift tier exposes the boundary condition: with a≈\\approx95% sentence\-level base error rate there is no low\-risk subset to find, and the top\-confidence self\-consistency bin inverts \(Figure[1](https://arxiv.org/html/2608.19558#S6.F1)\)\. Manual inspection shows the failure mode RockNER\[[18](https://arxiv.org/html/2608.19558#bib.bib30)\]predicts: the model consistently extracts salient capitalized tokens as ORG/PER across all five samples — consistency measures stability of a bias, not correctness\. This motivates a staged deployment: \(1\) detect severe domain shift upstream, \(2\) apply confidence\-gated selective prediction within acceptable domains, \(3\) route severe\-shift inputs directly to human review\. We do not implement or evaluate such a domain detector here; distribution\-level tests for harmful covariate shift\[[8](https://arxiv.org/html/2608.19558#bib.bib38)\]are a natural candidate, and building and benchmarking this routing stage is future work\. More broadly, this reflects a growing view of reliable LLM deployment: outputs enter downstream workflows only when an explicit, auditable reliability criterion is satisfied, deferring uncertain cases to a safer fallback or human review\[[19](https://arxiv.org/html/2608.19558#bib.bib39)\]\.

#### Sensitivity to TweetNER7 filtering

Table[VI](https://arxiv.org/html/2608.19558#S7.T6)quantifies the far\-tier filtering choice; the generative comparison uses seed 42 under both protocols\. The relaxed variant leaves the picture unchanged: the generative model scores 29\.1 F1 with span AUROC 0\.715 \(vs\. 29\.7 / 0\.725 on the main filtered set\), and the encoder drops slightly to 19\.5±\\pm1\.0 F1 \(vs\. 24\.1±\\pm2\.5\) with statistically indistinguishable MSP AUROC \(0\.650±\\pm0\.047 vs\. 0\.615±\\pm0\.054\)\. If anything, the main filtered set slightly*understates*far\-shift difficulty; the headline conclusions are not artifacts of the filtering rule\.

TABLE VI:TweetNER7 filtering sensitivity, main \(drop\-sentence\) vs\. relaxed \(ignore\-entity\) protocol\. Encoder: 3\-seed mean; Qwen: seed 42\.
#### Hallucination and abstention interact favorably

Hallucinated entities \(4–8% of predictions\) skew toward the low\-confidence tail: the bottom span\-confidence tertile contains 67% of hallucinated spans on tweets and 61% in\-domain, versus 33% under a uniform spread \(39% on news\)\. A confidence gate therefore preferentially removes fabricated entities — the highest\-risk error class in a financial pipeline — exactly where fabrication is most frequent\.

## VIIIConclusion

This study evaluated financial NER as a reliability component of applied AI rather than only as an extraction benchmark\. Confidence rankings change as inputs move from filings to news and beyond the financial domain: complete\-output likelihood loses discrimination under shift, entity\-span probability is more robust, and self\-consistency is better calibrated — a pattern that holds across three seeds and recurs qualitatively in a single\-seed 1\.5B check\. Selective prediction nearly eliminates error on the highest\-confidence in\-domain subset and remains useful on financial news, but cannot recover a usable clean subset when base extraction error is extreme\. A reliable financial AI pipeline should therefore screen for domain shift upstream — distribution\-level shift tests\[[8](https://arxiv.org/html/2608.19558#bib.bib38)\]are a natural building block — apply confidence gating only within supported domains, and route the rest to human review; implementing such a routing stage is important future work\.

## IXLimitations and Ethical Considerations

Scale\.Our generative models \(0\.5B and 1\.5B\) and training set \(∼\\sim1k sentences\) are small by production standards; absolute F1 values are lower bounds for the setting\. The 1\.5B scale check rests on a single seed, so its agreement with the 0\.5B results is qualitative rather than a demonstrated replication; whether the shift\-degradation patterns attenuate at much larger scale\[[13](https://arxiv.org/html/2608.19558#bib.bib21)\]is open\.

Far\-shift construction\.TweetNER7 is deliberately general\-topic rather than financial social media, and our main protocol retains 628 of 2,807 tweets \(evaluating 300\), so the far tier identifies an extreme out\-of\-domain boundary but cannot isolate register shift or directly represent financial social\-media deployment; a matched, human\-annotated financial social\-media test set would be the definitive fix\. Our tiers also vary jointly in register, topic, annotation, time period, and entity density\.

Calibration measurement\.ECE is reported with 15 equal\-width bins on≈\\approx300\-sentence tiers; such estimates can be unstable and can conceal high\-confidence failures, which is why we pair them with reliability diagrams\. ECE confidence intervals and adaptive or classwise calibration analyses are worthwhile in future, larger\-scale studies\.

Signals and routing\.We exclude verbalized confidence \(known to be overconfident for small models\[[38](https://arxiv.org/html/2608.19558#bib.bib26)\]\) and P\(True\)\-style self\-verification\[[13](https://arxiv.org/html/2608.19558#bib.bib21)\], and semantic entropy\[[15](https://arxiv.org/html/2608.19558#bib.bib22)\]is only approximated by surface\-form vote share\. Entity\-level handoff, learned deferral\[[22](https://arxiv.org/html/2608.19558#bib.bib36)\], and the upstream domain detector our far\-tier results motivate are not implemented here and remain future work\.Ethical considerations\.All datasets are publicly released research corpora; we collect no new user data and do not attempt to identify social\-media users\. Because confidence can remain high under shift, consequential deployments require domain monitoring, human review, and institution\-specific validation; our results do not support automated profiling or financial decisions about individuals\.

## References

- \[1\]\(1970\)On optimum recognition error and reject tradeoff\.IEEE Trans\. Inf\. Theory16\(1\),pp\. 41–46\.Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]A\. Culotta and A\. McCallum\(2004\)Confidence estimation for information extraction\.InProc\. HLT\-NAACL,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px3.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px4.p1.1)\.
- \[3\]S\. Desai and G\. Durrett\(2020\)Calibration of pre\-trained transformers\.InProc\. EMNLP,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1)\.
- \[4\]Y\. Ding, J\. Li, P\. Wang, Z\. Tang, B\. Yan, and M\. Zhang\(2024\)Rethinking negative instances for generative named entity recognition\.InFindings of ACL,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§VI\-A](https://arxiv.org/html/2608.19558#S6.SS1.p1.1)\.
- \[5\]R\. El\-Yaniv and Y\. Wiener\(2010\)On the foundations of noise\-free selective classification\.J\. Mach\. Learn\. Res\.11,pp\. 1605–1641\.Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px4.p1.1)\.
- \[6\]S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal\(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630,pp\. 625–630\.Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1)\.
- \[7\]Y\. Geifman and R\. El\-Yaniv\(2017\)Selective classification for deep neural networks\.InProc\. NeurIPS,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px4.p1.1)\.
- \[8\]T\. Ginsberg, Z\. Liang, and R\. G\. Krishnan\(2023\)A learning based hypothesis test for harmful covariate shift\.InProc\. ICLR,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p2.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1),[§VII](https://arxiv.org/html/2608.19558#S7.SS0.SSS0.Px3.p1.1),[§VIII](https://arxiv.org/html/2608.19558#S8.p1.1)\.
- \[9\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InProc\. ICML,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px3.p1.1)\.
- \[10\]D\. Hendrycks and K\. Gimpel\(2017\)A baseline for detecting misclassified and out\-of\-distribution examples in neural networks\.InProc\. ICLR,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px3.p1.1),[§VI\-B](https://arxiv.org/html/2608.19558#S6.SS2.p2.1)\.
- \[11\]E\. J\. Huet al\.\(2022\)LoRA: low\-rank adaptation of large language models\.InProc\. ICLR,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§V](https://arxiv.org/html/2608.19558#S5.SS0.SSS0.Px1.p1.1)\.
- \[12\]A\. Jagannatha and H\. Yu\(2020\)Calibrating structured output predictors for natural language processing\.InProc\. ACL,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1)\.
- \[13\]S\. Kadavathet al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§IX](https://arxiv.org/html/2608.19558#S9.p1.1),[§IX](https://arxiv.org/html/2608.19558#S9.p4.1)\.
- \[14\]A\. Kamath, R\. Jia, and P\. Liang\(2020\)Selective question answering under domain shift\.InProc\. ACL,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p2.1),[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1),[§V](https://arxiv.org/html/2608.19558#S5.SS0.SSS0.Px2.p1.1),[§VI\-B](https://arxiv.org/html/2608.19558#S6.SS2.p4.1),[§VII](https://arxiv.org/html/2608.19558#S7.SS0.SSS0.Px2.p1.1)\.
- \[15\]L\. Kuhn, Y\. Gal, and S\. Farquhar\(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InProc\. ICLR,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§IX](https://arxiv.org/html/2608.19558#S9.p4.1)\.
- \[16\]M\. Liet al\.\(2023\)CoAnnotating: uncertainty\-guided work allocation between human and large language models for data annotation\.InProc\. EMNLP,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1)\.
- \[17\]Z\. Lian\(2026\)Instruction finetuning llama\-3\-8b model using lora for financial named entity recognition\.arXiv preprint arXiv:2601\.10043\.Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p1.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]B\. Y\. Lin, W\. Gao, J\. Yan, R\. Moreno, and X\. Ren\(2021\)RockNER: a simple method to create adversarial examples for evaluating the robustness of named entity recognition models\.InProc\. EMNLP,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p2.1),[§VII](https://arxiv.org/html/2608.19558#S7.SS0.SSS0.Px3.p1.1)\.
- \[19\]G\. Liuet al\.\(2026\)CARE: controlling LLM\-generated policies through auditable review of evidence in scientific experimentation\.arXiv preprint arXiv:2606\.14581\.Cited by:[§VII](https://arxiv.org/html/2608.19558#S7.SS0.SSS0.Px3.p1.1)\.
- \[20\]Y\. Lu and Y\. Huo\(2025\)Financial named entity recognition: how far can llm go?\.InProc\. FinNLP–FNP–LLMFinLegal Workshop,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p1.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]P\. Manakul, A\. Liusie, and M\. J\. F\. Gales\(2023\)SelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProc\. EMNLP,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px2.p1.1)\.
- \[22\]H\. Mozannar and D\. Sontag\(2020\)Consistent estimators for learning to defer to an expert\.InProc\. ICML,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§IX](https://arxiv.org/html/2608.19558#S9.p4.1)\.
- \[23\]K\. Murray and D\. Chiang\(2018\)Correcting length bias in neural machine translation\.InProc\. WMT,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px2.p1.1)\.
- \[24\]M\. P\. Naeini, G\. F\. Cooper, and M\. Hauskrecht\(2015\)Obtaining well calibrated probabilities using bayesian binning\.InProc\. AAAI,Cited by:[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px4.p1.1)\.
- \[25\]Y\. Ovadiaet al\.\(2019\)Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift\.InProc\. NeurIPS,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p2.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1),[§VI\-B](https://arxiv.org/html/2608.19558#S6.SS2.p3.1)\.
- \[26\]Qwen Team\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§V](https://arxiv.org/html/2608.19558#S5.SS0.SSS0.Px1.p1.1)\.
- \[27\]J\. C\. Salinas Alvarado, K\. Verspoor, and T\. Baldwin\(2015\)Domain adaption of named entity recognition to support credit risk assessment\.InProc\. ALTA Workshop,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1),[§III](https://arxiv.org/html/2608.19558#S3.SS0.SSS0.Px1.p1.1),[TABLE I](https://arxiv.org/html/2608.19558#S3.T1.1.2.2.1),[TABLE I](https://arxiv.org/html/2608.19558#S3.T1.1.3.2.1),[TABLE I](https://arxiv.org/html/2608.19558#S3.T1.1.4.2.1)\.
- \[28\]A\. Shah, A\. Gullapalli, R\. Vithani, M\. Galarnyk, and S\. Chava\(2023\)FiNER\-ord: financial named entity recognition open research dataset\.arXiv preprint arXiv:2302\.11157\.Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1),[§III](https://arxiv.org/html/2608.19558#S3.SS0.SSS0.Px2.p1.1),[TABLE I](https://arxiv.org/html/2608.19558#S3.T1.1.5.2.1)\.
- \[29\]R\. S\. Shahet al\.\(2022\)When flue meets flang: benchmarks and large pretrained language model for financial domain\.InProc\. EMNLP,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1)\.
- \[30\]A\. Ushio, L\. Neves, V\. Silva, F\. Barbieri, and J\. Camacho\-Collados\(2022\)Named entity recognition in twitter: a dataset and analysis on short\-term temporal shifts\.InProc\. AACL\-IJCNLP,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§III](https://arxiv.org/html/2608.19558#S3.SS0.SSS0.Px3.p1.1),[TABLE I](https://arxiv.org/html/2608.19558#S3.T1.1.6.2.1)\.
- \[31\]N\. Varshney, S\. Mishra, and C\. Baral\(2022\)Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings\.InFindings of ACL,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px2.p1.1),[§VI\-B](https://arxiv.org/html/2608.19558#S6.SS2.p2.1)\.
- \[32\]S\. Wanget al\.\(2025\)GPT\-ner: named entity recognition via large language models\.InFindings of NAACL,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1)\.
- \[33\]X\. Wanget al\.\(2023\)Self\-consistency improves chain of thought reasoning in language models\.InProc\. ICLR,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p3.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§IV](https://arxiv.org/html/2608.19558#S4.SS0.SSS0.Px2.p1.1)\.
- \[34\]S\. Wuet al\.\(2023\)BloombergGPT: a large language model for finance\.arXiv preprint arXiv:2303\.17564\.Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p1.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1)\.
- \[35\]Y\. Wu and M\. Luo\(2026\)Instruction finetuning deepseek\-r1\-8b model using lora and neftune for financial named entity recognition\.arXiv preprint arXiv:2606\.10392\.Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p1.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1)\.
- \[36\]Q\. Xieet al\.\(2023\)PIXIU: a comprehensive benchmark, instruction dataset and large language model for finance\.InNeurIPS Datasets and Benchmarks,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p1.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1)\.
- \[37\]Q\. Xieet al\.\(2024\)FinBen: a holistic financial benchmark for large language models\.InNeurIPS Datasets and Benchmarks,Cited by:[§I](https://arxiv.org/html/2608.19558#S1.p1.1),[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px1.p1.1)\.
- \[38\]M\. Xionget al\.\(2024\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.InProc\. ICLR,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§IX](https://arxiv.org/html/2608.19558#S9.p4.1)\.
- \[39\]W\. Zhou, S\. Zhang, Y\. Gu, M\. Chen, and H\. Poon\(2024\)UniversalNER: targeted distillation from large language models for open named entity recognition\.InProc\. ICLR,Cited by:[§II](https://arxiv.org/html/2608.19558#S2.SS0.SSS0.Px3.p1.1),[§VI\-A](https://arxiv.org/html/2608.19558#S6.SS1.p1.1)\.