From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders

Hugging Face Daily Papers Papers
biomedical-nlp sentence-encoders typed-decisions calibration text-classification reward-learning

Summary

The paper introduces SBERT2S1, which converts Sentence-Transformers retrieval encoders into calibrated typed decision models, along with the BIODECIDE benchmark and MEDLINE-S1 dataset (243k decisions from NLM indexing), showing that retrieval training helps prior-fused residual heads but often hurts cross-heads, and that RLCD reward training underperforms cross-entropy due to biased reward normalization.

Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask whether biomedical sentence encoders trained for retrieval are good starting points for such models. We present SBERT2S1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models, together with BIODECIDE, a biomedical typed-decision suite, and MEDLINE-S1, 243k training decisions derived from NLM indexing. Across six parent-retriever pairs, retrieval training improves zero-shot matching of content-bearing options. After fine-tuning, its effect depends on the head: across five pairs and three training-set sizes, retrieval training significantly helps PFR, which keeps the retrieval prior, in 10 of 15 comparisons, but helps C in one and hurts it in five. A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective, and that the released RLCD recipe of open System One models trails cross-entropy by 2.5-3.0 points. The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6-15-fold; an unbiased leave-one-out estimator recovers most of the gap. After temperature scaling, no objective is clearly better calibrated than cross-entropy. We release the code, the MEDLINE-S1 labels and a model.
Original Article
View Cached Full Text

Cached at: 10/05/26, 08:46 AM

Paper page - From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders

Source: https://huggingface.co/papers/2610.02486

Abstract

Typeddecisionmodelsanswerschema-constrainedquestionsaboutatextinoneforwardpassandreturnprobabilitiesmeanttobethresholded.Weaskwhetherbiomedicalsentenceencoderstrainedforretrievalaregoodstartingpointsforsuchmodels.WepresentSBERT2S1,whichconvertsSentence-Transformersencodersintobi-encoder,cross-head(C)andprior-fusedresidual(PFR)decisionmodels,togetherwithBIODECIDE,abiomedicaltyped-decisionsuite,andMEDLINE-S1,243ktrainingdecisionsderivedfromNLMindexing.Acrosssixparent-retrieverpairs,retrievaltrainingimproveszero-shotmatchingofcontent-bearingoptions.Afterfine-tuning,itseffectdependsonthehead:acrossfivepairsandthreetraining-setsizes,retrievaltrainingsignificantlyhelpsPFR,whichkeepstheretrievalprior,in10of15comparisons,buthelpsCinoneandhurtsitinfive.AmatchedgridoftwoheadsandfivetrainingobjectivesshowsthatCoutperformsPFRundereveryobjective,andthatthereleasedRLCDrecipeofopenSystemOnemodelstrailscross-entropyby2.5-3.0points.Thedeficitstemsmainlyfromitsrewardnormalisation,whichinflatesthenoisyscore-functionterm3.6-15-fold;anunbiasedleave-one-outestimatorrecoversmostofthegap.Aftertemperaturescaling,noobjectiveisclearlybettercalibratedthancross-entropy.Wereleasethecode,theMEDLINE-S1labelsandamodel.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2610\.02486

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### pritamdeka/S1-PubMedBERT Text Classification• 0.1B• Updatedabout 2 hours ago

Datasets citing this paper1

#### pritamdeka/MEDLINE-S1 Viewer• Updatedabout 2 hours ago • 47.4k • 38

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.02486 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SanSi: A Looped Typed Decision Model for System 1.5 Thinking

Hugging Face Daily Papers

SanSi 将预训练的循环语言模型转化为类型化决策模型，通过在单次读出前递归复用相同层数层（即 System 1.5 思考）提升决策准确率，在 10,027 个测试决策上达到 72.0% 准确率，并可作为 RL 策略优化的判别器提升生成器 F1。

A Causal Language Modeling Detour Improves Encoder Continued Pretraining

Hugging Face Daily Papers

This paper demonstrates that switching from Masked Language Modeling to Causal Language Modeling during encoder adaptation improves downstream performance on biomedical texts. The authors release ModernBERT-bio and ModernCamemBERT-bio as state-of-the-art biomedical encoders.