[P] Pecision models that score every allowed label from the logits: Jebadiah v2.1 (27B, 9B), open weights and self-run benchmark results [P]

Reddit r/MachineLearning Models

Summary

作者发布 Jebadiah v2.1(27B 与 9B)开源模型,将决策建模为封闭集打分问题,直接从候选标签 logits 输出各选项概率,并公布在 Decision Index 0.3 上的自测成绩、污染检查与量化一致性结果。

I've been building open models that treat a decision as a closed-set scoring problem rather than text generation. The input is structured context plus a typed question with a fixed set of options. The output is a probability for each option, taken from the candidate-label logits, so there's no generation step to parse and no sampling. v2.1 updates the 27B and the 9B. On the Decision Index 0.3 public suite (my own runs with the unchanged official scorer, submitted to the board, which adds private tests before ranking anything): - 27B: 57.03 vs 55.11 for v2. Knowledge +2.34, Language +3.89, Retrieval +2.81, Arts +1.31, Tools -2.10 - 9B: 47.20 vs 44.09. Knowledge +3.67, Language +4.34, Retrieval +5.96, Arts +0.01, Tools -0.84 The regressions are concentrated: When2Call fell 6.38 (27B) and 7.44 (9B). I haven't found the cause yet and the cards don't claim statistical significance for any of these deltas. Contamination check: the benchmark families that overlap public training sources were scanned against the training data with zero hits. It's a text-match scanner, so paraphrased overlap would get through. Quantization: I measured agreement with the full weights on 260 held-out questions per build (for example 27B Q8_0 259/260, 9B MLX 4-bit 237/260) and publish the per-question records. Weights (Apache-2.0, Qwen base): https://huggingface.co/frontier-infra/jebadiah-27b-v2-1 and https://huggingface.co/frontier-infra/jebadiah-9b-v2-1 Results: https://huggingface.co/datasets/frontier-infra/jebadiah-v2-1-decision-index-results Code and evals: https://github.com/getainode/jebadiah I'd welcome critique of the evaluation setup, especially the tool-use regression.
Original Article

Similar Articles

Perplexity Decider 27B: Open weights decision model fine tune of Qwen3.8 27B

Reddit r/LocalLLaMA

Perplexity has released pplx-decider-v1-27b, an open-weights decision model fine-tuned from Qwen3-27B that scores 85.71% overall across 11 benchmarks, notably improving on tasks like RAGTruth (88.80%), FinancialPhraseBank (84.18%), and TabFact (90.60%). The model can be run locally via Hugging Face and the Perplexity API for structured choice and yes/no probability predictions.

Benchmarking small confidence scoring decision (Jev, Laya) models [P]

Reddit r/MachineLearning

A self-run evaluation of two small confidence-scoring classification models — TypeSafe AI's hosted Jev and the open-source Laya — finds that label descriptions phrased around input features, consistent time windows, and calibration (ECE) matter more than model scale for downstream thresholding performance.

jevman: AI decision models play Pac-Man

Reddit r/LocalLLaMA

作者将六个流行的决策模型(jev 1.13、Kev 4B、Clef、Clef Flash、GPT-6 Luna 和 Laya)放在《吃豆人》游戏中进行实时对决,jev 1.13 以平均 2,750 分领跑排行榜。团队已公开排行榜和开源仓库,任何人提交自己的模型即可参赛。

Jiwo - Small decision models topping decision index leaderboard

Reddit r/ArtificialInteligence

A developer released Jiwo, two Qwen3.5 fine-tuned 'decision models' (a sub-1B 0.8B model and a 4B model) that topped the Decision Index leaderboard for their size categories, achieving 45.7% improvement over the best other Qwen3.5-0.8B fine-tune.