Tag
该论文提出一套区分「记忆依赖性、实用性和内容收益」的评估框架,指出仅通过置零记忆或与无文档模型比较,无法真正证明压缩记忆中的内容带来了性能提升,并在九个系统和 TriviaQA 等基准上进行了实证验证。
This paper investigates how different evaluation setup choices affect base language model benchmarking by examining 8 design aspects across 26 2B-parameter models and larger LLMs in four languages, proposing monotonicity, non-randomness, and ranking consistency as meta-measures to guide reliable evaluation setup selection during pretraining.
Oxford researchers directly measure the inference energy efficiency of 86 open-weight LLM checkpoints across seven benchmarks and six hardware platforms, finding large energy disparities among similarly capable models and arguing benchmark scores should be reported alongside measured energy consumption.
PMO-Dock introduces a standardized benchmark of 25 docking-based molecular optimization tasks—covering hit generation, lead optimization, and a new specificity task—with strict validation/test splits and oracle budgets, evaluating methods like Saturn, GenMol, Genetic-guided GFlowNet, and ChemLlama to reveal no universally best approach.
The paper introduces 'Artificiality's First Exam', a C-test benchmark based on Levin's Kt complexity that uses LLMs as reference machines, enabling certification of ultra-hard test items beyond current frontier model capabilities, and evaluates twelve LLMs to map performance into a 'supra-intelligence' difficulty window.
Jev 1.14.0 achieves strong overall calibration on binary questions (ECE of 0.023), but a breakdown by difficulty reveals it is strongly over-confident on hard questions by about 22 percentage points, exceeding the calibration threshold for that slice.
A user with a 128GB Mac Studio M1 Ultra compares running Qwen 3.8 Flash Next versus DeepSeek V4 locally, noting that Flash Next benchmarks higher but takes dramatically longer per task, questioning the practical tradeoff between quality and speed.
作者将 Gevva0 升级为级联决策引擎(静态规则 → 吸收马尔可夫链 → 快速路径 logits → 有界 CoT),并诚实地发布了对先前"100% GTO"说法的更正:修复提示泄漏 bug 后,零样本 GTO 一致率实为约 15%-29%,但延迟仍保持约 2ms。
The author plans to try out Gemini 4 Argon today and share how good it performs.
A technical blog post comparing causal-decoder (v19) and encoder-based designs for strands-decider-2B, finding that encoders like e1b offer lower latency with similar accuracy, while encoder-decoder models scale worse in prompt length.
Gergely Orosz asks engineering leaders to share their team's PR output before and after adopting AI tools, along with per-engineer AI spend, to compile and share anonymized industry benchmarks on AI's impact on developer productivity.
这篇论文将多选题评测中的选择偏差和顺序敏感性分解为翻转率(flip rate)和位置偏斜(positional skew)两个因素,并发现先验校正主要降低位置偏斜,而顺序聚合主要降低翻转率,建议同时报告这两个因素以区分缓解效果。
A deep-dive debugging writeup showing how antcc's -O2 pessimization made a Lua benchmark 1.4x slower than the unoptimized build, tracing the 2.4x code-size blowup in luaV_execute past inlining heuristics and IR expansion into instruction selection and register allocation.
用户分享对 Muse Code (muse-wonder-full-1.0 high) 的体验测试,认为其表现优于 Astra、接近 Claude Code Fable 和 5.5 Opus,且速度更快,能处理 Codex 无法胜任的复杂编译器开发任务。
该论文提出将战略选择建模为叠加在基础博弈之上的元博弈,评估了5种博弈和10个大模型,发现多臂老虎机方法比基于嵌入的选择器更可靠,但仍未超过固定模型基线,揭示了当前方法应对动态对手的不足。
一篇系统综述,回顾了119篇关于大语言模型基准压缩的论文,提出了涵盖单基准内压缩、跨基准套件压缩及自适应评估的分类体系,旨在降低LLM评估成本并提升可靠性。
Daniel Lemire 演示了如何在无 GIL 的 Python 3.14 中,使用 fastsimdjson 的 map_many 多线程与惰性解析,在数秒内解析 7.6 GB 的 GH Archive JSON 数据,性能远超标准库 json、orjson 与 msgspec。
一项预注册研究考察了 GEPA、MIPROv2 和 TextGrad 等提示优化器在提升目标指标时是否会破坏未被评分的输出要求,发现主要回归源于输出截断导致的提取失败,增大输出预算即可消除该副作用。
perturber 是一个无状态服务与 Python 库,提供跨 11 个类别的 68 种语义保持文本扰动,每次调用仅应用一种扰动,以实现可复现、可比较的 LLM 鲁棒性评估。
The paper shows that random data splits in perovskite photovoltaics ML benchmarks systematically leak information due to massive internal duplication in the Perovskite Database, and proposes a provenance-grouped evaluation protocol. Benchmarking thirteen models reveals tree ensembles fail on truly unseen institutions while composition-aware neural architectures excel, establishing new state-of-the-art results.