Tag
perturber 是一个无状态服务与 Python 库,提供跨 11 个类别的 68 种语义保持文本扰动,每次调用仅应用一种扰动,以实现可复现、可比较的 LLM 鲁棒性评估。
The paper shows that random data splits in perovskite photovoltaics ML benchmarks systematically leak information due to massive internal duplication in the Perovskite Database, and proposes a provenance-grouped evaluation protocol. Benchmarking thirteen models reveals tree ensembles fail on truly unseen institutions while composition-aware neural architectures excel, establishing new state-of-the-art results.
An empirical study from Nanjing University investigating whether integrating faster, lower-cost decision models (Jev by TypeSafe AI) into LLM agent workflows actually improves task-level efficiency and reliability, evaluating five agents across ALFWorld and AndroidWorld with 4,500 task executions.
A tweet highlights a benchmark showing a big gap in computer-use ability between large general models and specialized models, which mostly fail at pointing tasks due to lacking the necessary vocabulary.
A 20-year software engineer compares Gemma4-31B and Qwen3.8-27B (Q4, via Codex CLI and OpenCode CLI) on codebase analysis and greenfield C#/WPF app authoring, finding both workable locally but clearly behind GPT 6.1-Sol, which delivered a one-shot A+ result.
作者对比了 Jev 与三种 LLM 评审(Sonnet、Haiku、Gemini Flash Lite)在智能体评测中的表现:Jev 成本低 180 倍(每 1k 条裁决 $0.07)但准确率低 2.6 个百分点(90.6% vs 93.2%),建议采用 Jev 先筛 + Sonnet 处理约 16% 不确定样本的级联方案,以 1/6 成本达到 Sonnet 级准确率。
该论文在统一可控的基准框架下系统评估了多模态嵌入学习中的各类对比训练策略,发现大多数策略相比标准对齐基线仅有边际收益甚至带来负面影响,并提出结合难负样本挖掘、假负样本缓解与难批次构建的协同训练方案,在MMEB上以Qwen2-VL-2B平均提升6.0%。
微软发布 Microsoft-Decision-1,一个专为快速决策打分设计的模型,可用于路由、分类、优先级排序与工作流控制,基于 Qwen3.5-9B 后训练,在 36 项基准测试中取得最高准确率,延迟比 GPT-6 Sol 快约 35 倍。
本文提出了一种名为 Interventional Transfer (IT) 的方法,用于评估 LLM 生成的评测 rubric 的质量,发现生成 rubric 与专家 rubric 之间存在不对称的迁移现象,对使用 LLM 生成 rubric 进行性能监控和模型优化具有重要启示。
Jane Street 分享了其内部消息框架 Aria 的扩展实践:通过为 topic partition 引入索引与树拆分策略优化 tip recovery 过滤流程,在生产负载上实现 30% 的 CPU 使用率下降。
hyperfine 2.0 has been released, adding support for alternative performance metrics such as peak memory usage and CPU cycles to the command-line benchmarking tool.
The paper introduces Market-1T, a ~1-trillion observation U.S. equity dataset at 1 Hz resolution, a rigorous evaluation protocol, and a large-scale study of 18 encoder-training strategies for financial representation learning across 2008–2025 market regimes, establishing a foundation for financial world models like DINO-WM, V-JEPA 2, and LeWM.
Cisco researchers evaluate semantic entropy as an escalation signal for routing queries from small to large LLMs across three benchmarks, showing it works on GSM8K but can be misleadingly matched by naive difficulty heuristics. The paper's main contribution is a checklist of evaluation pitfalls—question-only baselines, ambiguous success metrics, benchmark saturation, and hidden sampling costs—for assessing routing signals.
MemMux is a local runtime that treats resource governance for fleets of parallel coding agents as runtime verification, providing per-agent memory attribution, complete reclamation, escaped-process visibility, and bounded footprint. Benchmarks show it keeps a 7.5 GiB budget under zero swap and detects all injected escaped processes, with an open-sourced engine and benchmark.
该研究评测了轻量级开源语言模型在资源受限的边缘环境中对智能数据模型(SDM)进行分类的性能,涵盖通用、推理专用和代码专用架构,并与TF-IDF及轻量句子编码器基线进行对比,为边缘部署的模型选择和任务设计提供参考。
This paper presents the first end-to-end benchmark of a voice-first offline AI stack for African language communities, evaluating three instruction-tuned 2–5B models across four quantization formats on NVIDIA Jetson Orin NX and Raspberry Pi 5 hardware, finding that Q4_K_M offers the best size-to-quality trade-off.
A benchmarking write-up comparing multiple Qwen3.8-27B fine-tunes against frontier models like Opus 5.5 and Astra on a 469-question domain-specific eval, finding that a Q4_K_M quantization (Signal-3.8-27B-Terse-Coder) outperformed larger fine-tunes locally while remaining ~28x slower than frontier. The author also released the decoupled eval tool 'tuieval' on GitHub for reproducible, hardware-grounded comparisons.
This paper analyzes 127,833 real clinical queries sent by 6,342 healthcare providers to an institutional LLM assistant, finds that actual use (dominated by documentation and knowledge retrieval) diverges sharply from the task mix covered by 58 public clinical benchmarks, arguing that benchmark scores poorly predict real-world clinical AI performance.
提出 Agentic BBO-Bench 跨领域基准,在统一预算协议下评估 LLM 智能体做黑盒优化的能力;实验显示 agentic BBO 在五个领域均优于直接 LM 方法,并在四个领域超越最佳数值优化器,GPT-6Astra 与 DeepSeek-V4.1-Flash 位于性能-成本 Pareto 前沿。
LlamaIndex 发布 OpenDocRouter,一个统一的文档转 Markdown API 平台,聚合前沿与开源 OCR 模型,并通过 ParseBench 对质量与成本进行基准测试。