benchmarking

Tag

Cards List
#benchmarking

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

arXiv cs.CL ↗ · 2026-10-12 Cached

perturber 是一个无状态服务与 Python 库,提供跨 11 个类别的 68 种语义保持文本扰动,每次调用仅应用一种扰动,以实现可复现、可比较的 LLM 鲁棒性评估。

0 favorites 0 likes
#benchmarking

Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics

arXiv cs.LG ↗ · 2026-10-12

The paper shows that random data splits in perovskite photovoltaics ML benchmarks systematically leak information due to massive internal duplication in the Perovskite Database, and proposes a provenance-grouped evaluation protocol. Benchmarking thirteen models reveals tree ensembles fail on truly unseen institutions while composition-aware neural architectures excel, establishing new state-of-the-art results.

0 favorites 0 likes
#benchmarking

When Faster Decisions Do Not Make Better Agents: An Empirical Study of Jev in Agent Workflows

arXiv cs.AI ↗ · 2026-10-12 Cached

An empirical study from Nanjing University investigating whether integrating faster, lower-cost decision models (Jev by TypeSafe AI) into LLM agent workflows actually improves task-level efficiency and reliability, evaluating five agents across ALFWorld and AndroidWorld with 4,500 task executions.

0 favorites 0 likes
#benchmarking

@MaximeRivest: Lol, Lilly is an absolute savage when trying to benchmark computer use for AI. Link to her benchmark below.

X AI KOLs Following ↗ · 2026-10-11 Cached

A tweet highlights a benchmark showing a big gap in computer-use ability between large general models and specialized models, which mostly fail at pointing tasks due to lacking the necessary vocabulary.

0 favorites 0 likes
#benchmarking

Engineer / developer observations of Gemma4-31B, Qwen3.8-27B, and 6.1-Sol for software engineering work

Reddit r/LocalLLaMA ↗ · 2026-10-10

A 20-year software engineer compares Gemma4-31B and Qwen3.8-27B (Q4, via Codex CLI and OpenCode CLI) on codebase analysis and greenfield C#/WPF app authoring, finding both workable locally but clearly behind GPT 6.1-Sol, which delivered a one-shot A+ result.

0 favorites 0 likes
#benchmarking

Jev vs LLM judges on my agent's evals: 180x cheaper, 2.6 points less accurate

Reddit r/AI_Agents ↗ · 2026-10-10

作者对比了 Jev 与三种 LLM 评审(Sonnet、Haiku、Gemini Flash Lite)在智能体评测中的表现:Jev 成本低 180 倍(每 1k 条裁决 $0.07)但准确率低 2.6 个百分点(90.6% vs 93.2%),建议采用 Jev 先筛 + Sonnet 处理约 16% 不确定样本的级联方案,以 1/6 成本达到 Sonnet 级准确率。

0 favorites 0 likes
#benchmarking

What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

Hugging Face Daily Papers ↗ · 2026-10-10 Cached

该论文在统一可控的基准框架下系统评估了多模态嵌入学习中的各类对比训练策略,发现大多数策略相比标准对齐基线仅有边际收益甚至带来负面影响,并提出结合难负样本挖掘、假负样本缓解与难批次构建的协同训练方案,在MMEB上以Qwen2-VL-2B平均提升6.0%。

0 favorites 0 likes
#benchmarking

Microsoft-Decision-1, our model for fast decision-making

Hacker News Top ↗ · 2026-10-09 Cached

微软发布 Microsoft-Decision-1,一个专为快速决策打分设计的模型,可用于路由、分类、优先级排序与工作流控制,基于 Qwen3.5-9B 后训练,在 36 项基准测试中取得最高准确率,延迟比 GPT-6 Sol 快约 35 倍。

0 favorites 0 likes
#benchmarking

Evaluating Rubric Generation with Interventional Transfer

arXiv cs.LG ↗ · 2026-10-09 Cached

本文提出了一种名为 Interventional Transfer (IT) 的方法,用于评估 LLM 生成的评测 rubric 的质量,发现生成 rubric 与专家 rubric 之间存在不对称的迁移现象,对使用 LLM 生成 rubric 进行性能监控和模型优化具有重要启示。

0 favorites 0 likes
#benchmarking

Scaling and benchmarking a critical message bus using a new indexing strategy

Hacker News Top ↗ · 2026-10-08 Cached

Jane Street 分享了其内部消息框架 Aria 的扩展实践:通过为 topic partition 引入索引与树拆分策略优化 tip recovery 过滤流程,在生产负载上实现 30% 的 CPU 使用率下降。

0 favorites 0 likes
#benchmarking

@charliermarsh: hyperfine 2.0 is out now! It includes support for alternative performance metrics like peak memory usage, CPU cycles, a…

X AI KOLs Timeline ↗ · 2026-10-08 Cached

hyperfine 2.0 has been released, adding support for alternative performance metrics such as peak memory usage and CPU cycles to the command-line benchmarking tool.

0 favorites 0 likes
#benchmarking

Towards Financial World Modeling

arXiv cs.LG ↗ · 2026-10-08 Cached

The paper introduces Market-1T, a ~1-trillion observation U.S. equity dataset at 1 Hz resolution, a rigorous evaluation protocol, and a large-scale study of 18 encoder-training strategies for financial representation learning across 2008–2025 market regimes, establishing a foundation for financial world models like DINO-WM, V-JEPA 2, and LeWM.

0 favorites 0 likes
#benchmarking

Evaluating Escalation Signals for LLM Routing: Targets, Controls, and Five Ways to Fool Yourself

arXiv cs.AI ↗ · 2026-10-08 Cached

Cisco researchers evaluate semantic entropy as an escalation signal for routing queries from small to large LLMs across three benchmarks, showing it works on GSM8K but can be misleadingly matched by naive difficulty heuristics. The paper's main contribution is a checklist of evaluation pitfalls—question-only baselines, ambiguous success metrics, benchmark saturation, and hidden sampling costs—for assessing routing signals.

0 favorites 0 likes
#benchmarking

MemMux: Runtime Verification and Honest Resource Attribution for Fleets of Parallel Coding Agents

arXiv cs.AI ↗ · 2026-10-08 Cached

MemMux is a local runtime that treats resource governance for fleets of parallel coding agents as runtime verification, providing per-agent memory attribution, complete reclamation, escaped-process visibility, and bounded footprint. Benchmarks show it keeps a 7.5 GiB budget under zero swap and detects all injected escaped processes, with an open-sourced engine and benchmark.

0 favorites 0 likes
#benchmarking

Small Language Models for Smart Data Model Classification at the Edge: A Cost-Aware Hybrid Approach

arXiv cs.AI ↗ · 2026-10-08

该研究评测了轻量级开源语言模型在资源受限的边缘环境中对智能数据模型(SDM)进行分类的性能,涵盖通用、推理专用和代码专用架构,并与TF-IDF及轻量句子编码器基线进行对比,为边缘部署的模型选择和任务设计提供参考。

0 favorites 0 likes
#benchmarking

Offline AI Modules: Voice-First Offline Architecture, Hardware Reference Stack, Quantization and Benchmarking

arXiv cs.AI ↗ · 2026-10-08 Cached

This paper presents the first end-to-end benchmark of a voice-first offline AI stack for African language communities, evaluating three instruction-tuned 2–5B models across four quantization formats on NVIDIA Jetson Orin NX and Raspberry Pi 5 hardware, finding that Q4_K_M offers the best size-to-quality trade-off.

0 favorites 0 likes
#benchmarking

Comparing Qwen3.8-27B fine-tunes and baselining vs. frontier

Reddit r/LocalLLaMA ↗ · 2026-10-08

A benchmarking write-up comparing multiple Qwen3.8-27B fine-tunes against frontier models like Opus 5.5 and Astra on a 469-question domain-specific eval, finding that a Q4_K_M quantization (Signal-3.8-27B-Terse-Coder) outperformed larger fine-tunes locally while remaining ~28x slower than frontier. The author also released the decoupled eval tool 'tuieval' on GitHub for reproducible, hardware-grounded comparisons.

0 favorites 0 likes
#benchmarking

Clinician use of language models diverges from how the models are evaluated

Hugging Face Daily Papers ↗ · 2026-10-08 Cached

This paper analyzes 127,833 real clinical queries sent by 6,342 healthcare providers to an institutional LLM assistant, finds that actual use (dominated by documentation and knowledge retrieval) diverges sharply from the task mix covered by 58 public clinical benchmarks, arguing that benchmark scores poorly predict real-world clinical AI performance.

0 favorites 0 likes
#benchmarking

A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

Hugging Face Daily Papers ↗ · 2026-10-08 Cached

提出 Agentic BBO-Bench 跨领域基准,在统一预算协议下评估 LLM 智能体做黑盒优化的能力;实验显示 agentic BBO 在五个领域均优于直接 LM 方法,并在四个领域超越最佳数值优化器,GPT-6Astra 与 DeepSeek-V4.1-Flash 位于性能-成本 Pareto 前沿。

0 favorites 0 likes
#benchmarking

Introducing OpenDocRouter: every document model under one API (5 minute read)

TLDR AI ↗ · 2026-10-08 Cached

LlamaIndex 发布 OpenDocRouter,一个统一的文档转 Markdown API 平台,聚合前沿与开源 OCR 模型,并通过 ParseBench 对质量与成本进行基准测试。

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback