benchmarking

Tag

Cards List
#benchmarking

Evaluating Compressed Memory: Dependence, Utility, and Content Benefit

arXiv cs.CL ↗ · 2026-10-15 Cached

该论文提出一套区分「记忆依赖性、实用性和内容收益」的评估框架,指出仅通过置零记忆或与无文档模型比较,无法真正证明压缩记忆中的内容带来了性能提升,并在九个系统和 TriviaQA 等基准上进行了实证验证。

0 favorites 0 likes
#benchmarking

How to Evaluate When You Don't Know How to Evaluate? Choosing Reliable Setups for Base Language Model Benchmarking

arXiv cs.CL ↗ · 2026-10-15 Cached

This paper investigates how different evaluation setup choices affect base language model benchmarking by examining 8 design aspects across 26 2B-parameter models and larger LLMs in four languages, proposing monotonicity, non-randomness, and ranking consistency as meta-measures to guide reliable evaluation setup selection during pretraining.

0 favorites 0 likes
#benchmarking

Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware

arXiv cs.AI ↗ · 2026-10-15 Cached

Oxford researchers directly measure the inference energy efficiency of 86 open-weight LLM checkpoints across seven benchmarks and six hardware platforms, finding large energy disparities among similarly capable models and arguing benchmark scores should be reported alongside measured energy consumption.

0 favorites 0 likes
#benchmarking

PMO-Dock: Benchmarking Docking, Specificity, and Generalization in Molecular Optimization

arXiv cs.LG ↗ · 2026-10-15 Cached

PMO-Dock introduces a standardized benchmark of 25 docking-based molecular optimization tasks—covering hit generation, lead optimization, and a new specificity task—with strict validation/test splits and oracle budgets, evaluating methods like Saturn, GenMol, Genetic-guided GFlowNet, and ChemLlama to reveal no universally best approach.

0 favorites 0 likes
#benchmarking

Artificiality's First Exam: An LLM-Referenced C-test for Supraintelligence

arXiv cs.AI ↗ · 2026-10-15 Cached

The paper introduces 'Artificiality's First Exam', a C-test benchmark based on Levin's Kt complexity that uses LLMs as reference machines, enabling certification of ultra-hard test items beyond current frontier model capabilities, and evaluates twelve LLMs to map performance into a 'supra-intelligence' difficulty window.

0 favorites 0 likes
#benchmarking

[crosspost] Jev 1.14.0 is bluffing on the hard question

Reddit r/AI_Agents ↗ · 2026-10-15

Jev 1.14.0 achieves strong overall calibration on binary questions (ECE of 0.023), but a breakdown by difficulty reveals it is strongly over-confident on hard questions by about 22 percentage points, exceeding the calibration threshold for that slice.

0 favorites 0 likes
#benchmarking

Qwen 3.8 Flash Next, DeepSeekV4 Flash and Time

Reddit r/LocalLLaMA ↗ · 2026-10-14

A user with a 128GB Mac Studio M1 Ultra compares running Qwen 3.8 Flash Next versus DeepSeek V4 locally, noting that Flash Next benchmarks higher but takes dramatically longer per task, questioning the practical tradeoff between quality and speed.

0 favorites 0 likes
#benchmarking

Using Absorbing Markov Chains + EmbeddingGemma 2 as a Decision Engine: ~2ms inference with 100% GTO Parity on Vegas Blackjack

Reddit r/LocalLLaMA ↗ · 2026-10-14

作者将 Gevva0 升级为级联决策引擎(静态规则 → 吸收马尔可夫链 → 快速路径 logits → 有界 CoT),并诚实地发布了对先前"100% GTO"说法的更正:修复提示泄漏 bug 后,零样本 GTO 一致率实为约 15%-29%,但延迟仍保持约 2ms。

0 favorites 0 likes
#benchmarking

@_arohan_: I am looking to try Gemini 4 Argon today and will let you all know how good it is.

X AI KOLs Following ↗ · 2026-10-14

The author plans to try out Gemini 4 Argon today and share how good it performs.

0 favorites 0 likes
#benchmarking

Strands Decider: Why Not an Encoder?

Hacker News Top ↗ · 2026-10-14 Cached

A technical blog post comparing causal-decoder (v19) and encoder-based designs for strands-decider-2B, finding that encoders like e1b offer lower latency with similar accuracy, while encoder-decoder models scale worse in prompt length.

0 favorites 0 likes
#benchmarking

@GergelyOrosz: What is the PR output of your eng team, “before” and “after” adopting AI tools? And how much is your AI spend, per engi…

X AI KOLs Timeline ↗ · 2026-10-14 Cached

Gergely Orosz asks engineering leaders to share their team's PR output before and after adopting AI tools, along with per-engineer AI spend, to compile and share anonymized industry benchmarks on AI's impact on developer productivity.

0 favorites 0 likes
#benchmarking

One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

Hugging Face Daily Papers ↗ · 2026-10-14 Cached

这篇论文将多选题评测中的选择偏差和顺序敏感性分解为翻转率(flip rate)和位置偏斜(positional skew)两个因素,并发现先验校正主要降低位置偏斜,而顺序聚合主要降低翻转率,建议同时报告这两个因素以区分缓解效果。

0 favorites 0 likes
#benchmarking

Chasing a -O2 pessimization in antcc

Lobsters Hottest ↗ · 2026-10-13 Cached

A deep-dive debugging writeup showing how antcc's -O2 pessimization made a Lua benchmark 1.4x slower than the unoptimized build, tracing the 2.4x code-size blowup in luaV_execute past inlining heuristics and IR expansion into instruction selection and register allocation.

0 favorites 0 likes
#benchmarking

@QuixiAI: Muse Code (muse-wonder-full-1.0 high) is doing really good. By my vibe testing, It's better than Astra. Not quite as go…

X AI KOLs Timeline ↗ · 2026-10-13 Cached

用户分享对 Muse Code (muse-wonder-full-1.0 high) 的体验测试,认为其表现优于 Astra、接近 Claude Code Fable 和 5.5 Opus,且速度更快,能处理 Codex 无法胜任的复杂编译器开发任务。

0 favorites 0 likes
#benchmarking

Choosing LLMs for Strategic Agent Interactions

Hugging Face Daily Papers ↗ · 2026-10-13 Cached

该论文提出将战略选择建模为叠加在基础博弈之上的元博弈,评估了5种博弈和10个大模型,发现多臂老虎机方法比基于嵌入的选择器更可靠,但仍未超过固定模型基线,揭示了当前方法应对动态对手的不足。

0 favorites 0 likes
#benchmarking

Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

Hugging Face Daily Papers ↗ · 2026-10-13 Cached

一篇系统综述,回顾了119篇关于大语言模型基准压缩的论文,提出了涵盖单基准内压缩、跨基准套件压缩及自适应评估的分类体系,旨在降低LLM评估成本并提升可靠性。

0 favorites 0 likes
#benchmarking

Parsing JSON at tens of gigabytes per second… in Python

Lobsters Hottest ↗ · 2026-10-12 Cached

Daniel Lemire 演示了如何在无 GIL 的 Python 3.14 中,使用 fastsimdjson 的 map_many 多线程与惰性解析,在数秒内解析 7.6 GB 的 GH Archive JSON 数据,性能远超标准库 json、orjson 与 msgspec。

0 favorites 0 likes
#benchmarking

Do Prompt Optimizers Break Requirements They Are Not Scored On? A Preregistered Study

arXiv cs.CL ↗ · 2026-10-12 Cached

一项预注册研究考察了 GEPA、MIPROv2 和 TextGrad 等提示优化器在提升目标指标时是否会破坏未被评分的输出要求,发现主要回归源于输出截断导致的提取失败,增大输出预算即可消除该副作用。

0 favorites 0 likes
#benchmarking

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

arXiv cs.CL ↗ · 2026-10-12 Cached

perturber 是一个无状态服务与 Python 库,提供跨 11 个类别的 68 种语义保持文本扰动,每次调用仅应用一种扰动,以实现可复现、可比较的 LLM 鲁棒性评估。

0 favorites 0 likes
#benchmarking

Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics

arXiv cs.LG ↗ · 2026-10-12

The paper shows that random data splits in perovskite photovoltaics ML benchmarks systematically leak information due to massive internal duplication in the Perovskite Database, and proposes a provenance-grouped evaluation protocol. Benchmarking thirteen models reveals tree ensembles fail on truly unseen institutions while composition-aware neural architectures excel, establishing new state-of-the-art results.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback