benchmarking

The paper presents TSDS-Toolbox, a unified framework for measuring time-series dataset similarity, enabling systematic comparison, extensibility, and consistent evaluation of similarity methods.

This paper proposes a blockchain-based commit-reveal protocol to decentralize trust in LLM benchmarking, using anonymous multi-model verifiers to address identity-aware bias and manipulation in benchmark claims.

Tweet reports that Ling 3.0 Flash on AMD Strix Halo is significantly faster than Qwen-122b using ROCm-optimized formats, but notes tool calls are broken in certain harnesses.

A user shares local testing of Muse Glimmer (Q4 quant via Unsloth) on llama.cpp with OpenCode, noting it performs below Qwen3.6 27B but had reliable tool calls.

A PSA about Meta's Muse Glimmer: setting reasoning strength via system prompt is partially overridden by the chat template's high default, and the absent default is high, not off. Using chat_template_kwargs.reasoning_strength provides more control.

Dan Luu critiques claims that dynamic languages are more token-efficient for LLMs, pointing out flaws in existing evals and emphasizing the need for better benchmarking methods.

Introduces an operating-point-aware evaluation framework for adaptive data cleaning that corrects for removal-budget confounding. Experiments on CIFAR-10 and ImageNet-100 show that apparent performance gains vanish when operating points are matched, suggesting that many improvements reflect smaller removal budgets rather than genuine corruption discrimination.

Daniel Lemire benchmarks profile-guided optimization (PGO) in Go, showing modest speedups of 2-5% when parsing JSON, with the best gains when the training profile matches the workload.

Introduces A2E, an end-to-end evaluation engine for agent harnesses, using a standardized task protocol and execution traces to assess capabilities like efficiency, tool use, planning, and error recovery.

Matt Shumer shares a technique for adapting skills to Claude Opus 5: a loop that updates skills, benchmarks them against Opus 4.8 with old skills, and iterates until Opus 5 wins.

OrchestraBench is a new benchmark that evaluates multi-agent orchestration frameworks on failure modes, recovery, and decomposition quality, using failure-injection and cascade-radius metrics to diagnose where and why pipelines fail.

A user on a dual RTX 3090 setup discovers that llama.cpp's --split-mode tensor runs prompt processing on CPU (~400 t/s), while switching to --split-mode layer unlocks GPU prompt processing over 1600 t/s, with only a modest drop in token generation speed.

A developer benchmarks 32 local models on fact extraction for agent memory, showing that F1 hides a critical failure mode: models with similar scores differ greatly in how often they invent facts on inputs that should output nothing. The article argues agent memory evaluation must include empty-output and retraction cases.

A practical tutorial on measuring transformer inference performance, covering metrics like latency, TTFT, throughput, memory usage, and benchmarking techniques for LLMs.

This article investigates a LuaJIT pitfall where a Not Yet Implemented (NYI) operation like unpack silently causes trace blacklisting, leading to a 20x benchmark slowdown, and offers ways to guard against it in CI.

A side-by-side coding experiment comparing GPT-5.6 Luna and DeepSeek V4 Flash shows that DeepSeek's apparent 5x price advantage shrinks when retries are included. The article argues for more comprehensive benchmarks reporting cost per attempt and cost per verified success.

User asks for performance numbers on Deepseek V4 Flash running via Colibri, focusing on high VRAM setups, long context prefill, and token generation speed for agentic workloads.

This paper tests whether commonsense benchmark scores predict real-world downstream task performance by evaluating 23 LLMs across four benchmarks and their reworked variants, finding that revisions preserve rankings but only offer task-dependent predictive validity.

A user shares experiments running a 4-bit quantized DeepSeek v4 Flash on a 32GB M5 MacBook Air, achieving roughly 50 tokens/s prefill and 1 token/s decode using streamed experts and other tricks.

A detailed benchmark of DeepSeek V4 Flash on a dual GH200 workstation, comparing SGLang and vLLM at 1M context with DSpark speculative decoding, finding SGLang faster at ~317 vs ~276 tok/s.