benchmarking

This paper introduces Cross-Contextual Consistency (C3), a behavioral property for measuring LLM credibility by checking whether answers remain stable under topic-aligned, content-neutral perturbations. Across 26 models and six benchmarks, they find that higher consistency correlates with correctness, offering a complementary evaluation axis.

A hands-on benchmark of Nvidia CMP170HX mining cards repurposed as 64GB VRAM AI inference accelerators, showing they can run large local LLMs like DeepSeek V4-Flash and gpt-oss-120B at useful speeds, with caveats around Ampere-class throughput and PCIe Gen2 x4 connectivity.

The author quantizes DeepSeek V4 0731, fixing FP8 downconversion issues that skew baselines, and benchmarks 38 quant files on 8× RTX 5090 to show GPU-dependent results and file-size-based comparisons.

The paper presents TSDS-Toolbox, a unified framework for measuring time-series dataset similarity, enabling systematic comparison, extensibility, and consistent evaluation of similarity methods.

This paper proposes a blockchain-based commit-reveal protocol to decentralize trust in LLM benchmarking, using anonymous multi-model verifiers to address identity-aware bias and manipulation in benchmark claims.

Tweet reports that Ling 3.0 Flash on AMD Strix Halo is significantly faster than Qwen-122b using ROCm-optimized formats, but notes tool calls are broken in certain harnesses.

A user shares local testing of Muse Glimmer (Q4 quant via Unsloth) on llama.cpp with OpenCode, noting it performs below Qwen3.6 27B but had reliable tool calls.

A PSA about Meta's Muse Glimmer: setting reasoning strength via system prompt is partially overridden by the chat template's high default, and the absent default is high, not off. Using chat_template_kwargs.reasoning_strength provides more control.

Dan Luu critiques claims that dynamic languages are more token-efficient for LLMs, pointing out flaws in existing evals and emphasizing the need for better benchmarking methods.

Introduces an operating-point-aware evaluation framework for adaptive data cleaning that corrects for removal-budget confounding. Experiments on CIFAR-10 and ImageNet-100 show that apparent performance gains vanish when operating points are matched, suggesting that many improvements reflect smaller removal budgets rather than genuine corruption discrimination.

Daniel Lemire benchmarks profile-guided optimization (PGO) in Go, showing modest speedups of 2-5% when parsing JSON, with the best gains when the training profile matches the workload.

Introduces A2E, an end-to-end evaluation engine for agent harnesses, using a standardized task protocol and execution traces to assess capabilities like efficiency, tool use, planning, and error recovery.

Matt Shumer shares a technique for adapting skills to Claude Opus 5: a loop that updates skills, benchmarks them against Opus 4.8 with old skills, and iterates until Opus 5 wins.

OrchestraBench is a new benchmark that evaluates multi-agent orchestration frameworks on failure modes, recovery, and decomposition quality, using failure-injection and cascade-radius metrics to diagnose where and why pipelines fail.

A user on a dual RTX 3090 setup discovers that llama.cpp's --split-mode tensor runs prompt processing on CPU (~400 t/s), while switching to --split-mode layer unlocks GPU prompt processing over 1600 t/s, with only a modest drop in token generation speed.

A developer benchmarks 32 local models on fact extraction for agent memory, showing that F1 hides a critical failure mode: models with similar scores differ greatly in how often they invent facts on inputs that should output nothing. The article argues agent memory evaluation must include empty-output and retraction cases.

A practical tutorial on measuring transformer inference performance, covering metrics like latency, TTFT, throughput, memory usage, and benchmarking techniques for LLMs.

This article investigates a LuaJIT pitfall where a Not Yet Implemented (NYI) operation like unpack silently causes trace blacklisting, leading to a 20x benchmark slowdown, and offers ways to guard against it in CI.

A side-by-side coding experiment comparing GPT-5.6 Luna and DeepSeek V4 Flash shows that DeepSeek's apparent 5x price advantage shrinks when retries are included. The article argues for more comprehensive benchmarks reporting cost per attempt and cost per verified success.

User asks for performance numbers on Deepseek V4 Flash running via Colibri, focusing on high VRAM setups, long context prefill, and token generation speed for agentic workloads.