Tag
Miguel Grinberg benchmarks Python 3.15 (3.15.0rc3) against CPython 3.10–3.14, PyPy, Node.js, and Rust using recursive and loop-based test programs across single- and multi-threaded modes, including JIT and free-threading variants.
As the global AI race intensifies, European lab Mistral appears to have taken the lead on a benchmark that other major players weren't prioritizing.
MIT Lincoln Laboratory researchers have published a series of papers under the Lincoln AI Computing Survey (LAICS) documenting and comparing the performance and power of commercial AI accelerators — from a first study of 57 chips to a latest edition covering over 120.
Simon Willison tests Mistral Large 4 alongside frontier models (Claude Opus 5.5, GPT-6.1, Gemini 3.8 Flash) by generating an SVG of an armadillo in fishnet tights jaywalking on Mars, highlighting how saturated benchmarks push toward absurd creative prompts.
DAIR.AI shares a link on improving the reliability of agent evaluation using a Jev-as-a-Judge approach.
norsyx 报告 GLM 5.3 全量模型以 3bpw 量化在 4x DGX Sparks 上成功运行,并通过其修改过的 ZCode harness 截图展示了性能指标。
A Shanghai AI Laboratory paper introduces SkillGym, which turns human-written agent skill files into sandboxed pass-or-fail tasks and fine-tunes models on verified runs, improving agent performance (e.g., Terminal-Bench 2.1 +19.10 points in Claude Code) even without the skill files at prompt time.
Running Swift 1.5 (UkisAI's reasoning-efficient fine-tune of Qwen3.8-Flash-Next) on sf-stav's veloGB10 Rust/CUDA engine across 2× DGX Spark yields ~110 tok/s decode, letting xhigh reasoning effort finish faster than base Flash-Next at medium on an older vLLM NVFP4 setup — including a header-only safetensors fix so 12 of 14 EXL3 community packs load, since current exllamav3 writes the n-gram table as 128 shards that velo can't read.
A user benchmarks local agent models on dual RTX 5070 Tis, finding that Ornith 1.5 35B-A3B (a ~3B-active-parameter MoE) matches Qwen3.8 27B on their tests while running about 3x faster (~180 tok/s) and scaling to 512K/1M contexts with YaRN.
The paper introduces Calibrated User Embeddings (CUE), a training-free framework that encodes observed sessions and samples continuous persona embeddings decoded into prompts for LLM user simulators, achieving better outcome calibration and failure-mode reproduction than existing persona-based simulation methods on tau^2-Bench.
This paper argues that time series forecasting benchmarks need scenario-grounded stress testing, where test cases include semantic scenarios, explicit failure operators, and measurable difficulty levels to capture structured anomalies like regime shifts and sensor failures that i.i.d. perturbation-based robustness tests miss.
DeReAct decomposes ReAct-style agents into a task-solving Brain plus two gating policies (a Critic for action validation and a Context Manager for completion certification), improving Pass@1 on GAIA and SWE-bench Verified especially for weaker models like Qwen3-Coder-480B and Claude Sonnet 4.5.
Matthias Endler-style (matklad) blog post arguing micro benchmarks should be tuned to run around 300ms — fast enough to iterate quickly, slow enough to avoid startup overhead and to give humans an intuitive sense of speed — with the aim that benchmarking provides decision-making intuition rather than precise measurement.
An analysis of four years of AI progress across seven capability areas, with results adjusted for changes and drift in the underlying benchmarks to better reflect true model improvement.
分析了用户关于 Claude Opus 5.5 被“削弱”的说法:截至 2026 年 10 月 4 日没有公开证据表明 Anthropic 在权重层面降低了模型能力,性能波动更多与推理配置、系统指令及服务基础设施等多层因素有关。
LocalMaxxing is a community benchmarking platform for local LLM inference, where users submit real-world test results to compare tokens/sec, first-token latency, and VRAM usage across models, GPUs, inference engines, and quantization formats. You can also run benchmarks yourself via the CLI and contribute to the public leaderboards when picking hardware or tuning local setups.
本文研究 agentic RAG 评测中评估预算在问题数、搜索轨迹和重复读取之间的分配策略,通过 HotpotQA 和 MuSiQue 上的检索-反馈对比,发现更广的问题覆盖在固定 token 预算下能显著降低标准误差,并提出了预算分配的预测模型。
A plain-language, source-backed guide to LLM evaluations: how to build repeatable eval loops with inputs, gold answers, and graders, why unchecked judge models are the most common way AI teams fool themselves, and how to get honest numbers with error bars for code-based and model-based grading of retrieval and agent systems.
Course notes for Chapter 3 of AI Engineering, covering evaluation methodologies (Evals) in depth: starting from the challenges posed by open-ended outputs and black-box models, it introduces language modeling metrics such as cross-entropy, perplexity, BPC/BPB, and compares two mainstream evaluation approaches — AI-as-a-judge and comparative evaluation.
A practical comparison of Cursor, Codex, and Claude Code run on the same real SaaS codebase, revealing that shared home-folder configs (plugins, skills, MCP servers) silently follow agents between tools, and that differences between agents came down more to scope of responsibility than model quality.