benchmarking

Tag

Cards List
#benchmarking

How Fast is Python 3.15?

Hacker News Top ↗ · 2026-10-06 Cached

Miguel Grinberg benchmarks Python 3.15 (3.15.0rc3) against CPython 3.10–3.14, PyPy, Node.js, and Rust using recursive and loop-based test programs across single- and multi-threaded modes, including JIT and free-threading variants.

0 favorites 0 likes
#benchmarking

Europe finally takes the lead

Reddit r/ArtificialInteligence ↗ · 2026-10-06

As the global AI race intensifies, European lab Mistral appears to have taken the lead on a benchmark that other major players weren't prioritizing.

0 favorites 0 likes
#benchmarking

Supercomputing researchers document evolution of AI hardware

MIT News — Artificial Intelligence ↗ · 2026-10-06 Cached

MIT Lincoln Laboratory researchers have published a series of papers under the Lincoln AI Computing Survey (LAICS) documenting and comparing the performance and power of commercial AI accelerators — from a first study of 57 chips to a latest edition covering over 120.

0 favorites 0 likes
#benchmarking

Mistral Large 4

Simon Willison's Blog ↗ · 2026-10-06 Cached

Simon Willison tests Mistral Large 4 alongside frontier models (Claude Opus 5.5, GPT-6.1, Gemini 3.8 Flash) by generating an SVG of an armadillo in fishnet tights jaywalking on Mars, highlighting how saturated benchmarks push toward absurd creative prompts.

0 favorites 0 likes
#benchmarking

@dair_ai: Improve the reliability of agent evaluation with Jev-as-a-Judge.

X AI KOLs Following ↗ · 2026-10-06 Cached

DAIR.AI shares a link on improving the reliability of agent evaluation using a Jev-as-a-Judge approach.

0 favorites 0 likes
#benchmarking

@norsyx: GLM 5.3 Full 3bpw 4x DGX Sparks P.S: Screenshot from my patched ZCode harness that shows metrics

X AI KOLs Following ↗ · 2026-10-06 Cached

norsyx 报告 GLM 5.3 全量模型以 3bpw 量化在 4x DGX Sparks 上成功运行,并通过其修改过的 ZCode harness 截图展示了性能指标。

0 favorites 0 likes
#benchmarking

@rohanpaul_ai: New Shanghai AI Laboratory paper finds that training on verified runs of human-written agent skills makes a model a bet…

X AI KOLs Following ↗ · 2026-10-05 Cached

A Shanghai AI Laboratory paper introduces SkillGym, which turns human-written agent skill files into sandboxed pass-or-fail tasks and fine-tunes models on verified runs, improving agent performance (e.g., Terminal-Bench 2.1 +19.10 points in Claude Code) even without the skill files at prompt time.

0 favorites 0 likes
#benchmarking

Swift 1.5 on veloGB10, ~110 tok/s on 2× DGX Spark: xhigh beats base Flash-Next at medium on vLLM

Reddit r/LocalLLaMA ↗ · 2026-10-05

Running Swift 1.5 (UkisAI's reasoning-efficient fine-tune of Qwen3.8-Flash-Next) on sf-stav's veloGB10 Rust/CUDA engine across 2× DGX Spark yields ~110 tok/s decode, letting xhigh reasoning effort finish faster than base Flash-Next at medium on an older vLLM NVFP4 setup — including a header-only safetensors fix so 12 of 14 EXL3 community packs load, since current exllamav3 writes the n-gram table as 128 shards that velo can't read.

0 favorites 0 likes
#benchmarking

Switched my local agent from Qwen3.8 27B to Ornith 1.5 35B-A3B on two 5070 Tis: about 180 tok/s vs 60, same scores on my tests

Reddit r/LocalLLaMA ↗ · 2026-10-05

A user benchmarks local agent models on dual RTX 5070 Tis, finding that Ornith 1.5 35B-A3B (a ~3B-active-parameter MoE) matches Qwen3.8 27B on their tests while running about 3x faster (~180 tok/s) and scaling to 512K/1M contexts with YaRN.

0 favorites 0 likes
#benchmarking

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

arXiv cs.CL ↗ · 2026-10-05 Cached

The paper introduces Calibrated User Embeddings (CUE), a training-free framework that encodes observed sessions and samples continuous persona embeddings decoded into prompts for LLM user simulators, achieving better outcome calibration and failure-mode reproduction than existing persona-based simulation methods on tau^2-Bench.

0 favorites 0 likes
#benchmarking

Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing

arXiv cs.AI ↗ · 2026-10-05 Cached

This paper argues that time series forecasting benchmarks need scenario-grounded stress testing, where test cases include semantic scenarios, explicit failure operators, and measurable difficulty levels to capture structured anomalies like regime shifts and sensor failures that i.i.d. perturbation-based robustness tests miss.

0 favorites 0 likes
#benchmarking

DeReAct: Decomposed Reasoning and Acting for Reliable AI Agents

arXiv cs.AI ↗ · 2026-10-05 Cached

DeReAct decomposes ReAct-style agents into a task-solving Brain plus two gating policies (a Critic for action validation and a Context Manager for completion certification), improving Pass@1 on GAIA and SWE-bench Verified especially for weaker models like Qwen3-Coder-480B and Claude Sonnet 4.5.

0 favorites 0 likes
#benchmarking

Benchmark In Milliseconds

matklad ↗ · 2026-10-05 Cached

Matthias Endler-style (matklad) blog post arguing micro benchmarks should be tuned to run around 300ms — fast enough to iterate quickly, slow enough to avoid startup overhead and to give humans an intuitive sense of speed — with the aim that benchmarking provides decision-making intuition rather than precise measurement.

0 favorites 0 likes
#benchmarking

Four years of AI progress across seven capabilities, adjusted for changes in the benchmark

Reddit r/ArtificialInteligence ↗ · 2026-10-04

An analysis of four years of AI progress across seven capability areas, with results adjusted for changes and drift in the underlying benchmarks to better reflect true model improvement.

0 favorites 0 likes
#benchmarking

Has Claude Opus 5.5 Actually Been Nerfed? What the Last Three Days Show

Reddit r/artificial ↗ · 2026-10-04 Cached

分析了用户关于 Claude Opus 5.5 被“削弱”的说法:截至 2026 年 10 月 4 日没有公开证据表明 Anthropic 在权重层面降低了模型能力,性能波动更多与推理配置、系统指令及服务基础设施等多层因素有关。

0 favorites 0 likes
#benchmarking

@lxfater: After National Day, I bet everyone wants to try running large models locally. Because now, with 100 billion parameters,…

X AI KOLs Timeline ↗ · 2026-10-04 Cached

LocalMaxxing is a community benchmarking platform for local LLM inference, where users submit real-world test results to compare tokens/sec, first-token latency, and VRAM usage across models, GPUs, inference engines, and quantization formats. You can also run benchmarks yourself via the CLI and contribute to the public leaderboards when picking hardware or tuning local setups.

0 favorites 0 likes
#benchmarking

Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads

Hugging Face Daily Papers ↗ · 2026-10-04 Cached

本文研究 agentic RAG 评测中评估预算在问题数、搜索轨迹和重复读取之间的分配策略,通过 HotpotQA 和 MuSiQue 上的检索-反馈对比,发现更广的问题覆盖在固定 token 预算下能显著降低标准误差,并提出了预算分配的预测模型。

0 favorites 0 likes
#benchmarking

@sermakarevich: An article for everyone who ships, buys, or signs off on software built on large language models (LLMs): engineers, pro…

X AI KOLs Timeline ↗ · 2026-10-03 Cached

A plain-language, source-backed guide to LLM evaluations: how to build repeatable eval loops with inputs, gold answers, and graders, why unchecked judge models are the most common way AI teams fool themselves, and how to get honest numbers with error bars for code-based and model-based grading of retrieval and agent systems.

0 favorites 0 likes
#benchmarking

Evaluation Methodologies (Evals) in AI Engineering

Reddit r/ArtificialInteligence ↗ · 2026-10-03 Cached

Course notes for Chapter 3 of AI Engineering, covering evaluation methodologies (Evals) in depth: starting from the challenges posed by open-ended outputs and black-box models, it introduces language modeling metrics such as cross-entropy, perplexity, BPC/BPB, and compares two mainstream evaluation approaches — AI-as-a-judge and comparative evaluation.

0 favorites 0 likes
#benchmarking

tested three coding agents on the same real codebase. the biggest surprise was how much of your setup follows you between them

Reddit r/AI_Agents ↗ · 2026-10-02

A practical comparison of Cursor, Codex, and Claude Code run on the same real SaaS codebase, revealing that shared home-folder configs (plugins, skills, MCP servers) silently follow agents between tools, and that differences between agents came down more to scope of responsibility than model quality.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback