benchmark

Presents comparison results between Fable 5.5 and GPT 6.2 Astra when generating an SVG of a PS5 controller, routed via stealth routing.

Particle-1, a coding-agent-focused model emphasizing token efficiency (~45% fewer output tokens than DeepSeek V4.1 Flash with comparable accuracy), is now available in OpenClaw v2026.10.10, with zero data retention for standard inference.

balbal-v2 releases two CPU-efficient Turkish speech recognition models (313M and 45M params, CTC + KenLM) trained for under $250, ranking 2nd on the Mu-Bench TR benchmark behind Google Chirp 3 while running at up to 236x real time on modest hardware.

A developer compares running Qwen3.8-Flash-Next locally (UD-Q4_K_XL on llama-server with ROCm) against Claude Opus in Claude Code for reviewing a real Swift commit: Qwen produced 8 findings with zero false positives and caught 2 issues Claude missed, but missed 5 others, making it a solid private first-pass reviewer rather than a final authority.

Fable 5.5 reportedly generated a complete animation in a single pass using only code. Chetaslua said that standard Opus 5.5 and Fable 5.1 produced noticeably worse results with the same prompt and provided a method for verification, but the report has not yet been officially confirmed.

The authors replaced five notations (Make, init-d, Graphviz DOT, Gherkin, Belnap tables) in their AI agent control plane with 1980 Smalltalk-80 keyword cascades, reporting 100% compliance in a 360-trial benchmark versus 2.5% for prose/Gherkin and 91.7% for S-expressions.

A quantization report for LiquidAI's LFM3-1.7B shows it runs losslessly in 3.6 GB VRAM (2.2 GB with modest degradation) and outperforms MiniCPM5-2B on LiquidAI benchmarks while offering 2x context in half the memory.

An analysis points out that all four Unsloth quantizations of DeepSeek-V4.1-Flash-1017 plus the Q8 baseline simultaneously improved by 2–3 points on LiveCodeBench v6 in an updated chart, a shift statistically improbable under an independent-sampling model, while GSQ-RCO results behaved inconsistently.

Chris Parmer noted that Particle's new model landed on Dash's Pareto frontier for data analytics tasks, outperforming Terra-6.1 and Haiku 5.5 — the first new entrant to their frontier in years.

A blog post arguing that coding-agent session transcripts and trajectories are valuable records for debugging failures, understanding costs, and reusing fixes — and warning that Claude Code's 30-day cleanup can delete them permanently. It breaks down what a session archive contains and cites an RTK benchmark attempt that passed only after 339 consecutive errors at ~9x the baseline cost.

OpenAI announced a new internal benchmark evaluating how well its AI handles everyday human tasks like shopping and meal planning, reporting that GPT-6 Astra scored 93.1% on finding a kindergarten after a move and 90.4% on creating a diabetes-friendly meal plan.

OpenAI announced LifeVal, a new benchmark built from feedback by thousands of ChatGPT users on everyday tasks they most want help with, and has used it over the past year to measure and improve model performance on practical real-world workflows.

The organization introduces Hydrogen-1.0, an open-weight model fine-tuned from GLM-5.3 on internal scientific data, claiming 38.3% on FrontierScience-Research (vs GPT-6-Astra's 33.3%) and 93.0% on GPQA-Diamonds, in service of open scientific discovery.

LGW Lab open-sourced a Claude skill that makes publication-quality schematic figures for research papers, plus a benchmark showing Opus 5.5 and GPT-6 Astra achieve near-publication quality with the skill.

A 216-run benchmark compares Claude Code (Opus 5.5) and Codex (GPT 6 Astra) across agent-browser, Playwright, and Chrome DevTools via CLI, skill, and MCP on a browser task; all runs passed, tool choice depended on the agent, and a generated Python script vastly outperformed every agent run in speed and cost.

A hobbyist benchmark pits locally run quantized models (Qwen3.8-Flash-Next, Qwen3.8-27B, Gemma 4) against Claude models on 9 real C++/CUDA bug-fixing tasks, finding 3-bit quantization costs little in score but makes models think longer, and that local Flash-Next roughly matches Claude Haiku 5.5 while Opus 5.5 leads overall.

The author tested three local open-source models — DeepSeek-V4.1-Flash-1017, Qwen-3.8-Flash-Next, and Qwen-3.8-27b — on an RTX 5090, an RTX 4090, and dual V100s, comparing the real-world experience gap against cloud models like GPT-5.5 and Opus-4.8 through a Core-300 blind test and 200K+ long-context stress tests.

Lab2Sim introduces a benchmark of 218 science experiment videos with evidence-grounded specifications, along with EduVideo2Code, which encodes procedures as state machines to guide verified code generation for interactive virtual experiments, improving apparatus fidelity by 24.15 points.

The HarnessMM paper, a collaboration between Tsinghua University and Microsoft Research Asia, proposes improving agent harness token efficiency through protected decision reuse and recursive self-improvement. Experiments show DeepSeek Harness (DSH) benefits significantly: HumanEval success rate rises from 71.43% to 85.71% with 87.93% fewer tokens per problem, while Codex only saves 29.54%.

OpenAI announced LifeVal, an evaluation built to measure how well its models perform on practical everyday tasks—such as childcare, budgeting, shopping, and government services—derived from time-use studies and a new 3,995-respondent survey, and used over the past year to drive model improvements.