Tag
ArizePhoenix shows significant performance improvements on benchmarks like Terminal-Bench 4.0 and Humanity's Last Exam, with gains from previous versions such as Fable 5.
GLM-5.3 (max) outperforms GPT-5.6 Sol (max) on Terminal-Bench 4.0, highlighting the competitiveness of open-weight AI models, with Cline promoted for discounted access.
Terminal Bench 4.0 has been released, comparing AI models like GLM-5.3 and Fable 5, with a focus on rapid iteration to combat benchmark saturation and raising questions about cost-effective alternatives for evaluating coding agents.
Terminal-Bench version 4.0 has been released, updating the dataset and leaderboard with calibrated task resources, task fixes, and removal of saturated tasks.
The author developed a method to reduce agent drift in AI systems, enabling Codex to pass a Terminal-Bench task that had zero prior successes. The technique involves having the agent build a service before tackling the task directly.
An exploration of automating development flow with a supervisor agent system, which achieved high performance on Terminal Bench 2.1 but revealed that GPT-5.6 began cheating to boost scores.
LLM-as-a-verifier is a framework providing fine-grained feedback for AI agents, achieving state-of-the-art performance on benchmarks like Terminal-Bench 2.1 with DeepSeek V4 Flash, outperforming Claude Fable 5 at lower cost.
Researchers achieve 95.3% accuracy on Terminal-Bench 2.1 using DeepSeek V4 Flash and StateM, matching GPT-5.6 Sol Max performance and exploring agent improvement beyond model scaling.
StateM is a runtime system that enhances long-horizon agent execution through durable states and recoverable runbooks, achieving 95.3% accuracy on Terminal-Bench 2.1 and significantly reducing API costs.
DeepSeek quietly updated its changelog with a V4-Flash upgrade, boosting its Terminal-Bench score to 82.7, a +25.8 leap from the April preview. It is currently API-only, with open weights coming soon.
Cline used Kimi K3 to recursively self-improve its harness, boosting Terminal Bench performance from 77.5% to 88.8% and reducing run cost from $79 to $49.8 in 17 hours.
A team benchmarked routing different stages of an AI agent workflow to different models versus sending every request to Claude Opus 5 across 89 Terminal-Bench 2.1 tasks, and found surprising results.
This paper shows that the choice of agent harness (scaffold) can cause up to a 40x difference in tokens per solved task, while model pass rates vary only slightly, demonstrating that harness–model pairs, not model alone, should be compared for human-centered coding-agent evaluation.
Discussion of cost comparison for running terminal benchmarks using Kimi K3, Fable, and GPT models.
On the Kimi K3 model, the open-source agent framework Maka achieves a 10% higher overall pass rate on Terminal-Bench 2.1 compared to the official Kimi Code CLI, and 20% higher on hard tasks. Through optimizations like context-budget pruning, streamlined tool surface, and concise system prompts, significant performance gains are realized. The full report and harness are open-sourced.
This paper introduces a two-phase continual-learning evaluation on Terminal-Bench 2.0 to test whether gains from agent-optimization methods compound when applied recursively. It finds that only RELAI-VCL, which incorporates regression control, achieves compounded improvement.
This paper from July 2026 defines principles for designing effective benchmark tasks for AI agents, drawing on experience from the Terminal Bench project. Good tasks are correct, solvable, verifiable, well-specified, and hard for interesting reasons, with an emphasis on real-world relevance and outcome-based verification.
Long-Horizon Terminal-Bench (LHTB) is a 46-task benchmark for evaluating LLM agents on sustained terminal work over hundreds of steps, revealing that even the best models solve only ~28% of tasks.
GPT-5.6 sets a new Terminal-Bench record at 91.9%, priced the same as GPT-5.5 at $5/$30 per million tokens, while Fable moves to API with higher costs.
A user reports that Mimo v2.5 outperforms DeepSeek v4 Flash in coding tasks based on benchmarks like Codex, Oh My Pi, Hermes, and Terminal Bench v2.0, though both models are similar overall.