Tag
The article details a re-run of an AI benchmark test with 15 models, correcting previous errors and analyzing factors like accuracy, cost, and writing quality in multilingual pricing tasks.
The Artificial Analysis Image Editing Arena has been updated to expand editing tasks and use cases, providing a comprehensive leaderboard that ranks AI models on human preference for various image editing scenarios.
A benchmark comparison on MineBench shows Fable 5.1 has higher costs and inference times than Fable 5 due to longer reasoning, but it achieves more detailed 3D builds, including recognizable interiors.
The paper introduces VoiceLongMemEval (VLME), a benchmark that evaluates AI assistants' ability to remember and reason over paralinguistic metadata like emotion and prosody from voice in long-term conversations, revealing an 'affect gap' in current models.
ArizePhoenix shows significant performance improvements on benchmarks like Terminal-Bench 4.0 and Humanity's Last Exam, with gains from previous versions such as Fable 5.
Quasar 438B from Multiverse Computing is claimed to be the best European large language model, though it is proprietary and lacks transparency in its details.
CommerceAgentBench is a new benchmark with 107 real-world e-commerce tasks designed to test whether AI agents can actually complete jobs, moving beyond traditional answer-based AI benchmarks.
This paper introduces an expert-validated STEM QA dataset created to address gaps in existing benchmarks, demonstrating low performance from frontier AI models and potential for model training.
HunterBench introduces a benchmark to evaluate LLMs on autonomous penetration testing tasks, scoring them on coverage and exploitation capabilities across two simulated labs.
This article evaluates open weights VLMs for egocentric data processing using HFlow, showing that models like Gemma and Qwen achieve high accuracy compared to the Gemini baseline while being cheaper and suitable for self-hosting.
A personal benchmark comparing GLM5.3 Flash and Deepseek V4 Flash on a 2x DGX Spark setup shows GLM5.3 Flash has higher accuracy on HumanEval but with reduced context length and slower speed.
GLM-5.3 (max) outperforms GPT-5.6 Sol (max) on Terminal-Bench 4.0, highlighting the competitiveness of open-weight AI models, with Cline promoted for discounted access.
This article details experiments where various AI models were tested on the political compass test, with a focus on methodology, scoring validation, and results from multiple runs to ensure reproducibility.
Terminal-Bench version 4.0 has been released, updating the dataset and leaderboard with calibrated task resources, task fixes, and removal of saturated tasks.
The article describes a reinforcement learning method with verifiable rewards to fine-tune AI models for text-to-SQL, achieving human-level accuracy on the BIRD benchmark without scaffolding.
Terminal-Bench-Science is a new benchmark released as version 0.1.0 for evaluating AI agents on scientific research workflows, led by a Stanford community effort.
ProgramBench is a benchmark for agents to reproduce real software behavior from scratch. Factory is announced as the most advanced reverse-engineering system with breakthroughs in model-agnostic long-horizon agents.
The author argues for reviving 'Needle in a haystack' benchmarks to evaluate AI capabilities, sharing private test results that show many models performing poorly in remembering instructions, questioning their trustworthiness for real-world tasks.
A benchmark comparison of community quantized Qwen 3.8 27B models on RTX 6000 GPUs versus Claude Opus 4.6, evaluating token generation speed and efficiency for creating HTML games.
GLM-5.3-Flash is the first natively multimodal model in the GLM-5 series, with 320B total parameters and 18B active, utilizing a hybrid sparse-linear attention architecture to reduce costs while outperforming previous versions and approaching Claude Opus 4.8 on benchmarks.