Tag
An updated version of the Humanity's Last Exam, a benchmark for AI evaluation, named HLE-Diamond, has been announced and released.
The paper introduces FrontierMath Erdős, a benchmark of 68 open Erdős problems for evaluating AI models in mathematics using Lean proof assistant, aiming to address shortcomings in current AI research demonstrations.
The Vals AI RSI Index indicates that the Opus 5.5 AI model has beaten the published human baseline on one of five tasks, shifting the predicted date for achieving full RSI from August 2027 to July 2027.
Opus 5.5 dominates all three performance metrics from ArtificialAnalysis.ai and offers a cost-effective alternative to Fable 5.1.
GPT-6 Sol outperforms Claude Opus 5 on the Agents' Last Exam benchmark while offering 60% lower cost, indicating a major advancement in AI model efficiency.
The Remote Labor Index is updated with Fable and Astra, providing a benchmark for AI models on real-world projects from the remote labor economy, judged by human experts.
This article benchmarks AI models on playing Brood War, with Codex Astra leading the leaderboard. It discusses model performance, noting that newer models handle real-time strategy better than older ones that treated it as turn-based.
A tweet discusses concerns about an AI benchmark being included in training data, with Simon Willison sharing a related movie trivia fact.
This article compares the performance of Bonsai 2 27B, Gemma 4 12B, and Qwen 3.5 9B models on generating a Japanese voxel pagoda using an RTX 5090, concluding that Bonsai offers superior intelligence and detail for its memory footprint, benefiting the local AI community.
Released a new dataset called DeepSWE-mini, a 16-instance subset of DeepSWE designed for efficient benchmarking of local AI models by replicating leaderboard rankings.
RecreationWorld introduces a scalable and verifiable framework for hybrid computer-use agents, providing environments across five platforms and a benchmark for evaluation.
The article analyzes GPT-6 Astra's 99.9% score on the ARC-AGI-3 benchmark, revealing that the score varies based on testing harnesses and questioning OpenAI's AGI interpretation amid clarifications from the benchmark creators.
An AI agent using the Qwen3.8-27b model from Alibaba achieved a 56.7% success rate in the AndroidLife benchmark, running 60 real tasks on a OnePlus phone with detailed performance and efficiency metrics.
OmniExtractBench is a new benchmark with 620 documents from multiple vendors, designed for fair evaluation of structured extraction, highlighting top performance by Datalab and Reducto.
Benchmarked GPT-5.6 Luna vs GPT-6 Astra across 50 real PRs, showing Astra found 92 bugs vs Luna's 69, with Luna catching 75% of bugs at 3.6% of the cost. Includes detailed evaluation breakdown and upcoming comparison with Fable 5.1.
The article examines inconsistencies in benchmark scores for OpenAI's Astra model on the ARC Prize, highlighting discrepancies from different testing harnesses and changes to OpenAI's launch page, which prompts concerns about evaluation accuracy.
AA introduced a new benchmark in the Intelligence Index v4.3 update, and DeepSeek V4.1 Flash has outperformed Astra to take first place.
Luxobench is a benchmark for AI models to design an interactive desktop lamp, with criteria covering functionality, cost, manufacturability, and lead time.
ARC-AGI-4 is introduced as a benchmark for measuring autonomous open-ended innovation in AI, underscoring the importance of open-source development to advance AI capabilities and benefit humanity.
Benchmark results for the Nex-N2.5-mini-MLX-4bit model on Apple M5 Max hardware, achieving 133.6 tokens per second generation speed and quality scores up to 85.80 in research tasks.