ai-benchmark

Tag

Cards List
#ai-benchmark

Updated version of Humanity’s Last Exam: HLE-Diamond

Reddit r/singularity ↗ · 2026-09-23

An updated version of the Humanity's Last Exam, a benchmark for AI evaluation, named HLE-Diamond, has been announced and released.

0 favorites 0 likes
#ai-benchmark

FrontierMath Erd\H{o}s

arXiv cs.CL ↗ · 2026-09-23 Cached

The paper introduces FrontierMath Erdős, a benchmark of 68 open Erdős problems for evaluating AI models in mathematics using Lean proof assistant, aiming to address shortcomings in current AI research demonstrations.

0 favorites 0 likes
#ai-benchmark

Vals AI RSI Index: Opus 5.5 Beats the Published Human Baseline on One of Five Tasks; Shifts the Extrapolated Date for Full RSI from August 2027 to July 2027

Reddit r/singularity ↗ · 2026-09-23

The Vals AI RSI Index indicates that the Opus 5.5 AI model has beaten the published human baseline on one of five tasks, shifting the predicted date for achieving full RSI from August 2027 to July 2027.

0 favorites 0 likes
#ai-benchmark

Opus 5.5 dominates on all three performance metrics from ArtificialAnalysis.ai

Reddit r/artificial ↗ · 2026-09-22

Opus 5.5 dominates all three performance metrics from ArtificialAnalysis.ai and offers a cost-effective alternative to Fable 5.1.

0 favorites 0 likes
#ai-benchmark

GPT-6 Sol surpasses Claude Opus 5 on Agents’ Last Exam at 60% lower cost

Reddit r/singularity ↗ · 2026-09-22

GPT-6 Sol outperforms Claude Opus 5 on the Agents' Last Exam benchmark while offering 60% lower cost, indicating a major advancement in AI model efficiency.

0 favorites 0 likes
#ai-benchmark

Remote Labor Index updated with Fable and Astra

Reddit r/singularity ↗ · 2026-09-20

The Remote Labor Index is updated with Fable and Astra, providing a benchmark for AI models on real-world projects from the remote labor economy, judged by human experts.

0 favorites 0 likes
#ai-benchmark

Brood War Bench

Hacker News Top ↗ · 2026-09-19 Cached

This article benchmarks AI models on playing Brood War, with Codex Astra leading the leaderboard. It discusses model performance, noting that newer models handle real-time strategy better than older ones that treated it as turn-based.

0 favorites 0 likes
#ai-benchmark

@charles_irl: hate to see my favorite benchmark was actually in the training data all along

X AI KOLs Following ↗ · 2026-09-19 Cached

A tweet discusses concerns about an AI benchmark being included in training data, with Simon Willison sharing a related movie trivia fact.

0 favorites 0 likes
#ai-benchmark

RTX 5090 Bonsai 2 27B vs Gemma 4 12B vs Qwen 3.5 9B Japanese voxel pagoda

Reddit r/LocalLLaMA ↗ · 2026-09-18

This article compares the performance of Bonsai 2 27B, Gemma 4 12B, and Qwen 3.5 9B models on generating a Japanese voxel pagoda using an RTX 5090, concluding that Bonsai offers superior intelligence and detail for its memory footprint, benefiting the local AI community.

0 favorites 0 likes
#ai-benchmark

DeepSWE-mini a 16 instance subset of DeepSWE that replicates the rankings of the leaderboard

Reddit r/LocalLLaMA ↗ · 2026-09-18

Released a new dataset called DeepSWE-mini, a 16-instance subset of DeepSWE designed for efficient benchmarking of local AI models by replicating leaderboard rankings.

0 favorites 0 likes
#ai-benchmark

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

Hugging Face Daily Papers ↗ · 2026-09-18 Cached

RecreationWorld introduces a scalable and verifiable framework for hybrid computer-use agents, providing environments across five platforms and a benchmark for evaluation.

0 favorites 0 likes
#ai-benchmark

What GPT-6 Astra’s 99.9% ARC-AGI-3 Score Actually Measures

Reddit r/artificial ↗ · 2026-09-17 Cached

The article analyzes GPT-6 Astra's 99.9% score on the ARC-AGI-3 benchmark, revealing that the score varies based on testing harnesses and questioning OpenAI's AGI interpretation amid clarifications from the benchmark creators.

0 favorites 0 likes
#ai-benchmark

AndroidLife: Can an AI agent survive a day in the life of a real user? Qwen3.8-27b run: 56.7% SR

Reddit r/LocalLLaMA ↗ · 2026-09-17

An AI agent using the Qwen3.8-27b model from Alibaba achieved a 56.7% success rate in the AndroidLife benchmark, running 60 real tasks on a OnePlus phone with detailed performance and efficiency metrics.

0 favorites 0 likes
#ai-benchmark

@VikParuchuri: Structured extraction is hard to eval, and most benchmarks are biased, or have bad scoring/GT. That’s why we made OmniE…

X AI KOLs Timeline ↗ · 2026-09-16 Cached

OmniExtractBench is a new benchmark with 620 documents from multiple vendors, designed for fair evaluation of structured extraction, highlighting top performance by Datalab and Reducto.

0 favorites 0 likes
#ai-benchmark

GPT-5.6 Luna vs GPT-6 Astra: is a $1.20 model good enough for code review?

Reddit r/artificial ↗ · 2026-09-14

Benchmarked GPT-5.6 Luna vs GPT-6 Astra across 50 real PRs, showing Astra found 92 bugs vs Luna's 69, with Luna catching 75% of bugs at 3.6% of the cost. Includes detailed evaluation breakdown and upcoming comparison with Fable 5.1.

0 favorites 0 likes
#ai-benchmark

OpenAI's Astra scored 62.7% and 99.9% on the same benchmark, and I still don't fully know which one to trust

Reddit r/ArtificialInteligence ↗ · 2026-09-14

The article examines inconsistencies in benchmark scores for OpenAI's Astra model on the ARC Prize, highlighting discrepancies from different testing harnesses and changes to OpenAI's launch page, which prompts concerns about evaluation accuracy.

0 favorites 0 likes
#ai-benchmark

DeepSeek V4.1 Flash beats Astra on AA's new benchmark

Reddit r/LocalLLaMA ↗ · 2026-09-14

AA introduced a new benchmark in the Intelligence Index v4.3 update, and DeepSeek V4.1 Flash has outperformed Astra to take first place.

0 favorites 0 likes
#ai-benchmark

luxobench (Website)

TLDR AI ↗ · 2026-09-14 Cached

Luxobench is a benchmark for AI models to design an interactive desktop lamp, with criteria covering functionality, cost, manufacturability, and lead time.

0 favorites 0 likes
#ai-benchmark

ARC-AGI-4 (2 minute read)

TLDR AI ↗ · 2026-09-14 Cached

ARC-AGI-4 is introduced as a benchmark for measuring autonomous open-ended innovation in AI, underscoring the importance of open-source development to advance AI capabilities and benefit humanity.

0 favorites 0 likes
#ai-benchmark

Nex-N2.5-mini-MLX-4bit on Apple M5 Max — 133.6 tok/s — llm-bench.io

Reddit r/LocalLLaMA ↗ · 2026-09-12 Cached

Benchmark results for the Nex-N2.5-mini-MLX-4bit model on Apple M5 Max hardware, achieving 133.6 tokens per second generation speed and quality scores up to 85.80 in research tasks.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback