benchmark

MME-Safety is a rigorously verified benchmark for evaluating the safety of Multimodal Large Language Models, featuring a four-dimensional annotation schema and a hierarchical framework to assess risk scenarios, harm severity, and modality-specific stealth levels.

VisPath introduces a visual-intent-guided path reasoning framework for multimodal knowledge graph question answering, achieving significant improvements over baselines on a new benchmark, VisPath-Bench, and existing datasets.

PhysioBench introduces a unified benchmark for physiological signal question answering, harmonizing 22 datasets into 61.4 million questions across 30 tasks to evaluate the performance of various AI models.

HappyWorld-Bench is a comprehensive benchmark that evaluates the reliability of world models under interaction and modification across video, spatial, and embodied tracks.

The paper presents VideoGen-Agent, a reinforcement learning-based multimodal agent that coordinates tools for video generation, significantly improving performance on the new VABench benchmark.

The paper introduces GameHorizon Suite, a unified data and evaluation framework for assessing AI models' capabilities in gameplay across multiple temporal horizons, featuring an annotation pipeline, large-scale dataset, and reproducible benchmark.

A small lab from Switzerland and South Africa has open-sourced Hemmingway-1, a 27B Apache-2.0 licensed fine-tune of Qwen3.8-27B specialized for creative writing, achieving high scores on EQ-Bench and internal benchmarks.

The tweet expresses anticipation for community-created LoRAs following the announcement of an open-weight 7B model that reportedly surpasses Nano Banana 2.0 in performance.

Jev + Mercury 2.5 AI model completed all tasks on the WebMCP benchmark with significantly lower cost compared to GPT-6 Astra variants.

The article explains how to build a local decision engine using open-source LLMs and SGLang, enabling efficient scoring and probability distributions for fixed choices without full text generation, compared to systems like Jev.

An experiment where AI models including Jev, Laya, finetuned ModernCE-base-nli, and Qwen3.5-4B are given control in the game Doom, with performance metrics measured for kill counts and survival time in different scenarios.

OmniEcho introduces a spatially aware omni-modal model and a new benchmark for spatial audio-visual perception and navigation in embodied agents, achieving state-of-the-art performance.

PackLab introduces a comprehensive framework for robotic bin packing using multimodal large language models, including a simulation platform, a specialized model, and a benchmark that outperforms traditional methods.

JevBench is a new benchmark that evaluates AI models on bounded software decisions by combining intelligence, calibration, speed, and cost, as announced by @rohanpaul_ai.

The paper introduces Self-Improvement via Fast Tree-search (SIFT), a framework that uses an LLM-as-a-judge to efficiently evaluate self-modifications in coding agents, achieving better benchmark performance with significantly reduced CPU hours and API costs.

The article discusses a paper titled 'Reasoning-Intensive Regression' that proposes MENTAT, a lightweight method combining batch-reflective prompt optimization with neural ensemble learning to improve numerical score prediction from text in AI tasks, showing up to 65% improvement over baselines.

A skeptical deep dive finds numerous errors in the Humanity's Last Exam benchmark, with the official o3-mini grader incorrectly marking correct answers as wrong.

Apple's M6 Pro chip has achieved the highest single-core CPU score in the Geekbench 7 benchmark, demonstrating its leading performance capabilities.

A discussion about using Jev for query optimization, with one user sharing a 12% speed improvement in Postgres queries and another commenting on Jev's performance issues.

Android Bench 2.0 is an updated AI evaluation framework that assesses models on long-horizon engineering tasks for Android, such as building apps from scratch and migrating codebases, with continuous scoring.