benchmark

Introduces MoCA (Implicit Social Context Analysis), a new task and benchmark for modeling implicit social scenarios across affection, intent, and stance, along with a Conflict-Driven Abductive Reasoning (CoDAR) framework. Experiments show state-of-the-art multimodal LLMs struggle on this task, while CoDAR improves performance but still lags behind human reasoning.

This paper introduces M3R-Bench, a unified evidence-grounded benchmark for multimodal metaphor understanding with 1,000 image-text instances, and proposes M3R-Reasoner, an 8B-parameter model combining curriculum-based reasoning supervision and reinforcement learning that outperforms larger proprietary models.

This paper introduces DREAM, a structured memory framework for LLM-based role-playing agents that uses an Event-aware Memory Graph to maintain temporal and causal coherence, and proposes the TCM benchmark for evaluation.

This paper introduces a three-condition experimental framework and a benchmark of 24,300 prompts to study how biased user turns modulate cognitive bias expression in frontier LLMs under multi-turn interactions. It finds that biased conversational context amplifies bias in most models, while explicit bias cues can trigger alignment-related suppression.

Introduces PoolBench, a benchmark that isolates pooling strategies in decoder-only LLM concept representation, evaluating 19 strategies across 17 concepts and 3 models to provide a standardized protocol for comparing pooling choices.

This paper introduces MS-MLB, an open machine learning benchmark for classifying multiple sclerosis from whole blood RNA expression data using the public GSE17048 cohort. It provides a reproducible, leakage-controlled evaluation pipeline and reports Gradient Boosting as the top performer.

This paper introduces Unified Agent, a stateful AI agent that maintains compact interaction state across devices and time to handle cross-device, cross-time user requests, outperforming existing agent designs in a new benchmark.

Introduces SkillTV-Bench, a 681-case benchmark for evaluating skill-aware trajectory verification in LLM agents, along with SkillTV-Evolve, a method that externalizes verification knowledge as a reusable JudgeSkill, improving judge accuracy by 14.8 percentage points.

This paper introduces EcoAgent-Bench, a 304-task benchmark for evaluating LLM agents' economic decision-making under explicit budgets and priced actions, testing four cost-related decisions across five task families.

Introduces C3PO, a benchmark of 3,404 samples for evaluating cross-modal composition and counterfactual reasoning in multimodal LLMs. It finds modality dominance causes most failures, with even the best model (Gemini-3.1-Pro) far below human accuracy.

This paper introduces LUNAR, a benchmark for evaluating how large language models personalize responses from longitudinal app interaction histories across daily-life domains such as clothing, food, housing, and mobility. Experiments on 19 mainstream LLMs reveal that effective personalization depends on evidence selection and cross-domain integration, and that stronger personalization can come at the cost of privacy protection.

This paper introduces SearchAuditBench, a benchmark of 1,243 failed long-horizon search-agent trajectories with expert annotations, and SearchAuditor, a multi-perspective auditing framework that localizes, attributes, and repairs agent failures. Experiments show SearchAuditor outperforms baselines, achieving a 32.3% end-to-end pass rate with frontier models like GPT-5.5.

Factorio Learning Environment v0.3.0 is an open-source platform for evaluating AI agents in Factorio, adding headless scaling, OpenAI Gym compatibility, and Claude Code integration for live demonstrations.

Qwen3.8 Max is now ranked as the best overall model on Artificial Analysis's agentic index, surpassing other leading AI models in independent evaluations.

A developer benchmarks 13 search API configurations inside an AI agent, revealing that hidden token costs from reading payloads can dominate the total bill and vary by up to 67x across providers.

Technical post sharing performance stats for running Qwen 27B on 2x RTX 5070 Ti GPUs with vLLM cu129-nightly, achieving up to 94-87 tps decode and 170k GPU KV cache.

A benchmark comparing 8 PDF parsers across 14 capabilities, finding Chandra the most accurate while noting trade-offs like speed; LightOnOCR-1B impresses for its size but hallucinates on illegible text.

A head-to-head evaluation of 32 local language models on a fact-extraction corpus finds that most models are statistically indistinguishable, with LFM2.5 models performing significantly worse despite larger sizes.

A discussion about how Artificial Analysis ranks Gemma 4 above Qwen3.6 27b on the SciCode benchmark, questioning whether the ranking reflects real-world coding ability or reveals a benchmarking issue.

A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.