Tag
Introduces Autoregressive Mosaics (AM-Bench), a benchmark to evaluate whether text-only LLMs have genuine 2D spatial reasoning abilities, distinct from code generation. Findings show spatial reasoning varies among models and is influenced by output medium like SVG vs. code.
This article reports on running the Qwen3.8-Flash-Next model on a MacBook Pro M5 Max, benchmarking speed versus context depth over 100 turns, with insights into performance and issues like role confusion at long contexts.
DeflectBench introduces a benchmark for evaluating the generation of rhetorical fallacies by large language models, revealing that safety refusal is highly sensitive to prompt framing rather than claim content, with models showing different compliance signatures.
The article describes the optimization of AMD MI350X GPUs for running the Qwen3.6-35B-A3B LLM, achieving high output token throughput and open-sourcing the kernel to improve performance.
The article introduces BenchBench-Protocol, a benchmark of 149 real-world wet-lab protocol-modification tasks to evaluate large language models' reasoning, with Claude Opus 5 scoring highest at 59.2%.
Ahmad tweets that DeepSeek V4 Flash 0731 outperforms Qwen 3.8 27B, and lists other models like Kimi K3, GLM 5.2, and MiniMax H3.
Black Bench is a project that builds a benchmark to measure LLMs on political affinity, ethical values, and personality traits, with support sought for the initiative.
A data scientist has created a benchmark for reinforcement learning and large language models by transforming Pokelike.xyz into a game environment where AI bots can be trained and tested. The project is open-source and invites contributions to improve bot performance.
PTXBench is introduced as a benchmark to evaluate and adapt large language models for optimizing GPU kernels using architecture-specific PTX, showing uneven performance and fine-tuning insights.
HarmProfile is a benchmark dataset for analyzing harmful outputs from frontier LLMs, covering over 80,000 artifacts across 23 models and 15 harm categories to define model risk profiles.
A proposal to create a community-driven benchmark specifically for evaluating LLM harnesses, with a leaderboard measuring harness performance on diverse real-world tasks.
A developer describes building 'AI Squid Game', an LLM benchmark that places 12 AI agents in a survival gameshow arena, each inside its own Docker container, and turns the resulting logs into a YouTube video.
This paper presents AWARE-FX, an auditable AI/NLP decision-support system that extracts and scores corporate foreign-exchange hedging disclosures from annual reports, evaluated on 24,909 Hong Kong firm-years with FinBERT, ModernBERT, and Qwen3-8B comparisons.
This paper benchmarks LLM-simulated human survey responses across two large-scale datasets, finding that no model beats simple baselines at the individual level and that models systematically over-determine demographics, distorting segment differences. The failures persist across model scales and families, raising concerns about using synthetic users for decision support.
This study evaluates frontier and open-weight LLMs on IMO 2026 problems, demonstrating that specialized harnesses like AutoFyn significantly improve performance of sub-frontier models, though hallucination issues persist on the hardest problem.
This paper introduces SDR-Bench, a benchmark for evaluating the personalization capabilities of large language models in a two-party Bayesian Persuasion framework, finding a consistent plateau across frontier LLMs and validating the framework with a field deployment.
This paper reports on a collaboration between the European Commission's Directorate-General for Translation and the European Master's in Translation network to localize the MMLU dataset into 11 European languages, creating a more inclusive benchmark for LLM evaluation while providing authentic training for translation students.
A user shares their experience pushing Qwen 3.6 27B to 262K context with coherent results, and discusses using Rope/Yarn scaling to go higher, along with kv-cache swapping strategies for RTX 3090 Ti.
Introduces SDABench, a benchmark evaluating LLMs on six scientific analysis capabilities across five domains, finding models struggle with tasks requiring assumption selection and mechanistic reasoning.
Testing 7 frontier models on live prediction markets found only 2 made money; a forecaster tool built on the Limitless API flags markets where models have a real edge.