llm-benchmark

Tag

Cards List
#llm-benchmark

Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models

Hugging Face Daily Papers ↗ · 2026-09-01 Cached

Introduces Autoregressive Mosaics (AM-Bench), a benchmark to evaluate whether text-only LLMs have genuine 2D spatial reasoning abilities, distinct from code generation. Findings show spatial reasoning varies among models and is influenced by output medium like SVG vs. code.

0 favorites 0 likes
#llm-benchmark

Ran Qwen3.8-Flash-Next (79 GB, 2-bit) at 350K ctx for 3.5 hours on a 128 GB M5 Max — speed vs context depth, 100 turns, one graph

Reddit r/LocalLLaMA ↗ · 2026-08-30

This article reports on running the Qwen3.8-Flash-Next model on a MacBook Pro M5 Max, benchmarking speed versus context depth over 100 turns, with insights into performance and issues like role confusion at long contexts.

0 favorites 0 likes
#llm-benchmark

DeflectBench: A Benchmark for Evaluating Rhetorical Fallacy Generation in LLMs

arXiv cs.CL ↗ · 2026-08-28 Cached

DeflectBench introduces a benchmark for evaluating the generation of rhetorical fallacies by large language models, revealing that safety refusal is highly sensitive to prompt framing rather than claim content, with models showing different compliance signatures.

0 favorites 0 likes
#llm-benchmark

Open Source Kernel in Qwen3.6-35B-A3B for AMD MI350X: 78,498 output tok/s on 8 GPUs

Reddit r/LocalLLaMA ↗ · 2026-08-26

The article describes the optimization of AMD MI350X GPUs for running the Qwen3.6-35B-A3B LLM, achieving high output token throughput and open-sourcing the kernel to improve performance.

0 favorites 0 likes
#llm-benchmark

BenchBench-Protocol: Evaluating Real-World Wet-Lab Protocol Reasoning and Modification

arXiv cs.AI ↗ · 2026-08-26 Cached

The article introduces BenchBench-Protocol, a benchmark of 149 real-world wet-lab protocol-modification tasks to evaluate large language models' reasoning, with Claude Opus 5 scoring highest at 59.2%.

0 favorites 0 likes
#llm-benchmark

@TheAhmadOsman: DeepSeek V4 Flash 0731 beats Qwen 3.8 27B btw

X AI KOLs Following ↗ · 2026-08-20 Cached

Ahmad tweets that DeepSeek V4 Flash 0731 outperforms Qwen 3.8 27B, and lists other models like Kimi K3, GLM 5.2, and MiniMax H3.

0 favorites 0 likes
#llm-benchmark

I spent way too much building a benchmark that measures LLM's political affinity, ethical values, and personality traits

Reddit r/singularity ↗ · 2026-08-20 Cached

Black Bench is a project that builds a benchmark to measure LLMs on political affinity, ethical values, and personality traits, with support sought for the initiative.

0 favorites 0 likes
#llm-benchmark

I transformed Pokelike.xyz into a LLM and RL benchmark!

Reddit r/LocalLLaMA ↗ · 2026-08-19

A data scientist has created a benchmark for reinforcement learning and large language models by transforming Pokelike.xyz into a game environment where AI bots can be trained and tested. The project is open-source and invites contributions to improve bot performance.

0 favorites 0 likes
#llm-benchmark

PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX

arXiv cs.CL ↗ · 2026-08-19 Cached

PTXBench is introduced as a benchmark to evaluate and adapt large language models for optimizing GPU kernels using architecture-specific PTX, showing uneven performance and fine-tuning insights.

0 favorites 0 likes
#llm-benchmark

HarmProfile: Characterizing Harmful Distributions in Frontier LLMs

arXiv cs.CL ↗ · 2026-08-18 Cached

HarmProfile is a benchmark dataset for analyzing harmful outputs from frontier LLMs, covering over 80,000 artifacts across 23 models and 15 harm categories to define model risk profiles.

0 favorites 0 likes
#llm-benchmark

Anyone interested in building a harness-only benchmark?

Reddit r/LocalLLaMA ↗ · 2026-08-05

A proposal to create a community-driven benchmark specifically for evaluating LLM harnesses, with a leaderboard measuring harness performance on diverse real-world tasks.

0 favorites 0 likes
#llm-benchmark

AI Squid Game - Deadlock

Reddit r/AI_Agents ↗ · 2026-08-02

A developer describes building 'AI Squid Game', an LLM benchmark that places 12 AI agents in a survival gameshow arena, each inside its own Docker container, and turns the resulting logs into a YouTube video.

0 favorites 0 likes
#llm-benchmark

AWARE-FX: An Auditable Knowledge-Guided AI System for Measuring Corporate Foreign-Exchange Hedging Disclosure

arXiv cs.CL ↗ · 2026-07-31 Cached

This paper presents AWARE-FX, an auditable AI/NLP decision-support system that extracts and scores corporate foreign-exchange hedging disclosures from annual reports, evaluated on 24,909 Hong Kong firm-years with FinBERT, ModernBERT, and Qwen3-8B comparisons.

0 favorites 0 likes
#llm-benchmark

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

arXiv cs.CL ↗ · 2026-07-30 Cached

This paper benchmarks LLM-simulated human survey responses across two large-scale datasets, finding that no model beats simple baselines at the individual level and that models systematically over-determine demographics, distorting segment differences. The failures persist across model scales and families, raising concerns about using synthetic users for decision support.

0 favorites 0 likes
#llm-benchmark

We compared different LLMs on IMO 2026 [R]

Reddit r/MachineLearning ↗ · 2026-07-26

This study evaluates frontier and open-weight LLMs on IMO 2026 problems, demonstrating that specialized harnesses like AutoFyn significantly improve performance of sub-frontier models, though hallucination issues persist on the hardest problem.

0 favorites 0 likes
#llm-benchmark

Benchmarking the Personalization Capabilities of Large Language Models

arXiv cs.AI ↗ · 2026-07-24 Cached

This paper introduces SDR-Bench, a benchmark for evaluating the personalization capabilities of large language models in a two-party Bayesian Persuasion framework, finding a consistent plateau across frontier LLMs and validating the framework with a field deployment.

0 favorites 0 likes
#llm-benchmark

Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network

arXiv cs.CL ↗ · 2026-07-22 Cached

This paper reports on a collaboration between the European Commission's Directorate-General for Translation and the European Master's in Translation network to localize the MMLU dataset into 11 European languages, creating a more inclusive benchmark for LLM evaluation while providing authentic training for translation students.

0 favorites 0 likes
#llm-benchmark

Qwen 3.6 27B is solid up to 262K context. How high have you guys gone above that using Rope/Yarn scaling?

Reddit r/LocalLLaMA ↗ · 2026-07-16

A user shares their experience pushing Qwen 3.6 27B to 262K context with coherent results, and discusses using Rope/Yarn scaling to go higher, along with kv-cache swapping strategies for RTX 3090 Ti.

0 favorites 0 likes
#llm-benchmark

Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

Hugging Face Daily Papers ↗ · 2026-07-13 Cached

Introduces SDABench, a benchmark evaluating LLMs on six scientific analysis capabilities across five domains, finding models struggle with tasks requiring assumption selection and mechanistic reasoning.

0 favorites 0 likes
#llm-benchmark

@DeRonin_: everyone thinks you can point an LLM at prediction markets and "print money" i tested it on Limitless that's not what h…

X AI KOLs Following ↗ · 2026-07-06 Cached

Testing 7 frontier models on live prediction markets found only 2 made money; a forecaster tool built on the Limitless API flags markets where models have a real edge.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback