Tag
A developer built a Codenames-inspired browser game where LLMs play each other, running 376 games across 19 models and finding that guesser quality (68% win correlation) matters far more than clue quality (53%), with newer models guessing much better than older ones. Raw data and replayable games are shared publicly.
The Continual Learning Exam (CLE) introduces 120 expert-written professional tasks organized into 5-level ladders to test whether LLM agents can reuse knowledge gained earlier; even the best setup only clears 47.7% of levels.
OpenFrontiers introduces a cross-disciplinary corpus of 12,151 open research problems plus a 200-problem retrospective benchmark of recently resolved mathematics problems with Lean 4 statements, evaluating seven LLM agents and finding that closed-book rediscovery of research-level results remains rare while live-web success largely reflects retrieval of published solutions.
JEV Arena is an open-source 2D fighting game benchmark where decision models choose every move based on health, distance and opponent actions, featuring a hosted leaderboard, custom fighter configs, and a 'mental_attack' move that lets models inject messages into their opponent's prompt.
This paper introduces BoardSafety, a benchmark built from vendor datasheets to evaluate whether LLMs can judge if an operation is safe for a specific development board. Results show current models, even with access to board-specific rules, cannot reliably make safe determinations.
A developer reports that Gemini 4 Argon made many mistakes when building on the With compiler, while Meta Muse Wonder Full completed a similar test without errors, suggesting Muse is better suited for developing the compiler.
作者设计并开源了 KillSwitch，一种专为对抗 LLM 和编码代理而构建的栈式编程语言，利用隐藏谎言规范、字符计数规则、触发安全护栏的语句等特性，并对主流 LLM 进行了基准测试，Claude Opus 5 表现最佳。
A community member released Qwen3.8-27B-antislop, a Qwen 27B model reinforcement-learned in the style of Meta's RL XAR to reduce 'sloppy' LLM output, ranking 4th on EQbench (BF16+MXFP4). Weights and GGUF versions are available on Hugging Face.
StarSkirmish Bench v0.2 在一小时时限内让各旗舰 LLM 用 C++ 编写 StarCraft: Brood War 神族机器人并与人类选手机器人对战，GPT-6 Astra 以明显优势居首，Gemini 4 Argon、Claude Opus 5.5 和 Claude Fable 5.1 紧随其后，且结果显示人类顶尖机器人仍远超所有模型。
该论文提出VALCON框架，通过生成者/评判者双重角色变体来基准测试并提升大语言模型的价值一致性，实验揭示28个LLM普遍存在跨角色不一致问题，并提出基于表征的干预方法加以缓解。
该论文提出'standing'概念，研究多用户聊天中智能体的承诺能否被非债权人取消，并在六个模型上区分未授权释放与覆盖两类错误，发现自维护承诺账本加释放策略可显著提升严格准确率。
We present SmartSalesBench, the first multi-turn conversational benchmark designed to evaluate LLM sales agents against psychologically grounded, resistant buyer personas, with nine evaluation metrics and experiments across four models totaling 13,680 sessions.
RAG-Stress 提出一个受控诊断协议，通过编辑检索证据中的单一断言来诱导错误答案，量化模型在检索增强生成中被误导证据覆盖正确答案的比率（misleading rate），并发现严格遵循文档的指令会系统性提高误导率。
AI Monster Brawl sent the same prompt to all 325 OpenRouter text models to build a monster fighter within a 1000-point budget; a third failed on the first try, mostly by overspending, and 7% never produced a valid fighter. The post analyzes common failure modes like budget overruns, invalid parts, and malformed JSON.
The paper introduces sk-bench, a native-first benchmark for evaluating large language models in Slovak, covering 30 datasets (33 task variants) across ten skill categories to address the lack of native Slovak coverage in existing multilingual benchmarks.
Emo-Jev is a training-free framework that uses Jev's probabilistic decision interface for text classification, decomposing tasks into atomic judgments or aggregating multiple judgment paths to improve over direct prediction, evaluated across eight sentiment, emotion, sarcasm, and humor datasets against five state-of-the-art LLMs.
The paper proposes a pipeline that automatically generates ontology-grounded, reasoner-verified multiple-choice benchmarks for evaluating LLM logical reasoning in scientific AI, validated on three ontologies (Pizza, PMDco, DOID) across six LLMs.
The paper introduces Multilingual GSM-Symbolic, a symbolic-template dataset covering 15 languages and 30k item-matched QA pairs, to quantify what determines cross-lingual capability transfer in LLMs. It finds model size, language resource level, reasoning, and typological distance are key drivers, and shows a framework explaining 92% of between-language variation while predicting unseen-language performance within 6.0pp.
MedKIT is a new benchmark evaluating how well LLMs integrate and apply updated clinical knowledge across lexical, relational, compositional, and operational probes. A large-scale study of 12 knowledge integration strategies across 5 models reveals a persistent gap between factual recall and usable knowledge, with little improvement on compositional or operational reasoning.
A same-harness benchmark on a Mac Studio M5 Ultra 256GB compares Qwen3.8-Flash-Next (182GB oQ8e) vs Laguna-S-2.1 GGUF at up to 262K context. Qwen sustains ~4,200 tok/s linear prefill and stable decode, while Laguna degrades superlinearly; quality is a 4/4 draw with very different answer styles.