Tag
This paper presents SRBench, a benchmark dataset for evaluating LLM performance in systematic review screening, and introduces PromptSR, a tool for managing prompt experiments and analysis.
The article presents results from an 8-hour test comparing 9 LLMs on a web-development prompt, focusing on which local models can match frontier AI performance on an RTX 3060 12GB GPU, with detailed generation times and practical insights.
A benchmark release comparing 24 LLMs to human writers on 475 creative writing prompts, using a custom reward model to show that frontier models outperform amateurs but not professionals.
The paper introduces KoNeoBench, a curated benchmark dataset for evaluating large language models' understanding of Korean neologisms, based on 1,785 entries from online news since 2020. It reveals limitations in current LLMs in handling recent lexical changes and specific Korean linguistic properties.
The Neo-Classic benchmark evaluates linguistic-aesthetic reasoning in classical Chinese poetry using contemporary poems to test LLMs, revealing performance gaps and difficulties in global hierarchical planning.
GoBench is a benchmark for evaluating large language models on 9x9 Go games, demonstrating strong correlation with ARC-AGI and featuring a leaderboard with KataGo opponents from random to superhuman levels.
This paper introduces Japanese Stroke LLM Evaluation, a conversational benchmark for assessing large language models in safe Japanese stroke care, showing that Claude Fable 5 achieved the highest score and met safety thresholds while other models made critical mistakes.
IBBench-Light is a paired evaluation benchmark that tests how large language models handle external directives based on user requests, revealing that marginal accuracies can be misleading when assessing task-conditioned responses.
This paper introduces LogiMed-RoB, a benchmark for evaluating large language models' hierarchical logical consistency in medical risk-of-bias assessment, revealing that high atomic consistency can conceal critical reasoning flaws in clinical deployment.
SemVerBench is the first benchmark evaluating LLMs' comprehension of version-constraint semantics across npm, PEP 440, and Cargo, revealing systematic blind spots and suggesting tool delegation for accurate resolution.
This post discusses the recent release of the K2-Horizon-MoVA-36B-A4B MoE model and asks about its current usage and potential use cases, providing a link to model comparisons.
A developer built a benchmark called lolbench to evaluate LLMs on understanding and creating jokes, revealing models are strong at explaining real jokes but struggle with failed ones, and incorporating human voting for preference testing.
Φ-Bench is a new benchmark for evaluating large language models on engineering and optimizing AI infrastructure, covering tasks from kernel-level code completion to end-to-end system optimization.
The paper presents 5-Dialects-BN, a manually annotated benchmark dataset for five Bangla dialects with aligned transliterations and translations, designed to improve evaluation of dialect-aware LLMs.
SWORD introduces a Wikidata-based benchmark to evaluate cross-lingual inconsistencies in LLMs' rejection of factual errors, revealing that models rely on distributional familiarity and show performance degradation in Asian languages.
Φ-Bench is a benchmark designed to evaluate large language models on their ability to engineer and optimize the LLM infrastructure stack, covering tasks from kernel optimization to end-to-end system design.
The paper presents Chiaro, a new benchmark dataset for contrastive emotion recognition where two individuals experience opposing emotions from a shared event, grounded in appraisal theory. It evaluates seven LLMs and four emotion classifiers, revealing that current models fall short of human performance.
An LLM benchmark's scoring pipeline could erroneously treat unsupported tool-call responses as stable empty outputs due to preprocessing, highlighting the importance of distinguishing between parsing failures and genuine outputs in evaluation.
This article benchmarks various LLMs on their ability to identify mushroom species from images using a large dataset, highlighting accuracy issues and safety concerns for real-world applications.
RegDivergence-101 is a pilot benchmark for detecting cross-jurisdiction regulatory contradictions between FDA and EMA using LLMs, establishing baseline methods with varying performance in classifying regulatory relationships.