Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models
Summary
Researchers present SemanticQA, a benchmark for evaluating language models on semantic phrase processing tasks including idioms, noun compounds, and verbal constructions, revealing significant performance variation across model architectures and scales on semantic reasoning tasks.
View Cached Full Text
Cached at: 04/21/26, 07:03 AM
# Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models Source: [https://arxiv.org/abs/2604.16593](https://arxiv.org/abs/2604.16593) [View PDF](https://arxiv.org/pdf/2604.16593) > Abstract:We present SemanticQA, an evaluation suite designed to assess language models \(LMs\) in semantic phrase processing tasks\. The benchmark consolidates existing multiword expression \(MwE\) resources and reorganizes them into a unified testbed\. It covers both general lexical phenomena, such as lexical collocations, and three fine\-grained categories: idiomatic expressions, noun compounds, and verbal constructions\. Through SemanticQA, we assess LMs of diverse architectures and scales in extraction, classification, and interpretation tasks, as well as sequential task compositions\. We reveal substantial performance variation, particularly on tasks requiring semantic reasoning, highlighting differences in reasoning efficacy and semantic understanding of LMs, providing insights for pushing LMs with stronger comprehension on non\-trivial semantic phrases\. The evaluation harness and data of SemanticQA are available at[this https URL](https://github.com/jacklanda/SemanticQA)\. ## Submission history From: Yang Liu \[[view email](https://arxiv.org/show-email/ed5e06c8/2604.16593)\] **\[v1\]**Fri, 17 Apr 2026 17:56:21 UTC \(574 KB\)
Similar Articles
Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
This survey synthesizes recent advancements in mathematical reasoning with large language models, covering benchmarks, architectures, training strategies, and evaluation protocols. It identifies key challenges such as reasoning faithfulness and benchmark biases.
From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models
This paper introduces MPAR-Bench, a bilingual benchmark for evaluating multi-point associative reasoning in large language models, along with a perturbation suite and coarse-to-fine evaluation protocol. Results show that deeper reasoning does not automatically confer robust reasoning breadth.
Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment
This paper introduces UPBench, a benchmark to evaluate large language models on urban planning knowledge across four knowledge pillars and five cognitive levels, finding that models perform better on higher-order analysis than factual recall, and identifying epistemic limitations such as regulatory hallucination and phronetic deficit.
When Absence Is Evidence: Evaluating Completeness-Sensitive Negative Reasoning in Large Language Models
Introduces CROWN-QA, a benchmark for completeness-sensitive negative reasoning in LLMs, showing models struggle to distinguish justified negative answers from insufficient evidence, often over-closing.
ClosureBench: A Constructive Benchmark for Compositional Graph Reasoning
ClosureBench is a constructive benchmark for evaluating language models on compositional graph reasoning tasks, revealing that frontier models degrade with complexity while a fine-tuned model maintains performance through program synthesis.