FrameBench:A Language Understanding Benchmark Based on Frame Semantics

# FrameBench:A Language Understanding Benchmark Based on Frame Semantics
Source: [https://arxiv.org/html/2609.03370](https://arxiv.org/html/2609.03370)
\\setCJKmainfont

HaranoAjiMincho\-Regular\.otf\[ BoldFont = HaranoAjiMincho\-Bold\.otf \]

Ryohei SasanoAffiliation:Graduate School of Informatics, Nagoya UniversityEmail:[yano\.chihiro\.j3@s\.mail\.nagoya\-u\.ac\.jp](mailto:)sasano@i\.nagoya\-u\.ac\.jp

###### Abstract

In frame semantics, sentence comprehension is assumed to proceed by relating lexical meaning to background knowledge called semantic frames, thereby enabling readers to implicitly enrich the text with unstated information\. Recent large language models \(LLMs\) have achieved strong performance across a wide range of downstream tasks\. However, it remains unclear whether they can reproduce the kinds of implicit enrichment that humans naturally make during comprehension\. To address this question, we introduce FrameBench, a benchmark grounded in frame semantics\. FrameBench consists of multiple\-choice questions that test whether models distinguish the frames evoked by the same verb across contexts\. We construct the benchmark for English and Japanese using FrameNet\-style resources and a generation\-and\-verification pipeline with native\-speaker judgments\.Our experiments on a diverse set of models reveal challenges for small models, while several large modelssurpass the human reference scores\.We release the constructed FrameBench dataset and the code for dataset construction and evaluation at[https://github\.com/SasanoLab/FrameBench](https://github.com/SasanoLab/FrameBench)

## 1Introduction

The same verb can evoke different situations depending on context\. Consider these examples:

1. 1\.Heleftthe bank after talking with a friend\.
2. 2\.Heleftthe bank at the age of sixty\.

Although both sentences contain the same verb,left, they evoke different situations\. In Sentence[1](https://arxiv.org/html/2609.03370#S1.I1.i1),leftdescribes a physical departure from the bank as a location\. In Sentence[2](https://arxiv.org/html/2609.03370#S1.I1.i2), by contrast, it evokes a quitting interpretation rather than physical departure\. Under this interpretation, readers infer an employment relation that is not explicitly stated\. This kind of context\-dependent enrichment is central to frame semantics[Fillmore \(1982\)](https://arxiv.org/html/2609.03370#bib.bib4)\.

Figure 1:An example from FrameBench\.Despite the strong performance of recent large language models \(LLMs\) across a wide range of downstream tasks, it remains unclear whether they can reliably perform this kind of implicit, context\-dependent enrichment\. Most existing evaluations still rely on broad benchmark suites built from diverse downstream tasks, which do not directly test this capability[Wang et al\. \(2018\)](https://arxiv.org/html/2609.03370#bib.bib14);[Wang et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib17)\.To address this gap, we introduce FrameBench, a multiple\-choice benchmark that evaluates whether LLMs can distinguish context\-dependent frame\-semantic interpretations evoked by the same verb\. Rather than requiring explicit frame\-label prediction, FrameBench probes such distinctions indirectly through natural\-language questions about the situations implied by each sentence\.

In this work, we construct FrameBench for English and Japanese, two typologically distant languages, using their respective FrameNet resources[Ruppenhofer et al\. \(2016\)](https://arxiv.org/html/2609.03370#bib.bib5);[Ohara et al\. \(2004\)](https://arxiv.org/html/2609.03370#bib.bib3)\.FrameBench is built through LLM\-based generation and native\-speaker validation\. Importantly, the generation process is grounded in human\-authored resources rather than solely in the implicit knowledge of LLMs\.We also evaluate a diverse set of LLMs on the resulting benchmarks to analyze whether they can distinguish the background knowledge evoked by the same verb across contexts\.

Figure[1](https://arxiv.org/html/2609.03370#S1.F1)shows an example item from FrameBench\. The verbleftappears in both sentences, and the prompt asks the model to select all sentences that describe Noah quitting his job\. The correct answer is “1:SA\\text\{S\}\_\{\\text\{A\}\}” because onlySA\\text\{S\}\_\{\\text\{A\}\}expresses quitting, whereasSB\\text\{S\}\_\{\\text\{B\}\}describes physical departure\. The item cannot be solved by matching the predicate alone\. It requires context\-sensitive frame\-semantic interpretation\.

We make the following contributions:

- •We introduce FrameBench, a FrameNet\-grounded multiple\-choice benchmark that evaluates whether LLMs can distinguish context\-dependent frame\-semantic interpretations evoked by the same verb\.
- •We construct English and Japanese versions of FrameBench through LLM\-based generation and native\-speaker validation\. We release the English and Japanese FrameBench datasets, along with the construction and evaluation code\.
- •We benchmark a wide range of LLMs, showing that FrameBench performance is strongly affected by model scale, reasoning mode, and language\. Through behavioral analyses and case studies, we identify error patterns and fine\-grained frame\-semantic distinctions that can still challenge high\-performing models\.

## 2Related Work

### 2\.1Evaluation of Language Models

Evaluating LLMs has attracted substantial attention in recent years\. Because LLM capabilities are multifaceted, evaluations often rely on benchmarks that bundle diverse downstream tasks rather than a single task[Wang et al\. \(2018\)](https://arxiv.org/html/2609.03370#bib.bib14);[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.03370#bib.bib13);[Wang et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib17);[Srivastava et al\. \(2023\)](https://arxiv.org/html/2609.03370#bib.bib23)\.

Some benchmarks more directly test whether models can distinguish meanings based on context\. For instance, Word Sense Disambiguation \(WSD\) tasks and the WiC dataset[Pilehvar and Camacho\-Collados \(2019\)](https://arxiv.org/html/2609.03370#bib.bib24)ask whether the same word has the same meaning in two different contexts\. Other approaches focus on sentence\-level meaning representations, such as AMR[Knight et al\. \(2017\)](https://arxiv.org/html/2609.03370#bib.bib25), by evaluating how well models can parse sentences into graph\-structured semantics\. However, our goal is to test whether a model can discriminate context\-dependent event interpretations evoked by the same verb\. Unlike standard WSD, which focuses on dictionary senses, our evaluation is grounded in frame semantics, capturing broader conceptual situations, which differs from the objectives of these existing resources\.

### 2\.2Frame\-Semantic Resources and Datasets

FrameNet is a lexical knowledge base grounded in frame semantics[Fillmore \(1982\)](https://arxiv.org/html/2609.03370#bib.bib4), built through manual annotation of corpus sentences with evoked frames and participant roles[Baker et al\. \(1998\)](https://arxiv.org/html/2609.03370#bib.bib6)\. It has been widely used for frame\-semantic parsing and related tasks such as semantic role labeling, and provides structured semantic information, including relations between frames\.

Beyond English, FrameNet\-style resources have been developed for many languages[Ohara et al\. \(2004\)](https://arxiv.org/html/2609.03370#bib.bib3);[Hahm et al\. \(2020\)](https://arxiv.org/html/2609.03370#bib.bib30);[You and Liu \(2005\)](https://arxiv.org/html/2609.03370#bib.bib31);[Djemaa et al\. \(2016\)](https://arxiv.org/html/2609.03370#bib.bib27);[Lyngfelt et al\. \(2018\)](https://arxiv.org/html/2609.03370#bib.bib32)\. More recently, datasets have been proposed that annotate multimodal data with frames and semantic roles[Belcavello et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib28);[Viridiano et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib29)\.

![Refer to caption](https://arxiv.org/html/2609.03370v1/fig2_pipe_2.png)Figure 2:Overview of the benchmark construction\. The numbered steps correspond to those described in Section[3\.2](https://arxiv.org/html/2609.03370#S3.SS2)\.\(a\)Example of Entry\(b\)Evaluation Format
Figure 3:Overview of the FrameBench task\.
### 2\.3Frame Semantics and LLMs

A growing body of work explores the use of LLMs for frame\-semantic analysis[Chundru et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib18);[Devasier et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib21);[Garat et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib22);[Yano et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib10);[Rai et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib19)\. Other studies attempt to extend or assist FrameNet annotation with LLMs[Belcavello et al\. \(2026\)](https://arxiv.org/html/2609.03370#bib.bib20);[Han et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib38)\. Overall, these results indicate that LLMs may be able to leverage frame\-related information from context to some extent\.

In addition, frameworks have been proposed to directly evaluate how well LLMs acquire conceptual structures grounded in frame semantics\.[Guo et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib26)propose NutFrame, which evaluates whether LLMs can induce explicit frame\-semantic structures from FrameNet, and report that such induction remains challenging\. Moreover, evaluating generative or parsing tasks often suffers from formatting inconsistencies in LLM outputs\. In contrast, FrameBench evaluates the discrimination of context\-dependent event interpretations evoked by the same verb as a multiple\-choice task\. By avoiding tasks that require explicit prediction of frame labels or role inventories, FrameBench more directly evaluates whether models distinguish context\-dependent event interpretations evoked by the same verb\.

## 3FrameBench: Task Design and Dataset Construction

FrameBench is a four\-choice benchmark for evaluating whether a model can correctly distinguish between different semantic frames evoked by the same verb\. This section first defines the FrameBench task and its evaluation protocol, then describes the construction pipeline and its application to English and Japanese\.

### 3\.1Task Definition

As shown in Figure[3\(a\)](https://arxiv.org/html/2609.03370#S2.F3.sf1), each FrameBench entry consists of a questionQQand four candidate sentences \(SAS\_\{\\text\{A\}\},SBS\_\{\\text\{B\}\},SA’S\_\{\\text\{A'\}\},SB’S\_\{\\text\{B'\}\}\), constructed from a polysemous verb and a pair of semantic frames \(FrameA\\text\{Frame\}\_\{\\text\{A\}\},FrameB\\text\{Frame\}\_\{\\text\{B\}\}\)\. The questionQQtargetsFrameA\\text\{Frame\}\_\{\\text\{A\}\}:SAS\_\{\\text\{A\}\}andSA’S\_\{\\text\{A'\}\}are designed to evokeFrameA\\text\{Frame\}\_\{\\text\{A\}\}, whileSBS\_\{\\text\{B\}\}andSB’S\_\{\\text\{B'\}\}are designed to evoke the contrastingFrameB\\text\{Frame\}\_\{\\text\{B\}\}\.During dataset construction, we use human\-authored frame\-semantic resources that specify each frame’s name, definition, core frame elements, and example sentences\. Thus, the semantic distinctions in FrameBench are grounded in external frame\-semantic resources rather than in an LLM’s internal knowledge alone\. Since these resources are not provided at evaluation time, solving the task using only the model’s internal knowledge is non\-trivial\.Each entry also includes human evaluation scores for acceptability and correctness\.

For evaluation, the four sentences are split into two predefined pairs, each containing exactly one sentence that evokes the target frame: the base pair \(SAS\_\{\\text\{A\}\},SBS\_\{\\text\{B\}\}\), which is constructed to be surface\-similar and therefore more challenging, and the extended pair \(SA’S\_\{\\text\{A'\}\},SB’S\_\{\\text\{B'\}\}\), as illustrated in Figure[3\(b\)](https://arxiv.org/html/2609.03370#S2.F3.sf2)\.Given the questionQQand an evaluation pair, the model selects one of four labels:Sentence A,Sentence B,Both Sentences, orNeither Sentence\.Although exactly one sentence is correct in each pair, we include the two dummy options to reduce noise from forced binary choices\.

### 3\.2Construction Pipeline

Figure[2](https://arxiv.org/html/2609.03370#S2.F2)provides an overview of the benchmark construction pipeline, which consists of the following three steps:

#### Step 1: Construction of Questions andBaseSentence Pairs

We extract polysemous verbs and their evoked frame pairs from the target language’s frame\-semantic resource\. For each verbVVand frame pair \(FrameA\\text\{Frame\}\_\{\\text\{A\}\},FrameB\\text\{Frame\}\_\{\\text\{B\}\}\), we use an LLM to generate the base sentence pair \(SAS\_\{\\text\{A\}\},SBS\_\{\\text\{B\}\}\) and a questionQQfor which onlySAS\_\{\\text\{A\}\}is the correct answer\. The generation is conditioned on the frame names, definitions, core elements, and example usages of both frames\. To increase task difficulty, we instruct the LLM to makeSAS\_\{\\text\{A\}\}andSBS\_\{\\text\{B\}\}as lexically similar as possible while preserving their distinct frame assignments\.

#### Step 2: Expansion of Target Sentence Pairs

To diversify the evaluation set, we generate additional sentences to form the extended pair \(SA’S\_\{\\text\{A'\}\},SB’S\_\{\\text\{B'\}\}\)\. Unlike the base pair, the extended pair is not constrained to be surface\-similar\. For eachFramex\\text\{Frame\}\_\{\\text\{x\}\}\(x∈\{A, B\}x\\in\\\{\\text\{A, B\}\\\}\), we generate an additional sentenceSx’S\_\{\\text\{x'\}\}using theFramex\\text\{Frame\}\_\{\\text\{x\}\}information, the questionQQ, and the original sentenceSxS\_\{\\text\{x\}\}as input\.Sx’S\_\{\\text\{x'\}\}is constrained to evoke the same frame asSxS\_\{\\text\{x\}\}\. Thus,Sx’S\_\{\\text\{x'\}\}retains the same ground\-truth label asSxS\_\{\\text\{x\}\}with respect toQQ\. For example,SA’S\_\{\\text\{A'\}\}is a correct answer toQQ, matching the label ofSAS\_\{\\text\{A\}\}\.

#### Step 3: Human Validation and Filtering

To ensure benchmark quality, native speakers of the target language manually evaluated the constructed items\. The evaluation consisted of two components: a correctness judgment, implemented as a four\-choice task, and an acceptability judgment of the descriptions\.Model evaluation uses only single\-correct pairs, whereas human evaluation additionally includes auxiliary both\-correct pairs \(SA,SA’S\_\{\\text\{A\}\},S\_\{\\text\{A'\}\}\) and neither\-correct pairs \(SB,SB’S\_\{\\text\{B\}\},S\_\{\\text\{B'\}\}\) to reduce annotator bias toward selecting a single sentence\. These auxiliary pairs are used only for validation and are excluded from entry\-level scoring\. Correctness and acceptability scores are computed over the two single\-correct pairs by taking the minimum across pairs for each dimension\.

### 3\.3English and Japanese Versions

We constructed English and Japanese versions of FrameBench\.

#### Resources and Generation Setup

As sources of frame knowledge, we used FrameNet[Ruppenhofer et al\. \(2016\)](https://arxiv.org/html/2609.03370#bib.bib5)for English and Japanese FrameNet[Ohara et al\. \(2004\)](https://arxiv.org/html/2609.03370#bib.bib3)for Japanese\. In both languages, GPT\-5111Model version: 2025\-08\-07[OpenAI \(2025\)](https://arxiv.org/html/2609.03370#bib.bib11)was used for generation in Step 1 and Step 2\. Table[1](https://arxiv.org/html/2609.03370#S3.T1)summarizes the resource and dataset statistics\. Due to differences in resource scale, we randomly sampled 800 frame pairs for English, while using all 335 eligible frame pairs for Japanese\. For Japanese, we generated two entries per frame pair \(k=2k\{=\}2\) to ensure sufficient dataset size\. During construction, we removed invalid generations, yielding 731 final items for English and 549 for Japanese\.

MetricEnglishJapaneseSource FrameNet\# Frame Pairs \(PP\)800335\# Unique LUs430139\# Entries per Pair \(kk\)12FrameBench\# Candidate Items \(P×kP\\times k\)800670\# Final Items731549\# Unique LUs407128Table 1:Statistics of the FrameBench dataset\.\(a\)English\#Accpt\.0123Total\#Corr\.00381121241510312518486313431155201287554Total1880272361731
\(b\)Japanese\#Accpt\.0123Total\#Corr\.0001910101726342011211713030434337375Total0654489549

Table 2:Distribution of entries by the numbers of annotators who judged each entry correct and acceptable\. Highlighted cells indicate the high\-quality subset used for evaluation\.
#### Manual Revision

Compared with the English outputs, the Japanese outputs tended to contain less natural phrasing\. Therefore, one of the authors, a native Japanese speaker, manually revised the generated Japanese descriptions after Steps 1 and 2\. Low\-acceptability items were further revised after the initial human validation in Step 3 and re\-evaluated by annotators\.

#### Dataset Statistics and Human Validation Results

Table[2](https://arxiv.org/html/2609.03370#S3.T2)summarizes the distribution of human validation scores for FrameBench entries\. Each entry consists of a question and two sentence pairs: the base pair and the extended pair\. Following the filtering criterion described in Step 3, we retain only entries that at least two annotators judged correct and at least two judged acceptable\. This subset defines the main evaluation set used throughout our experiments\. As shown by the blue\-highlighted cells in Tables[2\(a\)](https://arxiv.org/html/2609.03370#S3.T2.st1)and[2\(b\)](https://arxiv.org/html/2609.03370#S3.T2.st2), the resulting set contains 599 English entries and 500 Japanese entries\. Detailed inter\-annotator agreement statistics are provided in Appendix[A\.3](https://arxiv.org/html/2609.03370#A1.SS3)\.

Modelℛ\\mathcal\{R\}EnglishJapaneseFrameBenchMMLU\-GPQA\-HLEFrameFrameBenchJamC\-MMLU\-FrameProDiamondIdent\.QAProXIdent\.Human\-96\.396\.3±\\pm1\.91\.9−\-\\phantom\{\{\}^\{\\text\{a\}\}\}\-−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}94\.994\.9±\\pm3\.73\.7−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}Closed\-Source ModelsGPT\-5 nano✓\\checkmark93\.593\.5±\\pm0\.80\.8−\-\\phantom\{\{\}^\{\\text\{a\}\}\}67\.0c67\.0^\{\\text\{c\}\}7\.6c7\.6^\{\\text\{c\}\}77\.877\.8\\phantom\{\{\}^\{\\text\{a\}\}\}83\.983\.9±\\pm4\.44\.4−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}76\.376\.3\\phantom\{\{\}^\{\\text\{a\}\}\}GPT\-5✓\\checkmark99\.199\.1±\\pm0\.20\.286\.5e86\.5^\{\\text\{e\}\}84\.2c84\.2^\{\\text\{c\}\}23\.5c23\.5^\{\\text\{c\}\}84\.084\.0\\phantom\{\{\}^\{\\text\{a\}\}\}96\.396\.3±\\pm2\.02\.085\.8e85\.8^\{\\text\{e\}\}84\.9e84\.9^\{\\text\{e\}\}78\.378\.3\\phantom\{\{\}^\{\\text\{a\}\}\}Gemini 3\.1 Flash\-Lite✓\\checkmark97\.697\.6±\\pm0\.40\.486\.2a86\.2^\{\\text\{a\}\}82\.2c82\.2^\{\\text\{c\}\}16\.2c16\.2^\{\\text\{c\}\}80\.080\.0\\phantom\{\{\}^\{\\text\{a\}\}\}97\.897\.8±\\pm0\.20\.2−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}72\.372\.3\\phantom\{\{\}^\{\\text\{a\}\}\}Gemini 3\.1 Pro✓\\checkmark99\.399\.3±\\pm0\.20\.291\.2b91\.2^\{\\text\{b\}\}94\.1c94\.1^\{\\text\{c\}\}44\.7c44\.7^\{\\text\{c\}\}82\.582\.5\\phantom\{\{\}^\{\\text\{a\}\}\}97\.597\.5±\\pm0\.70\.7−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}71\.571\.5\\phantom\{\{\}^\{\\text\{a\}\}\}Open\-Weight ModelsGemma 4 E2B✓\\checkmark86\.386\.3±\\pm1\.91\.960\.0a60\.0^\{\\text\{a\}\}43\.3c43\.3^\{\\text\{c\}\}4\.8c4\.8^\{\\text\{c\}\}74\.874\.8\\phantom\{\{\}^\{\\text\{a\}\}\}82\.382\.3±\\pm2\.82\.836\.1e36\.1^\{\\text\{e\}\}58\.0e58\.0^\{\\text\{e\}\}72\.372\.3\\phantom\{\{\}^\{\\text\{a\}\}\}Gemma 4 E4B✓\\checkmark94\.594\.5±\\pm1\.11\.169\.4a69\.4^\{\\text\{a\}\}57\.6c57\.6^\{\\text\{c\}\}3\.7c3\.7^\{\\text\{c\}\}80\.880\.8\\phantom\{\{\}^\{\\text\{a\}\}\}92\.292\.2±\\pm1\.01\.042\.4e42\.4^\{\\text\{e\}\}67\.0e67\.0^\{\\text\{e\}\}76\.876\.8\\phantom\{\{\}^\{\\text\{a\}\}\}Gemma 4 31B✓\\checkmark98\.698\.6±\\pm0\.30\.385\.2a85\.2^\{\\text\{a\}\}85\.7c85\.7^\{\\text\{c\}\}22\.7c22\.7^\{\\text\{c\}\}82\.082\.0\\phantom\{\{\}^\{\\text\{a\}\}\}99\.199\.1±\\pm0\.10\.168\.6e68\.6^\{\\text\{e\}\}84\.1e84\.1^\{\\text\{e\}\}73\.873\.8\\phantom\{\{\}^\{\\text\{a\}\}\}Gemma 4 E2B\-68\.668\.6±\\pm21\.521\.557\.9f57\.9^\{\\text\{f\}\}40\.5c40\.5^\{\\text\{c\}\}4\.5c4\.5^\{\\text\{c\}\}74\.074\.0\\phantom\{\{\}^\{\\text\{a\}\}\}41\.541\.5±\\pm11\.011\.033\.7f33\.7^\{\\text\{f\}\}50\.5f50\.5^\{\\text\{f\}\}67\.867\.8\\phantom\{\{\}^\{\\text\{a\}\}\}Gemma 4 E4B\-82\.582\.5±\\pm20\.120\.167\.0f67\.0^\{\\text\{f\}\}54\.9c54\.9^\{\\text\{c\}\}4\.7c4\.7^\{\\text\{c\}\}71\.571\.5\\phantom\{\{\}^\{\\text\{a\}\}\}84\.684\.6±\\pm5\.15\.143\.4f43\.4^\{\\text\{f\}\}63\.6f63\.6^\{\\text\{f\}\}68\.568\.5\\phantom\{\{\}^\{\\text\{a\}\}\}Gemma 4 31B\-97\.597\.5±\\pm0\.50\.583\.8f83\.8^\{\\text\{f\}\}76\.3c76\.3^\{\\text\{c\}\}11\.5c11\.5^\{\\text\{c\}\}81\.081\.0\\phantom\{\{\}^\{\\text\{a\}\}\}97\.697\.6±\\pm0\.50\.567\.6f67\.6^\{\\text\{f\}\}80\.0f80\.0^\{\\text\{f\}\}71\.571\.5\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-0\.8B✓\\checkmark33\.333\.3±\\pm7\.27\.242\.3a42\.3^\{\\text\{a\}\}11\.1c11\.1^\{\\text\{c\}\}1\.2c1\.2^\{\\text\{c\}\}41\.041\.0\\phantom\{\{\}^\{\\text\{a\}\}\}24\.224\.2±\\pm1\.51\.524\.5e24\.5^\{\\text\{e\}\}23\.6e23\.6^\{\\text\{e\}\}39\.339\.3\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-2B✓\\checkmark80\.380\.3±\\pm1\.11\.166\.5a66\.5^\{\\text\{a\}\}59\.8c59\.8^\{\\text\{c\}\}5\.1c5\.1^\{\\text\{c\}\}68\.068\.0\\phantom\{\{\}^\{\\text\{a\}\}\}60\.760\.7±\\pm3\.33\.328\.3e28\.3^\{\\text\{e\}\}39\.1e39\.1^\{\\text\{e\}\}54\.354\.3\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-4B✓\\checkmark95\.595\.5±\\pm0\.50\.579\.1a79\.1^\{\\text\{a\}\}68\.8c68\.8^\{\\text\{c\}\}6\.7c6\.7^\{\\text\{c\}\}81\.081\.0\\phantom\{\{\}^\{\\text\{a\}\}\}89\.989\.9±\\pm2\.52\.539\.5e39\.5^\{\\text\{e\}\}75\.0e75\.0^\{\\text\{e\}\}71\.071\.0\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-9B✓\\checkmark97\.397\.3±\\pm0\.20\.282\.5a82\.5^\{\\text\{a\}\}77\.8c77\.8^\{\\text\{c\}\}7\.5c7\.5^\{\\text\{c\}\}81\.381\.3\\phantom\{\{\}^\{\\text\{a\}\}\}91\.891\.8±\\pm2\.42\.448\.9e48\.9^\{\\text\{e\}\}78\.4e78\.4^\{\\text\{e\}\}72\.072\.0\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-27B✓\\checkmark98\.698\.6±\\pm0\.20\.286\.1a86\.1^\{\\text\{a\}\}87\.5c87\.5^\{\\text\{c\}\}16\.6c16\.6^\{\\text\{c\}\}80\.880\.8\\phantom\{\{\}^\{\\text\{a\}\}\}97\.397\.3±\\pm0\.90\.959\.1e59\.1^\{\\text\{e\}\}83\.5e83\.5^\{\\text\{e\}\}70\.870\.8\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-0\.8B\-47\.647\.6±\\pm8\.68\.629\.7a29\.7^\{\\text\{a\}\}23\.6c23\.6^\{\\text\{c\}\}4\.9c4\.9^\{\\text\{c\}\}49\.549\.5\\phantom\{\{\}^\{\\text\{a\}\}\}25\.425\.4±\\pm2\.42\.424\.0f24\.0^\{\\text\{f\}\}20\.9f20\.9^\{\\text\{f\}\}32\.532\.5\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-2B\-48\.548\.5±\\pm11\.011\.055\.3a55\.3^\{\\text\{a\}\}43\.8c43\.8^\{\\text\{c\}\}4\.9c4\.9^\{\\text\{c\}\}53\.353\.3\\phantom\{\{\}^\{\\text\{a\}\}\}30\.030\.0±\\pm3\.33\.328\.5f28\.5^\{\\text\{f\}\}36\.2f36\.2^\{\\text\{f\}\}40\.040\.0\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-4B\-81\.181\.1±\\pm1\.91\.969\.8f69\.8^\{\\text\{f\}\}71\.2c71\.2^\{\\text\{c\}\}7\.5c7\.5^\{\\text\{c\}\}45\.545\.5\\phantom\{\{\}^\{\\text\{a\}\}\}56\.756\.7±\\pm4\.84\.837\.7f37\.7^\{\\text\{f\}\}60\.1f60\.1^\{\\text\{f\}\}45\.045\.0\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-9B\-85\.885\.8±\\pm3\.23\.273\.1f73\.1^\{\\text\{f\}\}78\.6c78\.6^\{\\text\{c\}\}8\.6c8\.6^\{\\text\{c\}\}64\.064\.0\\phantom\{\{\}^\{\\text\{a\}\}\}72\.972\.9±\\pm3\.93\.946\.9f46\.9^\{\\text\{f\}\}65\.5f65\.5^\{\\text\{f\}\}47\.847\.8\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\.5\-27B\-96\.196\.1±\\pm0\.50\.578\.3f78\.3^\{\\text\{f\}\}84\.2c84\.2^\{\\text\{c\}\}13\.2c13\.2^\{\\text\{c\}\}74\.874\.8\\phantom\{\{\}^\{\\text\{a\}\}\}83\.783\.7±\\pm3\.13\.155\.9f55\.9^\{\\text\{f\}\}74\.5f74\.5^\{\\text\{f\}\}56\.556\.5\\phantom\{\{\}^\{\\text\{a\}\}\}Japanese\-Oriented ModelsLLM\-jp 4 8B✓\\checkmark−\-−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}84\.584\.5±\\pm5\.85\.851\.4e51\.4^\{\\text\{e\}\}62\.3e62\.3^\{\\text\{e\}\}75\.075\.0\\phantom\{\{\}^\{\\text\{a\}\}\}LLM\-jp 4 8B\-−\-−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}69\.2\{\\color\[rgb\]\{0,0,0\}69\.2\}±\\pm8\.88\.847\.1f47\.1^\{\\text\{f\}\}46\.1f46\.1^\{\\text\{f\}\}33\.333\.3\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3 Swallow 8B✓\\checkmark−\-−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}84\.084\.0±\\pm4\.44\.446\.9e46\.9^\{\\text\{e\}\}70\.8e70\.8^\{\\text\{e\}\}78\.878\.8\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3 Swallow 32B✓\\checkmark−\-−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}93\.293\.2±\\pm1\.11\.151\.8e51\.8^\{\\text\{e\}\}76\.1e76\.1^\{\\text\{e\}\}76\.376\.3\\phantom\{\{\}^\{\\text\{a\}\}\}Models with Shared Base ModelsQwen3\-8B✓\\checkmark−\-−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}85\.085\.0±\\pm6\.46\.440\.1e40\.1^\{\\text\{e\}\}71\.1e71\.1^\{\\text\{e\}\}69\.069\.0\\phantom\{\{\}^\{\\text\{a\}\}\}Qwen3\-32B✓\\checkmark−\-−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}−\-\\phantom\{\{\}^\{\\text\{a\}\}\}93\.993\.9±\\pm0\.80\.847\.2e47\.2^\{\\text\{e\}\}75\.9e75\.9^\{\\text\{e\}\}71\.371\.3\\phantom\{\{\}^\{\\text\{a\}\}\}Table 3:FrameBench results with comparison benchmark scores\. Model scores on FrameBench are reported as mean±\\pmstandard deviation over five prompt templates, while human scores are reported as mean±\\pmstandard deviation across three annotators\.ℛ\\mathcal\{R\}denotes reasoning mode, and Frame Ident\. denotes the frame identification task\. Rows under “Models with Shared Base Models” share base models with the Qwen3 Swallow models\. For non\-FrameBench scores, superscripts indicate the source of each score:amodel\-provider reports,b[MMLU\-Pro leaderboard](https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro),c[Artificial Analysis](https://artificialanalysis.ai/),e[Swallow LLM Leaderboard](https://swallow-llm.github.io/leaderboard/index-post.en.html), andfour own evaluations\.

## 4Evaluation on FrameBench

We evaluate LLMs on the English and Japanese versions of FrameBench and compare their performance with existing benchmarks\.

### 4\.1Experimental Setup

We evaluate models on the English and Japanese main evaluation sets described in Section[3\.3](https://arxiv.org/html/2609.03370#S3.SS3)\. For each entry, we evaluate both the base pair and the extended pair\. To mitigate position bias, each pair is tested in both original and swapped orders\. We report accuracy as the mean and standard deviation across five prompt templates per language\. Answer options are randomly ordered for each question\.

We test closed\-source, open\-weight, and Japanese\-oriented models, using constrained or structured decoding for reliable answer extraction\. As the human score, we report accuracy computed from the human validation results described in Section[3\.3](https://arxiv.org/html/2609.03370#S3.SS3)\. Since the evaluation subset is filtered to entries answered correctly by at least two annotators, the human score may be upwardly biased\. Note that an independently measured human score may be lower if the same task were administered in a separate evaluation round\.

For comparison, we report results on widely used LLM evaluation benchmarks: MMLU\-Pro[Wang et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib17), GPQA\-Diamond[Rein et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib16), Humanity’s Last Exam \(HLE\)[Center for AI Safety et al\. \(2026\)](https://arxiv.org/html/2609.03370#bib.bib15), JamC\-QA[Oka et al\. \(2026\)](https://arxiv.org/html/2609.03370#bib.bib1), and MMLU\-ProX[Xuan et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib2)\. We additionally evaluate a frame identification task as a complementary measure of models’ ability to identify frames from sentences\. Additional details of the experimental setup are provided in Appendix[B](https://arxiv.org/html/2609.03370#A2)\.

### 4\.2Results

Table[3](https://arxiv.org/html/2609.03370#S3.T3)shows the results of the English and Japanese evaluations\. Across both languages, FrameBench performance increases with model scale and reasoning\. Several high\-performing models surpass the human reference scores, despite the potential upward bias in the human scores\. The scale and reasoning trends are broadly consistent with the results on the comparison benchmarks\. However, reasoning gains tend to be larger on FrameBench than on several of these benchmarks\.

We first focus on the English results\. Among closed\-source models, Gemini 3\.1 Pro achieves the highest score, reaching 99\.3, followed by GPT\-5 at 99\.1\. Among open\-weight models, Qwen3\.5\-27B and Gemma 4 31B perform best, both reaching 98\.6 with reasoning\. All of these scores exceed the human reference score of 96\.3\. Across the results, larger models consistently obtain higher scores, and reasoning improves performance for all matched open\-weight models except Qwen3\.5\-0\.8B\. The gains are particularly large for smaller and mid\-sized models\. For example, enabling reasoning improves Qwen3\.5\-2B from 48\.5 to 80\.3 and Qwen3\.5\-4B from 81\.1 to 95\.5\.

Compared with existing LLM benchmarks, FrameBench shows broadly consistent model rankings\. Frame identification scores also show a broadly similar trend to FrameBench, but with several reversals relative to the comparison benchmarks\. Together, these patterns suggest that FrameBench reflects general language\-understanding ability while remaining grounded in frame\-semantic interpretation\. This makes FrameBench a more suitable evaluation setting than direct frame identification for assessing frame\-semantic language understanding in LLMs\. At the same time, FrameBench shows larger performance gains than the comparison benchmarks as model capability improves through scaling or reasoning\. For example, Qwen3\.5 with reasoning improves from 33\.3 at 0\.8B to 80\.3 at 2B in English, a much larger jump than the corresponding increase on MMLU\-Pro\. This pattern suggests that FrameBench is particularly sensitive to the ability to discriminate context\-dependent frame\-semantic interpretations\.

The LLM used in dataset construction could affect the resulting FrameBench entries and the evaluation results\. To examine this possibility, we constructed an additional 100 English FrameBench entries using Gemini 3\.1 Pro in place of GPT\-5 and evaluated them\. The overall performance pattern was similar to that observed on the original FrameBench\. Detailed results are provided in Appendix[C](https://arxiv.org/html/2609.03370#A3)\.

Turning to the Japanese results, we observe similar trends, with performance generally improving with model scale and reasoning\. This is in line with the results on MMLU\-ProX\. Gemma 4 31B achieves the highest score of 99\.1, followed by Gemini 3\.1 Flash\-Lite at 97\.8, Gemini 3\.1 Pro at 97\.5, Qwen3\.5\-27B at 97\.3, and GPT\-5 at 96\.3\. All of these scores exceed the human reference score of 94\.9\. Reasoning also yields sharp gains, improving Qwen3\.5\-2B from 30\.0 to 60\.7, Qwen3\.5\-4B from 56\.7 to 89\.9, Gemma 4 E2B from 41\.5 to 82\.3, and Gemma 4 E4B from 84\.6 to 92\.2\.

For Japanese\-oriented models, we compare Qwen3 Swallow models with the corresponding Qwen3 models, since they are based on the same Qwen3\-Base models\. Although the Swallow models outperform their Qwen3 counterparts on JamC\-QA, they perform comparably to or slightly below the corresponding Qwen3 models on FrameBench\. Specifically, Qwen3 Swallow 8B scores 84\.0 compared with 85\.0 for Qwen3\-8B, and Qwen3 Swallow 32B scores 93\.2 compared with 93\.9 for Qwen3\-32B\. This contrast suggests that target\-language training does not necessarily improve frame\-semantic interpretation\.

Comparing the English and Japanese results, the human reference score decreases only slightly, from 96\.3 to 94\.9, whereas many LLMs show larger drops\. Moreover, the Japanese results separate models more clearly than the English results\. For example, strong reasoning models show small cross\-lingual gaps: Gemma 4 31B changes only from 98\.6 to 99\.1, and Qwen3\.5\-27B from 98\.6 to 97\.3\. In contrast, weaker models show larger drops, with GPT\-5 nano falling from 93\.5 to 83\.9 and Qwen3\.5\-9B without reasoning from 85\.8 to 72\.9\. These results suggest that Japanese FrameBench cannot be solved by the cross\-lingual transfer ability of lower\-capability models alone, and instead requires robust context\-dependent semantic interpretation in Japanese\.

\(i\) Sent\. Pair Type\(ii\) Correct Sent\. Position\(iii\) Error BreakdownModelℛ\\mathcal\{R\}Acc\.BaseExt\.ΔB−E\\Delta\_\{B\-E\}1st2ndΔ1−2\\Delta\_\{1\-2\}BothNeitherOppositeGemma 4 E2B\-68\.668\.666\.966\.970\.3−3\.3\-3\.377\.160\.160\.1\+17\.0\+17\.011\.711\.77\.77\.712\.0Gemma 4 E4B\-82\.582\.580\.880\.884\.1−3\.3\-3\.384\.280\.880\.8\+3\.4\+3\.46\.36\.34\.64\.66\.6Gemma 4 31B\-97\.597\.596\.896\.898\.2−1\.4\-1\.497\.597\.50\.00\.02\.20\.20\.20\.10\.1Gemma 4 E2B✓86\.386\.385\.485\.487\.1−1\.7\-1\.789\.283\.483\.4\+5\.7\+5\.710\.82\.12\.10\.80\.8Gemma 4 E4B✓94\.594\.593\.893\.895\.1−1\.4\-1\.495\.693\.493\.4\+2\.2\+2\.24\.60\.60\.60\.30\.3Gemma 4 31B✓98\.698\.698\.298\.299\.1−0\.9\-0\.998\.998\.398\.3\+0\.6\+0\.61\.10\.10\.10\.20\.2Qwen3\.5\-0\.8B\-47\.647\.646\.246\.249\.0−2\.7\-2\.752\.942\.442\.4\+10\.5\+10\.524\.124\.12\.92\.925\.4Qwen3\.5\-2B\-48\.548\.546\.446\.450\.5−4\.2\-4\.254\.442\.642\.6\+11\.9\+11\.915\.315\.322\.513\.713\.7Qwen3\.5\-4B\-81\.181\.180\.580\.581\.6−1\.1\-1\.190\.871\.471\.4\+19\.4\+19\.47\.94\.04\.07\.17\.1Qwen3\.5\-9B\-85\.885\.885\.485\.486\.2−0\.7\-0\.791\.979\.779\.7\+12\.3\+12\.37\.71\.61\.64\.94\.9Qwen3\.5\-27B\-96\.196\.195\.895\.896\.3−0\.6\-0\.696\.895\.495\.4\+1\.4\+1\.42\.80\.70\.70\.40\.4Qwen3\.5\-0\.8B✓33\.333\.332\.332\.334\.2−1\.9\-1\.930\.630\.636\.0−5\.4\-5\.425\.820\.120\.120\.920\.9Qwen3\.5\-2B✓80\.380\.378\.278\.282\.3−4\.1\-4\.181\.379\.379\.3\+2\.0\+2\.013\.14\.34\.32\.32\.3Qwen3\.5\-4B✓95\.595\.594\.794\.796\.3−1\.6\-1\.696\.294\.894\.8\+1\.4\+1\.43\.80\.40\.40\.20\.2Qwen3\.5\-9B✓97\.397\.396\.796\.797\.9−1\.2\-1\.298\.296\.596\.5\+1\.7\+1\.72\.40\.20\.20\.10\.1Qwen3\.5\-27B✓98\.698\.698\.198\.199\.1−1\.0\-1\.099\.198\.198\.1\+1\.0\+1\.01\.00\.30\.30\.10\.1Table 4:Breakdown of English FrameBench performance by sentence\-pair type, position of the correct sentence, and error type\.ℛ\\mathcal\{R\}denotes reasoning mode, and Acc\. denotes overall accuracy\. Base and Ext\. report accuracy on base and extended sentence pairs, while 1st and 2nd report accuracy when the correct sentence appears first or second\.ΔB−E\\Delta\_\{B\-E\}andΔ1−2\\Delta\_\{1\-2\}indicate performance differences\. Both denotes selecting both sentences, Neither denotes selecting neither sentence, and Opposite denotes selecting the incorrect sentence\. Acc\. and the three error\-type rates sum to 100% up to rounding\.Modelℛ\\mathcal\{R\}ℳ\\mathcal\{M\}FrameBenchMMLU\-ProQwen3\-1\.7B✓77\.835\.6cQwen3\-VL\-2B✓✓67\.762\.3aQwen3\-4B✓90\.952\.2cQwen3\-VL\-4B✓✓92\.373\.6aQwen3\-8B✓92\.572\.1cQwen3\-VL\-8B✓✓96\.377\.3aQwen3\-32B✓96\.966\.8cQwen3\-VL\-32B✓✓95\.782\.1aQwen3\-1\.7B53\.537\.3fQwen3\-VL\-2B✓71\.449\.0aQwen3\-4B84\.657\.9fQwen3\-VL\-4B✓89\.867\.1aQwen3\-8B87\.159\.9fQwen3\-VL\-8B✓94\.371\.6aQwen3\-32B92\.572\.7fQwen3\-VL\-32B✓95\.778\.6aPhi\-3\.5\-mini59\.734\.5fPhi\-3\.5\-vision✓72\.132\.2fPhi\-4\-mini78\.543\.3fPhi\-4\-multimodal✓79\.839\.1fTable 5:Performance comparison of matched text\-only and multimodal models\.ℛ\\mathcal\{R\}denotes reasoning mode, andℳ\\mathcal\{M\}denotes multimodality\. For MMLU\-Pro scores, superscripts indicate the source of each score:amodel\-provider reports,c[Artificial Analysis](https://artificialanalysis.ai/), andfour own evaluations\.

## 5Analysis

We analyze model behavior on the English subset of FrameBench to better understand error patterns and remaining challenges\.

Question and Sentence PairQwen3\.5Gemma 40\.8B2B4B9B27BE2BE4B31BQ1\.Select all sentences where the speaker gets hurt\.SAS\_\{\\text\{A\}\}\.Rounding the corner too fast, Igrazedthe wall with my knee\. \(\[Impact\]\)SBS\_\{\\text\{B\}\}\.Rounding the corner too fast, Igrazedmy knee on the wall\. \(\[Body\_injury\]\)––––––✓\\checkmark–Q2\.Select all sentences where someone carries out a ceremonial rite\.SAS\_\{\\text\{A\}\}\.Theychristenedtheir new boat the Sea Breeze at the pier\. \(\[Name\_conferral\]\)SBS\_\{\\text\{B\}\}\.Theychristenedtheir new boat with a bottle of champagne at the pier\. \(\[Rite\]\)✓\\checkmark–––––––Q3\.Select all sentences where someone is trying to follow the person\.SAS\_\{\\text\{A\}\}\.Ilostthe guards in the crowd\. \(\[Losing\_track\_of\_perceiver\]\)SBS\_\{\\text\{B\}\}\.Ilostmy wallet in the crowd\. \(\[Losing\]\)––✓\\checkmark✓\\checkmark✓\\checkmark–✓\\checkmark✓\\checkmarkQ4\.Select all sentences where a person takes on a character in a play\.SAS\_\{\\text\{A\}\}\.He and MayaplayedRomeo and Juliet at the festival\. \(\[Performers\_and\_roles\]\)SBS\_\{\\text\{B\}\}\.He and Mayaplayedtennis at the festival\. \(\[Competition\]\)✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmarkTable 6:Examples from FrameBench\. Bold text marks the correct answer, underlining marks the frame\-evoking word, and parenthetical tags show the corresponding frame\. Each model column reports whether the model answered the item correctly in the reasoning setting, where ‘✓’ denotes a correct prediction and ‘–’ denotes an incorrect prediction\.### 5\.1Error and Bias Analysis

We further analyze model behavior on FrameBench along three dimensions: sentence\-pair type, sentence position, and error type\. Japanese results for the same analyses are provided in Appendix[D](https://arxiv.org/html/2609.03370#A4); they show similar tendencies except for a weaker sentence\-pair type effect\.

#### Difficulty by Sentence Pair Type

Since base pairs are designed to be lexically similar, they are expected to be harder for models than extended pairs, which are not constrained to be surface\-similar\. Columns under\(i\) Sent\. Pair Typein Table[4](https://arxiv.org/html/2609.03370#S4.T4)show the accuracy for each pair type and the performance gap\. All models achieved higher accuracy on extended pairs than on base pairs, as indicated by the negative values ofΔB−E\\Delta\_\{B\-E\}\. The human score gap shows the same tendency, with extended pairs outperforming base pairs by 2\.57 points\. These results indicate that lexical similarity makes base pairs harder for LLMs\. In the Japanese subset, however, this pair\-type effect is weaker: the human score gap decreases to 0\.93 points, and the LLM results show a less consistent advantage for extended pairs\.

#### Analysis of Sentence Position Bias

We also investigated order sensitivity by comparing performance when the correct sentence appears in the first versus second position\. Columns under\(ii\) Correct Sent\. Positionin Table[4](https://arxiv.org/html/2609.03370#S4.T4)report accuracy for these two positions and the corresponding performance gap\. Since each sentence pair is evaluated in both original and swapped orders, the same items appear in both position conditions, controlling for item difficulty\. Most models show a noticeable absolute difference between the two positions, indicating that sentence order can affect model predictions\. This supports the main evaluation protocol described in Section[4](https://arxiv.org/html/2609.03370#S4), where each pair is evaluated in both possible orders to mitigate order sensitivity\.

#### Error Tendencies in Incorrect Choices

Columns under\(iii\) Error Breakdownin Table[4](https://arxiv.org/html/2609.03370#S4.T4)show the distribution of model errors\. For models scoring above 85%, “Opposite” errors become rare\. For high\-performing models, errors therefore tend to reflect over\-selection or under\-selection rather than outright selection of the opposite frame, suggesting that the remaining difficulty lies in borderline frame\-semantic distinctions\.

### 5\.2Impact of Multimodality

Frame semantics is a linguistic framework, but the situations it describes often involve visually grounded event knowledge\.This motivates examining whether multimodal training helps models interpret frame\-semantic distinctions\.Table[5](https://arxiv.org/html/2609.03370#S4.T5)reports matched comparisons between multimodal models and the language models on which they are based\.Overall, multimodal models outperform the corresponding language models in most matched comparisons on FrameBench\.However, these gains should be interpreted cautiously: many multimodal models also achieve higher MMLU\-Pro scores, while the Phi models show FrameBench improvements despite lower MMLU\-Pro scores\.Thus, visually grounded training may help frame\-semantic interpretation, but its effect cannot be isolated from other training differences\.

### 5\.3Case Study

We examine examples selected to span different levels of model accuracy\. Because we consider only items for which at least two of the three human annotators selected the correct answer, these examples are intended to be answerable by humans rather than inherently ambiguous\. Table[6](https://arxiv.org/html/2609.03370#S5.T6)reports per\-model correctness for reasoning\-enabled Qwen3\.5 and Gemma 4 models using the first prompt template in Table[9](https://arxiv.org/html/2609.03370#A2.T9)\. The scores are shown across model sizes, allowing us to examine whether each example is solved consistently, requires larger models, or remains difficult even for stronger models\.

In Questions 1 and 2 of Table[6](https://arxiv.org/html/2609.03370#S5.T6), even the largest Qwen3\.5 and Gemma 4 models fail, incorrectly selecting both sentences\. These examples require distinguishing subtle frame\-semantic differences despite strong lexical or contextual overlap\. In Question 1, a small change in argument structure shifts the interpretation from contact with an object to injury to a body part\. In Question 2,christencarries a ceremonial nuance in both sentences, but only Sentence B explicitly evokes the\[Rite\]frame through the use of a bottle of champagne\. These results suggest that even strong models can struggle when the relevant frame distinction depends on fine\-grained semantic cues\.

Question 3 is easier, but still challenging for smaller models\. Both sentences uselostin closely related senses, but only Sentence A describes a situation in which someone trying to follow the speaker loses track of them\. The score pattern suggests that distinguishing related senses within a close semantic domain requires a moderate level of semantic sensitivity\.

By contrast, Question 4 is solved by all models in the table\. Here, the two meanings ofplaybelong to clearly different semantic domains, and the distinction can be identified from salient contextual cues such asRomeo and Julietandtennis\.

## 6Conclusion

In this study, we introduced FrameBench, a FrameNet\-grounded multiple\-choice benchmark for evaluating whether LLMs can distinguish the frames evoked by the same verb across contexts\. We constructed English and Japanese versions of FrameBench using FrameNet resources, a generation\-and\-verification pipeline, and native\-speaker judgments\. Together, these datasets provide a new resource for evaluating frame\-semantic interpretation in LLMs across two typologically distant languages\.

Experiments across a wide range of LLMs showed that several frontier models reached or exceeded the human reference score, while performance varied substantially with model scale, reasoning mode, and language\. Our analyses further characterized model behavior in terms of sentence\-pair type, sentence position, and error type, and examined whether multimodal training may support frame\-semantic interpretation\.

## Limitations

First, FrameBench focuses on a specific aspect of semantic competence, namely frame\-semantic discrimination for context\-dependent verb interpretations in a four\-choice setting\. It does not directly evaluate broader language understanding, open\-ended generation, or explicit frame prediction\. Second, the human reference scores reported in our experiments are not based on an independent evaluation\. They are calculated from the judgments of the same annotators whose responses were used to validate and filter FrameBench items\. Because our experiments use only items that were answered correctly by at least two annotators during validation, the reported human scores may overestimate performance relative to an evaluation conducted with an independent group of annotators\.

## Ethical considerations

Our dataset construction involved human annotation in both English and Japanese\. English annotations were conducted by three expert native English annotators recruited through an annotation vendor and compensated according to the vendor’s standard rates\. Japanese evaluation was conducted by three native Japanese\-speaking university students, who were compensated at the university\-defined hourly rate for research assistants\. Manual revision of Japanese generations was performed by one of the authors as part of the research process\. All annotators were informed in advance that their judgments would be used for research purposes and reported in a paper\.

## Acknowledgments

We would like to express our gratitude to Dr\. Kyoko Ohara of Keio University for providing the Japanese FrameNet data used in this study\. This research was supported by JST FOREST Program JPMJFR216N and JST SPRING Program JPMJSP2125\.

## References

- Abdinet al\.\(2024\)M\. Abdin, J\. Aneja, H\. Awadalla, A\. Awadallah, A\. A\. Awan, N\. Bach, A\. Bahree, A\. Bakhtiari, J\. Bao, H\. Behl, A\. Benhaim, M\. Bilenko, J\. Bjorck, S\. Bubeck, M\. Cai, Q\. Cai, V\. Chaudhary, D\. Chen, D\. Chen, W\. Chen, Y\. Chen, Y\. Chen, H\. Cheng, P\. Chopra, X\. Dai, M\. Dixon, R\. Eldan, V\. Fragoso, J\. Gao, M\. Gao, M\. Gao, A\. Garg, A\. D\. Giorno, A\. Goswami, S\. Gunasekar, E\. Haider, J\. Hao, R\. J\. Hewett, W\. Hu, J\. Huynh, D\. Iter, S\. A\. Jacobs, M\. Javaheripi, X\. Jin, N\. Karampatziakis, P\. Kauffmann, M\. Khademi, D\. Kim, Y\. J\. Kim, L\. Kurilenko, J\. R\. Lee, Y\. T\. Lee, Y\. Li, Y\. Li, C\. Liang, L\. Liden, X\. Lin, Z\. Lin, C\. Liu, L\. Liu, M\. Liu, W\. Liu, X\. Liu, C\. Luo, P\. Madan, A\. Mahmoudzadeh, D\. Majercak, M\. Mazzola, C\. C\. T\. Mendes, A\. Mitra, H\. Modi, A\. Nguyen, B\. Norick, B\. Patra, D\. Perez\-Becker, T\. Portet, R\. Pryzant, H\. Qin, M\. Radmilac, L\. Ren, G\. de Rosa, C\. Rosset, S\. Roy, O\. Ruwase, O\. Saarikivi, A\. Saied, A\. Salim, M\. Santacroce, S\. Shah, N\. Shang, H\. Sharma, Y\. Shen, S\. Shukla, X\. Song, M\. Tanaka, A\. Tupini, P\. Vaddamanu, C\. Wang, G\. Wang, L\. Wang, S\. Wang, X\. Wang, Y\. Wang, R\. Ward, W\. Wen, P\. Witte, H\. Wu, X\. Wu, M\. Wyatt, B\. Xiao, C\. Xu, J\. Xu, W\. Xu, J\. Xue, S\. Yadav, F\. Yang, J\. Yang, Y\. Yang, Z\. Yang, D\. Yu, L\. Yuan, C\. Zhang, C\. Zhang, J\. Zhang, L\. L\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, and X\. ZhouPhi\-3 Technical Report: A Highly Capable Language Model Locally on Your Phone\.arXiv preprint arXiv:2404\.14219\.External Links:[Link](https://arxiv.org/abs/2404.14219),2404\.14219Cited by:[4th item](https://arxiv.org/html/2609.03370#A2.I2.i4.p1.1),[2nd item](https://arxiv.org/html/2609.03370#A2.I4.i2.p1.1)\.
- Aboueleninet al\.\(2025\)A\. Abouelenin, A\. Ashfaq, A\. Atkinson, H\. Awadalla, N\. Bach, J\. Bao, A\. Benhaim, M\. Cai, V\. Chaudhary, C\. Chen, D\. Chen, D\. Chen, J\. Chen, W\. Chen, Y\. Chen, Y\. Chen, Q\. Dai, X\. Dai, R\. Fan, M\. Gao, M\. Gao, A\. Garg, A\. Goswami, J\. Hao, A\. Hendy, Y\. Hu, X\. Jin, M\. Khademi, D\. Kim, Y\. J\. Kim, G\. Lee, J\. Li, Y\. Li, C\. Liang, X\. Lin, Z\. Lin, M\. Liu, Y\. Liu, G\. Lopez, C\. Luo, P\. Madan, V\. Mazalov, A\. Mitra, A\. Mousavi, A\. Nguyen, J\. Pan, D\. Perez\-Becker, J\. Platin, T\. Portet, K\. Qiu, B\. Ren, L\. Ren, S\. Roy, N\. Shang, Y\. Shen, S\. Singhal, S\. Som, X\. Song, T\. Sych, P\. Vaddamanu, S\. Wang, Y\. Wang, Z\. Wang, H\. Wu, H\. Xu, W\. Xu, Y\. Yang, Z\. Yang, D\. Yu, I\. Zabir, J\. Zhang, L\. L\. Zhang, Y\. Zhang, and X\. ZhouPhi\-4\-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture\-of\-LoRAs\.arXiv preprint arXiv:2503\.01743\.External Links:[Link](https://arxiv.org/abs/2503.01743),2503\.01743Cited by:[4th item](https://arxiv.org/html/2609.03370#A2.I2.i4.p1.1),[2nd item](https://arxiv.org/html/2609.03370#A2.I4.i2.p1.1)\.
- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. ZhuQwen3\-VL Technical Report\.arXiv preprint arXiv:2511\.21631\.External Links:2511\.21631,[Link](https://arxiv.org/abs/2511.21631)Cited by:[1st item](https://arxiv.org/html/2609.03370#A2.I4.i1.p1.1)\.
- Bakeret al\.\(1998\)C\. F\. Baker, C\. J\. Fillmore, and J\. B\. LoweThe Berkeley FrameNet project\.InProceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics \(ACL\-COLING 1998\),pp\. 86–90\.Cited by:[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p1.1)\.
- Belcavelloet al\.\(2026\)F\. Belcavello, E\. E\. Matos, A\. Lorenzi, L\. Bonoto, L\. Pádua Ruiz, L\. F\. Pereira, V\. Herbst, Y\. L\. Navarro, H\. d\. A\. Abreu, L\. Vicente Dutra, and T\. T\. TorrentEvaluating the Impact of LLM\-Assisted Annotation in a Perspectivized Setting: The Case of FrameNet Annotation\.InProceedings of the 22nd Joint ACL \- ISO Workshop on Interoperable Semantic Annotation and Representation \(ISA\-22\) @ LREC 2026,pp\. 77–87\.External Links:[Link](https://aclanthology.org/2026.isa-1.9/),[Document](https://dx.doi.org/10.63317/36ot28es26u8)Cited by:[§2\.3](https://arxiv.org/html/2609.03370#S2.SS3.p1.1)\.
- Belcavelloet al\.\(2024\)F\. Belcavello, T\. Timponi Torrent, E\. E\. Matos, A\. S\. Pagano, M\. Gamonal, N\. Sigiliano, L\. V\. Dutra, H\. de Andrade Abreu, M\. Samagaio, M\. Carvalho, F\. Campos, G\. Azalim, B\. Mazzei, M\. F\. de Oliveira, A\. C\. Loçasso Luz, L\. Pádua Ruiz, J\. Bellei, A\. Pestana, J\. Costa, I\. Rabelo, A\. B\. Silva, R\. Roza, M\. Souza, and I\. OliveiraFrame2: A FrameNet\-based Multimodal Dataset for Tackling Text\-image Interactions in Video\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 7429–7437\.External Links:[Link](https://aclanthology.org/2024.lrec-main.655/)Cited by:[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p2.1)\.
- Center for AI Safetyet al\.\(2026\)Center for AI Safety, Scale AI, and HLE Contributors ConsortiumA benchmark of expert\-level academic questions to assess AI capabilities\.Nature649\(8099\),pp\. 1139–1146\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09962-4),[Document](https://dx.doi.org/10.1038/s41586-025-09962-4)Cited by:[3rd item](https://arxiv.org/html/2609.03370#A2.I5.i3.p1.1.1),[§4\.1](https://arxiv.org/html/2609.03370#S4.SS1.p3.1.1)\.
- Chundruet al\.\(2025\)J\. K\. Chundru, R\. Poddar, J\. Cao, and T\. JiangDo LLMs Encode Frame Semantics? Evidence from Frame Identification\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP 2025\),pp\. 29488–29500\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1499/)Cited by:[§2\.3](https://arxiv.org/html/2609.03370#S2.SS3.p1.1)\.
- Devasieret al\.\(2025\)J\. Devasier, R\. Mediratta, and C\. LiCan LLMs Extract Frame\-Semantic Arguments?\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP 2025\),pp\. 30609–30622\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1557/)Cited by:[§2\.3](https://arxiv.org/html/2609.03370#S2.SS3.p1.1)\.
- Djemaaet al\.\(2016\)M\. Djemaa, M\. Candito, P\. Muller, and L\. VieuCorpus Annotation within the French FrameNet: a Domain\-by\-domain Methodology\.InProceedings of the Tenth International Conference on Language Resources and Evaluation \(LREC’16\),pp\. 3794–3801\.External Links:[Link](https://aclanthology.org/L16-1601/)Cited by:[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p2.1)\.
- Farabet and Lacombe \(2026\)C\. Farabet and O\. LacombeGemma 4: Byte for byte, the most capable open models\.External Links:[Link](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4)Cited by:[1st item](https://arxiv.org/html/2609.03370#A2.I2.i1.p1.1)\.
- Fillmore \(1982\)C\. J\. FillmoreFrame Semantics\.InLinguistics in the Morning Calm,pp\. 111–137\.External Links:[Link](https://brenocon.com/Fillmore%201982_2up.pdf)Cited by:[§1](https://arxiv.org/html/2609.03370#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p1.1)\.
- Garatet al\.\(2025\)D\. Garat, G\. Moncecchi, and D\. WonseverExploring in\-context learning for frame\-semantic parsing\.arXiv preprint arXiv:2507\.23082\.External Links:2507\.23082,[Link](https://arxiv.org/abs/2507.23082)Cited by:[§2\.3](https://arxiv.org/html/2609.03370#S2.SS3.p1.1)\.
- Guoet al\.\(2024\)S\. Guo, Y\. Chen, K\. Liu, R\. Li, and J\. ZhaoNutFrame: Frame\-based Conceptual Structure Induction with LLMs\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 12330–12335\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1079/)Cited by:[§2\.3](https://arxiv.org/html/2609.03370#S2.SS3.p2.1)\.
- Hahmet al\.\(2020\)Y\. Hahm, Y\. Noh, J\. Y\. Han, T\. H\. Oh, H\. Choe, H\. Kim, and K\. ChoiCrowdsourcing in the Development of a Multilingual FrameNet: A Case Study of Korean FrameNet\.InProceedings of the Twelfth Language Resources and Evaluation Conference \(LREC 2020\),pp\. 236–244\.External Links:[Link](https://aclanthology.org/2020.lrec-1.30/),ISBN 979\-10\-95546\-34\-4Cited by:[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p2.1)\.
- Hanet al\.\(2024\)Y\. Han, R\. Sasano, and K\. TakedaDefinition Generation for Automatically Induced Semantic Frame\.InFindings of the Association for Computational Linguistics: ACL 2024 \(ACL2024 Findings\),pp\. 11112–11118\.External Links:[Link](https://aclanthology.org/2024.findings-acl.661/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.661)Cited by:[§2\.3](https://arxiv.org/html/2609.03370#S2.SS3.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring Massive Multitask Language Understanding\.InInternational Conference on Learning Representations \(ICLR 2021\),External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§2\.1](https://arxiv.org/html/2609.03370#S2.SS1.p1.1)\.
- Knightet al\.\(2017\)K\. Knight, B\. Badarau, L\. Baranescu, C\. Bonial, M\. Bardocz, K\. Griffitt, U\. Hermjakob, D\. Marcu, M\. Palmer, T\. O’Gorman, and N\. SchneiderAbstract Meaning Representation \(AMR\) Annotation Release 2\.0\.Note:LDC2017T10DatasetExternal Links:[Link](https://catalog.ldc.upenn.edu/LDC2017T10),[Document](https://dx.doi.org/10.35111/s444-np87)Cited by:[§2\.1](https://arxiv.org/html/2609.03370#S2.SS1.p2.1)\.
- LLM\-jp \(2024\)LLM\-jpLLM\-jp: A Cross\-organizational Project for the Research and Development of Fully Open Japanese LLMs\.arXiv preprint arXiv:2407\.03963\.External Links:[Link](https://arxiv.org/abs/2407.03963)Cited by:[1st item](https://arxiv.org/html/2609.03370#A2.I3.i1.p1.1)\.
- B\. Lyngfelt, L\. Borin, K\. Ohara, and T\. T\. Torrent \(Eds\.\) \(2018\)B\. Lyngfelt, L\. Borin, K\. Ohara, and T\. T\. Torrent \(Eds\.\)Constructicography: constructicon development across languages\.John Benjamins Publishing Company\.External Links:[Document](https://dx.doi.org/10.1075/CAL.22),[Link](https://chooser.crossref.org/?doi=10.1075%2Fcal.22)Cited by:[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p2.1)\.
- Oharaet al\.\(2004\)K\. H\. Ohara, S\. Fujii, T\. Ohori, R\. Suzuki, H\. Saito, and S\. IshizakiThe Japanese FrameNet Project: an introduction\.InProceedings of the LREC 2004 Satellite Workshop “Building Lexical Resources from Semantically Annotated Corpora”,pp\. 9–11\.Cited by:[§1](https://arxiv.org/html/2609.03370#S1.p5.1),[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p2.1),[§3\.3](https://arxiv.org/html/2609.03370#S3.SS3.SSS0.Px1.p1.1)\.
- Okaet al\.\(2026\)T\. Oka, T\. Shibata, and N\. YoshidaJamC\-QA: A Multiple\-Choice Question Answering Benchmark for Japan\-Specific Knowledge\.InProceedings of the Fifteenth Language Resources and Evaluation Conference\(LREC 2026\),pp\. 4536–4546\.External Links:[Link](https://aclanthology.org/2026.lrec-1.356/),[Document](https://dx.doi.org/10.63317/33xwat5joobd)Cited by:[1st item](https://arxiv.org/html/2609.03370#A2.I6.i1.p1.1.1),[§4\.1](https://arxiv.org/html/2609.03370#S4.SS1.p3.1.1)\.
- OpenAI \(2025\)OpenAIIntroducing GPT\-5\.Note:August 7, 2025External Links:[Link](https://openai.com/index/introducing-gpt-5/)Cited by:[1st item](https://arxiv.org/html/2609.03370#A2.I1.i1.p1.1),[§3\.3](https://arxiv.org/html/2609.03370#S3.SS3.SSS0.Px1.p1.1)\.
- Pilehvar and Camacho\-Collados \(2019\)M\. T\. Pilehvar and J\. Camacho\-ColladosWiC: the Word\-in\-Context Dataset for Evaluating Context\-Sensitive Meaning Representations\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long and Short Papers\),pp\. 1267–1273\.External Links:[Link](https://aclanthology.org/N19-1128/),[Document](https://dx.doi.org/10.18653/v1/N19-1128)Cited by:[§2\.1](https://arxiv.org/html/2609.03370#S2.SS1.p2.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[2nd item](https://arxiv.org/html/2609.03370#A2.I2.i2.p1.1)\.
- Raiet al\.\(2025\)S\. I\. Rai, D\. Croce, and R\. BasiliInjecting Frame Semantics into Large Language Models via Prompt\-Based Fine\-Tuning\.InProceedings of the 14th Joint Conference on Lexical and Computational Semantics \(\*SEM 2025\),pp\. 31–47\.External Links:[Link](https://aclanthology.org/2025.starsem-1.3/)Cited by:[§2\.3](https://arxiv.org/html/2609.03370#S2.SS3.p1.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: A Graduate\-Level Google\-Proof Q&A Benchmark\.InFirst Conference on Language Modeling \(CoLM 2024\),External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[2nd item](https://arxiv.org/html/2609.03370#A2.I5.i2.p1.1.1),[§4\.1](https://arxiv.org/html/2609.03370#S4.SS1.p3.1.1)\.
- Ruppenhoferet al\.\(2016\)J\. Ruppenhofer, M\. Ellsworth, M\. Schwarzer\-Petruck, C\. R\. Johnson, and J\. ScheffczykFrameNet II: Extended theory and practice\.External Links:[Link](https://api.semanticscholar.org/CorpusID:62163005)Cited by:[§1](https://arxiv.org/html/2609.03370#S1.p5.1),[§3\.3](https://arxiv.org/html/2609.03370#S3.SS3.SSS0.Px1.p1.1)\.
- Srivastavaet al\.\(2023\)A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso, A\. Kluska, A\. Lewkowycz, A\. Agarwal, A\. Power, A\. Ray, A\. Warstadt, A\. W\. Kocurek, A\. Safaya, A\. Tazarv, A\. Xiang, A\. Parrish, A\. Nie, A\. Hussain, A\. Askell, A\. Dsouza, A\. Slone, A\. Rahane, A\. S\. Iyer, A\. J\. Andreassen, A\. Madotto, A\. Santilli, A\. Stuhlmüller, A\. M\. Dai, A\. La, A\. K\. Lampinen, A\. Zou, A\. Jiang, A\. Chen, A\. Vuong, A\. Gupta, A\. Gottardi, A\. Norelli, A\. Venkatesh, A\. Gholamidavoodi, A\. Tabassum, A\. Menezes, A\. Kirubarajan, A\. Mullokandov, A\. Sabharwal, A\. Herrick, A\. Efrat, A\. Erdem, A\. Karakaş, B\. R\. Roberts, B\. S\. Loe, B\. Zoph, B\. Bojanowski, B\. Özyurt, B\. Hedayatnia, B\. Neyshabur, B\. Inden, B\. Stein, B\. Ekmekci, B\. Y\. Lin, B\. Howald, B\. Orinion, C\. Diao, C\. Dour, C\. Stinson, C\. Argueta, C\. Ferri, C\. Singh, C\. Rathkopf, C\. Meng, C\. Baral, C\. Wu, C\. Callison\-Burch, C\. Waites, C\. Voigt, C\. D\. Manning, C\. Potts, C\. Ramirez, C\. E\. Rivera, C\. Siro, C\. Raffel, C\. Ashcraft, C\. Garbacea, D\. Sileo, D\. Garrette, D\. Hendrycks, D\. Kilman, D\. Roth, C\. D\. Freeman, D\. Khashabi, D\. Levy, D\. M\. González, D\. Perszyk, D\. Hernandez, D\. Chen, D\. Ippolito, D\. Gilboa, D\. Dohan, D\. Drakard, D\. Jurgens, D\. Datta, D\. Ganguli, D\. Emelin, D\. Kleyko, D\. Yuret, D\. Chen, D\. Tam, D\. Hupkes, D\. Misra, D\. Buzan, D\. C\. Mollo, D\. Yang, D\. Lee, D\. Schrader, E\. Shutova, E\. D\. Cubuk, E\. Segal, E\. Hagerman, E\. Barnes, E\. Donoway, E\. Pavlick, E\. Rodolà, E\. Lam, E\. Chu, E\. Tang, E\. Erdem, E\. Chang, E\. A\. Chi, E\. Dyer, E\. Jerzak, E\. Kim, E\. E\. Manyasi, E\. Zheltonozhskii, F\. Xia, F\. Siar, F\. Martínez\-Plumed, F\. Happé, F\. Chollet, F\. Rong, G\. Mishra, G\. I\. Winata, G\. de Melo, G\. Kruszewski, G\. Parascandolo, G\. Mariani, G\. X\. Wang, G\. Jaimovitch\-Lopez, G\. Betz, G\. Gur\-Ari, H\. Galijasevic, H\. Kim, H\. Rashkin, H\. Hajishirzi, H\. Mehta, H\. Bogar, H\. F\. A\. Shevlin, H\. Schuetze, H\. Yakura, H\. Zhang, H\. M\. Wong, I\. Ng, I\. Noble, J\. Jumelet, J\. Geissinger, J\. Kernion, J\. Hilton, J\. Lee, J\. F\. Fisac, J\. B\. Simon, J\. Koppel, J\. Zheng, J\. Zou, J\. Kocon, J\. Thompson, J\. Wingfield, J\. Kaplan, J\. Radom, J\. Sohl\-Dickstein, J\. Phang, J\. Wei, J\. Yosinski, J\. Novikova, J\. Bosscher, J\. Marsh, J\. Kim, J\. Taal, J\. Engel, J\. Alabi, J\. Xu, J\. Song, J\. Tang, J\. Waweru, J\. Burden, J\. Miller, J\. U\. Balis, J\. Batchelder, J\. Berant, J\. Frohberg, J\. Rozen, J\. Hernandez\-Orallo, J\. Boudeman, J\. Guerr, J\. Jones, J\. B\. Tenenbaum, J\. S\. Rule, J\. Chua, K\. Kanclerz, K\. Livescu, K\. Krauth, K\. Gopalakrishnan, K\. Ignatyeva, K\. Markert, K\. Dhole, K\. Gimpel, K\. Omondi, K\. W\. Mathewson, K\. Chiafullo, K\. Shkaruta, K\. Shridhar, K\. McDonell, K\. Richardson, L\. Reynolds, L\. Gao, L\. Zhang, L\. Dugan, L\. Qin, L\. Contreras\-Ochando, L\. Morency, L\. Moschella, L\. Lam, L\. Noble, L\. Schmidt, L\. He, L\. Oliveros\-Colón, L\. Metz, L\. K\. Senel, M\. Bosma, M\. Sap, M\. T\. Hoeve, M\. Farooqi, M\. Faruqui, M\. Mazeika, M\. Baturan, M\. Marelli, M\. Maru, M\. J\. Ramirez\-Quintana, M\. Tolkiehn, M\. Giulianelli, M\. Lewis, M\. Potthast, M\. L\. Leavitt, M\. Hagen, M\. Schubert, M\. O\. Baitemirova, M\. Arnaud, M\. McElrath, M\. A\. Yee, M\. Cohen, M\. Gu, M\. Ivanitskiy, M\. Starritt, M\. Strube, M\. Swędrowski, M\. Bevilacqua, M\. Yasunaga, M\. Kale, M\. Cain, M\. Xu, M\. Suzgun, M\. Walker, M\. Tiwari, M\. Bansal, M\. Aminnaseri, M\. Geva, M\. Gheini, M\. V\. T, N\. Peng, N\. A\. Chi, N\. Lee, N\. G\. Krakover, N\. Cameron, N\. Roberts, N\. Doiron, N\. Martinez, N\. Nangia, N\. Deckers, N\. Muennighoff, N\. S\. Keskar, N\. S\. Iyer, N\. Constant, N\. Fiedel, N\. Wen, O\. Zhang, O\. Agha, O\. Elbaghdadi, O\. Levy, O\. Evans, P\. A\. M\. Casares, P\. Doshi, P\. Fung, P\. P\. Liang, P\. Vicol, P\. Alipoormolabashi, P\. Liao, P\. Liang, P\. W\. Chang, P\. Eckersley, P\. M\. Htut, P\. Hwang, P\. Miłkowski, P\. Patil, P\. Pezeshkpour, P\. Oli, Q\. Mei, Q\. Lyu, Q\. Chen, R\. Banjade, R\. E\. Rudolph, R\. Gabriel, R\. Habacker, R\. Risco, R\. Millière, R\. Garg, R\. Barnes, R\. A\. Saurous, R\. Arakawa, R\. Raymaekers, R\. Frank, R\. Sikand, R\. Novak, R\. Sitelew, R\. L\. Bras, R\. Liu, R\. Jacobs, R\. Zhang, R\. Salakhutdinov, R\. A\. Chi, S\. R\. Lee, R\. Stovall, R\. Teehan, R\. Yang, S\. Singh, S\. M\. Mohammad, S\. Anand, S\. Dillavou, S\. Shleifer, S\. Wiseman, S\. Gruetter, S\. R\. Bowman, S\. S\. Schoenholz, S\. Han, S\. Kwatra, S\. A\. Rous, S\. Ghazarian, S\. Ghosh, S\. Casey, S\. Bischoff, S\. Gehrmann, S\. Schuster, S\. Sadeghi, S\. Hamdan, S\. Zhou, S\. Srivastava, S\. Shi, S\. Singh, S\. Asaadi, S\. S\. Gu, S\. Pachchigar, S\. Toshniwal, S\. Upadhyay, S\. S\. Debnath, S\. Shakeri, S\. Thormeyer, S\. Melzi, S\. Reddy, S\. P\. Makini, S\. Lee, S\. Torene, S\. Hatwar, S\. Dehaene, S\. Divic, S\. Ermon, S\. Biderman, S\. Lin, S\. Prasad, S\. Piantadosi, S\. Shieber, S\. Misherghi, S\. Kiritchenko, S\. Mishra, T\. Linzen, T\. Schuster, T\. Li, T\. Yu, T\. Ali, T\. Hashimoto, T\. Wu, T\. Desbordes, T\. Rothschild, T\. Phan, T\. Wang, T\. Nkinyili, T\. Schick, T\. Kornev, T\. Tunduny, T\. Gerstenberg, T\. Chang, T\. Neeraj, T\. Khot, T\. Shultz, U\. Shaham, V\. Misra, V\. Demberg, V\. Nyamai, V\. Raunak, V\. V\. Ramasesh, vinay uday prabhu, V\. Padmakumar, V\. Srikumar, W\. Fedus, W\. Saunders, W\. Zhang, W\. Vossen, X\. Ren, X\. Tong, X\. Zhao, X\. Wu, X\. Shen, Y\. Yaghoobzadeh, Y\. Lakretz, Y\. Song, Y\. Bahri, Y\. Choi, Y\. Yang, S\. Hao, Y\. Chen, Y\. Belinkov, Y\. Hou, Y\. Hou, Y\. Bai, Z\. Seid, Z\. Zhao, Z\. Wang, Z\. J\. Wang, Z\. Wang, and Z\. WuBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=uyTL5Bvosj)Cited by:[§2\.1](https://arxiv.org/html/2609.03370#S2.SS1.p1.1)\.
- Swallow LLM Teamet al\.\(2025\)Swallow LLM Team, S\. Mizuki, K\. Saito, M\. Oi, T\. Ichinose, N\. Matsushita, S\. Miyamoto, T\. D\. Nguyen, and S\. Moonswallow\-evaluation\-instruct: Evaluation Framework for Large Language Models \(in Japanese\)\.External Links:[Link](https://github.com/swallow-llm/swallow-evaluation-instruct)Cited by:[§B\.4](https://arxiv.org/html/2609.03370#A2.SS4.p1.1)\.
- Swallow LLM Team \(2026\)Swallow LLM TeamQwen3 Swallow\.External Links:[Link](https://swallow-llm.github.io/qwen3-swallow.en.html)Cited by:[2nd item](https://arxiv.org/html/2609.03370#A2.I3.i2.p1.1)\.
- Viridianoet al\.\(2024\)M\. Viridiano, A\. Lorenzi, T\. Timponi Torrent, E\. E\. Matos, A\. S\. Pagano, N\. Sathler Sigiliano, M\. Gamonal, H\. de Andrade Abreu, L\. Vicente Dutra, M\. Samagaio, M\. Carvalho, F\. Campos, G\. Azalim, B\. Mazzei, M\. Fonseca de Oliveira, A\. C\. Luz, L\. Padua Ruiz, J\. Bellei, A\. Pestana, J\. Costa, I\. Rabelo, A\. B\. Silva, R\. Roza, M\. Souza Mota, I\. Oliveira, and M\. H\. Pelegrino de FreitasFramed Multi30K: A Frame\-Based Multimodal\-Multilingual Dataset\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 7438–7449\.External Links:[Link](https://aclanthology.org/2024.lrec-main.656/)Cited by:[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p2.1)\.
- Wanget al\.\(2018\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. BowmanGLUE: A Multi\-Task Benchmark and Analysis Platform for Natural Language Understanding\.InProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,pp\. 353–355\.External Links:[Link](https://aclanthology.org/W18-5446/),[Document](https://dx.doi.org/10.18653/v1/W18-5446)Cited by:[§1](https://arxiv.org/html/2609.03370#S1.p4.1),[§2\.1](https://arxiv.org/html/2609.03370#S2.SS1.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-Pro: A More Robust and Challenging Multi\-Task Language Understanding Benchmark\.InAdvances in Neural Information Processing Systems \(NeurIPS 2024\),Vol\.37,pp\. 95266–95290\.External Links:[Document](https://dx.doi.org/10.52202/079017-3018),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/ad236edc564f3e3156e1b2feafb99a24-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[1st item](https://arxiv.org/html/2609.03370#A2.I5.i1.p1.1.1),[§1](https://arxiv.org/html/2609.03370#S1.p4.1),[§2\.1](https://arxiv.org/html/2609.03370#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.03370#S4.SS1.p3.1.1)\.
- Xuanet al\.\(2025\)W\. Xuan, R\. Yang, H\. Qi, Q\. Zeng, Y\. Xiao, A\. Feng, D\. Liu, Y\. Xing, J\. Wang, F\. Gao, J\. Lu, Y\. Jiang, H\. Li, X\. Li, K\. Yu, R\. Dong, S\. Gu, Y\. Li, X\. Xie, F\. Juefei\-Xu, F\. Khomh, O\. Yoshie, Q\. Chen, D\. Teodoro, N\. Liu, R\. Goebel, L\. Ma, E\. Marrese\-Taylor, S\. Lu, Y\. Iwasawa, Y\. Matsuo, and I\. LiMMLU\-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP 2025\),pp\. 1513–1532\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.79/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.79)Cited by:[2nd item](https://arxiv.org/html/2609.03370#A2.I6.i2.p1.1.1),[§4\.1](https://arxiv.org/html/2609.03370#S4.SS1.p3.1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 Technical Report\.arXiv preprint arXiv:2505\.09388\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[3rd item](https://arxiv.org/html/2609.03370#A2.I2.i3.p1.1)\.
- Yanoet al\.\(2025\)C\. Yano, K\. Yamada, H\. Tsukagoshi, R\. Sasano, and K\. TakedaFrameEOL: Semantic Frame Induction using Causal Language Models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 11620–11632\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.623/)Cited by:[§2\.3](https://arxiv.org/html/2609.03370#S2.SS3.p1.1)\.
- You and Liu \(2005\)L\. You and K\. LiuBuilding Chinese FrameNet database\.In2005 International Conference on Natural Language Processing and Knowledge Engineering \(IEEE NLP\-KE 2005\),pp\. 301–306\.External Links:[Document](https://dx.doi.org/10.1109/NLPKE.2005.1598752)Cited by:[§2\.2](https://arxiv.org/html/2609.03370#S2.SS2.p2.1)\.

## Appendix ADetails of Human Annotation and Manual Revision

### A\.1Human Annotation Protocol

We manually evaluated correctness and acceptability for the finalized items in both English and Japanese FrameBench\. For correctness, annotators answered the same four\-choice question format used in model evaluation\. For acceptability, Japanese items were judged with a binary naturalness label, whereas English items were judged with a three\-way scale \(Unacceptable,Acceptable,Natural\) to reduce rater drift\.

### A\.2Acceptability scale and binarization

To facilitate cross\-lingual analysis, we convert acceptability judgments into a binary label\. For English, we mapNaturalto the positive label and treatAcceptableandUnacceptableas negative\. This mapping yields a more informative label distribution than treating onlyUnacceptableas negative, sinceUnacceptableis extremely rare in our data, and it aligns with our goal of using a stricter notion of linguistic naturalness\. We report the three\-way label distribution for English in Table[7](https://arxiv.org/html/2609.03370#A1.T7)\.

NaturalAcceptableUnacceptableAnn183\.814\.91\.3Ann291\.48\.30\.2Ann382\.616\.80\.6Table 7:Distribution of 3\-level acceptability labels for English\. Values are percentages\.Positive label \(%\)AgreementFleiss’Ann1Ann2Ann3\(%\)κ\\kappaEnglishCorrectnessBase88\.892\.593\.682\.50\.2406Extended93\.896\.096\.089\.30\.2054AcceptabilityBase82\.991\.881\.067\.60\.1417Extended84\.591\.983\.268\.80\.1070JapaneseCorrectnessBase86\.287\.396\.277\.20\.1670Extended91\.191\.696\.484\.50\.2054AcceptabilityBase96\.099\.197\.192\.90\.0687Extended98\.099\.597\.194\.90\.0494Table 8:Inter\-annotator agreement for binary judgments\. Agreement is the percentage of pairs where all three annotators made the same judgment\.
### A\.3Inter\-annotator agreement

Table[8](https://arxiv.org/html/2609.03370#A1.T8)reports positive\-label rates, Fleiss’κ\\kappa, and the 3/3 agreement rate for the binary judgments\. Becauseκ\\kappais sensitive to skewed label distributions, especially for acceptability, we report these additional statistics to aid interpretation\. In English, acceptability contains more borderline cases than correctness, which can lower agreement\. For the main experiments, we therefore use a high\-quality subset defined by a strict human validation criterion\. Under this criterion, an entry is included only if at least two annotators judged it correct and at least two judged it acceptable\.

## Appendix BExperimental Details

### B\.1Evaluation Prompts

Table[9](https://arxiv.org/html/2609.03370#A2.T9)and Table[10](https://arxiv.org/html/2609.03370#A2.T10)present the prompts used for English and Japanese evaluation\. The variables \{question\}, \{verb\}, \{sentence\_a\}, \{sentence\_b\}, and \{choices\_text\} serve as placeholders for the question text, the target verb, the target sentence pair, and the options, respectively\. To eliminate the influence of specific choice number output probabilities, the mapping between options and numbers was randomized for each question\.

Prompt templates\{question\}Sentence A: \{sentence\_a\}Sentence B: \{sentence\_b\}Choices: \{choices\_text\}Instruction: Please make your decision by focusing on the verb “\{verb\}”\.Respond using only the choice number: “1”, “2”, “3”, or “4”\.Please make your decision by focusing on the verb “\{verb\}”\.\{question\}Sentence A: \{sentence\_a\}Sentence B: \{sentence\_b\}Choices: \{choices\_text\}Respond using only the choice number: “1”, “2”, “3”, or “4”\.\{question\}Sentence A: \{sentence\_a\}Sentence B: \{sentence\_b\}Choices: \{choices\_text\}Respond using only the choice number: “1”, “2”, “3”, or “4”\.Your task is to compare the two sentences through the usage of “\{verb\}”\.\{question\}Sentence A: \{sentence\_a\}Sentence B: \{sentence\_b\}Available choices:\{choices\_text\}Output strictly one number \(1–4\)\. No explanation\.Problem: \{question\}\[Sentence A\]\{sentence\_a\}\[Sentence B\]\{sentence\_b\}Choices:\{choices\_text\}Final answer format: just the option number\.Table 9:Prompt templates used for English evaluation\.Prompt templates\{question\}文A: \{sentence\_a\}文B: \{sentence\_b\}選択肢: \{choices\_text\}回答する際は、文の最後の動詞に注目して判断してください。\(When answering, focus on the sentence\-final verb in each sentence\.\)回答は選択肢の番号「1」「2」「3」「4」のいずれかで答えてください。\(Respond using only the option number 1, 2, 3, or 4\.\)\{question\}文A: \{sentence\_a\}文B: \{sentence\_b\}選択肢: \{choices\_text\}選択肢の番号「1」「2」「3」「4」のいずれかで答えてください。\(Answer with the option number 1, 2, 3, or 4\.\)それぞれの文の述語動詞に注意して、\{question\}\(Pay attention to the predicate verb in each sentence:\{question\}\)文A: \{sentence\_a\}文B: \{sentence\_b\}選択肢: \{choices\_text\}選択肢の番号「1」「2」「3」「4」のいずれかで答えてください。\(Answer with the option number 1, 2, 3, or 4\.\)それぞれの文の述語動詞に注意して回答してください。\(Pay attention to the predicate verb in each sentence\.\)\{question\}文A: \{sentence\_a\}文B: \{sentence\_b\}選択肢: \{choices\_text\}選択肢の番号「1」「2」「3」「4」のいずれかで答えてください。\(Answer with the option number 1, 2, 3, or 4\.\)選択肢の番号で回答してください。\(Please answer using the number of the correct option\.\)\{question\}文A: \{sentence\_a\}文B: \{sentence\_b\}選択肢: \{choices\_text\}Table 10:Prompt templates used for Japanese evaluation\.
### B\.2Evaluated Models List

Closed\-source Models

- •GPT\-5 Series[OpenAI \(2025\)](https://arxiv.org/html/2609.03370#bib.bib11):gpt\-5\-2025\-08\-07,gpt\-5\-nano\-2025\-08\-07
- •

Open\-weight Models

- •
- •
- •
- •

Japanese\-oriented Models

- •
- •Swallow Series[Swallow LLM Team \(2026\)](https://arxiv.org/html/2609.03370#bib.bib35):[Qwen3\-Swallow\-8B\-RL](https://huggingface.co/tokyotech-llm/Qwen3-Swallow-8B-RL-v0.2),[32B\-RL](https://huggingface.co/tokyotech-llm/Qwen3-Swallow-32B-RL-v0.2) Qwen3\-Swallow models are developed through continual pre\-training, SFT, and RL based on Qwen3\-Base to enhance their Japanese language capabilities\.

Multimodal Models

- •
- •Phi Series:[Phi\-3\.5\-vision\-instruct](https://huggingface.co/microsoft/Phi-3.5-vision-instruct)[Abdin et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib36),[Phi\-4\-multimodal\-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct)[Abouelenin et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib37) Both Phi\-3\.5\-vision\-instruct and Phi\-4\-multimodal\-instruct are multimodal Phi\-series models that support text and image inputs, using language backbones derived from Phi\-3\.5\-mini and Phi\-4\-Mini\-Instruct, respectively\.

### B\.3Decoding Procedure

For open\-weight models evaluated in the non\-reasoning setting, generation is restricted to the tokens corresponding to the answer choices\.In the reasoning setting, models with reasoning capabilities generate tokens freely until they output a specific tag marking the end of the reasoning phase\. After this tag is generated, decoding is restricted in the same way as in the non\-reasoning setting, allowing the model to generate only the tokens corresponding to the valid answer choices\. For models with a native reasoning process, such as the GPT\-5 series, we utilize JSON\-based structured output to facilitate reliable answer extraction\.

### B\.4Details of the Comparison Benchmarks

We describe the LLM benchmarks used for comparison in each experiment\. Unless otherwise noted, we adopt the evaluation protocol inswallow\-evaluation\-instruct222Version: v202510[Swallow LLM Team et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib12)\.

English Benchmarks

- •MMLU\-Pro[Wang et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib17):A harder variant of MMLU that evaluates broad domain knowledge via multiple\-choice questions\. It spans 14 subject areas, including humanities, social sciences, and STEM\.
- •GPQA\-Diamond[Rein et al\. \(2024\)](https://arxiv.org/html/2609.03370#bib.bib16):A four\-choice, graduate\-level science QA benchmark written by domain experts\. It covers biology, physics, and chemistry and is designed to be difficult to solve via web search\.
- •Humanity’s Last Exam[Center for AI Safety et al\. \(2026\)](https://arxiv.org/html/2609.03370#bib.bib15):A cross\-disciplinary benchmark comprising 2,500 questions across dozens of subjects, including mathematics, the humanities, and the natural sciences\. It features a mix of multiple\-choice and short\-answer questions designed to evaluate expert\-level reasoning across broad academic domains\.

Japanese Benchmarks

- •JamC\-QA[Oka et al\. \(2026\)](https://arxiv.org/html/2609.03370#bib.bib1):A high\-difficulty four\-choice QA benchmark specialized for knowledge of Japan\-specific culture and customs\. Questions cover eight categories including culture, customs, local environment, geography, administration, law, medicine, and related topics, requiring niche and diverse knowledge\.
- •MMLU\-ProX[Xuan et al\. \(2025\)](https://arxiv.org/html/2609.03370#bib.bib2):A multilingual extension of MMLU\-Pro covering 29 languages\. We use its Japanese subset\.

### B\.5Details of the Frame Identification Evaluation

This section describes the frame identification evaluation\. Unlike FrameBench, this evaluation directly provides models with candidate frame names and their definitions from the corresponding frame\-semantic resources: FrameNet for English and Japanese FrameNet for Japanese\. It therefore tests whether models can identify the frame evoked by a target word when the relevant candidate frames are explicitly given\.

Each test instance consists of a sentence containing a target frame\-evoking verb and a set of answer options\. The answer options include two candidate frames that the same verb can evoke, as well as a negative option, “Neither frame is evoked\.” The order of the answer options is randomized\.

We construct this evaluation from the same verb–frame pairs used in FrameBench\. When a verb appears in multiple FrameBench pairs because it can evoke three or more frames, we randomly select one pair for that verb to avoid redundancy\. For each selected verb–frame pair, we sample one sentence for each of the two candidate frames from the corresponding frame\-semantic resource\. Thus, each pair yields two independent frame identification instances, one for each candidate frame\.

We evaluate 200 verb–frame pairs in a 3\-shot setting\. Because each pair contributes two independently evaluated target sentences, the resulting evaluation set contains 400 test instances in total\.

## Appendix CEffect of the LLM Used for Dataset Construction

To assess the effect of the LLM used in dataset construction, we generated an additional 100 English candidate FrameBench entries using Gemini 3\.1 Pro in place of GPT\-5 for Steps 1 and 2\. The verb–frame pairs used for these entries were selected from those used in the original GPT\-5\-based construction\. To limit additional annotation costs, we did not conduct the human validation and filtering in Step 3 for these additional entries\. We evaluated them using the same model evaluation protocol as in the main experiment\.

Modelℛ\\mathcal\{R\}FrameBench\(Gemini\-generated\)GPT\-5 nano✓92\.7GPT\-5✓97\.9Gemini 3\.1 Flash\-Lite✓98\.8Gemini 3\.1 Pro✓100\.0Gemma 4 E2B✓88\.6Gemma 4 E4B✓95\.5Gemma 4 31B✓98\.9Gemma 4 E2B–66\.2Gemma 4 E4B–85\.5Gemma 4 31B–97\.9Qwen3\.5\-0\.8B✓33\.8Qwen3\.5\-2B✓80\.8Qwen3\.5\-4B✓96\.6Qwen3\.5\-9B✓96\.9Qwen3\.5\-27B✓99\.1Qwen3\.5\-0\.8B–47\.6Qwen3\.5\-2B–47\.6Qwen3\.5\-4B–82\.4Qwen3\.5\-9B–86\.9Qwen3\.5\-27B–97\.1Table 11:English FrameBench results on 100 entries generated using Gemini 3\.1 Pro instead of GPT\-5 in Steps 1 and 2\.ℛ\\mathcal\{R\}denotes reasoning mode\.Table[11](https://arxiv.org/html/2609.03370#A3.T11)reports the results\. We compared the results on entries constructed using Gemini 3\.1 Pro with the original FrameBench results reported in Table[3](https://arxiv.org/html/2609.03370#S3.T3)\. Performance tended to be slightly higher for models from the same family as the LLM used to construct the benchmark\. However, the overall performance pattern was similar across the two sets, with a Spearman rank correlation ofρ=0\.986\\rho=0\.986\.

\(i\) Sent\. Pair Type\(ii\) Correct Sent\. Position\(iii\) Error BreakdownModelℛ\\mathcal\{R\}Acc\.BaseExt\.ΔB−E\\Delta\_\{B\-E\}1st2ndΔ1−2\\Delta\_\{1\-2\}BothNeitherOppositeGemma 4 E2B\-41\.541\.538\.838\.844\.2−5\.5\-5\.553\.429\.629\.6\+23\.8\+23\.820\.120\.122\.116\.416\.4Gemma 4 E4B\-84\.684\.684\.384\.384\.8−0\.4\-0\.484\.084\.085\.0−1\.0\-1\.05\.94\.04\.05\.65\.6Gemma 4 31B\-97\.697\.697\.797\.597\.5\+0\.2\+0\.298\.596\.896\.8\+1\.6\+1\.60\.70\.71\.20\.40\.4Gemma 4 E2B✓82\.382\.378\.678\.686\.0−7\.5\-7\.580\.780\.784\.0−3\.3\-3\.38\.58\.38\.30\.80\.8Gemma 4 E4B✓92\.292\.290\.990\.993\.5−2\.6\-2\.692\.092\.092\.5−0\.5\-0\.54\.03\.13\.10\.70\.7Gemma 4 31B✓99\.199\.199\.299\.099\.0\+0\.2\+0\.299\.598\.698\.6\+0\.9\+0\.90\.40\.30\.30\.10\.1Qwen3\.5\-0\.8B\-25\.425\.425\.525\.425\.4\+0\.1\+0\.127\.823\.123\.1\+4\.8\+4\.828\.820\.420\.425\.425\.4Qwen3\.5\-2B\-30\.030\.028\.828\.831\.3−2\.5\-2\.534\.525\.525\.5\+8\.9\+8\.921\.921\.928\.020\.120\.1Qwen3\.5\-4B\-56\.756\.757\.655\.755\.7\+1\.9\+1\.966\.546\.846\.8\+19\.6\+19\.612\.112\.119\.012\.312\.3Qwen3\.5\-9B\-72\.972\.973\.772\.172\.1\+1\.6\+1\.674\.271\.671\.6\+2\.6\+2\.610\.48\.38\.38\.48\.4Qwen3\.5\-27B\-83\.783\.785\.481\.981\.9\+3\.5\+3\.588\.578\.978\.9\+9\.6\+9\.610\.22\.82\.83\.33\.3Qwen3\.5\-0\.8B✓24\.224\.223\.223\.225\.1−1\.9\-1\.924\.124\.124\.3−0\.3\-0\.326\.025\.125\.124\.724\.7Qwen3\.5\-2B✓60\.760\.758\.258\.263\.0−4\.8\-4\.860\.660\.660\.7−0\.1\-0\.119\.012\.612\.67\.77\.7Qwen3\.5\-4B✓89\.989\.989\.089\.090\.8−1\.8\-1\.889\.989\.990\.0−0\.1\-0\.13\.63\.64\.32\.22\.2Qwen3\.5\-9B✓91\.891\.890\.890\.892\.8−1\.9\-1\.991\.591\.592\.2−0\.7\-0\.72\.82\.84\.01\.41\.4Qwen3\.5\-27B✓97\.397\.396\.996\.997\.7−0\.8\-0\.897\.597\.197\.1\+0\.4\+0\.41\.11\.11\.20\.40\.4Table 12:Breakdown of Japanese FrameBench performance by sentence\-pair type, position of the correct sentence, and error type\.ℛ\\mathcal\{R\}denotes reasoning mode, and Acc\. denotes overall accuracy\. Base and Ext\. report accuracy on base and extended sentence pairs, while 1st and 2nd report accuracy when the correct sentence appears first or second\.ΔB−E\\Delta\_\{B\-E\}andΔ1−2\\Delta\_\{1\-2\}indicate performance differences\. Both denotes selecting both sentences, Neither denotes selecting neither sentence, and Opposite denotes selecting the incorrect sentence\. Acc\. and the three error\-type rates sum to 100% up to rounding\.These results suggest that the overall performance trends are largely preserved when a different LLM is used in dataset construction\.

## Appendix DAnalysis on the Japanese Subset

Table[12](https://arxiv.org/html/2609.03370#A3.T12)reports the behavioral analysis on the Japanese subset of FrameBench, covering sentence\-pair type, correct sentence position, and error type\. The Japanese results are generally consistent with the English analysis for correct sentence position and error type, but differ in sentence\-pair type\. In English, extended pairs show the expected advantage over base pairs, with a gap of 2\.57 points in human scores\. In Japanese, this effect is weaker: the corresponding gap decreases to 0\.93 points, suggesting that the surface\-level contrast between base and extended pairs is less pronounced\. Consistent with this weaker contrast, the LLM results in Japanese show a less consistent advantage for extended pairs\.