How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing

# How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing
Source: [https://arxiv.org/html/2610.13236](https://arxiv.org/html/2610.13236)
Anuj KumarAffiliation:Indian Institute of Technology Jammu, IndiaAffiliation:Nara Institute of Science and Technology \(NAIST\), JapanEmail:[ozaki\.shintaro\.ou6@naist\.ac\.jp](mailto:)Shintaro OzakiAffiliation:Nara Institute of Science and Technology \(NAIST\), JapanEmail:[kamigaito\.h@naist\.ac\.jp](mailto:)Yamuna PrasadKatsuhiko HayashiAffiliation:Nara Institute of Science and Technology \(NAIST\), JapanAffiliation:The University of Tokyo, Japan\{anuj,yamuna\.prasad\}@iitjammu\.ac\.in, katsuhiko\-hayashi@g\.ecc\.u\-tokyo\.ac\.jpHidetaka KamigaitoAffiliation:Nara Institute of Science and Technology \(NAIST\), JapanTaro WatanabeAffiliation:Nara Institute of Science and Technology \(NAIST\), Japan

###### Abstract

Recent advances have significantly improved the performance of Abstract Meaning Representation \(AMR\) parsers, yet determining when their outputs can be trusted remains largely unexplored\. This work presents the first study of uncertainty estimation for AMR parsers in a black\-box setting using top\-kkbeam\-search candidates\. Uncertainty is estimated from pairwise candidate similarities and evaluated using the Prediction Rejection Ratio\. This work characterises the relationship between parsing performance and uncertainty, evaluates the reliability of beam candidate distributions, analyses the effect of evaluation metrics on uncertainty assessment, and identifies parsers that provide the strongest balance between performance and reliability\. Experimental results show the value of uncertainty estimation as a complementary dimension for evaluating AMR parsers\.

## 1Introduction

Abstract Meaning Representation \(AMR\) represents sentence meaning as a rooted semantic graph that captures concepts and their relations beyond surface text[Banarescu et al\. \(2013\)](https://arxiv.org/html/2610.13236#bib.bib19)\. By modelling predicate–argument structure, AMR supports meaning\-centric NLP applications including information extraction[Han et al\. \(2025\)](https://arxiv.org/html/2610.13236#bib.bib20), question answering[Rakshit and Flanigan \(2025\)](https://arxiv.org/html/2610.13236#bib.bib21), summarisation[Qiu et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib22), multilingual semantic parsing[Vanroy and Van de Cruys \(2024\)](https://arxiv.org/html/2610.13236#bib.bib23), and trustworthy AI[Regan et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib24);[Pust et al\. \(2015\)](https://arxiv.org/html/2610.13236#bib.bib18)\.

However, manually constructing AMR graphs is expensive and difficult to scale, motivating the development of automatic AMR parsers that convert text into structured semantic representations\. Advances in pretrained language models have substantially improved parsers such as SPRING[Bevilacqua et al\. \(2021\)](https://arxiv.org/html/2610.13236#bib.bib1), IBM\-Parser[Fernandez Astudillo et al\. \(2020\)](https://arxiv.org/html/2610.13236#bib.bib2), HMG\-AMR[Sataer et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib3), MBSE[Lee et al\. \(2022\)](https://arxiv.org/html/2610.13236#bib.bib4), and IBM\-Neural[Drozdov et al\. \(2022\)](https://arxiv.org/html/2610.13236#bib.bib5), making AMR parsing increasingly important for downstream applications where graph quality directly affects reasoning and decision\-making\.

Despite recent advances in AMR parsing, an important question remains unresolved: how confident should a parser be in its prediction? Recent studies have explored uncertainty estimation for large language models using probabilistic, diversity\-based, and graph\-based approaches[Kuhn et al\. \(2023\)](https://arxiv.org/html/2610.13236#bib.bib12);[Lin et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib13);[Hashimoto et al\. \(2025\)](https://arxiv.org/html/2610.13236#bib.bib7);[Vashurin et al\. \(2025\)](https://arxiv.org/html/2610.13236#bib.bib25), highlighting the importance of estimating prediction reliability beyond output generation\. However, Uncertainty Estimation \(UE\) for AMR parsing remains largely unexplored\. Existing evaluation protocols rely on graph similarity metrics[Cai and Knight \(2013\)](https://arxiv.org/html/2610.13236#bib.bib8);[Opitz et al\. \(2020\)](https://arxiv.org/html/2610.13236#bib.bib9);[Anchiêta et al\. \(2019\)](https://arxiv.org/html/2610.13236#bib.bib10);[Song and Gildea \(2019\)](https://arxiv.org/html/2610.13236#bib.bib15);[Opitz \(2023\)](https://arxiv.org/html/2610.13236#bib.bib14);[Opitz et al\. \(2021\)](https://arxiv.org/html/2610.13236#bib.bib11), which compare predicted graphs against gold annotations and measure aggregate parser performance\. Nevertheless, these metrics provide limited insight into the reliability of individual predictions and into whether a parser’s confidence aligns with its actual performance\. To the best of our knowledge, this work presents the first systematic study of uncertainty estimation for AMR parsing in a black\-box setting and addresses the following research questions:

RQ1:Do performance and uncertainty correlate in AMR parsing?

RQ2:How reliable are beam candidates?

RQ3:Is relying on a single metric enough to judge the uncertainty of AMR parsers?

RQ4:Which parser is best in terms of both performance and reliability?

Our experiments111[https://github\.com/naist\-nlp/UE\_Method\_AMR\_Parser](https://github.com/naist-nlp/UE_Method_AMR_Parser)evaluate uncertainty estimation across multiple AMR parsers in a black\-box setting\. For each input, the parser generates a set of beam candidates, from which an uncertainty estimator assigns a scalar uncertainty score; higher scores indicate greater uncertainty\. In this work,Uncertainty Estimation\(UE\) refers to the process of assigning these scores, whileReliabilityrefers to how well the scores reflect the actual quality of predictions\. The reliability of these uncertainty estimates is evaluated using the Prediction Rejection Ratio \(PRR\)[Malinin et al\. \(2017\)](https://arxiv.org/html/2610.13236#bib.bib16), which measures how effectively the uncertainty scores identify low\-quality predictions: a reliable estimator should assign higher uncertainty to predictions with lower parsing quality, such that rejecting the most uncertain predictions improves the remaining average quality\. Results show a general tendency for higher\-performing parsers to produce more reliable uncertainty estimates, although parser architecture and candidate\-generation behaviour also influence reliability\. Moderate beam sizes yield the most reliable uncertainty estimates, while different evaluation metrics can lead to different conclusions about their reliability\. These findings demonstrate that uncertainty estimation provides a complementary perspective beyond parsing accuracy for evaluating AMR parsers\.

## 2Problem Definition

Let𝒟=\{\(si,gi∗\)\}i=1n\\mathcal\{D\}=\\\{\(s\_\{i\},g\_\{i\}^\{\*\}\)\\\}\_\{i=1\}^\{n\}denote a dataset of sizenn, wheresis\_\{i\}denotes an input sentence andgi∗g\_\{i\}^\{\*\}its corresponding gold AMR graph\. Given sentencesis\_\{i\}, an AMR parser generateskkbeam candidates

𝒞i=\{gi\(1\),…,gi\(k\)\},gi\(1\)=g^i,\\mathcal\{C\}\_\{i\}=\\\{g\_\{i\}^\{\(1\)\},\\ldots,g\_\{i\}^\{\(k\)\}\\\},\\qquad g\_\{i\}^\{\(1\)\}=\\hat\{g\}\_\{i\},\(1\)where𝒞i\\mathcal\{C\}\_\{i\}denotes the set ofkkcandidate AMR graphs,gi\(j\)g\_\{i\}^\{\(j\)\}thejj\-th beam prediction, andg^i\\hat\{g\}\_\{i\}the top\-1 prediction\. In the black\-box setting, parsing is defined asP:si↦𝒞iP:s\_\{i\}\\mapsto\\mathcal\{C\}\_\{i\}without access to model probabilities or internal states\. The objective is to learn an uncertainty estimatorUUthat maps beam candidates to a scalar uncertainty score,

U:𝒞i↦ui,ui∈ℝU:\\mathcal\{C\}\_\{i\}\\mapsto u\_\{i\},\\qquad u\_\{i\}\\in\\mathbb\{R\}\(2\)yielding one uncertainty score per input sentence and uncertainty vector𝒖=\(u1,…,un\)∈ℝn\\bm\{u\}=\(u\_\{1\},\\ldots,u\_\{n\}\)\\in\\mathbb\{R\}^\{n\}\. Prediction quality is computed between the top\-1 prediction and gold graph producing quality vector𝒒=\(q1,…,qn\)∈ℝn\\bm\{q\}=\(q\_\{1\},\\ldots,q\_\{n\}\)\\in\\mathbb\{R\}^\{n\}\. A desirable estimator satisfiesui∝\(1−qi\)u\_\{i\}\\propto\(1\-q\_\{i\}\)and maximises PRR\.

## 3Methodology

This work estimates uncertainty for AMR parsing in a black\-box setting, where model probabilities and internal states are unavailable\. The key intuition is that a set of beam candidates with high semantic agreement indicates lower uncertainty, whereas greater disagreement among candidates indicates higher uncertainty\. For each input sentence, pairwise similarities among the candidate AMR graphs are computed to construct a similarity matrix \(§[3\.1](https://arxiv.org/html/2610.13236#S3.SS1)\), which is then converted into a scalar uncertainty score \(§[3\.2](https://arxiv.org/html/2610.13236#S3.SS2)\)\. The quality of these uncertainty estimates is evaluated using the Prediction Rejection Ratio \(PRR\) \(§[3\.3](https://arxiv.org/html/2610.13236#S3.SS3)\), which measures how effectively the uncertainty scores identify low\-quality predictions and whether rejecting the most uncertain predictions improves the quality of the remaining predictions\.

### 3\.1Pairwise Similarity Matrix

For each candidate set𝒞i\\mathcal\{C\}\_\{i\}, pairwise candidate agreement is represented by

\[𝐒i\]j​l=Fsim​\(gi\(j\),gi\(l\)\),𝐒i∈ℝk×k,\[\\mathbf\{S\}\_\{i\}\]\_\{jl\}=F\_\{\\mathrm\{sim\}\}\(g\_\{i\}^\{\(j\)\},g\_\{i\}^\{\(l\)\}\),\\qquad\\mathbf\{S\}\_\{i\}\\in\\mathbb\{R\}^\{k\\times k\},\(3\)whereFsimF\_\{\\mathrm\{sim\}\}denotes an AMR similarity function,\[𝐒i\]j​l∈\[0,1\]\[\\mathbf\{S\}\_\{i\}\]\_\{jl\}\\in\[0,1\]measures similarity between beam candidatesgi\(j\)g\_\{i\}^\{\(j\)\}andgi\(l\)g\_\{i\}^\{\(l\)\}, and\[𝐒i\]j​j=1\[\\mathbf\{S\}\_\{i\}\]\_\{jj\}=1\. This procedure produces one similarity matrix𝐒i\\mathbf\{S\}\_\{i\}per sentence\.

### 3\.2Uncertainty Estimation

Each uncertainty estimator maps beam candidates to an uncertainty score asU:𝒞i↦𝐒i↦uiU:\\mathcal\{C\}\_\{i\}\\mapsto\\mathbf\{S\}\_\{i\}\\mapsto u\_\{i\}, where𝐒i∈ℝk×k\\mathbf\{S\}\_\{i\}\\in\\mathbb\{R\}^\{k\\times k\}is the pairwise similarity matrix,ui∈ℝu\_\{i\}\\in\\mathbb\{R\}is the scalar uncertainty score for sentencesis\_\{i\}, and𝒖=\(u1,…,un\)∈ℝn\\bm\{u\}=\(u\_\{1\},\\ldots,u\_\{n\}\)\\in\\mathbb\{R\}^\{n\}is the uncertainty vector\. We use the following estimators:

#### LexSim

Following[Kuhn et al\. \(2023\)](https://arxiv.org/html/2610.13236#bib.bib12), lexical uncertainty is estimated from pairwise ROUGE\-L[Lin \(2004\)](https://arxiv.org/html/2610.13236#bib.bib28)similarity:

uiLexSim=−1k⁡\(k−1\)∑j≠l\[𝐑i\]j​lu\_\{i\}^\{\\mathrm\{LexSim\}\}=\-\\frac\{1\}\{k\(k\-1\)\}\\sum\_\{j\\neq l\}\[\\mathbf\{R\}\_\{i\}\]\_\{jl\}\(4\)where\[𝐑i\]j​l∈\[0,1\]\[\\mathbf\{R\}\_\{i\}\]\_\{jl\}\\in\[0,1\]denotes the ROUGE\-L similarity between beam candidatesgi\(j\)g\_\{i\}^\{\(j\)\}andgi\(l\)g\_\{i\}^\{\(l\)\}\.

#### Graph\-Based Methods

Following[Lin et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib13),𝐒i\\mathbf\{S\}\_\{i\}is interpreted as a weighted graph\. The symmetrized adjacency matrix𝐀∈ℝk×k\\mathbf\{A\}\\in\\mathbb\{R\}^\{k\\times k\}and degree matrix𝐃∈ℝk×k\\mathbf\{D\}\\in\\mathbb\{R\}^\{k\\times k\}are defined as

\[𝐀\]j​l=\[𝐒i\]j​l\+\[𝐒i\]l​j2,\[𝐃\]j​j=∑l\[𝐀\]j​l\[\\mathbf\{A\}\]\_\{jl\}=\\frac\{\[\\mathbf\{S\}\_\{i\}\]\_\{jl\}\+\[\\mathbf\{S\}\_\{i\}\]\_\{lj\}\}\{2\},\\qquad\[\\mathbf\{D\}\]\_\{jj\}=\\sum\_\{l\}\[\\mathbf\{A\}\]\_\{jl\}\(5\)and transformed into the normalised Laplacian

𝐋=𝐈−𝐃−1/2𝐀𝐃−1/2,\\mathbf\{L\}=\\mathbf\{I\}\-\\mathbf\{D\}^\{\-1/2\}\\mathbf\{A\}\\mathbf\{D\}^\{\-1/2\},\(6\)where𝐋∈ℝk×k\\mathbf\{L\}\\in\\mathbb\{R\}^\{k\\times k\}andλ1,…,λk\\lambda\_\{1\},\\ldots,\\lambda\_\{k\}denote its eigenvalues\. The normalised Laplacian provides a spectral representation of candidate connectivity: the multiplicity of its zero eigenvalue equals the number of connected components in the graph\([von Luxburg, 2007](https://arxiv.org/html/2610.13236#bib.bib17), Proposition 4\)\. This property motivates the use of the Laplacian eigenvalues and eigenvectors in the followingEigVof Equation[7](https://arxiv.org/html/2610.13236#S3.E7)andEccof Equation[8](https://arxiv.org/html/2610.13236#S3.E8)\.

EigVestimates uncertainty from eigenvalue deviations, where larger deviations\(1−λj\)\(1\-\\lambda\_\{j\}\)correspond to greater candidate disagreement:

uiEigV=∑j=1kmax⁡\(0,1−λj\)\.u\_\{i\}^\{\\mathrm\{EigV\}\}=\\sum\_\{j=1\}^\{k\}\\max\(0,1\-\\lambda\_\{j\}\)\.\(7\)
Eccestimates uncertainty from candidate spread in the Laplacian eigenspace\. Let𝒗1,…,𝒗k∈ℝk\\bm\{v\}\_\{1\},\\ldots,\\bm\{v\}\_\{k\}\\in\\mathbb\{R\}^\{k\}denote the eigenvectors of𝐋\\mathbf\{L\}\. Each candidategi\(j\)g\_\{i\}^\{\(j\)\}is embedded as𝒆j=\(v1,j,…,vk,j\)⊤∈ℝk\\bm\{e\}\_\{j\}=\(v\_\{1,j\},\\ldots,v\_\{k,j\}\)^\{\\top\}\\in\\mathbb\{R\}^\{k\}and mean\-centred as𝒆~j=𝒆j−1k​∑l=1k𝒆l\\tilde\{\\bm\{e\}\}\_\{j\}=\\bm\{e\}\_\{j\}\-\\frac\{1\}\{k\}\\sum\_\{l=1\}^\{k\}\\bm\{e\}\_\{l\}; uncertainty is the Frobenius norm of the centred embeddings:

uiEcc=∑j=1k‖𝒆~j‖2\.u\_\{i\}^\{\\mathrm\{Ecc\}\}=\\sqrt\{\\sum\_\{j=1\}^\{k\}\\\|\\tilde\{\\bm\{e\}\}\_\{j\}\\\|^\{2\}\}\.\(8\)
DegMatdoes not rely on the normalized Laplacian and estimates uncertainty from average candidate agreement, where smaller∑j,l\[𝐀\]j​l\\sum\_\{j,l\}\[\\mathbf\{A\}\]\_\{jl\}indicates higher uncertainty:

uiDegMat=1−tr⁡\(𝐃\)k2=1−1k2​∑j,l\[𝐀\]j​l\.u\_\{i\}^\{\\mathrm\{DegMat\}\}=1\-\\frac\{\\mathrm\{tr\}\(\\mathbf\{D\}\)\}\{k^\{2\}\}=1\-\\frac\{1\}\{k^\{2\}\}\\sum\_\{j,l\}\[\\mathbf\{A\}\]\_\{jl\}\.\(9\)

### 3\.3Prediction Rejection Evaluation

An uncertainty estimator should assign a low uncertainty score to reliable predictions and a high uncertainty score to unreliable predictions\. PRR[Malinin et al\. \(2017\)](https://arxiv.org/html/2610.13236#bib.bib16)evaluates this behaviour by measuring quality improvement after rejecting uncertain outputs\.

Prediction quality is computed between the top\-1 predictiong^i\\hat\{g\}\_\{i\}and the gold graphgi∗g\_\{i\}^\{\*\}:

qi=ℱqual​\(g^i,gi∗\)∈ℝq\_\{i\}=\\mathcal\{F\}\_\{\\mathrm\{qual\}\}\(\\hat\{g\}\_\{i\},g\_\{i\}^\{\*\}\)\\in\\mathbb\{R\}\(10\)yielding quality vector𝒒=\(q1,…,qn\)∈ℝn\\bm\{q\}=\(q\_\{1\},\\ldots,q\_\{n\}\)\\in\\mathbb\{R\}^\{n\}\. Given uncertainty vector𝒖∈ℝn\\bm\{u\}\\in\\mathbb\{R\}^\{n\}, predictions are sorted by increasing uncertainty𝝅=argsort⁡\(𝒖\)∈ℤn\\bm\{\\pi\}=\\mathrm\{argsort\}\(\\bm\{u\}\)\\in\\mathbb\{Z\}^\{n\}\. At rejection rateα∈\[0,1\]\\alpha\\in\[0,1\], the⌊α​n⌋\\lfloor\\alpha n\\rfloormost uncertain predictions are removed, and the remaining mean quality is

Q⁡\(α\)=1n−⌊α​n⌋​∑i=1n−⌊α​n⌋qπi∈ℝQ\(\\alpha\)=\\frac\{1\}\{n\-\\lfloor\\alpha n\\rfloor\}\\sum\_\{i=1\}^\{n\-\\lfloor\\alpha n\\rfloor\}q\_\{\\pi\_\{i\}\}\\in\\mathbb\{R\}\(11\)whereπi\\pi\_\{i\}is theii\-th element of𝝅\\bm\{\\pi\}\. The area under the resulting prediction–rejection curve \(AUCPR\) is normalised against oracle and random rejection:

PRR=AUCPRunc−AUCPRrandomAUCPRoracle−AUCPRrandom∈ℝ\\mathrm\{PRR\}=\\frac\{\\mathrm\{AUCPR\}\_\{\\mathrm\{unc\}\}\-\\mathrm\{AUCPR\}\_\{\\mathrm\{random\}\}\}\{\\mathrm\{AUCPR\}\_\{\\mathrm\{oracle\}\}\-\\mathrm\{AUCPR\}\_\{\\mathrm\{random\}\}\}\\in\\mathbb\{R\}\(12\)wherePRR=1\\mathrm\{PRR\}=1denotes oracle performance,PRR=0\\mathrm\{PRR\}=0random rejection, andPRR<0\\mathrm\{PRR\}<0harmful uncertainty estimates\. Further details on the algorithmic framework and AUCPR derivation are provided in Appendix[A](https://arxiv.org/html/2610.13236#A1)\.

## 4Experiments and Results

### 4\.1Experimental Setup

Experiments are conducted on the AMR 3\.0 test set[Knight et al\. \(2020\)](https://arxiv.org/html/2610.13236#bib.bib6), comprisingn=1,898n=1,898sentence\-\-graph pairs, using six AMR parsers222Parsers were selected based on reported performance and code availability\. LLM baselines were excluded due to substantially lower parsing performance\. Additional details are provided in Appendix[C\.1](https://arxiv.org/html/2610.13236#A3.SS1): SPRING[Bevilacqua et al\. \(2021\)](https://arxiv.org/html/2610.13236#bib.bib1), IBM Transition Parser[Fernandez Astudillo et al\. \(2020\)](https://arxiv.org/html/2610.13236#bib.bib2), HMG\-AMR\-BART[Sataer et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib3), HMG\-AMR\-T5[Sataer et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib3), MBSE[Lee et al\. \(2022\)](https://arxiv.org/html/2610.13236#bib.bib4), and IBM\-Neural[Drozdov et al\. \(2022\)](https://arxiv.org/html/2610.13236#bib.bib5)\. For each parser, beam search generates candidate sets withk∈\{3,5,8,10\}k\\in\\\{3,5,8,10\\\}\. Fewer than 1% of generated beam candidates are invalid across parsers and beam sizes and are assigned zero similarity when computing graph\-based similarity measures\.

AMR similarity function defined in Equation \([3](https://arxiv.org/html/2610.13236#S3.E3)\) is measured using Smatch[Cai and Knight \(2013\)](https://arxiv.org/html/2610.13236#bib.bib8), Smatch\+\+[Opitz \(2023\)](https://arxiv.org/html/2610.13236#bib.bib14), S2Match[Opitz et al\. \(2020\)](https://arxiv.org/html/2610.13236#bib.bib9), SEMA[Anchiêta et al\. \(2019\)](https://arxiv.org/html/2610.13236#bib.bib10), and WLK[Opitz et al\. \(2021\)](https://arxiv.org/html/2610.13236#bib.bib11), while prediction quality defined in Equation \([10](https://arxiv.org/html/2610.13236#S3.E10)\) is evaluated against gold graphs using Smatch\+\+[Opitz \(2023\)](https://arxiv.org/html/2610.13236#bib.bib14), S2Match, and SemBLEU[Song and Gildea \(2019\)](https://arxiv.org/html/2610.13236#bib.bib15)\. Additional implementation details, parser checkpoints, similarity metrics, and hardware configuration are provided in Appendix[B](https://arxiv.org/html/2610.13236#A2)\.

Table 1:Top\-1 AMR parsing performance measured using Smatch and Smatch\+\+\.Figure 1:Mean PRR for each parser and prediction quality metric, averaged across all beam sizes and uncertainty estimators\.
### 4\.2Results and Analysis

#### RQ1: Do performance and uncertainty correlate in AMR parsing?

Table[1](https://arxiv.org/html/2610.13236#S4.T1)and Figure[1](https://arxiv.org/html/2610.13236#S4.F1)compare parsing performance and PRR score across parsers\. Overall, parsers with higher top\-1 Smatch scores tend to achieve higher PRR, indicating a positive relationship between parsing accuracy and reliability\. However, parsers with comparable Smatch scores exhibit substantially different PRR values; for instance, MBSE and HMG\-AMR\-BART have mean PRRs of 0\.183 and 0\.154, respectively, under Smatch\+\+, despite near\-identical parsing accuracy, suggesting that candidate\-generation behaviour and architectural design also independently influence reliability\. Additional results are provided in Appendix[C\.1](https://arxiv.org/html/2610.13236#A3.SS1)and Appendix[C\.2](https://arxiv.org/html/2610.13236#A3.SS2), where positive correlations are observed consistently across evaluation metrics\. Quantitatively, the Spearman correlation between parsing performance and mean PRR isρ=0\.388\\rho=0\.388for Smatch\+\+,ρ=0\.317\\rho=0\.317for S2Match, andρ=0\.360\\rho=0\.360for SemBLEU, with the strongest correlation observed under Smatch\+\+\. These results further support the positive relationship between parsing performance and uncertainty\-estimation reliability\.

Figure 2:Mean PRR averaged across uncertainty estimation methods and prediction quality metrics for beam sizesk∈\{3,5,8,10\}k\\in\\\{3,5,8,10\\\}\.
#### RQ2: How reliable are beam candidates?

Beam candidates provide meaningful uncertainty estimates, as shown in Figure[2](https://arxiv.org/html/2610.13236#S4.F2)\. However, PRR declines across most parsers as beam size increases, suggesting that larger beams introduce a more diverse set of candidates, weakening the relationship between uncertainty and prediction quality\. The rate of degradation varies substantially: HMG\-AMR\-T5 falls below zero atk=10k=10, indicating that the uncertainty estimates are actively harmful, whereas transition\-based parsers remain comparatively stable, potentially because their architecture explores structured action sequences rather than relying solely on graph linearisation\. Overall, moderate beam sizes provide the strongest trade\-off between candidate diversity and uncertainty quality\. Additional analysis across beam sizes and uncertainty estimators is provided in Appendix[C\.3](https://arxiv.org/html/2610.13236#A3.SS3), whereLexSimis the only estimator whose PRR consistently increases with beam size, whereasEccexhibits the steepest degradation\. These results show that increasing the beam size does not affect all uncertainty estimators equally, as their ability to capture candidate agreement varies\.

Figure 3:Each value shows the mean PRR averaged across uncertainty estimation methods and prediction quality metrics\.
#### RQ3: Is relying on a single metric enough to judge the uncertainty of AMR parsers?

No, the uncertainty of AMR parsers cannot be judged by using a single evaluation metric\. As illustrated in Figure[3](https://arxiv.org/html/2610.13236#S4.F3), PRR varies across metrics, with transition\-based parsers achieving stronger PRR under S2Match than Smatch\+\+, indicating that uncertainty assessment depends on the notion of graph similarity being measured\. Additional analysis is provided in Appendix[C\.4](https://arxiv.org/html/2610.13236#A3.SS4), where graph\-based estimators show strong mutual agreement under Smatch\+\+, whileLexSimshows near\-zero or negative correlation with most graph\-based methods\. Under SemBLEU, estimator agreement becomes more uniform, indicating that the relationship between uncertainty estimators also changes with the evaluation metric\. These observations reinforce that reliability is jointly determined by the uncertainty estimator and the metric used to assess prediction quality, making multi\-metric evaluation important for a robust assessment\.

#### RQ4: Which parser is best in terms of both performance and reliability?

SPRING[Bevilacqua et al\. \(2021\)](https://arxiv.org/html/2610.13236#bib.bib1)achieves the strongest overall balance, obtaining the highest mean PRR \(0\.265\) while maintaining positive uncertainty estimates across all beam sizes and evaluation metrics as shown in Figure[1](https://arxiv.org/html/2610.13236#S4.F1)and[2](https://arxiv.org/html/2610.13236#S4.F2)\. Its seq2seq formulation with graph\-isomorphic linearisation reduces reliance on heuristic processing, preserving semantically coherent candidate sets as beam diversity increases\. Additional parser reliability results across beam sizes are provided in Appendix[C\.5](https://arxiv.org/html/2610.13236#A3.SS5), where IBM\-Parser remains the most stable across all metrics\. At the same time, HMG\-AMR\-T5 exhibits the steepest degradation, approaching zero or negative PRR atk=8k=8andk=10k=10\. Detailed estimation results atk=10k=10are provided in Appendix[C\.6](https://arxiv.org/html/2610.13236#A3.SS6), whereDegMat\-SEMAachieves the strongest PRR for SPRING and HMG variants, whileLexSimoutperforms graph\-based methods for IBM\-Parser and IBM\-Neural\. These results highlight that parser selection involves a trade\-off between overall parsing performance and the stability of uncertainty estimation across beam sizes\. While SPRING provides the strongest overall balance, IBM\-Parser may be preferable when stable uncertainty estimation across different beam sizes is prioritised\.

### 4\.3Additional Analysis

To further investigate factors affecting uncertainty estimation and reliability, the analysis examines sentence length and agreement between uncertainty estimators\. Short sentences \(≤9\\leq 9tokens\) receive substantially higher mean uncertainty scores than medium\- and long\-length sentences; however, PRR indicates that uncertainty estimates are most reliable for medium\-length sentences, followed by long and short sentences\. The analysis further shows strong agreement among graph\-based estimators\. Additional analysis is provided in Appendix[C\.3](https://arxiv.org/html/2610.13236#A3.SS3)and[C\.6](https://arxiv.org/html/2610.13236#A3.SS6)showing that parser reliability varies substantially with beam size: some parsers remain stable as the beam expands, whereas others exhibit pronounced degradation\. At larger beam sizes, the most reliable UE also varies across parsers, indicating that no single estimator consistently provides the strongest reliability\.

## 5Conclusion

This work presents UE for AMR parsers in a black\-box setting\. Across multiple AMR parsers, beam sizes, and evaluation metrics, it is observed that more accurate parsers generally yield more reliable uncertainty estimates\. At the same time, parser reliability also varies with beam size and parser architecture\. The results further show that the reliability of UE depends on the evaluation metric and that moderate beam sizes tend to yield more reliable UE than larger beams\. Importantly, the experiments demonstrate that parsing performance and uncertainty estimation with reliability provide complementary information for comparing AMR parsers\. Parsing accuracy measures the quality of the predicted AMRs, whereas PRR evaluates how effectively the estimated uncertainty identifies potentially low\-quality predictions\. Since parsers with similar parsing accuracy can exhibit substantially different PRR, considering both performance and reliability provides a more comprehensive assessment of parser behaviour\. Overall, the findings support the use of both conventional accuracy\-based evaluation and uncertainty\-based reliability evaluation when comparing AMR parsers\.

## Limitations

This work studies uncertainty estimation for AMR parsing under a black\-box setting using beam\-search candidates as the sole source of uncertainty\. Although beam decoding is adopted by most publicly available AMR parsers and therefore provides a consistent evaluation setting across models, the behaviour of uncertainty under alternative decoding strategies \(e\.g\., sampling\-based or diverse decoding methods\) remains unexplored and may yield different candidate distributions and reliability characteristics\.

A second limitation concerns evaluation\. Uncertainty quality in this study is measured using the PRR alongside existing AMR quality metrics\. However, current AMR evaluation metrics capture distinct aspects of graph quality and can yield different conclusions about uncertainty, as observed in our experiments\. This suggests that uncertainty\-aware evaluation for AMR remains an open problem and that future work may benefit from developing reliability\-oriented evaluation protocols that complement conventional parsing metrics\.

Finally, this study focuses on post\-hoc uncertainty estimation without modifying parser architectures or training objectives\. Integrating uncertainty directly into AMR parser training and decoding may further improve reliability\.

## Acknowledgment

The authors would like to thank the Japan Science and Technology Agency \(JST\) for supporting the research activity for this project at the Nara Institute of Science and Technology \(NAIST\) through the LOTUS Programme \(India\-Japan Circulation of Talented Youths in Science Programme\)\.

## Ethical Considerations

AI\-assisted tools were used to support parts of the research workflow\. ChatGPT333[https://chat\.openai\.com/](https://chat.openai.com/)was used for writing support and language refinement, Claude Code444[https://www\.anthropic\.com/claude\-code](https://www.anthropic.com/claude-code)was used to accelerate research workflows, and Grammarly555[https://www\.grammarly\.com/](https://www.grammarly.com/)was used for grammar correction and proofreading\. All experiments, analyses, research decisions, and final manuscript content were conducted and verified by the authors\.

## References

- Anchiêtaet al\.\(2019\)R\. T\. Anchiêta, M\. A\. S\. Cabezudo, and T\. A\. S\. PardoSEMA: an extended semantic evaluation metric for amr\.ArXivabs/1905\.12069\.External Links:[Link](https://api.semanticscholar.org/CorpusID:168169627)Cited by:[3rd item](https://arxiv.org/html/2610.13236#A2.I2.i3.p1.1),[§1](https://arxiv.org/html/2610.13236#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p2.1)\.
- Banarescuet al\.\(2013\)L\. Banarescu, C\. Bonial, S\. Cai, M\. Georgescu, K\. Griffitt, U\. Hermjakob, K\. Knight, P\. Koehn, M\. Palmer, and N\. SchneiderAbstract Meaning Representation for sembanking\.InProceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse,pp\. 178–186\.External Links:[Link](https://aclanthology.org/W13-2322/)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p1.1)\.
- Bevilacquaet al\.\(2021\)M\. Bevilacqua, R\. Blloshmi, and R\. NavigliOne spring to rule them both: symmetric amr semantic parsing and generation without a complex pipeline\.Proceedings of the AAAI Conference on Artificial Intelligence35\(14\),pp\. 12564–12573\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v35i14.17489)Cited by:[1st item](https://arxiv.org/html/2610.13236#A2.I1.i1.p1.1),[Table 3](https://arxiv.org/html/2610.13236#A3.T3.2.1.2.1),[§1](https://arxiv.org/html/2610.13236#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2610.13236#S4.SS2.SSS0.Px4.p1.1),[Table 1](https://arxiv.org/html/2610.13236#S4.T1.2.1.2.1)\.
- Cai and Knight \(2013\)S\. Cai and K\. KnightSmatch: an evaluation metric for semantic feature structures\.InProceedings of the 51st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 748–752\.External Links:[Link](https://aclanthology.org/P13-2131/)Cited by:[1st item](https://arxiv.org/html/2610.13236#A2.I2.i1.p1.1),[§1](https://arxiv.org/html/2610.13236#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p2.1)\.
- Drozdovet al\.\(2022\)A\. Drozdov, J\. Zhou, R\. Florian, A\. McCallum, T\. Naseem, Y\. Kim, and R\. AstudilloInducing and using alignments for transition\-based AMR parsing\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 1086–1098\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.80)Cited by:[4th item](https://arxiv.org/html/2610.13236#A2.I1.i4.p1.1),[Table 3](https://arxiv.org/html/2610.13236#A3.T3.2.1.6.1),[§1](https://arxiv.org/html/2610.13236#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p1.1),[Table 1](https://arxiv.org/html/2610.13236#S4.T1.2.1.6.1)\.
- Fernandez Astudilloet al\.\(2020\)R\. Fernandez Astudillo, M\. Ballesteros, T\. Naseem, A\. Blodgett, and R\. FlorianTransition\-based parsing with stack\-transformers\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1001–1007\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.89)Cited by:[2nd item](https://arxiv.org/html/2610.13236#A2.I1.i2.p1.1),[Table 3](https://arxiv.org/html/2610.13236#A3.T3.2.1.5.1),[§1](https://arxiv.org/html/2610.13236#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p1.1),[Table 1](https://arxiv.org/html/2610.13236#S4.T1.2.1.5.1)\.
- Hanet al\.\(2025\)P\. Han, L\. Pereira, F\. Cheng, W\. J\. She, and E\. AramakiAMR\-RE: Abstract Meaning Representations for retrieval\-based in\-context learning in relation extraction\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 4: Student Research Workshop\),External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-srw.33)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p1.1)\.
- Hashimotoet al\.\(2025\)W\. Hashimoto, H\. Kamigaito, and T\. WatanabeDecoding uncertainty: the impact of decoding strategies for uncertainty estimation in large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 14601–14613\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.788)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p3.1)\.
- Knightet al\.\(2020\)K\. Knight, B\. Badarau, L\. Baranescu, C\. Bonial, M\. Bardocz, K\. Griffitt, U\. Hermjakob, D\. Marcu, M\. Palmer, T\. O’Gorman, and N\. SchneiderAbstract Meaning Representation \(AMR\) Annotation Release 3\.0\.Linguistic Data Consortium,Philadelphia\.Note:LDC2020T02External Links:[Document](https://dx.doi.org/10.35111/44cy-bp51)Cited by:[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p1.1)\.
- Kuhnet al\.\(2023\)L\. Kuhn, Y\. Gal, and S\. FarquharSemantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VD-AYtP0dve)Cited by:[1st item](https://arxiv.org/html/2610.13236#A2.I4.i1.p1.1),[§1](https://arxiv.org/html/2610.13236#S1.p3.1),[§3\.2](https://arxiv.org/html/2610.13236#S3.SS2.SSS0.Px1.p1.1)\.
- Leeet al\.\(2022\)Y\. Lee, R\. Astudillo, H\. Thanh Lam, T\. Naseem, R\. Florian, and S\. RoukosMaximum Bayes Smatch ensemble distillation for AMR parsing\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 5379–5392\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.393)Cited by:[3rd item](https://arxiv.org/html/2610.13236#A2.I1.i3.p1.1),[Table 3](https://arxiv.org/html/2610.13236#A3.T3.2.1.3.1),[§1](https://arxiv.org/html/2610.13236#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p1.1),[Table 1](https://arxiv.org/html/2610.13236#S4.T1.2.1.3.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§3\.2](https://arxiv.org/html/2610.13236#S3.SS2.SSS0.Px1.p1.1)\.
- Linet al\.\(2024\)Z\. Lin, S\. Trivedi, and J\. SunGenerating with confidence: uncertainty quantification for black\-box large language models\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=DWkJCSxKU5)Cited by:[2nd item](https://arxiv.org/html/2610.13236#A2.I4.i2.p1.1),[3rd item](https://arxiv.org/html/2610.13236#A2.I4.i3.p1.1),[4th item](https://arxiv.org/html/2610.13236#A2.I4.i4.p1.1),[§1](https://arxiv.org/html/2610.13236#S1.p3.1),[§3\.2](https://arxiv.org/html/2610.13236#S3.SS2.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)A\. H\. Liu, K\. Khandelwal, S\. Subramanian, V\. Jouault, A\. Rastogi, A\. Sadé, A\. Jeffares, A\. Jiang, A\. Cahill, A\. Gavaudan, A\. Sablayrolles, A\. Héliou, A\. You, A\. Ehrenberg, A\. Lo, A\. Eliseev, A\. Calvi, A\. Sooriyarachchi, B\. Bout, B\. Rozière, B\. D\. Monicault, C\. Lanfranchi, C\. Barreau, C\. Courtot, D\. Grattarola, D\. Dabert, D\. de las Casas, E\. Chane\-Sane, F\. Ahmed, G\. Berrada, G\. Ecrepont, G\. Guinet, G\. Novikov, G\. Kunsch, G\. Lample, G\. Martin, G\. Gupta, J\. Ludziejewski, J\. Rute, J\. Studnia, J\. Amar, J\. Delas, J\. S\. Roberts, K\. Yadav, K\. Chandu, K\. Jain, L\. Aitchison, L\. Fainsin, L\. Blier, L\. Zhao, L\. Martin, L\. Saulnier, L\. Gao, M\. Buyl, M\. Jennings, M\. Pellat, M\. Prins, M\. Poirée, M\. Guillaumin, M\. Dinot, M\. Futeral, M\. Darrin, M\. Augustin, M\. Chiquier, M\. Schimpf, N\. Grinsztajn, N\. Gupta, N\. Raghuraman, O\. Bousquet, O\. Duchenne, P\. Wang, P\. von Platen, P\. Jacob, P\. Wambergue, P\. Kurylowicz, P\. R\. Muddireddy, P\. Chagniot, P\. Stock, P\. Agrawal, Q\. Torroba, R\. Sauvestre, R\. Soletskyi, R\. Menneer, S\. Vaze, S\. Barry, S\. Gandhi, S\. Waghjale, S\. Gandhi, S\. Ghosh, S\. Mishra, S\. Aithal, S\. Antoniak, T\. L\. Scao, T\. Cachet, T\. S\. Sorg, T\. Lavril, T\. N\. Saada, T\. Chabal, T\. Foubert, T\. Robert, T\. Wang, T\. Lawson, T\. Bewley, T\. Bewley, T\. Edwards, U\. Jamil, U\. Tomasini, V\. Nemychnikova, V\. Phung, V\. Maladière, V\. Richard, W\. Bouaziz, W\. Li, W\. Marshall, X\. Li, X\. Yang, Y\. E\. Ouahidi, Y\. Wang, Y\. Tang, and Z\. RamziMinistral 3\.External Links:2601\.08584,[Link](https://arxiv.org/abs/2601.08584)Cited by:[Table 3](https://arxiv.org/html/2610.13236#A3.T3.2.1.9.1)\.
- Malininet al\.\(2017\)A\. Malinin, A\. Ragni, K\. Knill, and M\. GalesIncorporating uncertainty into deep learning for spoken language assessment\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 45–50\.External Links:[Document](https://dx.doi.org/10.18653/v1/P17-2008)Cited by:[§B\.4](https://arxiv.org/html/2610.13236#A2.SS4.p3.1),[§1](https://arxiv.org/html/2610.13236#S1.p5.1),[§3\.3](https://arxiv.org/html/2610.13236#S3.SS3.p1.1)\.
- Opitzet al\.\(2021\)J\. Opitz, A\. Daza, and A\. FrankWeisfeiler\-leman in the bamboo: novel AMR graph metrics and a benchmark for AMR graph similarity\.Transactions of the Association for Computational Linguistics9,pp\. 1425–1441\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00435)Cited by:[4th item](https://arxiv.org/html/2610.13236#A2.I2.i4.p1.1),[§1](https://arxiv.org/html/2610.13236#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p2.1)\.
- Opitzet al\.\(2020\)J\. Opitz, L\. Parcalabescu, and A\. FrankAMR similarity metrics from principles\.Transactions of the Association for Computational Linguistics8,pp\. 522–538\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00329)Cited by:[2nd item](https://arxiv.org/html/2610.13236#A2.I2.i2.p1.1),[2nd item](https://arxiv.org/html/2610.13236#A2.I3.i2.p1.1),[§B\.4](https://arxiv.org/html/2610.13236#A2.SS4.p3.1),[§1](https://arxiv.org/html/2610.13236#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p2.1)\.
- Opitz \(2023\)J\. OpitzSMATCH\+\+: standardized and extended evaluation of semantic graphs\.InFindings of the Association for Computational Linguistics: EACL 2023,pp\. 1595–1607\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-eacl.118)Cited by:[1st item](https://arxiv.org/html/2610.13236#A2.I3.i1.p1.1),[§B\.4](https://arxiv.org/html/2610.13236#A2.SS4.p3.1),[§1](https://arxiv.org/html/2610.13236#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p2.1)\.
- Pustet al\.\(2015\)M\. Pust, U\. Hermjakob, K\. Knight, D\. Marcu, and J\. MayParsing English into Abstract Meaning Representation using syntax\-based machine translation\.InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,pp\. 1143–1154\.External Links:[Document](https://dx.doi.org/10.18653/v1/D15-1136)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p1.1)\.
- Qiuet al\.\(2024\)H\. Qiu, K\. Huang, J\. Qu, and N\. PengAMRFact: enhancing summarization factuality evaluation with AMR\-driven negative samples generation\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.33)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p1.1)\.
- Rakshit and Flanigan \(2025\)G\. Rakshit and J\. FlaniganASQ: automatically generating question\-answer pairs using AMRs\.InProceedings of the 15th International Conference on Recent Advances in Natural Language Processing \- Natural Language Processing in the Generative AI Era,pp\. 1003–1011\.External Links:[Link](https://aclanthology.org/2025.ranlp-1.115/)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p1.1)\.
- Reganet al\.\(2024\)M\. Regan, S\. Wein, G\. Baker, and E\. MontiMASSIVE multilingual Abstract Meaning Representation: a dataset and baselines for hallucination detection\.InProceedings of the 13th Joint Conference on Lexical and Computational Semantics \(\*SEM 2024\),External Links:[Document](https://dx.doi.org/10.18653/v1/2024.starsem-1.1)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p1.1)\.
- Sataeret al\.\(2024\)Y\. Sataer, Y\. Fan, B\. Li, M\. Gao, C\. Shi, and Z\. GaoHierarchical information matters\! improving amr parsing with multi\-granularity representation interactions\.Information Processing & Management61\(3\),pp\. 103698\.External Links:ISSN 0306\-4573,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.ipm.2024.103698)Cited by:[5th item](https://arxiv.org/html/2610.13236#A2.I1.i5.p1.1),[6th item](https://arxiv.org/html/2610.13236#A2.I1.i6.p1.1),[Table 3](https://arxiv.org/html/2610.13236#A3.T3.2.1.4.1),[Table 3](https://arxiv.org/html/2610.13236#A3.T3.2.1.7.1),[§1](https://arxiv.org/html/2610.13236#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p1.1),[Table 1](https://arxiv.org/html/2610.13236#S4.T1.2.1.4.1),[Table 1](https://arxiv.org/html/2610.13236#S4.T1.2.1.7.1)\.
- Song and Gildea \(2019\)L\. Song and D\. GildeaSemBleu: a robust metric for AMR parsing evaluation\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4547–4552\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1446)Cited by:[3rd item](https://arxiv.org/html/2610.13236#A2.I3.i3.p1.1),[§B\.4](https://arxiv.org/html/2610.13236#A2.SS4.p3.1),[§1](https://arxiv.org/html/2610.13236#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.13236#S4.SS1.p2.1)\.
- Vanroy and Van de Cruys \(2024\)B\. Vanroy and T\. Van de CruysLess is enough: less\-resourced multilingual AMR parsing\.InProceedings of the 20th Joint ACL \- ISO Workshop on Interoperable Semantic Annotation @ LREC\-COLING 2024,pp\. 82–92\.External Links:[Link](https://aclanthology.org/2024.isa-1.11/)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p1.1)\.
- Vashurinet al\.\(2025\)R\. Vashurin, E\. Fadeeva, A\. Vazhentsev, L\. Rvanova, D\. Vasilev, A\. Tsvigun, S\. Petrakov, R\. Xing, A\. Sadallah, K\. Grishchenkov, A\. Panchenko, T\. Baldwin, P\. Nakov, M\. Panov, and A\. ShelmanovBenchmarking uncertainty quantification methods for large language models with LM\-polygraph\.Transactions of the Association for Computational Linguistics13\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00737)Cited by:[§1](https://arxiv.org/html/2610.13236#S1.p3.1)\.
- von Luxburg \(2007\)U\. von LuxburgA tutorial on spectral clustering\.External Links:0711\.0189,[Link](https://arxiv.org/abs/0711.0189)Cited by:[§3\.2](https://arxiv.org/html/2610.13236#S3.SS2.SSS0.Px2.p1.3)\.
- Wolfet al\.\(2020\)T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. RushTransformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[§B\.2](https://arxiv.org/html/2610.13236#A2.SS2.p3.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Table 3](https://arxiv.org/html/2610.13236#A3.T3.2.1.8.1)\.

## Appendix AAlgorithm

Algorithm 1Black\-box Uncertainty Estimation for AMR ParsingInput:Dataset

𝒟=\{\(si,gi∗\)\}i=1n\\mathcal\{D\}=\\\{\(s\_\{i\},g\_\{i\}^\{\*\}\)\\\}\_\{i=1\}^\{n\}, parser

PP, beam size

kk
Output:PRR

PRR∈ℝ\\mathrm\{PRR\}\\in\\mathbb\{R\}
foreach*\(si,gi∗\)∈𝒟\(s\_\{i\},g\_\{i\}^\{\*\}\)\\in\\mathcal\{D\}*do

Generate

kkbeam candidates via beam search;

𝒞i=P⁡\(si\)=\{gi\(1\),…,gi\(k\)\}\\mathcal\{C\}\_\{i\}=P\(s\_\{i\}\)=\\\{g\_\{i\}^\{\(1\)\},\\ldots,g\_\{i\}^\{\(k\)\}\\\}
Compute pairwise candidate similarity matrix;

𝐒i=Fsim​\(𝒞i\)∈ℝk×k,\[𝐒i\]j​j=1,\\mathbf\{S\}\_\{i\}=F\_\{\\mathrm\{sim\}\}\(\\mathcal\{C\}\_\{i\}\)\\in\\mathbb\{R\}^\{k\\times k\},\\qquad\[\\mathbf\{S\}\_\{i\}\]\_\{jj\}=1,\[𝐒i\]j​l∈\[0,1\]\[\\mathbf\{S\}\_\{i\}\]\_\{jl\}\\in\[0,1\]
UE methods;

uiLexSim\\displaystyle u\_\{i\}^\{\\mathrm\{LexSim\}\}=1−1k⁡\(k−1\)​∑j≠lROUGE⁡\(gi\(j\),gi\(l\)\),\\displaystyle=1\-\\frac\{1\}\{k\(k\-1\)\}\\sum\_\{j\\neq l\}\\mathrm\{ROUGE\}\(g\_\{i\}^\{\(j\)\},g\_\{i\}^\{\(l\)\}\),uiDegMat\\displaystyle u\_\{i\}^\{\\mathrm\{DegMat\}\}=1−2k⁡\(k−1\)​∑j<l\[𝐒i\]j​l,\\displaystyle=1\-\\frac\{2\}\{k\(k\-1\)\}\\sum\_\{j<l\}\[\\mathbf\{S\}\_\{i\}\]\_\{jl\},uiEigV\\displaystyle u\_\{i\}^\{\\mathrm\{EigV\}\}=1k​∑r=1k\(λr−1\)2,\\displaystyle=\\frac\{1\}\{k\}\\sum\_\{r=1\}^\{k\}\(\\lambda\_\{r\}\-1\)^\{2\},uiEcc\\displaystyle u\_\{i\}^\{\\mathrm\{Ecc\}\}=\(∑j=1k‖𝐞j−1k​∑l=1k𝐞l‖22\)1/2\.\\displaystyle=\\left\(\\sum\_\{j=1\}^\{k\}\\left\\\|\\mathbf\{e\}\_\{j\}\-\\frac\{1\}\{k\}\\sum\_\{l=1\}^\{k\}\\mathbf\{e\}\_\{l\}\\right\\\|\_\{2\}^\{2\}\\right\)^\{1/2\}\.Higher

uiu\_\{i\}indicates greater uncertainty;

Compute prediction quality against the gold graph;

qi=ℱqual​\(gi\(1\),gi∗\)∈ℝq\_\{i\}=\\mathcal\{F\}\_\{\\mathrm\{qual\}\}\(g\_\{i\}^\{\(1\)\},g\_\{i\}^\{\*\}\)\\in\\mathbb\{R\}
Construct uncertainty and quality vectors;

𝒖=\(u1,…,un\)∈ℝn,𝒒=\(q1,…,qn\)∈ℝn\\bm\{u\}=\(u\_\{1\},\\ldots,u\_\{n\}\)\\in\\mathbb\{R\}^\{n\},\\qquad\\bm\{q\}=\(q\_\{1\},\\ldots,q\_\{n\}\)\\in\\mathbb\{R\}^\{n\}
Sort by increasing uncertainty \(

π1\\pi\_\{1\}= least uncertain\);

𝝅=argsort⁡\(𝒖\)∈ℤn\\bm\{\\pi\}=\\mathrm\{argsort\}\(\\bm\{u\}\)\\in\\mathbb\{Z\}^\{n\}
At rejection rate

α\\alpha, remove

⌊α​n⌋\\lfloor\\alpha n\\rfloormost uncertain predictions and compute remaining mean quality;

Q⁡\(α\)=1n−⌊α​n⌋​∑r=1n−⌊α​n⌋qπrQ\(\\alpha\)=\\frac\{1\}\{n\-\\lfloor\\alpha n\\rfloor\}\\sum\_\{r=1\}^\{n\-\\lfloor\\alpha n\\rfloor\}q\_\{\\pi\_\{r\}\}
Compute area under the prediction\-\-rejection curve;

AUCPR=∫01Q⁡\(α\)​𝑑α\\mathrm\{AUCPR\}=\\int\_\{0\}^\{1\}Q\(\\alpha\)\\,\\mathrm\{d\}\\alpha
Normalise against oracle and random baselines;

PRR=AUCPRunc−AUCPRrandomAUCPRoracle−AUCPRrandom\\mathrm\{PRR\}=\\frac\{\\mathrm\{AUCPR\}\_\{\\mathrm\{unc\}\}\-\\mathrm\{AUCPR\}\_\{\\mathrm\{random\}\}\}\{\\mathrm\{AUCPR\}\_\{\\mathrm\{oracle\}\}\-\\mathrm\{AUCPR\}\_\{\\mathrm\{random\}\}\}
return

PRR\\mathrm\{PRR\}

### A\.1Algorithm

Algorithm[1](https://arxiv.org/html/2610.13236#algorithm1)summarises the complete procedure\.

### A\.2Computational Complexity

The dominant cost of the proposed framework is pairwise similarity computation, which requires𝒪⁡\(k2\)\\mathcal\{O\}\(k^\{2\}\)graph comparisons per sentence\. Alignment\-based metrics \(Smatch, Smatch\+\+, S2Match\) scale as𝒪⁡\(\|V\|2​r\)\\mathcal\{O\}\(\|V\|^\{2\}r\)per comparison, where\|V\|\|V\|is the number of graph nodes andrrthe number of alignment restarts; SEMA scales as𝒪⁡\(\|V\|\+\|E\|\)\\mathcal\{O\}\(\|V\|\+\|E\|\); and WLK scales as𝒪⁡\(\|V\|​\|E\|​h\)\\mathcal\{O\}\(\|V\|\|E\|h\), where\|E\|\|E\|is the number of edges andhhthe number of Weisfeiler–Lehman iterations\. The overall runtime acrossPPparsers,BBbeam configurations, andNNsentences is therefore𝒪⁡\(P⋅B⋅N⋅k2⋅Csim\)\\mathcal\{O\}\(P\\cdot B\\cdot N\\cdot k^\{2\}\\cdot C\_\{\\mathrm\{sim\}\}\), whereCsimC\_\{\\mathrm\{sim\}\}denotes the per\-comparison cost of the chosen metric\. Uncertainty estimation adds negligible overhead:DegMatandEccoperate in𝒪⁡\(N​k2\)\\mathcal\{O\}\(Nk^\{2\}\),EigVin𝒪⁡\(N​k3\)\\mathcal\{O\}\(Nk^\{3\}\), and PRR evaluation in𝒪⁡\(N​log⁡N\)\\mathcal\{O\}\(N\\log N\)per estimator–metric pair\. To avoid redundant computation, all pairwise similarity matrices are cached as compressed\.npzarrays after the first computation\.

## Appendix BExperiment Setup

This Appendix provides implementation details required to reproduce all experiments\.

### B\.1Dataset

Experiments are conducted on the AMR 3\.0 \([https://catalog\.ldc\.upenn\.edu/LDC2020T02](https://catalog.ldc.upenn.edu/LDC2020T02)\) test set containing 1,898 sentence–graph pairs\. Each example consists of an input sentence and its corresponding gold AMR graph represented in PENMAN notation\. All uncertainty estimation experiments are performed on parser outputs generated from this test split\.

### B\.2AMR Parsers

Six publicly available AMR parsers are evaluated in this study\. All experiments use official implementations and publicly released pretrained checkpoints without modifying model architectures or retraining\.

- •
- •
- •
- •
- •
- •

For reference, two instruction\-tuned LLM baselines \(Qwen3\-32B and Ministral\-14B\) were additionally evaluated using direct AMR generation without task\-specific training using the transformer library[Wolf et al\. \(2020\)](https://arxiv.org/html/2610.13236#bib.bib29)\. Due to substantially lower performance on AMR 3\.0 as shown in Table[3](https://arxiv.org/html/2610.13236#A3.T3)and the absence of dedicated AMR parsing capability, these models were excluded from uncertainty estimation experiments\.

### B\.3Similarity Construction and Evaluation

Pairwise candidate similarity matrices𝐒i∈ℝk×k\\mathbf\{S\}\_\{i\}\\in\\mathbb\{R\}^\{k\\times k\}are constructed using four AMR\-specific similarity metrics:

- •
- •
- •
- •

Prediction quality for PRR evaluation is measured using three metrics:

- •
- •
- •

### B\.4Uncertainty Estimation and Evaluation

Uncertainty is estimated using one lexical and three graph\-based approaches:

- •LexSim[Kuhn et al\. \(2023\)](https://arxiv.org/html/2610.13236#bib.bib12): estimates uncertainty from mean pairwise ROUGE\-L similarity across beam candidates; lower lexical overlap indicates greater candidate diversity and higher uncertainty\.
- •EigV[Lin et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib13): constructs a similarity graph over candidates and measures uncertainty from the eigenvalues of the normalised Laplacian; larger eigenvalue deviations from one indicate weaker graph connectivity and greater candidate disagreement\.
- •DegMat[Lin et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib13): measures uncertainty as the complement of the normalised total pairwise similarity; a lower sum of edge weights in the similarity graph indicates higher uncertainty\.
- •Ecc[Lin et al\. \(2024\)](https://arxiv.org/html/2610.13236#bib.bib13): embeds each candidate in the Laplacian eigenspace and measures uncertainty as the spread of embeddings around their centroid; a larger Frobenius norm indicates greater candidate dispersion and higher uncertainty\.

Prediction quality for PRR evaluation is measured using Smatch\+\+[Opitz \(2023\)](https://arxiv.org/html/2610.13236#bib.bib14), S2Match[Opitz et al\. \(2020\)](https://arxiv.org/html/2610.13236#bib.bib9), and SemBLEU[Song and Gildea \(2019\)](https://arxiv.org/html/2610.13236#bib.bib15)\. Uncertainty quality is evaluated using the PRR[Malinin et al\. \(2017\)](https://arxiv.org/html/2610.13236#bib.bib16)\.

### B\.5Hardware and Runtime

All uncertainty estimation and PRR calculations were performed in a CPU\-only environment without GPU acceleration\. Each job was allocated 4 CPU cores and 64 GB of RAM\.

Table[2](https://arxiv.org/html/2610.13236#A2.T2)reports the observed runtime measured from execution logs\. Generating beam candidates and computing uncertainty metrics required approximately 25 hours per parser across all beam sizes \(k∈\{3,5,8,10\}k\\in\\\{3,5,8,10\\\}\)\. Processing a single beam configuration required approximately 6 hours\. Running the complete evaluation pipeline across all six parsers took approximately 150 hours of wall\-clock time\.

Table 2:Hardware allocation and observed execution time were measured from job logs\.

## Appendix CAdditional Results

This section presents additional analyses supporting the claims in the main paper, including parser\-level performance comparisons, the correlation between AMR parsing quality and uncertainty estimation, and the stability of uncertainty estimators under varying beam sizes\.

Table 3:Top\-1 AMR parsing performance measured using Smatch\.Figure 4:Relationship between parser accuracy \(Smatch\) and uncertainty quality \(PRR\) across beam sizes and evaluation metrics\.### C\.1AMR Parsing Performance

Table[3](https://arxiv.org/html/2610.13236#A3.T3)reports parser Smatch score performance on the gold dataset reproduced using the publicly released implementations and default hyperparameter settings provided in the original papers\. To ensure consistent comparison across parsers, all results are evaluated before BLINKification\. Under this setting, SPRING achieves the highest parsing accuracy \(0\.8449\), followed by MBSE \(0\.8374\) and HMG\-AMR\-BART \(0\.8358\), indicating that recent encoder–decoder approaches remain highly competitive for AMR generation\. IBM\-Parser and IBM\-Neural obtain comparable performance around 0\.81 Smatch, whereas HMG\-AMR\-T5 shows lower accuracy relative to other specialised AMR parsers\. For reference, two instruction\-tuned LLM baselines \(Qwen3\-32B and Ministral\-14B\) are additionally evaluated using direct AMR generation without task\-specific training and exhibit substantially lower performance, highlighting the difficulty of generating valid AMR structures through general\-purpose language modelling alone\. Due to their poor performance on the AMR 3\.0 test set, these models are excluded from subsequent analysis\.

![Refer to caption](https://arxiv.org/html/2610.13236v1/ue_stability.png)

Figure 5:Mean PRR of uncertainty estimators across beam sizes aggregated over all parsers\. Stable estimators exhibit less variation in performance across beam sizes\.![Refer to caption](https://arxiv.org/html/2610.13236v1/ue_corr_heatmap_S++.png)

Figure 6:Pairwise Spearman correlation between uncertainty estimators evaluated using Smatch\+\+\. Graph\-based estimators exhibit strong agreement, whileLexSimcaptures a distinct uncertainty signal\.Table 4:PRR at beam sizek=10k=10for all uncertainty estimation methods across parsers and quality metrics \(Smatch\+\+ \(S\+\+\) and SemBLEU \(SB\)\)\. Higher PRR indicates better alignment between uncertainty estimates and prediction quality\. Green cells denote the best\-performing uncertainty estimator within each parser–metric pair\. Consistent with the main findings, SPRING achieves the strongest overall reliability, whileDegMat\-SEMAfrequently emerges as the most effective graph\-based uncertainty estimator at larger beam sizes\.Figure 7:Parser\-level PRR as a function of beam size evaluated using SemBLEU\. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes\.
### C\.2Relationship Between Parsing Performance and Reliability

Figure[4](https://arxiv.org/html/2610.13236#A3.F4)examines whether AMR parsing accuracy translates into more reliable uncertainty estimates\. Each point represents a parser–beam combination \(6 parsers×\\times4 beam sizes = 24 points per panel\), where the x\-axis shows the parser’s fixed top\-1 Smatch accuracy and the y\-axis shows mean PRR averaged across all 13 uncertainty estimators for that parser and beam size\. Because Smatch accuracy is fixed per parser, the four points per parser are stacked vertically, reflecting how PRR varies across beam sizes at the same accuracy level\.

Across all evaluation metrics, a moderate positive relationship is observed \(ρ=0\.388\\rho=0\.388for Smatch\+\+,ρ=0\.317\\rho=0\.317for S2Match, andρ=0\.360\\rho=0\.360for SemBLEU\), indicating that parsers with stronger top\-1 accuracy tend to achieve higher mean PRR across beam configurations\. However, the correlations confirm that accuracy alone does not fully determine the quality of uncertainty\. Parsers with comparable Smatch scores often occupy different vertical regions of the PRR space, suggesting that beam decoding behaviour and architectural design also independently influence reliability, beyond accuracy\.

The vertical spread of points within each parser further illustrates the effect of beam size on uncertainty quality: some parsers maintain stable PRR across beams, while others experience substantial degradation as beam size increases, reinforcing the findings from RQ2\. Finally, the variation inρ\\rhoacross evaluation metrics indicates that the strength of the performance–uncertainty relationship depends on how prediction quality is defined, consistent with the findings reported in RQ3\.

Overall, parser accuracy provides only a partial explanation of uncertainty quality, and reliability emerges as a complementary evaluation dimension that is not captured by conventional AMR parsing metrics alone\.

![Refer to caption](https://arxiv.org/html/2610.13236v1/ue_corr_heatmap_S2.png)

Figure 8:Pairwise Spearman correlation between uncertainty estimators evaluated using S2Match\.
### C\.3Uncertainty Estimator Stability Across Beam Sizes

Figure[5](https://arxiv.org/html/2610.13236#A3.F5)reports the mean PRR aggregated across all six parsers for each uncertainty estimator and beam size, directly supporting RQ2\. The overall pattern is consistent with the main results: uncertainty\-estimation reliability is highest atk=3k=3and generally declines as the beam size increases, with the sharpest drops occurring betweenk=5k=5andk=8k=8\.

Degradation rates differ markedly across estimators\.Ecc\-based methods start among the strongest atk=3k=3, withEcc\-S2matchreaching 0\.345 under S2Match, yet show the steepest decline, falling to 0\.141 atk=10k=10\.EigVvariants follow a similar trajectory, dropping from approximately0\.300\.30–0\.340\.34atk=3k=3to0\.140\.14\-0\.200\.20atk=10k=10, depending on the metric\. Thus, both estimator families show substantial degradation in reliability as the beam size increases\.

In contrast,LexSimis the only estimator whose mean PRR increases with beam size, rising from 0\.149 to 0\.171 under Smatch\+\+ and from 0\.187 to 0\.215 under S2Match, suggesting that its uncertainty estimates are less sensitive to increasing candidate diversity\. Among graph\-based methods, SEMA\-based variants show greater robustness:DegMat\-SEMAretains PRR values of 0\.182 and 0\.220 atk=10k=10under Smatch\+\+ and S2Match, respectively, substantially outperforming other graph\-based methods at larger beam sizes\.

Notably, S2Match consistently yields higher mean PRR than Smatch\+\+ across all estimators and beam sizes\. For example,EigV\-S2matchobtains 0\.335 compared with 0\.301 atk=3k=3\. This indicates that the observed effect of increasing beam size also depends on how prediction quality is measured, with S2Match generally producing stronger uncertainty\-reliability scores in this analysis\. Overall, moderate beam sizes \(k=3k=3tok=5k=5\) provide the strongest trade\-off between candidate diversity and uncertainty\-estimation reliability\.

Figure 9:Parser\-level PRR as a function of beam size evaluated using Smatch\+\+\. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes\.Figure 10:Parser\-level PRR as a function of beam size evaluated using S2Match\. Each subplot corresponds to one parser and compares uncertainty estimators across beam sizes\.![Refer to caption](https://arxiv.org/html/2610.13236v1/ue_corr_heatmap_SB.png)

Figure 11:Pairwise Spearman correlation between uncertainty estimators evaluated using SemBLEU\.
### C\.4Agreement Between UE

Figures[6](https://arxiv.org/html/2610.13236#A3.F6),[8](https://arxiv.org/html/2610.13236#A3.F8), and[11](https://arxiv.org/html/2610.13236#A3.F11)report pairwise Spearman correlations between uncertainty estimators across all parser and beam configurations\. These results provide additional evidence for RQ3 by examining whether uncertainty estimators produce consistent rankings under different evaluation metrics\.

Across all metrics,EigV,DegMat, andEccexhibit consistently strong mutual agreement\. Under Smatch\+\+, most pairwise correlations exceedρ=0\.87\\rho=0\.87, with estimators using the same similarity function approaching near\-perfect agreement \(e\.g\.EigV\-SmatchandDegMat\-Smatch:ρ=1\.00\\rho=1\.00\)\. Agreement is similarly high under S2Match, where several pairs reachρ=1\.00\\rho=1\.00, and remains strong under SemBLEU \(ρ≥0\.85\\rho\\geq 0\.85for most graph\-based pairs\)\. This indicates that graph\-based estimators generally preserve similar uncertainty rankings, although their agreement depends on the similarity function used\.

In contrast,LexSimshows substantially weaker correlation with graph\-based estimators across metrics\. Under Smatch\+\+, correlations range fromρ=−0\.16\\rho=\-0\.16\(EigV\-WLK\) toρ=0\.31\\rho=0\.31\(DegMat\-SEMA\), while under SemBLEU they decrease to as low asρ=−0\.14\\rho=\-0\.14\(DegMat\-WLK\)\. This indicates that lexical overlap captures an uncertainty signal that differs substantially from structural graph agreement, and that the degree of this difference varies with the evaluation metric\.

Metric\-dependent differences are also observed in the PRR results\. Figure[5](https://arxiv.org/html/2610.13236#A3.F5)shows consistently higher mean PRR under S2Match than Smatch\+\+ across estimators and beam sizes; for example,EigV\-S2matchobtains 0\.335 compared with 0\.301 atk=3k=3\. Under SemBLEU, WLK\-based estimators show stronger agreement with other graph\-based methods \(ρ≥0\.85\\rho\\geq 0\.85\), reducing the separation between estimator families observed under Smatch\+\+ and S2Match\. Overall, these results show that changing the evaluation metric can alter both uncertainty reliability and agreement between estimators, indicating that no single metric provides a complete view of uncertainty reliability\.

### C\.5Parser Reliability Across Beam Sizes

Figures[7](https://arxiv.org/html/2610.13236#A3.F7),[9](https://arxiv.org/html/2610.13236#A3.F9), and[10](https://arxiv.org/html/2610.13236#A3.F10)visualise parser\-level PRR across beam sizes, with each curve reporting the mean PRR for a given uncertainty estimator\.

SPRING maintains the strongest and most consistent reliability profile across all metrics\. Under Smatch\+\+, PRR remains above0\.180\.18atk=10k=10for all estimators exceptEcc, which converges with others at larger beams\. Under S2Match and SemBLEU, SPRING similarly retains positive PRR across all estimators and beam sizes, confirming its robustness to candidate expansion\.

IBM\-Parser is the most stable parser across beam sizes\. Under all three metrics, PRR remains nearly flat fromk=3k=3tok=10k=10, withLexSimconsistently outperforming graph\-based methods and remaining the strongest estimator throughout\. IBM\-Neural shows a similar stability pattern, withLexSimagain achieving the highest PRR at larger beams\.

HMG\-AMR\-T5 exhibits the steepest degradation across all metrics\. Under Smatch\+\+, graph\-based estimators drop below zero byk=8k=8, andLexSimalso approaches zero byk=10k=10\. Under SemBLEU, all estimators reach negative PRR atk=8k=8andk=10k=10, indicating that beam candidates become actively misleading\. HMG\-AMR\-BART shows similar sensitivity, particularly under SemBLEU, whereEccdrops below zero atk=10k=10\.

MBSE achieves a competitive PRR atk=3k=3under all metrics\. However, it degrades noticeably at larger beams, particularly under S2Match whereEccfalls to≈0\.12\\approx 0\.12atk=10k=10, substantially belowLexSimandDegMat\. Overall, these results reinforce RQ4: the most reliable parser is the one that maintains reliable uncertainty estimates as candidate diversity increases not the one with the highest parsing accuracy\.

### C\.6Detailed Uncertainty Estimation Results

Table[4](https://arxiv.org/html/2610.13236#A3.T4)reports complete PRR results atk=10k=10across all parsers, uncertainty estimators, and evaluation metrics\. These results provide a more detailed view of RQ4 by examining whether the reliability of a parser is consistent across different uncertainty estimators at a large beam size\.DegMat\-SEMAachieves the strongest PRR for SPRING under both Smatch\+\+ \(0\.271\) and SemBLEU \(0\.247\), and also leads for HMG\-AMR\-BART \(0\.137, 0\.109\) and HMG\-AMR\-T5 \(0\.069, 0\.023\)\. In contrast,LexSimachieves the strongest PRR for IBM\-Parser \(0\.266, 0\.162\), IBM\-Neural \(0\.266, 0\.169\), and MBSE \(0\.229, 0\.133\)\. Thus, the estimator providing the most reliable uncertainty signal atk=10k=10varies across parsers, indicating that parser comparison based on reliability is also influenced by the interaction between the parser and the uncertainty estimator\.

Ecc\-based methods perform weakest overall atk=10k=10, with several parsers recording negative PRR under SemBLEU, such as HMG\-AMR\-BART \(−0\.025\-0\.025\) and HMG\-AMR\-T5 \(−0\.148\-0\.148\)\. This suggests that eigenspace\-based candidate spread becomes less informative as beam size increases for these parsers\. Overall, the detailed results show that although SPRING provides the strongest overall reliability, its advantage does not imply that a single uncertainty estimator is optimal across all parsers\.