Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents

# Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
Source: [https://arxiv.org/html/2608.12342](https://arxiv.org/html/2608.12342)
Ying He1,Zhouhong Gu1,Zhecheng Hu1,Yubo Zhou1, Hao Shen1,Jiaqing Liang2,Zhaoqian Dai3,Shuguang Ma3, Fei Yu3,Yanghua Xiao1,Zhixu Li411footnotemark:1 1College of Computer Science and Artificial Intelligence, Fudan University, 2School of Data Science, Fudan University,3Ant Group, 4School of Information and School of Smart Governance, Renmin University of China \{yinghe23,zhouyb24,zchu24,hshen22\}@m\.fudan\.edu\.cn, \{zhgu20,liangjiaqing,shawyh\}@fudan\.edu\.cn,zhixuli@ruc\.edu\.cn \{daizhaoqian\.dzq,liangxiao\.msg\}@antgroup\.com,feiyu\.fyyu@gmail\.com

###### Abstract

Ensuring the accuracy of financial documents is critical for economic analysis, regulatory compliance, and corporate decision\-making\. Several studies have shown that Large Language Models \(LLMs\) perform well in many financial tasks, such as stock price movements and financial analytics\. However, a critical task remains unexplored: the ability of LLMs to identify errors in financial documents\. In this paper, we introduceFinED\-Bench, the first publiclyBenchmark forFinancialErrorDetection across three levels of cognitive complexity\. FinED\-Bench covers nine real\-world financial scenarios, and includes over 900 documents reported in 2025 that are unseen by existing language models\. We detail the benchmark construction process and evaluate several advanced LLMs \(e\.g\., GPT\-4o, Qwen3\-14B\) on this tasks, which requires both financial domain knowledge and reasoning capabilities\. Experimental results show that current LLMs still struggle with this task, especially in high\-complexity cases\. Besides, supervised fine\-tuning can significantly improve the performance of weaker LLMs on this task\. Our data and code are available at https://github\.com/hedyHe/FinED\-Bench\.

Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents

Ying He1, Zhouhong Gu1, Zhecheng Hu1, Yubo Zhou1,Hao Shen1,Jiaqing Liang2,Zhaoqian Dai3,Shuguang Ma3,Fei Yu3,Yanghua Xiao1††thanks:Corresponding authors\.,Zhixu Li411footnotemark:11College of Computer Science and Artificial Intelligence, Fudan University,2School of Data Science, Fudan University,3Ant Group,4School of Information and School of Smart Governance, Renmin University of China\{yinghe23,zhouyb24,zchu24,hshen22\}@m\.fudan\.edu\.cn,\{zhgu20,liangjiaqing,shawyh\}@fudan\.edu\.cn,zhixuli@ruc\.edu\.cn\{daizhaoqian\.dzq,liangxiao\.msg\}@antgroup\.com,feiyu\.fyyu@gmail\.com

## 1Introduction

Errors in financial documents have serious impacts on economic analysisWuet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib34)\), regulatory complianceXieet al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib23)\), and corporate decision\-makingPenget al\.\([2025](https://arxiv.org/html/2608.12342#bib.bib38)\)\. A notable example is the 2012 “London Whale” scandal, where JPMorgan Chase’s $6 billion loss was not caused by market volatility but by an Excel error, highlighting the urgent need for error detection in the financial industry\. A surveyGartner \([2024](https://arxiv.org/html/2608.12342#bib.bib25)\)further reveals that 18% of financial practitioners make errors daily, one\-third make errors several times weekly, and 59% make errors several times monthly\. These errors not only may lead to huge economic losses but also affect market confidence and the accuracy of regulatory decisions\.

![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/intro1.png)Figure 1:An illustrative example of three types of errors in financial documents\. The figure demonstrates \(1\) General Knowledge Errors such as illegal times, \(2\) Financial Domain Knowledge Errors including terminology misuse, and \(3\) Financial Reasoning Errors requiring cross\-paragraph verification, where passenger volume growth claims in the text contradict numerical data in tables\.Compared to general documents, error detection in financial documents faces unique challenges\. These errors can be categorized into three levels: general knowledge errors, financial domain\-specific knowledge errors, and financial reasoning errors that require complex inference\. And, each level comprises several subcategories, as illustrated in Figure[2](https://arxiv.org/html/2608.12342#S1.F2)\. For example, general knowledge errors include illegal times like “February 30, 2025”; domain\-specific errors involve terminology misuse such as incorrectly stating “price\-to\-earnings ratio” as “price\-to\-book ratio”; reasoning errors manifest as cross\-paragraph inconsistencies where a report’s front section claims 10\.9% growth but the subsequent table shows 5\.7%, as illustrated in Figure[1](https://arxiv.org/html/2608.12342#S1.F1)\. Detecting these errors requires multi\-hop reasoning and a firmer grasp of financial domain knowledge compared to general\-domain error detection\.

Traditional automated error detection methodsGuoet al\.\([2021](https://arxiv.org/html/2608.12342#bib.bib55)\); Liet al\.\([2022](https://arxiv.org/html/2608.12342#bib.bib54)\)mainly rely on rule matching and statistical models, which have obvious limitations when handling semantic understanding, multi\-hop reasoning, and complex contextual dependencies\. In recent years, Large Language Models \(LLMs\) have demonstrated remarkable capabilities in natural language understanding, mathematical reasoning, and complex text analysisAhnet al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib52)\); Namet al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib53)\), providing new solutions for financial error detection\. The strong contextual understanding, cross\-domain knowledge integration, and multi\-step reasoning capabilities of LLMs make them particularly suitable for handling complex errors in financial documents\. However, a critical question arises:To what extent can LLMs understand complex financial documents and accurately detect subtle errors within them?

Although previous benchmarksOnoeet al\.\([2021](https://arxiv.org/html/2608.12342#bib.bib48)\); Abachaet al\.\([2024b](https://arxiv.org/html/2608.12342#bib.bib4),[a](https://arxiv.org/html/2608.12342#bib.bib18)\)have explored grammatical errors \(e\.g\., “I goes to school”\) and hallucination detection \(e\.g\., inconsistency between a summary and its source paragraph\) in generated texts, they typically focus on short, sentence\-level texts in general domains\. Consequently, they fall short in evaluating the numerical and long\-context reasoning capabilities required for financial error detection\. The field of professional error detection in financial documents still lacks systematic research and evaluation benchmarks\. The market urgently needs a specialized financial error detection benchmark that should have the following characteristics: 1\) realistic financial document scenarios; 2\) coverage of multi\-level error types; 3\) evaluation of long document reasoning capabilities\. To fill this gap, we propose theFinancialErrorDetectionBenchmark \(FinED\-Bench\), the first comprehensive benchmark specifically designed to evaluate LLMs’ capabilities in financial document error detection\. FinED\-Bench contains 973 realistic financial documents \(average length: 3,784\.6 words\) with 4,123 annotated error instances, covering the three levels of error types mentioned above\. All error instances are annotated and verified by experienced financial experts, ensuring the accuracy and reliability of the benchmark\. To further challenge LLMs’ long\-context reasoning abilities, we construct the FinED\-Bench\-Hard subset, which contains 24 documents ranging from 32K to 120K \(where K=1,000\) words with 83 error instances\.

We conduct comprehensive experiments to evaluate various LLMs on FinED\-Bench\. Results reveal that recent competitive LLMs face significant challenges in financial document error detection\. The best\-performing model, GPT\-4o, achieves an overall F1 score of 48\.34%, with performance dropping substantially across error categories: from 52\.33% for general knowledge errors to 38\.00% for financial reasoning errors\. This performance degradation is consistent across all models, highlighting limitations in handling complex financial logic\. Additionally, document length severely impacts performance, with F1 scores declining from 40\.16% to 16\.66% as document length increases from 2\.5K to 50\.2K words\. Notably, financial domain fine\-tuning substantially enhances performance, with Qwen3\-14B showing a 10\.70% improvement in overall F1 score, demonstrating the importance of domain\-specific adaptation for this task\.

![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/taxn.png)Figure 2:Overview of FinED\-Bench error taxonomy and document scenes\. The benchmark categorizes financial document errors into three hierarchical levels: \(1\) General Knowledge Errors \(GKEs\) including format and missing value issues, \(2\) Financial Domain Knowledge Errors \(FKEs\) covering terminology and regulatory violations, and \(3\) Financial Reasoning Errors \(FREs\) requiring complex inference across document sections\. These 15 error subcategories are evaluated across 9 realistic financial scenes ranging from research reports to legal contracts\.
## 2Related works

### 2\.1Financial Evaluation Benchmarks

LLMs have gained significant attention in the financial domain, being adapted for tasks such as financial text analysisZhanget al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib31),[2024a](https://arxiv.org/html/2608.12342#bib.bib58)\), market sentiment predictionDelgadilloet al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib32)\), and automated trading strategiesDinget al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib33)\)\. Many domain\-specific models have been developed, including BloombergGPTWuet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib34)\), FinGPTWanget al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib36)\), FinMAXieet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib35)\), OpenFinLLMsHuanget al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib37)\), and PlutusPenget al\.\([2025](https://arxiv.org/html/2608.12342#bib.bib38)\)\. Researchers have also proposed various LLM\-based agent systems such as FinAgentZhanget al\.\([2024b](https://arxiv.org/html/2608.12342#bib.bib43)\), FinMemYuet al\.\([2024a](https://arxiv.org/html/2608.12342#bib.bib44)\), FinConYuet al\.\([2024b](https://arxiv.org/html/2608.12342#bib.bib22)\), FinVisionFatemi and Hu \([2024](https://arxiv.org/html/2608.12342#bib.bib45)\), and FinRobotYanget al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib46)\)\. Additionally, general\-purpose LLMs like Qwen3 seriesTeam \([2025](https://arxiv.org/html/2608.12342#bib.bib27)\)and Deepseek\-R1DeepSeek\-AI \([2025](https://arxiv.org/html/2608.12342#bib.bib71)\)have shown strong performance on financial tasks\. Existing financial benchmarksXieet al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib23),[2023](https://arxiv.org/html/2608.12342#bib.bib35)\); Koncel\-Kedziorskiet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib56)\); Sinhaet al\.\([2022](https://arxiv.org/html/2608.12342#bib.bib57)\); Arunet al\.\([2025](https://arxiv.org/html/2608.12342#bib.bib72)\); Tatarinovet al\.\([2025](https://arxiv.org/html/2608.12342#bib.bib73)\)typically evaluate model performance across seven core tasks: Information Extraction \(IE\), Textual Analysis \(TA\), Question Answering \(QA\), Text Generation \(TG\), Risk Management \(RM\), Forecasting \(FO\), and Decision\-Making \(DM\), assessing capabilities in understanding, reasoning, and generation\.

Unlike existing financial benchmarks that focus on standard tasks, FinED\-Bench specifically targets error detection in lengthy financial documents, providing a unique perspective on model reliability in professional financial contexts\.

### 2\.2Error Detection Benchmarks

Errors are prevalent in daily communication, academic writing, and professional workflows, arising from human oversight, data processing issues, or insufficient knowledge\. Before the advent of LLMs, the Nature Language Processing \(NLP\) community primarily focused on formal errors such as spelling, punctuation, grammar, and word choice, which can be detected by rule\-based approachesBryantet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib3)\); Näther \([2020](https://arxiv.org/html/2608.12342#bib.bib51)\)\. Recent work has begun to identify semantic and mathematical errors requiring deeper contextual understanding\. Existing benchmarksOnoeet al\.\([2021](https://arxiv.org/html/2608.12342#bib.bib48)\); Abachaet al\.\([2024b](https://arxiv.org/html/2608.12342#bib.bib4),[a](https://arxiv.org/html/2608.12342#bib.bib18)\)have explored hallucination detection and mathematical errors in generated text\. For example, MEDECAbachaet al\.\([2024b](https://arxiv.org/html/2608.12342#bib.bib4)\)provides a benchmark for detecting and correcting errors in clinical notes, though it is limited to short medical texts with an average length of 126\.5 words\. ErrorRadarYanet al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib59)\)introduces a multimodal benchmark for detecting errors in K\-12 mathematical problem\-solving, but it neglects the real\-world relevance of mathematical reasoning in everyday or professional contexts\.

To address the gap in document\-level financial error detection, FinED\-Bench provides the first comprehensive benchmark for detecting semantic errors in lengthy Chinese financial documents, focusing on detection rather than correction across three distinct error levels\.

## 3FinED\-Bench

![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/frame.png)Figure 3:The Semi\-Automated Pipeline for Benchmark Construction\. The pipeline segments financial documents, injects errors from 15 subcategories \(abbreviated as IT, RS, NM, VFE, NVM, TM, ILR, AE, NUE, OFE, CE, TC, NI, CaE, CC\), and applies two\-stage filtering through model\-based verification and manual annotation to ensure benchmark quality\.The benchmark is built upon a two\-tier error taxonomy which includes 3 categories and 15 subcategories, developed through collaboration between finance domain experts and LLMs\.

### 3\.1Error Definition

To ensure systematic and reliable evaluation, we design a two\-tier error taxonomy grounded in cognitive\-linguistic theory and real\-world financial practices\.

Drawing inspiration from theDiscourse Representation ModelKintsch and Van Dijk \([1978](https://arxiv.org/html/2608.12342#bib.bib5)\), which describes three levels of text comprehension, we classify 15 common types of financial errors into the following three major categories:

\(1\)General Knowledge Errors: Errors at this level affect the readability and surface structure of the text, including redundant statements, incorrect dates, and missing numerical data and so on\. Such errors can be detected by non\-experts\. \(2\)Financial Domain Knowledge Errors: Errors at this level, such as misuse of terminology, incorrect legal reference, or ambiguous expression, distort the intended meaning\. Detecting them typically requires domain\-specific knowledge\. \(3\)Financial Reasoning Errors: Logical inconsistencies, such as conflicting expressions, time contradiction, or contradictory clauses, fall into this category\. Detecting these errors requires document\-level comprehension and reasoning\.

### 3\.2Dataset Construction

Given the absence of publicly available erroneous financial documents and concerns regarding training data contamination, we propose a semi\-automated construction pipeline in Figure[3](https://arxiv.org/html/2608.12342#S3.F3)\.

#### 3\.2\.1Financial Documents Collection & Parsing

To avoid overlap with LLM training corpora and minimize the chance of memorization, we collected a set of financial documents published after February 2025, postdating the knowledge cutoffs of evaluated models\. To unify the format of the input for LLMs, all documents were converted to plain text\. A basic data cleaning process was conducted, including the removal of special characters \(e\.g\., useless HTML tags\), blank lines, and other noise using regular expressions\. Additionally, documents under 200 words were excluded to ensure substantive content\.

Although FinED\-Bench focuses on assessing LLMs’ error detection capabilities in textual data, tabular information, an essential component of financial documents, was also retained to preserve factual integrity\. Tables were converted into Markdown format using tools such as OCR or openpyxl, and then integrated into the textual data\. Structural errors within tables were manually corrected, while only content\-related errors were preserved for evaluation\.

Hard Set:Due to the context length limitations of most LLMs \(typically 16kkor 32kktokens, wherekk=1024\), we collect documents below 16K words for the main dataset\. However, some financial documents, such as prospectuses, substantially exceed this length yet are important in this domain\. We therefore construct a challenging subset,FinED\-Bench\-Hard, comprising 24 documents ranging from 32K to 120K words, to evaluate model performance under long\-context scenarios\.

#### 3\.2\.2Error Generation

Combining domain expertise with LLM generation capabilities, we generate errors for each document automatically\. The process begins with domain experts identifying and formulating error seeds, which are common instances observed in real\-world financial documents\.

Considering the length of many financial documents, especially those exceeding 32K words, injecting errors directly across the entire document poses challenges for LLMs\. To address this, we segment documents into shorter, semantically complete fragments\. The segmentation strategy varies by error types: \(1\) For general knowledge and financial domain knowledge errors, documents are divided by chapter, and the obtained fragments serve as candidate contexts\. \(2\) For calculation errors, we select paragraphs with more than three numerical values as the candidates\. \(3\) For other reasoning errors, like time contradiction and numerical inconsistency, we merge adjacent paragraphs sharing overlapping numerical values or terminology as the candidates\. To increase the difficulty of the benchmark, we join paragraphs that are originally far apart in the document, for example, merging the first and last paragraphs as a candidate fragment\.

This segmentation strategy serves three key purposes: \(1\) reducing the input length for the generator \(GPT\-4o\), \(2\) preserving the semantic integrity of candidate fragments, and \(3\) ensuring an even distribution of error instances across the document\.

Following segmentation, we apply In\-Context Learning \(ICL\) to generate contextually relevant error instances within candidate fragments\. Note that error types vary by financial scenes\. For example, in contracts and legal documents, we focus on errors such as clause conflicts, incorrect legal references, and ambiguous expression, while for research reports, which often contain dense numerical and temporal data, we emphasize errors like time inconsistencies, incorrect calculations, and numerical unit errors\.

#### 3\.2\.3Error Filtering

To ensure the quality of the benchmark, all generated errors undergo a two\-stage filtering process\.

Model\-based Filtering:Generated errors are re\-evaluated by GPT\-4o\. Specifically, each error\-injected sentence replaces its original version within the fragment\. Then, the modified fragments, along with their surrounding context, are input into the generator to verify whether the injected errors are contextually appropriate\. For calculation errors, we prompt the model to output the corresponding calculation formulas rather than directly judging whether the error is appropriate, as shown in Figure[10](https://arxiv.org/html/2608.12342#A4.F10)in Appendix[E](https://arxiv.org/html/2608.12342#A5)\.

Manual Verification:To eliminate LLMs’ knowledge bias, a team of five experts manually verifies and selects the remained errors\. This process includes removing errors that may introduce unintended errors and refining error seeds for LLM generation\. Besides, the number of error instances per document is controlled to ensure realism\. Details of annotation guidelines and consistency are provided in Appendix[C](https://arxiv.org/html/2608.12342#A3)\.

Name\#DocsAvg\. \#Length\#ErrorsAvg \#ErrorsError Type \(%\)\(Doc\)CKEFKEFREReal\-world Data504,727\.81873\.754\.527\.717\.8FinED\-Bench9734,544\.74,1234\.242\.930\.326\.8FinED\-Bench\-Hard2471,536\.0833\.541\.730\.228\.1Table 1:Statistical comparison between FinED\-Bench and real\-world financial documents\.![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/rate1.png)Figure 4:Number of instances per error type in FinED\-Bench\.

### 3\.3Statistics

FinED\-Bench is primarily Chinese\-centric, with its statistical information summarized in Figure[4](https://arxiv.org/html/2608.12342#S3.F4), and its distributional comparison with real\-world financial documents reported in Table[1](https://arxiv.org/html/2608.12342#S3.T1)\. To ensure a more balanced and fair evaluation across different error categories, We slightly reduce the proportion of general knowledge errors while increasing the proportion of financial reasoning errors\. To further assess the performance of models beyond Chinese, we additionally construct a small English\-centric dataset \(i\.e\., FinED\-Bench\-EN\)\. Detailed statistic and evaluation results on this dataset are presented in Section[F\.4](https://arxiv.org/html/2608.12342#A6.SS4)\. Each document is stored in JSON format as illustrated in Figure[8](https://arxiv.org/html/2608.12342#A2.F8)in Appendix[B](https://arxiv.org/html/2608.12342#A2)\.

## 4Experiments

### 4\.1Overall Setup

LLMs:We evaluate FinED\-Bench using a diverse set of baseline models spanning both general\-purpose and domain\-specific LLMs from API\-based and open\-source domains\. Our evaluation includes models from the Qwen series \(Qwen3\-8B/14B, Qwen2\.5\-7B\-Instruct, DeepSeek\-R1\-0528\-Qwen3\-8B\), specialized financial models \(Dianjin\-R1\-7B, Fin\-R1\), and commercial models \(GPT\-4o, GPT\-4o\-mini, GPT\-3\.5\-turbo\)\. Detailed specifications for all baseline models are provided in Appendix[A](https://arxiv.org/html/2608.12342#A1)\.

Metrics:To evaluate the model performance in recognizing financial errors in documents, we use three metrics:Precision \(Pre\.\),Recall \(Rec\.\),F1\-score \(F1\)\. Specifically, a sentence extracted by LLMs is considered correctly, if: 1\) it either exactly matches or contains the standard sentence, and 2\) the error type is accurately classified\.

ModelGeneral KnowledgeErrorsFinancial DomainKnowledge ErrorsFinancial ReasoningErrorsOverallPre\.Rec\.F1Pre\.Rec\.F1Pre\.Rec\.F1F1General Large Language ModelsQwen2\.5\-7B\-Instruct20\.8711\.3214\.6713\.206\.098\.335\.312\.943\.799\.85Qwen3\-8B \(no thinking\)29\.6721\.5024\.9314\.8922\.9518\.067\.0710\.958\.6017\.44Qwen3\-8B54\.2142\.4347\.6050\.5231\.2238\.6037\.3322\.3627\.9639\.99DeepSeek\-R1\-0528\-Qwen3\-8B55\.3919\.7229\.0950\.2814\.7622\.8236\.088\.4613\.7123\.31Qwen3\-14B \(no thinking\)31\.2036\.6833\.7224\.2129\.6026\.6316\.3216\.9616\.6327\.19Qwen3\-14B53\.1549\.7751\.4153\.5933\.3341\.1040\.7322\.6329\.1043\.15GPT\-3\.5\-turbo20\.1310\.7914\.055\.692\.683\.6510\.001\.372\.417\.17GPT\-4o\-mini24\.9436\.6929\.6913\.984\.987\.3418\.117\.8810\.9818\.90GPT\-4o39\.1678\.8352\.3343\.2352\.3647\.3628\.4257\.3538\.0048\.34Financial Large Language ModelsFin\-R116\.623\.395\.635\.040\.971\.633\.490\.550\.953\.19Dianjin\-R1\-7B26\.4714\.4718\.7236\.1011\.2817\.1918\.945\.618\.6615\.81Human83\.3360\.6170\.1885\.0044\.7458\.6284\.6244\.0057\.8963\.63

- •Note:Details for the human baseline are provided in Appendix[C\.4](https://arxiv.org/html/2608.12342#A3.SS4)\.

Table 2:Performance of Different LLMs on FinED\-Bench \(%\)\.Boldindicates the best performance andunderlinedindicates the second\-best performance within each metric\.![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/box.png)Figure 5:Document Length Distribution and Model Performance \(F1\) Across Different Scenes \(%\)\. Red lines indicate the median document length for each scene\. Black numbers above each box represent the highest F1 scores achieved by the evaluated models\.![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/heatmap1_1.png)Figure 6:Heatmap of F1\-scores \(0\-100%\) for Each Model \(Columns\) Across Different Scenes \(Rows\) in FinED\-Bench\. Dark blue indicates high performance, while light green indicates low performance\.![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/ft_1.png)Figure 7:Impact of Supervised Fine\-Tuning on Qwen3\-14B Across Different Error Types\.TasksMetricsbeforeafterMCQsACC73\.8673\.08 \(↓\\downarrow0\.78\)BLEU\-414\.9215\.27 \(↑\\uparrow0\.35\)Fin\_MTBLEU\-419\.9621\.38 \(↑\\uparrow1\.42\)Fin\_TCACC66\.6768\.89 \(↑\\uparrow2\.22\)Fin\_REF147\.8745\.87\(↓\\downarrow2\.00 \)Fin\_TGROUGE\-L22\.5421\.91 \(↓\\downarrow0\.63\)Table 3:Comparison of Performances on Other Financial Tasks from CFLUE before and after Supervised Fine\-Tuning\. MCQs = multiple\-choice questions; Fin\_MT = Financial Translation from English to Chinese; Fin\_TC = Financial Text Classification; Fin\_RE = Financial Relation Extraction; Fin\_TG = Financial Text Generation\.
### 4\.2Main Results

#### 4\.2\.1Overall Performance

Table[2](https://arxiv.org/html/2608.12342#S4.T2)presents a comparative evaluation of general\-purpose and domain\-specific LLMs across three categories of financial document errors: General Knowledge, Financial Domain Knowledge, and Financial Reasoning Errors\. Details on 15 subtypes of errors are shown in Table[7](https://arxiv.org/html/2608.12342#A6.T7)in Appendix\. Key findings are summarized as follows:

Finding 1Current LLMs struggle with financial reasoning\. Even GPT\-4o show a significant performance drop in this category compare to other categories\.

GPT\-4o achieves the best performance on General Knowledge Errors \(F1 = 52\.33%\) but declines sharply on Financial Reasoning Errors \(F1 = 38\.00%\)\. This trend is consistent across other models: Qwen3\-14B drops from 51\.41% to 29\.10%, and Qwen3\-8B from 47\.60% to 27\.96%\. Financial reasoning errors are more challenging for LLMs because they require multi\-step calculations and the integration of financial\-specific regulations, whereas the other two types of errors rely primarily on commonsense reasoning and factual recall\.

Finding 2The performance of domain\-specific LLMs is largely constrained by the capabilities of their base models\.

Dianjin\-R1\-7B, a financial\-domain LLM fine\-tuned from Qwen2\.5\-7B\-Instruct, achieves a higher overall F1 score than its base model \(15\.81% vs\. 9\.85%\)\. In contrast, Fin\-R1 exhibits a decline in performance, with its overall F1 score dropping from 9\.85% to 3\.19%\. Despite these changes, both Dianjin\-R1\-7B and Fin\-R1 remain substantially below the performance of recent general\-purpose LLMs such as Qwen3\-8B \(39\.99%\) and Qwen3\-14B \(43\.15%\)\. These results indicate that \(1\) the foundational capabilities of base models largely determine the upper bound of performance achievable through domain\-specific fine\-tuning; \(2\) fine\-tuning on related tasks, such as QA, text summarization, or classification, does not improve the error detection ability of models\.

Finding 3GPT\-4o shows a high\-recall but low\-precision pattern on this task, primarily due to its overly sensitive detection\.

Compared with other LLMs, GPT\-4o tends to identify a much larger set of candidate errors, which greatly improves its recall but at the cost of precision\. As a no\-thinking model, GPT\-4o struggles to ensure internal inconsistency in its response, leading to numerous false positive and misclassified error types\. For example, in “The scale of waste incineration has reached 975\.9 million square meters per day”, the span “square meters per day” should be identified as a unit error, but GPT\-4o classifies it as a format error; and in “Recently, Xiangyuan Culture and Tourism released its 2024 annual report and the first\-quarter report for 2025”, GPT\-4o flags “2025” as an illegal time error\.

Finding 4Reasoning capabilities significantly enhance LLM performance on error detection by enabling them to infer implicit relationships within context\.

The reasoning\-ablated \(“no\-thinking”\) variants of Qwen3\-8B and Qwen3\-14B consistently underperform their full counterparts across all error categories\. Specifically, Qwen3\-8B without reasoning achieves 24\.93% F1 on General Knowledge Errors compared to 47\.60% with reasoning, while Qwen3\-14B shows a similar pattern \(33\.72% vs\. 51\.41%\)\. Distilling DeepSeek\-R1\-0528DeepSeek\-AI \([2025](https://arxiv.org/html/2608.12342#bib.bib71)\)reasoning\-chain data into Qwen3\-8B improves performance over its no\-thinking variant across all categories, yet it still lags behind the full Qwen3\-8B, suggesting that fine\-tuning may have partially compromised the model’s original capabilities\.

#### 4\.2\.2Performance by Scenes

To investigate the influence of document characteristics on model performance, we compare LLM performance across different financial scenarios\. Figure[5](https://arxiv.org/html/2608.12342#S4.F5)and[6](https://arxiv.org/html/2608.12342#S4.F6)present the performance distribution by scenes, highlighting key observations:

Longer documents significantly degrade model performance:As shown in Figure[5](https://arxiv.org/html/2608.12342#S4.F5), the performance declines sharply on longer documents\. The highest F1 scores occur in Stock Research Reports \(58\.07%\) and Industry Research Reports \(59\.68%\), both very short\. Performance drops to 44\.61% for Tender Announcements, then further to 38\.85% for Insurance Contracts and 29\.17% for Legal Documents\. An exception is Regulatory Documents, which achieve 50\.0%, likely because some regulatory documents in the training data overlap with current regulations, which have changed little\. The longest document types perform worst: Company Bylaws \(16\.66%\), Listing Prospectuses \(14\.76%\), and Bond Prospectuses \(9\.52%\)\. Figure[6](https://arxiv.org/html/2608.12342#S4.F6)confirms this trend across models\. For example, Qwen3\-14B achieves 50\.84% and 50\.31% F1 scores in the two shortest\-document scenes, while scores drop substantially for longer documents, as do Qwen3\-8B and GPT\-4o\.

#### 4\.2\.3Performance of Supervised Fine\-tuning

To enhance model performance on this task, we construct a supervised fine\-tuning dataset using the same pipeline as our benchmark, but without human verification\. Further details are provided in Appendix[D](https://arxiv.org/html/2608.12342#A4)\.

Fine\-tuning on financial error detection data substantially enhances the models’ ability to detect financial errors:As shown in Figure[7](https://arxiv.org/html/2608.12342#S4.F7), after fine\-tuning on financial error detection data, Qwen3\-14B achieves a 10\.70% improvement in overall F1 score \(from 43\.15% to 53\.85%\)\. The improvements are particularly notable for challenging error types: Conflicting Expression increases from 29\.21% to 51\.06%, and Ambiguous Expression rises from 2\.13% to 26\.42%\. Fine\-tuning also enhances performance on format\-based errors, with Illegal Time improving from 70\.53% to 80\.00% and Numerical Missing increasing from 52\.48% to 65\.99%\. Even complex reasoning errors show gains, with Time Contradiction improving from 46\.07% to 67\.32% and Clause Conflict rising from 25\.53% to 29\.36%\.

Fine\-tuning preserves the model’s generalization ability:We further evaluate the fine\-tuned model’s generalization through two dimensions: knowledge and application, with five financial tasks from CFLUEZhuet al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib70)\): MCQs, Fin\_MT, Fin\_TC, Fin\_RE, and Fin\_TG\. MCQs represent the knowledge dimension, while the remaining four tasks assess application\-oriented capabilities\. The results in Table[3](https://arxiv.org/html/2608.12342#S4.T3)show that the fine\-tuned model maintains comparable performance across these tasks, and in several cases even shows slight improvements over the original model\.

## 5Conclusion

In this paper, we present FinED\-Bench, a benchmark specifically designed to assess the error detection capabilities of LLMs within financial documents\. To avoid data contamination, we also propose a semi\-automatic construction pipeline that involves crawling raw financial documents from the internet and injecting domain\-specific errors\. The experimental results show that, while incorporating reasoning significantly enhances model performance, even recent LLMs, like GPT\-4o and Qwen3\-14B, still struggle with detecting errors in financial documents\. Furthermore, we observe that existing domain\-specific LLMs, which are often fine\-tuned for specific downstream tasks, such as stock movement prediction, demonstrate limited improvements in the error detection tasks, compared with their base models\.

## Limitations

While FinED\-Bench provides a structured evaluation framework, it does not fully capture the complexity of real\-world financial scenarios\. Several key limitations remain: 1\) Diversity of Financial Documents: Many financial documents, such as balance sheets, income statements, are not yet covered\. Errors in these documents often originate from underlying data sources, and verifying their correctness often requires a thorough review of extensive historical data\. Therefore, we excluded them from the current benchmark\. 2\) Multimodal Elements: Real\-world financial documents often contain visual elements, such as seals and signatures\. Accurately interpreting and validating these components requires multimodal capabilities, which are beyond the scope of text\-only models evaluated in this paper\. Therefore, such types of errors are not considered in the current benchmark\.

## Ethical Concerns

Considering that FinED\-Bench may contain sensitive information, such as contact details, even they are publicly available, supervised fine\-tuning LLMs on such data could inadvertently amplify security vulnerabilities\. To mitigate ethical dilemmas associated with this benchmark, we have invested significant effort and resources to replace real data with carefully crafted synthetic alternatives\.

## References

- A\. B\. Abacha, W\. Yim, Y\. Fu, Z\. Sun, F\. Xia, and M\. Yetisgen\-Yildiz \(2024a\)Overview of the mediqa\-corr 2024 shared task on medical error detection and correction\.InProceedings of the 6th Clinical Natural Language Processing Workshop,pp\. 596–603\.Cited by:[§1](https://arxiv.org/html/2608.12342#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.12342#S2.SS2.p1.1)\.
- A\. B\. Abacha, W\. Yim, Y\. Fu, Z\. Sun, M\. Yetisgen, F\. Xia, and T\. Lin \(2024b\)Medec: a benchmark for medical error detection and correction in clinical notes\.arXiv preprint arXiv:2412\.19260\.Cited by:[Table 12](https://arxiv.org/html/2608.12342#A6.T12.1.4.1),[§1](https://arxiv.org/html/2608.12342#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.12342#S2.SS2.p1.1)\.
- J\. Ahn, R\. Verma, R\. Lou, D\. Liu, R\. Zhang, and W\. Yin \(2024\)Large language models for mathematical reasoning: progresses and challenges\.arXiv preprint arXiv:2402\.00157\.Cited by:[§1](https://arxiv.org/html/2608.12342#S1.p3.1)\.
- A\. Arun, R\. R\. Harsh, B\. Sarmah, and S\. Pasquali \(2025\)FinReflectKG\-multihop: financial qa benchmark for reasoning with knowledge graph evidence\.arXiv preprint arXiv:2510\.02906\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- C\. Bryant, M\. Felice, Ø\. E\. Andersen, and T\. Briscoe \(2019\)The bea\-2019 shared task on grammatical error correction\.InProceedings of the fourteenth workshop on innovative use of NLP for building educational applications,pp\. 52–75\.Cited by:[Table 12](https://arxiv.org/html/2608.12342#A6.T12.1.3.1)\.
- C\. Bryant, Z\. Yuan, M\. R\. Qorib, H\. Cao, H\. T\. Ng, and T\. Briscoe \(2023\)Grammatical error correction: a survey of the state of the art\.Computational Linguistics49\(3\),pp\. 643–701\.Cited by:[§2\.2](https://arxiv.org/html/2608.12342#S2.SS2.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[2nd item](https://arxiv.org/html/2608.12342#A1.I1.i2.p1.1),[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1),[§4\.2\.1](https://arxiv.org/html/2608.12342#S4.SS2.SSS1.p9.1)\.
- J\. Delgadillo, J\. Kinyua, and C\. Mutigwe \(2024\)Finsosent: advancing financial market sentiment analysis through pretrained large language models\.Big Data and Cognitive Computing8\(8\),pp\. 87\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- H\. Ding, Y\. Li, J\. Wang, and H\. Chen \(2024\)Large language model agent in financial trading: a survey\.arXiv preprint arXiv:2408\.06361\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- S\. Fatemi and Y\. Hu \(2024\)FinVision: a multi\-agent framework for stock market prediction\.InProceedings of the 5th ACM International Conference on AI in Finance,pp\. 582–590\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- I\. Gartner \(2024\)Gartner survey shows that a third of accountants make several financial errors per week due to capacity constraints\.Cited by:[§1](https://arxiv.org/html/2608.12342#S1.p1.1)\.
- Z\. Guo, Y\. Ni, K\. Wang, W\. Zhu, and G\. Xie \(2021\)Global attention decoder for chinese spelling error correction\.InFindings of the association for computational linguistics: ACL\-IJCNLP 2021,pp\. 1419–1428\.Cited by:[§1](https://arxiv.org/html/2608.12342#S1.p3.1)\.
- S\. D\. Hernandez and H\. Calvo \(2014\)Conll 2014 shared task: grammatical error correction with a syntactic n\-gram language model from a big corpora\.InProceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task,pp\. 53–59\.Cited by:[Table 12](https://arxiv.org/html/2608.12342#A6.T12.1.2.1)\.
- J\. Huang, M\. Xiao, D\. Li, Z\. Jiang, Y\. Yang, Y\. Zhang, L\. Qian, Y\. Wang, X\. Peng, Y\. Ren,et al\.\(2024\)Open\-finllms: open multimodal large language models for financial applications\.arXiv preprint arXiv:2408\.11878\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- P\. Islam, A\. Kannappan, D\. Kiela, R\. Qian, N\. Scherrer, and B\. Vidgen \(2023\)Financebench: a new benchmark for financial question answering\.arXiv preprint arXiv:2311\.11944\.Cited by:[Table 11](https://arxiv.org/html/2608.12342#A6.T11.1.4.1)\.
- W\. Kintsch and T\. A\. Van Dijk \(1978\)Toward a model of text comprehension and production\.\.Psychological review85\(5\),pp\. 363\.Cited by:[§3\.1](https://arxiv.org/html/2608.12342#S3.SS1.p2.1)\.
- R\. Koncel\-Kedziorski, M\. Krumdick, V\. Lai, V\. Reddy, C\. Lovering, and C\. Tanner \(2023\)Bizbench: a quantitative reasoning benchmark for business and finance\.arXiv preprint arXiv:2311\.06602\.Cited by:[Table 11](https://arxiv.org/html/2608.12342#A6.T11.1.5.1),[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- Y\. Kong, Y\. Nie, X\. Dong, J\. M\. Mulvey, H\. V\. Poor, Q\. Wen, and S\. Zohren \(2024\)Large language models for financial and investment management: models, opportunities, and challenges\.\.Journal of Portfolio Management51\(2\)\.Cited by:[Table 10](https://arxiv.org/html/2608.12342#A6.T10.1.5.1)\.
- A\. Koyama, T\. Kiyuna, K\. Kobayashi, M\. Arai, and M\. Komachi \(2020\)Construction of an evaluation corpus for grammatical error correction for learners of japanese as a second language\.InProceedings of the Twelfth Language Resources and Evaluation Conference,pp\. 204–211\.Cited by:[Table 12](https://arxiv.org/html/2608.12342#A6.T12.1.8.1)\.
- Y\. Lei, J\. Li, D\. Cheng, Z\. Ding, and C\. Jiang \(2023\)Cfbenchmark: chinese financial assistant benchmark for large language model\.arXiv preprint arXiv:2311\.05812\.Cited by:[Table 11](https://arxiv.org/html/2608.12342#A6.T11.1.8.1)\.
- F\. Li, Y\. Shan, J\. Duan, X\. Mao, and M\. Huang \(2022\)WSpeller: robust word segmentation for enhancing chinese spelling check\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 1179–1188\.Cited by:[§1](https://arxiv.org/html/2608.12342#S1.p3.1)\.
- Z\. Liu, X\. Guo, F\. Lou, L\. Zeng, J\. Niu, Z\. Wang, J\. Xu, W\. Cai, Z\. Yang, X\. Zhao,et al\.\(2025\)Fin\-r1: a large language model for financial reasoning through reinforcement learning\.arXiv preprint arXiv:2503\.16252\.Cited by:[Table 10](https://arxiv.org/html/2608.12342#A6.T10.1.8.1)\.
- D\. Nam, A\. Macvean, V\. Hellendoorn, B\. Vasilescu, and B\. Myers \(2024\)Using an llm to help with code understanding\.InProceedings of the IEEE/ACM 46th International Conference on Software Engineering,pp\. 1–13\.Cited by:[§1](https://arxiv.org/html/2608.12342#S1.p3.1)\.
- C\. Napoles, M\. Nădejde, and J\. Tetreault \(2019\)Enabling robust grammatical error correction in new domains: data sets, metrics, and analyses\.Transactions of the Association for Computational Linguistics7,pp\. 551–566\.Cited by:[Table 12](https://arxiv.org/html/2608.12342#A6.T12.1.5.1)\.
- M\. Näther \(2020\)An in\-depth comparison of 14 spelling correction tools on a common benchmark\.InProceedings of the Twelfth Language Resources and Evaluation Conference,pp\. 1849–1857\.Cited by:[§2\.2](https://arxiv.org/html/2608.12342#S2.SS2.p1.1)\.
- Y\. Onoe, M\. J\. Zhang, E\. Choi, and G\. Durrett \(2021\)Creak: a dataset for commonsense reasoning over entity knowledge\.arXiv preprint arXiv:2109\.01653\.Cited by:[§1](https://arxiv.org/html/2608.12342#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.12342#S2.SS2.p1.1)\.
- OpenAI \(2023\)Gpt\-3\.5\-turbo\.External Links:[Link](https://platform.openai.com/docs/models#gpt-3-5-turbo)Cited by:[7th item](https://arxiv.org/html/2608.12342#A1.I1.i7.p1.1)\.
- OpenAI \(2024\)Gpt\-4o mini\.External Links:[Link](https://platform.openai.com/docs/models#gpt-4o-mini)Cited by:[6th item](https://arxiv.org/html/2608.12342#A1.I1.i6.p1.1),[8th item](https://arxiv.org/html/2608.12342#A1.I1.i8.p1.1)\.
- X\. Peng, T\. Papadopoulos, E\. Soufleri, P\. Giannouris, R\. Xiang, Y\. Wang, L\. Qian, J\. Huang, Q\. Xie, and S\. Ananiadou \(2025\)Plutus: benchmarking large language models in low\-resource greek finance\.arXiv preprint arXiv:2502\.18772\.Cited by:[Table 10](https://arxiv.org/html/2608.12342#A6.T10.1.2.1),[§1](https://arxiv.org/html/2608.12342#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- G\. Rao, E\. Yang, and B\. Zhang \(2020\)Overview of nlptea\-2020 shared task for chinese grammatical error diagnosis\.InProceedings of the 6th workshop on natural language processing techniques for educational applications,pp\. 25–35\.Cited by:[Table 12](https://arxiv.org/html/2608.12342#A6.T12.1.6.1)\.
- A\. Shah, S\. Paturi, and S\. Chava \(2023\)Trillion dollar words: a new financial dataset, task & market analysis\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 6664–6679\.External Links:[Link](https://aclanthology.org/2023.acl-long.368),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.368)Cited by:[Table 11](https://arxiv.org/html/2608.12342#A6.T11.1.6.1)\.
- R\. S\. Shah, K\. Chawla, D\. Eidnani, A\. Shah, W\. Du, S\. Chava, N\. Raman, C\. Smiley, J\. Chen, and D\. Yang \(2022\)When flue meets flang: benchmarks and large pre\-trained language model for financial domain\.arXiv preprint arXiv:2211\.00083\.Cited by:[Table 11](https://arxiv.org/html/2608.12342#A6.T11.1.2.1)\.
- A\. Sinha, S\. Kedas, R\. Kumar, and P\. Malo \(2022\)SEntFiN 1\.0: entity\-aware sentiment analysis for financial news\.Journal of the Association for Information Science and Technology73\(9\),pp\. 1314–1335\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- O\. Syvokon and O\. Nahorna \(2021\)UA\-gec: grammatical error correction and fluency corpus for the ukrainian language\.arXiv preprint arXiv:2103\.16997\.Cited by:[Table 12](https://arxiv.org/html/2608.12342#A6.T12.1.9.1)\.
- N\. Tatarinov, V\. Kannan, H\. Srinivasa, A\. Raj, H\. S\. Anand, V\. Singh, A\. Luthra, R\. Lade, A\. Shah, and S\. Chava \(2025\)KG\-qagen: a knowledge\-graph\-based framework for systematic question generation and long\-context llm evaluation\.arXiv preprint arXiv:2505\.12495\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- Q\. Team \(2024\)Qwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[3rd item](https://arxiv.org/html/2608.12342#A1.I1.i3.p1.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[1st item](https://arxiv.org/html/2608.12342#A1.I1.i1.p1.1),[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- N\. Wang, H\. Yang, and C\. D\. Wang \(2023\)Fingpt: instruction tuning benchmark for open\-source large language models in financial datasets\.arXiv preprint arXiv:2310\.04793\.Cited by:[Table 10](https://arxiv.org/html/2608.12342#A6.T10.1.6.1),[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- S\. Wu, O\. Irsoy, S\. Lu, V\. Dabravolski, M\. Dredze, S\. Gehrmann, P\. Kambadur, D\. Rosenberg, and G\. Mann \(2023\)Bloomberggpt: a large language model for finance\.arXiv preprint arXiv:2303\.17564\.Cited by:[Table 10](https://arxiv.org/html/2608.12342#A6.T10.1.3.1),[§1](https://arxiv.org/html/2608.12342#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- Q\. Xie, W\. Han, Z\. Chen, R\. Xiang, X\. Zhang, Y\. He, M\. Xiao, D\. Li, Y\. Dai, D\. Feng,et al\.\(2024\)Finben: a holistic financial benchmark for large language models\.Advances in Neural Information Processing Systems37,pp\. 95716–95743\.Cited by:[Table 11](https://arxiv.org/html/2608.12342#A6.T11.1.7.1),[§1](https://arxiv.org/html/2608.12342#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- Q\. Xie, W\. Han, X\. Zhang, Y\. Lai, M\. Peng, A\. Lopez\-Lira, and J\. Huang \(2023\)Pixiu: a comprehensive benchmark, instruction dataset and large language model for finance\.Advances in Neural Information Processing Systems36,pp\. 33469–33484\.Cited by:[Table 10](https://arxiv.org/html/2608.12342#A6.T10.1.4.1),[Table 11](https://arxiv.org/html/2608.12342#A6.T11.1.3.1),[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- Y\. Yan, S\. Wang, J\. Huo, H\. Li, B\. Li, J\. Su, X\. Gao, Y\. Zhang, T\. Xu, Z\. Chu,et al\.\(2024\)Errorradar: benchmarking complex mathematical reasoning of multimodal large language models via error detection\.arXiv preprint arXiv:2410\.04509\.Cited by:[§2\.2](https://arxiv.org/html/2608.12342#S2.SS2.p1.1)\.
- H\. Yang, B\. Zhang, N\. Wang, C\. Guo, X\. Zhang, L\. Lin, J\. Wang, T\. Zhou, M\. Guan, R\. Zhang,et al\.\(2024\)Finrobot: an open\-source ai agent platform for financial applications using large language models\.arXiv preprint arXiv:2405\.14767\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- Y\. Yu, H\. Li, Z\. Chen, Y\. Jiang, Y\. Li, D\. Zhang, R\. Liu, J\. W\. Suchow, and K\. Khashanah \(2024a\)Finmem: a performance\-enhanced llm trading agent with layered memory and character design\.InProceedings of the AAAI Symposium Series,Vol\.3,pp\. 595–597\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- Y\. Yu, Z\. Yao, H\. Li, Z\. Deng, Y\. Jiang, Y\. Cao, Z\. Chen, J\. Suchow, Z\. Cui, R\. Liu,et al\.\(2024b\)Fincon: a synthesized llm multi\-agent system with conceptual verbal reinforcement for enhanced financial decision making\.Advances in Neural Information Processing Systems37,pp\. 137010–137045\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- B\. Zhang, H\. Yang, T\. Zhou, M\. Ali Babar, and X\. Liu \(2023\)Enhancing financial sentiment analysis via retrieval augmented large language models\.InProceedings of the fourth ACM international conference on AI in finance,pp\. 349–356\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- C\. Zhang, Y\. Mao, Y\. Fan, Y\. Mi, Y\. Gao, L\. Chen, D\. Lou, and J\. Lin \(2024a\)Finsql: model\-agnostic llms\-based text\-to\-sql framework for financial analysis\.InCompanion of the 2024 International Conference on Management of Data,pp\. 93–105\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- W\. Zhang, L\. Zhao, H\. Xia, S\. Sun, J\. Sun, M\. Qin, X\. Li, Y\. Zhao, Y\. Zhao, X\. Cai,et al\.\(2024b\)A multimodal foundation agent for financial trading: tool\-augmented, diversified, and generalist\.InProceedings of the 30th acm sigkdd conference on knowledge discovery and data mining,pp\. 4314–4325\.Cited by:[§2\.1](https://arxiv.org/html/2608.12342#S2.SS1.p1.1)\.
- Y\. Zhang, Z\. Li, Z\. Bao, J\. Li, B\. Zhang, C\. Li, F\. Huang, and M\. Zhang \(2022\)MuCGEC: a multi\-reference multi\-source evaluation dataset for Chinese grammatical error correction\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Seattle, United States,pp\. 3118–3130\.Cited by:[Table 12](https://arxiv.org/html/2608.12342#A6.T12.1.7.1)\.
- J\. Zhu, Q\. Chen, H\. Dou, J\. Li, L\. Guo, F\. Chen, and C\. Zhang \(2025\)Dianjin\-r1: evaluating and enhancing financial reasoning in large language models\.arXiv preprint arXiv:2504\.15716\.Cited by:[Table 10](https://arxiv.org/html/2608.12342#A6.T10.1.7.1)\.
- J\. Zhu, J\. Li, Y\. Wen, and L\. Guo \(2024\)Benchmarking large language models on cflue\-a chinese financial language understanding evaluation dataset\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 5673–5693\.Cited by:[§4\.2\.3](https://arxiv.org/html/2608.12342#S4.SS2.SSS3.p3.1)\.

## Appendix ALLM Baseline Models Specifications

We experiment with a number of eminent general\-purpose and domain\-specific models from both the API\-based and the open\-source domains:

- •Qwen3\-8B/14B: the latest generation in the Qwen series, offering a comprehensive suite of dense and MoE modelsTeam \([2025](https://arxiv.org/html/2608.12342#bib.bib27)\)\.
- •DeepSeek\-R1\-0528\-Qwen3\-8B: a distilled model obtained by post\-training Qwen3\-8B using chain\-of\-thought traces from DeepSeek\-R1\-0528DeepSeek\-AI \([2025](https://arxiv.org/html/2608.12342#bib.bib71)\)\.
- •Qwen2\.5\-7B\-Instruct: an improved version of Qwen2, with enhanced capabilities in instruction following, long\-text generation, and structured outputs \(e\.g\., JSON\)Team \([2024](https://arxiv.org/html/2608.12342#bib.bib28)\)\.
- •Dianjin\-R1\-7B: a financial domain model built on Qwen2\.5\-7B\-Instruct, incorporating reasoning\-augmented supervision and reinforcement learning to enhance financial reasoning111https://modelscope\.cn/models/tongyi\_dianjin/DianJin\-R1\-7B\.
- •Fin\-R1: a specialized LLM for financial reasoning, also based on Qwen2\.5\-7B\-Instruct222https://modelscope\.cn/models/AI\-ModelScope/Fin\-R1\.
- •GPT\-4o\-mini\(∼\\sim8B parameters\): a fast, affordable small model for focused tasksOpenAI \([2024](https://arxiv.org/html/2608.12342#bib.bib29)\)\.
- •GPT\-3\.5\-turbo\(∼\\sim175B parameters\): a “high\-intelligence” modelOpenAI \([2023](https://arxiv.org/html/2608.12342#bib.bib30)\)\.
- •GPT\-4o\(∼\\sim200B parameters\): a versatile, high\-intelligence flagship modelOpenAI \([2024](https://arxiv.org/html/2608.12342#bib.bib29)\)\.

Note on Parameter Counts:The exact numbers of parameters for several LLMs \(e\.g\., GPT series\) have not been publicly disclosed yet\. The model size estimates reported here are mined from public articles333https://www\.thealgorithmicbridge\.com/p/openai\-o1\-a\-new\-paradigm\-for\-ai\.

Experimental Timeline:The experimentation was conducted using the official APIs for GPT series between May 11 and May 25, 2025\.

The parameter settings used for inference across different LLMs are presented in Table[4](https://arxiv.org/html/2608.12342#A1.T4)\.

ModelsMax TokensContext WindowTemperatureTopPTopKQwen2\.5\-7B\-Instruct120,00032,7680\.70\.820Qwen3\-8B \(no thinking\)120,00032,7680\.60\.9520Qwen3\-8B120,00032,7680\.70\.820DeepSeek\-R1\-0528\-Qwen3\-8B120,00032,7680\.60\.9520Qwen3\-14B \(no thinking\)120,00032,7680\.60\.9520Qwen3\-14B120,00032,7680\.70\.820GPT\-3\.5\-turbo18,00016,3840\.9\-\-GPT\-4o\-mini18,000200,0000\.9\-\-GPT\-4o18,000128,0000\.9\-\-Fin\-R1120,00032,7680\.70\.820Dianjin\-R1\-7B120,00032,7680\.70\.820Table 4:Parameter Setting for the inference stage\.
## Appendix BDefinitions of Different Errors

In the real world, the types of errors found in financial documents are diverse and virtually limitless\. Drawing on insights from financial experts and the cognitive\-linguistic theory, FinED\-Bench focuses on 15 representative subcategories of errors\. Below, we will provide specific definitions and examples for each\.

### B\.1General Knowledge Errors

Common mistakes that violate general knowledge include the following five types:

- •Illegal Time: Dates or times mentioned in the document are inconsistent with known facts\. Examples✗: … OnApril 31, Hangzhou initiated the fourth round of land supply\.✔: …OnApril 30, Hangzhou initiated the fourth round of land supply\.
- •Redundant Statements: Repeating the same information or including unnecessary repetition in the document\. Examples✗: …construction control area: 84,979 square meters,floor area ratio: 2\.8, height limit: 60 meters,floor area ratio: 2\.8…✔: …construction control area: 84,979 square meters,floor area ratio: 2\.8, height limit: 60 meters…
- •Value Format Errors: The format of attribute values \(such as phone numbers, dates, etc\.\) does not meet standard specifications or expectations\. Examples✗: …Contact number:010\.✔: …Contact number:010\-57365240\.
- •Numerical Missing: This error refers to the absence of a required numerical figure in a financial document\. Examples✗: The land transfer area of plot JG0404\-11 in Dingqiao Unit issquare meters\.✔: The land transfer area of plot JG0404\-11 in Dingqiao Unit is30,514 square meters\.
- •Non\-Numerical Attribute Value Missing: This error occurs when a non\-numerical attribute \(such as names, categories, etc\.\) misses its corresponding value\. Examples✗: \(1\)Procurement project name: ;\(2\)…✔: \(1\) Procurement project name:Office Software Procurement; \(2\)…

### B\.2Financial Domain Knowledge Errors

- •Terminology Misuse: Inaccurate or inappropriate use of financial or industry\-specific terms, resulting in incorrect or misleading expressions\. Examples✗: In 2024, the company obtained substantial equity funding throughdebt financing, thus strengthening its capital structure\.✔: In 2024, the company obtained substantial equity funding throughequity financing, thus strengthening its capital structure\.
- •Incorrect Legal Reference: Incorrect references to clauses, regulations, or legal documents\. Examples✗: The term “basic medical insurance” in this contract refers to the basic medical insurance stipulated in theRegulations on Government Investment of the People’s Republic of China\.✔: The term “basic medical insurance” in this contract refers to the basic medical insurance stipulated in theSocial Insurance Law of the People’s Republic of China\.
- •Ambiguous Expression: The use of vague or unclear language that may lead to multiple interpretations\. Examples✗: The purchaser shall not impose any unreasonable conditions on the winning bidder as prerequisites for contract signing,except in special circumstances\.✔: The purchaser shall not impose any unreasonable conditions on the winning bidder as prerequisites for contract signing\.
- •Numerical Unit Error: Use of incorrect or non\-standard units for numerical values\. Examples✗: Centralized drinking water source project… planned duration:180 hours\.✔: Centralized drinking water source project… planned duration:180 calendar days\.
- •Omitted Financial Element: Omission of commonly required financial elements, such as bidder qualification criteria, comparative growth rates, etc\. Examples✗: In terms of pricing, the price of Grade I metallurgical coke at major ports was RMB 2,540/ton\.✔: In terms of pricing, the price of Grade I metallurgical coke at major ports was RMB 2,540/ton,down 3\.79% week\-over\-week\.

### B\.3Financial Reasoning Errors

- •Conflicting Expression: Statements in a document that contradict with each other in meaning or logic\. Examples✗: The project isnot eligible for bidding, and isnow open for public bidding\.✔: The project iseligible for bidding, and is now open for public bidding\.
- •Time Contradiction: Inconsistent or logically conflicting time\-related information within a document\. Examples✗:Submission deadline: May 23…Bid opening: May 22…✔: Submission deadline:May 22… Bid opening: May 22…
- •Numerical Inconsistency: Discrepancies in numerical values cited in different sections of the same document\. Examples✗:Tax rate: 6%…Tax rate: 8%✔: Tax rate: 6%…Tax rate: 6%
- •Calculation Error: Incorrect numerical computations or total values\. Examples✗: Annual demand for hip joint systems:285,995 units \(Ceramic\-Ceramic: 102,264; Ceramic\-Polyethylene: 173,303; Alloy\-Polyethylene: 1,042\)\.✔: Annual demand for hip joint systems:285,995 units \(Ceramic\-Ceramic: 102,264; Ceramic\-Polyethylene: 173,303; Alloy\-Polyethylene: 10,428\)\.
- •Clause Conflict : Conflicting stipulations across different clauses or sections of the document\. Examples✗: Article 10…Under no circumstances shall the contract be terminated early… Article 11…The contract may be terminated early\.✔: Article 10… Under no circumstances shall the contract be terminated early…\[Remove Article 11\]

![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/example1.png)Figure 8:An Example in FinED\-Bench\.An example document in FinED\-Bench is shown in Figure[8](https://arxiv.org/html/2608.12342#A2.F8)\.

## Appendix CDetails about Manual Annotation

### C\.1Details about Annotators

Five financial experts served as annotators for this paper\. The entire annotation process was conducted under stringent supervision and scrutiny of the first author of this paper\.

TaskRequirementsError verification1\. Delete if the sentence is actually correct\.2\. Delete if its addition would introduce new errors,for example, injecting numerical unit errors may introduce numerical inconsistency\.3\. Delete if its incorrectness cannot be inferred from the context\.4\. Ensure that each document contains no more than four errors\.Seeds Update1\. Standardize and simplify the definitions of errors\.2\. Update the examples of each error type so that LLMs can generate more qualified error instances\.Table 5:Annotation Requirements for Each Tasks\.
### C\.2Annotation Tasks and Goals

The purpose of the manual annotation tasks was twofold\. The first goal was to obtain a comprehensive annotated dataset that could be used for model evaluation\. The second goal was to modify the definitions of errors and error seeds, enabling LLMs to generate error instances that better reflect real\-world scenarios and reducing human participation in the data construction process\. All the detailed annotation tasks and targets are list in Table[5](https://arxiv.org/html/2608.12342#A3.T5)\.

### C\.3Annotation Consistency

To ensure the quality of our benchmark dataset, we adopted a majority voting mechanism among five annotators is adopted, with each generated error instance reviewed by three annotators\. A high inter\-annotator agreement \(Fleiss’κ=0\.89\\kappa=0\.89\) indicates that the annotations are consistent and of high quality\.

### C\.4Human Performance on the FinED\-Bench

In addition to evaluating model performance, we conduct a human baseline study involving two sophomore students majoring in finance on FinED\-Bench\. Owing to time and cost constraints, we randomly sample 100 documents for manual evaluation and report their average performance in Table[7](https://arxiv.org/html/2608.12342#A6.T7)\. The results indicate that even for human with relevant domain background, accurately identifying all errors in long financial documents remains highly challenging\.

## Appendix DDetails of Supervised Fine\-tuning Data

To improve the performance of weaker LLMs in the task of financial error detection, we construct a supervised fine\-tuning dataset following the same pipeline as our benchmark, but without human verification\. Specifically, after injecting errors, we employ two models \(i\.e\., Qwen3\-32B and GPT\-4o\) as judges to identify both the error type and the error span within the target fragment\. An error instance is retained only if both judges correctly detect its span and type\. Besides, we preserve the reasoning process generated by Qwen3\-32B, located between the <think\> and </think\> tags\.

The resulting dataset contains 9,515 error\-free fragments and 8,697 error\-containing fragments, each annotated with the error type, error span and corresponding reasoning process\. This dataset is used to fine\-tune the target LLM, enabling it to acquire relevant knowledge and improve detection capabilities\.

![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/prompt.png)Figure 9:The prompt that guides LLM to perform error detection in financial documents\.![Refer to caption](https://arxiv.org/html/2608.12342v1/figure/cal.png)Figure 10:The example of the prompt to find mathematical formulas within a text\.
## Appendix EPrompt Details

This section primarily showcases two prompts used for the evaluation \(Figure[9](https://arxiv.org/html/2608.12342#A4.F9)\) and error generation \(Figure[10](https://arxiv.org/html/2608.12342#A4.F10)\)\.

## Appendix FAdditional Experimental Results

### F\.1Prompting Strategies

StrategiesMetricswholechunkmultisinglemultisinglefewzerofewzerofewzerofewzeroQwen3\-8B\(no thinking\)Pre\.16\.8715\.708\.717\.398\.006\.5713\.3012\.78Rec\.20\.4919\.3533\.8236\.1033\.9835\.9321\.3018\.86F118\.5017\.3313\.8512\.2712\.9511\.1116\.3815\.24Qwen3\-8BPre\.46\.9439\.0719\.5813\.2517\.1911\.9839\.1235\.83Rec\.34\.9633\.9842\.7643\.9043\.7445\.2034\.8035\.77F140\.0736\.3526\.8620\.3624\.6818\.9436\.8335\.80DeepSeek\-R1\-0528\-Qwen3\-8BPre\.53\.2744\.9019\.0913\.7918\.3314\.2148\.3143\.75Rec\.18\.5417\.8917\.7214\.9617\.8917\.4018\.5420\.49F127\.5125\.5918\.3814\.3518\.1115\.6426\.8027\.91Qwen3\-14B\(no thinking\)Pre\.21\.3521\.4011\.1210\.4210\.719\.7317\.3218\.16Rec\.28\.7829\.2739\.5143\.2540\.1643\.4131\.5429\.59F124\.5124\.7217\.3616\.7916\.9115\.9022\.3622\.51Qwen3\-14BPre\.46\.0441\.2020\.6315\.5018\.9713\.1643\.1534\.18Rec\.39\.6739\.1949\.7653\.5053\.9853\.8240\.9839\.19F142\.6240\.1729\.1724\.0428\.0721\.1542\.0436\.51Avg\.Pre\.36\.8932\.4515\.8312\.0714\.6411\.1332\.2428\.94Rec\.28\.4927\.9436\.7138\.3437\.9539\.1529\.4328\.78F130\.6428\.8321\.1217\.5620\.1416\.5528\.8827\.59Table 6:Performance Comparisons under Different Prompt Strategies\.Boldindicates the best performance\. Note: \(1\) whole: the entire document is provided in a prompt; \(2\) chunk: the document is segmented into small chunks and processed sequentially\. \(3\) multi: the LLM is asked to detect all error types at once\. \(4\) single: the LLM is asked to detect one specified error type per query\. \(5\) zero: no in\-context examples are provided\. \(6\) few: a small number of examples are given\.ModelsMetricsGeneral Knowledge ErrorsFinancial Domain Knowledge ErrorsFinancial Reasoning ErrorsOverallITRSVFENMNNMAEOFEUETMILRTCCaENICECCQwen2\.5\-7BPre\.31\.0318\.0121\.2825\.012\.240\.542\.536\.364\.352\.1318\.876\.253\.163\.7918\.1814\.37Rec\.18\.6924\.943\.974\.541\.821\.540\.6415\.740\.871\.253\.940\.66\.491\.225\.637\.49F123\.3320\.926\.697\.683\.170\.81\.0221\.971\.451\.576\.511\.094\.251\.858\.69\.85Qwen3\-8B\(no thinking\)Pre\.57\.3459\.3911\.2434\.759\.170\.188\.9341\.9112\.6431\.2531\.080\.03\.249\.4226\.0916\.03Rec\.43\.122\.7923\.4118\.553\.331\.5415\.7142\.5912\.796\.2518\.111\.219\.467\.078\.4519\.12F149\.2132\.9415\.1924\.194\.890\.3311\.3842\.2512\.7210\.4222\.890\.05\.568\.0812\.7717\.44Qwen3\-8BPre\.76\.8962\.1131\.6765\.5731\.390\.027\.1271\.9546\.9646\.4373\.1728\.8912\.1346\.0441\.9449\.24Rec\.63\.1056\.0527\.7836\.226\.060\.020\.5158\.8015\.7016\.2547\.247\.7817\.8415\.6118\.3133\.66F169\.3258\.9229\.6046\.6528\.480\.023\.3664\.7123\.5324\.0757\.4212\.2614\.4423\.3225\.4939\.99Deepseek\-R1\-0528\-Qwen3\-8BPre\.70\.8765\.1441\.5458\.3928\.210\.017\.9566\.0752\.5075\.0063\.4935\.2912\.1245\.7641\.1849\.92Rec\.31\.0326\.5110\.7118\.110\.00\.02\.2434\.266\.17\.515\.753\.596\.496\.599\.8615\.21F143\.1737\.6917\.0327\.6314\.770\.03\.9945\.1210\.9413\.6425\.240\.018\.4511\.5115\.9123\.31Qwen3\-14B\(no thinking\)Pre\.51\.8433\.4919\.0668\.0717\.5423\.1816\.3646\.3230\.773\.7927\.020\.010\.0820\.178\.025\.28Rec\.53\.250\.039\.2919\.6121\.0230\.7230\.8446\.588\.766\.2529\.261\.717\.5617\.3111\.2729\.41F152\.5140\.1125\.6630\.4519\.1326\.4221\.3846\.4513\.644\.7228\.090\.012\.8118\.639\.3627\.19Qwen3\-14BPre\.69\.756\.4931\.5473\.7738\.113\.4521\.8978\.769\.2352\.2773\.9142\.8617\.5549\.4252\.1750\.77Rec\.71\.3857\.6730\.1640\.7247\.581\.5418\.5961\.5718\.3128\.7533\.4612\.5723\.2420\.7316\.9037\.52F170\.5357\.0830\.8352\.4842\.322\.1320\.169\.0928\.9737\.1046\.0719\.4420\.0029\.2125\.5343\.15GPT\-3\.5\-turboPre\.32\.860\.009\.380\.0025\.000\.000\.007\.320\.00100\.000\.000\.0016\.670\.0014\.2913\.14Rec\.46\.940\.0010\.910\.001\.750\.000\.0016\.220\.001\.250\.000\.008\.820\.001\.414\.93F138\.660\.0010\.080\.003\.280\.000\.0010\.080\.000\.020\.000\.0011\.540\.002\.567\.17GPT\-4o\-miniPre\.43\.2111\.2415\.4827\.1830\.770\.000\.0023\.210\.000\.0023\.810\.007\.5850\.0075\.0021\.94Rec\.71\.4318\.1823\.6445\.1628\.070\.000\.0035\.140\.000\.0015\.380\.0014\.711\.038\.4516\.61F153\.8513\.8918\.7133\.9429\.360\.000\.0027\.960\.000\.0018\.690\.0010\.002\.0215\.1918\.90GPT\-4oPre\.43\.0245\.8717\.2254\.9228\.050\.0019\.1773\.8560\.0060\.4738\.896\.259\.4931\.9056\.7937\.76Rec\.90\.6982\.7849\.2174\.4468\.000\.0048\.5790\.0066\.6727\.0880\.0022\.2260\.0067\.6843\.4067\.13F158\.3659\.0325\.5163\.2139\.720\.0027\.4981\.1363\.1637\.4152\.349\.1616\.3943\.3749\.2048\.34Fin\-R1Pre\.5\.547\.930\.791\.360\.30\.00\.322\.550\.00\.00\.790\.61\.080\.240\.01\.9Rec\.10\.5614\.451\.522\.640\.60\.00\.644\.960\.00\.01\.561\.181\.980\.470\.03\.19F110\.5614\.451\.522\.640\.60\.00\.644\.960\.00\.01\.561\.181\.980\.470\.03\.19Dianjin\-R1\-7BPre\.53\.9620\.2918\.9438\.678\.558\.333\.260\.9818\.9233\.3330\.018\.755\.6527\.0313\.3327\.25Rec\.25\.9519\.589\.9213\.153\.031\.541\.2928\.942\.032\.515\.351\.83\.782\.442\.8211\.13F135\.0519\.9313\.0219\.634\.472\.61\.8339\.253\.674\.6520\.313\.284\.534\.474\.6515\.81

Table 7:Performance Comparison of different Large Language Models in FinED\-Bench across 15 errors categories\. \(%\)Boldindicates the best performance andunderlinedindicates the second\-best performance within each metric\.ModelBeginningMiddleEndingQwen3\-8B\(no thinking\)7\.766\.019\.11Qwen3\-8B13\.049\.1916\.75DeepSeek\-R1\-0528\-Qwen3\-8B6\.524\.957\.23Qwen3\-14B\(no thinking\)9\.3212\.0116\.75Qwen3\-14B22\.6713\.0720\.63GPT\-3\.5\-turbo1\.5511\.063\.56GPT\-4o\-mini0\.000\.0013\.70DianJin\-R1\-7B2\.482\.125\.45Fin\-R10\.620\.000\.63Table 8:Performance different in potions of errors appear in the document\.To find an effective prompting strategy for detecting errors in long financial documents, we evaluate several prompt designs, with the results summarized in Table[6](https://arxiv.org/html/2608.12342#A6.T6)\. The findings reveal three key observations\.

\(1\)Whole\-document prompting is more effective than chunking\-based prompting\.As noted in Section[3\.2\.2](https://arxiv.org/html/2608.12342#S3.SS2.SSS2), many generated errors are designed to span the entire document to increase the difficulty of the benchmark\. Such cross\-document inconsistencies cannot be reliably captured when the text is divided into isolated chunks, making whole\-document prompts more effective\.

\(2\)Multi\-error prompting outperforms single\-error prompting\.Prompting the model to detect all error types simultaneously \(‘multi’\) consistently yields higher F1 scores than prompting for one error type at a time \(‘single’\)\. This indicates that LLMs benefit from cross\-error contextual cues when performing multi\-type detection\.

\(3\)The combination of ‘whole \+ multi \+ few’ prompting achieves the best performance\.This strategy attains the highest average F1 \(30\.64%\) across all tested models, as it provide complete document context, multiple error types, and informative demonstrations\.

### F\.2Fine\-grained Performance on 15 Error Subcategories

We show the performance of all models across 15 subcategories of errors in Table[7](https://arxiv.org/html/2608.12342#A6.T7)\.

≤\\leq2K2K\-4K4K\-6K6K\-8K8K\-10K10K\-12K12K\-14K14K\-16K\>32K\-64K64K\-120K020020040040048548522022072728080434328281717292912121212Document Length Range \(K=1,000\)CountFigure 11:Document length distribution of FinED\-Bench\.
### F\.3Position\-aware Error Detection Performance

Besides, we analyze model performance with respect to the positional distribution of errors within a document\. Specifically, each document is divided into three segments \(i\.e\., beginning, middle, and ending\), as shown in Table[8](https://arxiv.org/html/2608.12342#A6.T8)\. The results show that models are more effective at detecting errors located in the ending section, followed by the beginning, while errors in the middle section are the most difficult to identify\. This pattern aligns with well\-established cognitive phenomena in psychology, namely the Primacy Effect and the Recency Effect\. Information presented at the beginning of a document is often repeatedly attended to as the model builds a global understanding of the context, whereas information near the end remains salient due to its proximity to the prediction step\. In contrast, content in the middle is less likely to be revisited or emphasized during inference, making errors in this region easier to overlook\.

ModelsPre\.Rec\.F1Qwen3\-8B \(no thinking\)13\.6524\.6717\.58Qwen3\-8B39\.3138\.0038\.64DeepSeek\-R1\-0528\-Qwen3\-8B54\.4124\.6733\.94Qwen3\-14B \(no thinking\)17\.1238\.0023\.60Qwen3\-14B47\.3742\.0044\.52Llama3\.1\-8B12\.5013\.3312\.90GPT\-4o\-mini22\.5620\.0021\.20GPT\-557\.2550\.0053\.38Dianjin\-R1\-7B14\.066\.008\.41Table 9:Performance of Different LLMs on English dataset \(%\)\.Boldindicates the best performance andunderlinedindicates the second\-best performance within each metric\.
### F\.4Evaluation on the English Benchmark

To address concerns about generalization beyond Chinese, we additionally construct an English financial error\-detection dataset sourced from English\-language financial websites\. This dataset comprises 56 documents with 198 annotated errors, with 91 general knowledge errors, 37 financial domain knowledge errors and 70 financial reasoning errors\. The evaluation results are summarized in Table[9](https://arxiv.org/html/2608.12342#A6.T9)\. Overall, the findings are consistent with those observed on the Chinese benchmark:

\(1\)Reasoning\-enabled models substantially outperform their non\-reasoning versions\.For example, Qwen3\-14B improves from 23\.60% → 44\.52% F1, and Qwen3\-8B improves from 17\.58% → 38\.64% F1 once thinking is enabled\.

\(2\)Recall remains a major challenging across models\.Despite improvements in precision and overall F1, recall values generally remain below 50%, indicating persistent difficulty in comprehensively identifying all errors\.

\(3\)Financial\-domain LLMs remain constrained by the capacity of their base models\. Although these domain\-specific LLMs are fine\-tuned for tasks like financial QA, text summarization, and classification tasks, only a limited amount of domain\-specific knowledge is learned\.

ModelsBase ModelParas\.LengthTasksTechniquesChineseYearOpenPlutusPenget al\.\([2025](https://arxiv.org/html/2608.12342#bib.bib38)\)Llama8B42000SMPIFT✗03/03/2025✔BloomberGPTWuet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib34)\)BLOOM50B2048SA, HC,NER, QAPT, PE✗03/30/2023✗FinMAXieet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib35)\)Llama7B/13B4096SA, HC, NER,QA, SMPIFT, PE✗06/01/2023✔InvestLMKonget al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib40)\)Llama65B4096SA, HC,QA, SummIFT, PE,PEFT✗09/15/2023✔FinGPT\-v3Wanget al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib36)\)Llama27B4096SA, HC,NER, REIFT, PE,PEFT✗10/12/2023✔DianJin\-R1Zhuet al\.\([2025](https://arxiv.org/html/2608.12342#bib.bib41)\)Qwen2\.57B/32B131072FS, IRIFT, RL✔04/23/2025✔Fin\-R1Liuet al\.\([2025](https://arxiv.org/html/2608.12342#bib.bib42)\)Qwen2\.57B131072FC, FSIFT, RL✔03/22/2025✔Table 10:A Summary of FinLLMs\. The abbreviations correspond to Para\. = Parameters, PT = Pre\-Training, PE = Prompt Engineering, IFT= Instruction Fine\-Tuning, PEFT = Parameter Efficient Fine\-Tuning, RL = Reinforcement Learning; \[SA\] Sentiment Analysis, \[HC\] Headline Classification, \[NER\] Named Entity Recognition, \[QA\] Question Answering, \[SMP\] Stock Movement Prediction, \[Summ\] Text Summarization, \[RE\] Relation Extraction, \[FS\] Financial Services, \[IR\] Investment Research, \[FC\] Financial Coding\.DatasetTasksLanguageYearFLUEShahet al\.\([2022](https://arxiv.org/html/2608.12342#bib.bib39)\)TA, IE, QAEnglish10/31/2022PIXIUXieet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib35)\)TA, IE, QA, FO, RMEnglish01/08/2023FinanceBenchIslamet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib49)\)QAEnglish12/20/2023BizBenchKoncel\-Kedziorskiet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib56)\)QA, IE, TGEnglish03/12/2024FOMCShahet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib61)\)TAEnglish2023FinBenXieet al\.\([2024](https://arxiv.org/html/2608.12342#bib.bib23)\)TA, IE, QA, FO, RM, DMEnglish, Chinese2024CFBenchmarkLeiet al\.\([2023](https://arxiv.org/html/2608.12342#bib.bib60)\)IE, TA, TGChinese05/21/2024Table 11:Comparison of Different Financial Models\. The abbreviations correspond to IE = Information Extraction, TA = Textual Analysis, QA = Question Answering, TG = Text Generation, RM = Risk Management, FO = Forecasting, DM = Decision\-Making\.DatasetsDomainLanguageAvg\. TokensError TypesYearCoNLL\-2014Hernandez and Calvo \([2014](https://arxiv.org/html/2608.12342#bib.bib63)\)GeneralEnglish602\.9grammatical, syntactic05/2014BEA\-2019Bryantet al\.\([2019](https://arxiv.org/html/2608.12342#bib.bib62)\)GeneralEnglish244\.9grammatical08/02/2019MEDECAbachaet al\.\([2024b](https://arxiv.org/html/2608.12342#bib.bib4)\)MedicalEnglish126\.5semantic01/02/2025GMEGNapoleset al\.\([2019](https://arxiv.org/html/2608.12342#bib.bib64)\)GeneralEnglish20\.7grammatical04/2019NLPTEA\-2020Raoet al\.\([2020](https://arxiv.org/html/2608.12342#bib.bib65)\)GeneralChinese35\.4grammatical12/04/2020MuCGECZhanget al\.\([2022](https://arxiv.org/html/2608.12342#bib.bib68)\)GeneralChinese38\.5grammatical07/10/2022TEC\-JLKoyamaet al\.\([2020](https://arxiv.org/html/2608.12342#bib.bib66)\)GeneralJapanese21\.8grammatical05/11/2020UA\-GECSyvokon and Nahorna \([2021](https://arxiv.org/html/2608.12342#bib.bib67)\)GeneralUkrainian15\.9grammatical11/08/2022Table 12:Comparison of Different Error Detection Benchmarks\.

## Appendix GSupplements to Related Work

We list financial models and their targeted tasks in Table[10](https://arxiv.org/html/2608.12342#A6.T10), and existing error detection benchmarks in Table[12](https://arxiv.org/html/2608.12342#A6.T12)\.