BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension

# BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension
Source: [https://arxiv.org/html/2607.05614](https://arxiv.org/html/2607.05614)
11institutetext:Wichita State University, USA22institutetext:Center for Computational & Data Sciences, Bangladesh33institutetext:University of Dhaka, Bangladesh44institutetext:Amazon GenAI, USAIshita Sur ApanFahim AhmedSumaiya Karim KathaEzharuddin JubaerArmun AlamPranjal Kumar NandiAmin Ahsan AliAman ChadhaMd Mofijul IslamAKM Mahbubur Rahman

###### Abstract

Document comprehension is a challenging yet impactful task for Multimodal Large Language Models, especially as these systems see growing adoption in real\-world, human\-centric applications\. However, this adoption is limited for low\-resource languages such as Bangla due to the scarcity of high\-quality annotated data\. To address this gap, we introduceBaFCo, a benchmark dataset for Bangla form comprehension with a focus on Document Layout Analysis \(DLA\) and Key Information Extraction \(KIE\)\. BaFCo curates 200 multi\-page complex Bangladeshi government forms, sourced from across diverse sectors including agriculture, education, banking, and land management\. To accurately capture the structural and contextual complexity of these forms, we define a fine\-grained annotation schema comprising 26 types of form entities, along with a separate coarse form entity set consisting of 5 types\. We evaluate the latest MLLMs from the ChatGPT, Gemini, Claude, Qwen, and Kimi series using zero\-shot and chain\-of\-thought prompts under both low and high reasoning setups\. Our results reveal limitations in current MLLMs’ ability in comprehending Bangla forms, particularly in accurately localizing highly granular form entities\. Our dataset and code is available at:[https://huggingface\.co/datasets/Mausul/bafco](https://huggingface.co/datasets/Mausul/bafco)

††footnotetext:⋆Equal contribution\.†Equal supervision\.‡Work done outside role at Amazon\.
\(🖂\) Corresponding author:††footnotetext:
Accepted at the 19th European Conference on Computer Vision \(ECCV\), 2026\.## 1Introduction

Information retrieval from documents such as forms underpins many real\-world systems including banking, education, and public administration\[un\_report\]\.Document Layout Analysis\(DLA\) andKey Information Extraction\(KIE\)\[dla,kie\_survey\_1,kie\_survey\_2\]are tasks at the core of Document Understanding\. DLA identifies the structural elements of a page and the relationships between them\. KIE involves locating and extracting values from form fields filled digitally or by hand\. Both DLA and KIE are foundational precursors to downstream document tasks such as document question answering, entity linking, and summarization\.

Bangla remains a low\-resource language in document understanding despite being the world’s 7th most spoken language, with 284 million speakers\[zaban\]\. Government forms are also underrepresented in research, despite their semantic diversity and practical importance in public services\. Benchmarks such as FUNSD\[funsd\]and XFUND\[xfund\]have advanced form understanding in English and other languages, but no comparable benchmark has been available for Bangla forms\. Consequently, the lack of high\-quality datasets and benchmarks continues to limit the development of Bangla document understanding systems\.

To address this gap, we introduceBaFCo, a curated benchmark for multi\-page Bangla form comprehension, focusing on government forms and the tasks of DLA and KIE\. BaFCo provides fine\-grained annotations spanning 26 entity types and labeled relationships between related fields\. For DLA, the dataset contains 16,382 entities and 8,771 relationships across 200 forms \(316 pages, with 1–5 pages per form\)\. For KIE, it further includes 1,926 key\-value pairs spanning 156 forms \(186 pages\)\. To maximize diversity, we prioritize complex and varied layouts over simpler forms\. All annotations are created by trained annotators and reviewed by experts to ensure structural and semantic quality\. In addition, we provide an end\-to\-end evaluation pipeline, standard document\-comprehension metrics, and a coarse label set with five entity types, enabling systematic evaluation of MLLMs on Bangla DLA and KIE tasks\.

With the advent of LLMs and MLLMs, document understanding workflows have increasingly shifted toward generative approaches, driven in part by the popularity of conversational interfaces\. MLLMs are now widely used for document understanding tasks\[genkie,genkie2,genkie3\], replacing earlier OCR\-based methods\[ofa\]\. Although document\-specialized models are available\[docllm,layoutllm,mplug,dockylin\], off\-the\-shelf flagship MLLMs offer a compelling alternative due to their lower development overhead through publicly accessible APIs, the scarcity of high\-quality domain\-specific training data, and the substantial computational resources required to train custom models\[llm\_cost,mllm\_cost\]\.

Consequently, we evaluate flagship MLLMs on BaFCo using tuning\-free prompt\-based methods to assess their layout\-grounding capabilities and their understanding of Bangla\-specific form elements\. In DLA,Gemini 3 Properforms the best across all experimental setup with average mAP scores of 0\.1177 and 0\.2646 for granular and coarse entity set, respectively\. For KIE,Gemini 3 Proagain outperforms other models for Bangla forms, but for English onesGPT\-5\.2is surpassing others\. We also evaluate the models on a set of English forms from the same domains to examine the effects of language\. In case of DLA \(particularly for granular entity set\), the effect is minimal \(performance difference≤0\.02\\leq 0\.02\), whereas for KIEGPT\-5\.2andClaude Opus 4\.6performs better on English forms andGemini 3 Proon Bangla forms\.

![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/data_diversity_final.png)Figure 1:Overview of BaFCo data diversity\. \(a\) Examples of annotated forms across three difficulty levels: easy, medium, and hard \(see[Sec\.˜3\.2](https://arxiv.org/html/2607.05614#S3.SS2)\); \(b\) BaFCo supports detailed layout annotations with 26 entity types, including titles, form key\-value pairs, and tables; \(c\) It contains links between related entities through three relationship types: key\-to\-key, key\-to\-value, and value\-to\-value; \(d\) Examples of form variations, including scanned, watermarked, and handwritten forms; \(e\) BaFCo includes government forms from 15 domains; \(f\) Each form is annotated with nine page\-level attributes based on annotation difficulty, image quality, and form background\.Our main contributions are as follows:

1. 1\.We introduce BaFCo, a high\-quality dataset of Bangladeshi government forms with 26 entity types and annotated relationships between related fields, validated through human annotation and expert review\.
2. 2\.We present the first benchmark for Document Layout Analysis \(DLA\) and Key Information Extraction \(KIE\) on Bangla forms, addressing a critical gap in low\-resource document understanding\.
3. 3\.We provide an empirical analysis of the performance of off\-the\-shelf flagship MLLMs on DLA and KIE tasks for Bangla documents, revealing limitations of the current models, particularly in handling fine\-grained layouts\.

## 2Related Works

Document Layout Analysis \(DLA\)aims to identify and localize structural elements, such as text blocks, tables, images, and form fields\. Early benchmarks spurred research from rule\-based systems\[ha1995recursive,journet2005text\]to machine learning\[wu2008machine,bukhari2010document\]for segmenting and classifying document regions\. However, they struggle with complex real\-world layouts\. Hence, recent approaches predominantly rely on deep learning models\[docopilot,doclayllm,layoutlmv3,mplug,publaynet\]that combine visual, textual, and structural cues, including CNN\-Transformer hybrids and multimodal architectures, which require high\-quality annotations to generalize across varied domains and document types\. Early foundational datasets like PubLayNet\[publaynet\]and DocBank\[li2020docbank\]scaled through leveraging scientific articles from PubMed Central and arXiv, respectively\. However, these born\-digital scientific datasets lack real\-world layout variability\. DocLayNet\[pfitzmann2022doclaynet\]addressed this gap by providing data from diverse sources; including financial reports, manuals, and legal documents\.

Key Information Extraction \(KIE\)focuses on identifying and extracting form key\-value pairs\. Among generative KIE approaches, GenKIE\[genkie\]uses an encoder\-decoder architecture with zero\-shot prompt based evaluation\. In BROS\[bros\]uses a multimodal, layout\-aware encoder\. LiLT\[lilt\]and OmniDocBench\[omnidocbench\]are multilingual, nonetheless Bangla is absent from their pool\. Apart from GPT4o in OmniDocBench, no work evaluates recent flagship MLLMs’ on KIE\.

Form Understandingis an important subfield of document intelligence, focusing on extracting, structuring, and linking information from forms\. FUNSD\[funsd\]was among the first public benchmarks in this area, providing annotations for questions, answers, headers, and other text entities, along with question–answer relationships\. It enabled research on semantic entity recognition and relation extraction, but its annotation schema is largely limited to flat question–answer structures\. Subsequent datasets have targeted specific domains and document types, including government business documents\[buddie\], financial forms\[formnlu\], receipts\[sroie,cord,docile\], and legal and financial documents\[kleister\]\. While these datasets contain challenging layouts and long\-form content, they generally lack fine\-grained text\-entity annotations and spatial relationships required for detailed form structure understanding, particularly in low\-resource and domain\-specific settings\.

Multilingual and Low\-Resource Document DatasetsMost document layout analysis benchmarks focus on high\-resource languages, leaving low\-resource settings underexplored\. XFUND\[xfund\]introduced a multilingual benchmark covering seven languages\. However, many low\-resource languages remain absent\. Bangla is one such example: despite its large speaker base, public benchmarks for Bangla document understanding have been scarce\. A first step was BaDLAD\[badlad\], which introduced a multi\-domain dataset spanning books, newspapers, government documents, and historical records\. Although it provides 710k polygon annotations, its labels are limited to coarse layout elements such as text regions, paragraphs, tables, and images\. However, it does not support fine\-grained form understanding with form\-specific entities, key–value relationships, or government\-form structures\.

Table 1:Comparison ofBaFCowith existing document understanding benchmarks\.Domaindenotes the no\. of document domains \(e\.g\., finance, education, healthcare\)\.Formindicates support for form\-style documents;LRL, low\-resource languages;Multipage, multi\-page annotations; andHuman, human\-created annotations\.BBoxandTextdenote bounding\-box and text annotations, respectively\.Tab,Img,Sig,Cb, andPhotoindicate annotations for tables, images \(e\.g\., figures, logos, diagrams\), signatures, checkboxes \(including single\-select, multi\-select, and radio buttons\), and photographs\.ELdenotes entity linking between related fields\.Taskslists supported tasks: Document Layout Analysis \(DLA\) and Key Information Extraction \(KIE\)\.BenchmarkDataAnnotation Attributes and TypesTasksDomainFormLRLMultipageHumanBBoxTextTabImgSigCbPhotoELDLAKIEDLA BenchmarksPubLayNet\[publaynet\]2––––✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark––––✓\\checkmark–DocBank\[li2020docbank\]1––✓\\checkmark–✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark––––✓\\checkmark–DocLayNet\[pfitzmann2022doclaynet\]5–––✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark––––✓\\checkmark–OmniDocBench\[omnidocbench\]9––✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark––––✓\\checkmark–Form BenchmarksBuDDIE\[buddie\]3✓\\checkmark––✓\\checkmark✓\\checkmark✓\\checkmark–––––––✓\\checkmarkFUNSD\[funsd\]1✓\\checkmark––✓\\checkmark✓\\checkmark✓\\checkmark–––––✓\\checkmark✓\\checkmark✓\\checkmarkXFUND\[xfund\]1✓\\checkmark––✓\\checkmark✓\\checkmark✓\\checkmark–––––✓\\checkmark–✓\\checkmarkFormNLU\[formnlu\]1✓\\checkmark––✓\\checkmark✓\\checkmark✓\\checkmark–––––✓\\checkmark✓\\checkmark✓\\checkmarkLow\-Resource Language BenchmarkBaDLAD\[badlad\]4–✓\\checkmark–✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark––––✓\\checkmark–\\rowcolorblack\!10BaFCo \(ours\)15✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark✓\\checkmark

Our work addresses this gap with BaFCo, the first publicly available benchmark for Bangla form comprehension and layout analysis\. BaFCo provides fine\-grained annotations spanning 26 entity types and labeled relationships between related entities \(e\.g\., linking field labels to values\), a capability not previously available for Bangla documents\.

## 3Dataset

### 3\.1Data Collection and Curation

We adopt a quality\-over\-quantity strategy, prioritizing careful selection, diversity, and annotation fidelity over scale\. The dataset is composed of publicly available Bangladeshi government forms††[https://forms\.portal\.gov\.bd/](https://forms.portal.gov.bd/), ensuring authenticity and real\-world relevance\. Government forms were chosen due to their structured yet highly variable layouts, dense semantic content, and practical importance\.

Forms were filtered to remove duplicates, incomplete scans, and low\-quality images that could introduce annotation noise\. The selected forms span multiple administrative domains, including taxation, healthcare, education, and civil services\. They exhibit varying degrees of layout complexity, including multi\-column structures, nested fields, tables, and handwritten or stamped regions\. We further categorize forms into three difficulty levels\-easy, medium, and difficult\-based on layout density, visual clutter, and semantic coupling between fields\.

Unlike large\-scale multilingual benchmarks such as OmniDocBench\[omnidocbench\]or BuDDIE\[buddie\], which emphasize breadth across document types and languages, our dataset is purpose\-built for Bangla forms\. Accordingly, comparisons in this work focus on Bangla\-specific and multilingual datasets that include Bangla content\. The dataset is intended as a high\-quality benchmark for low\-resource document layout analysis rather than a general\-purpose corpus\.

### 3\.2Dataset Description

Form Selection Criteria and Difficulty Levels:Forms containing 1–5 pages were first shortlisted\. The selected forms were then grouped into difficulty levels based on layout complexity and component diversity\.Easyforms contained basic elements such as form key–value pairs \(including one\-to\-many and key\-to\-key relationships\), inline and signature key–value pairs, headers, titles, section headings, text blocks, photo fields, and footers\. However, they did not include tables, checkboxes, or tick marks\.Mediumforms introduced additional structural complexity through checkboxes, tick marks, and tables consisting only of columns, while explicitly excluding tables with both rows and columns, sub\-columns, or embedded checkboxes and tick marks\.Hardforms represented the highest level of complexity and included tables with rows, columns, and sub\-columns; tables containing checkboxes or tick marks; and other unique, dense, or unconventional layout components\.

DLA Dataset Composition:Following the difficulty\-based grouping, BaFCo encompasses a diverse range of form structures, including application and non\-application forms, sparse and dense layouts \(ranging from forms with few fields to those containing crowded or heavily tabular regions\), and both guided forms \(with explicit bounding boxes\) and unguided forms \(without such cues\)\. These variations are representative of real\-world government forms\. BaFCo also exhibits a deliberately skewed entity\-class distribution, preserving the natural long\-tail distribution of entities found in real\-world documents rather than imposing artificial class balance\. Common elements \(e\.g\., form keys and values\) dominate the dataset, whereas specialized elements \(e\.g\., signatures and tabular sub\-fields\) are relatively rare, reflecting the inherent long\-tailed nature of government forms\. Overall, the dataset contains 16,382 entities and 8,771 relationships across 200 forms \(316 pages\), with each form comprising between 1 and 5 pages\.

KIE Dataset Composition:For KIE, we evaluate 156 forms \(186 pages and 1,926 key–value pairs\) spanning the same Easy, Medium, and Hard difficulty levels\. English forms are included to facilitate comparisons between a high\-resource language and a low\-resource language\. The Bangla\-to\-English ratio is 1\.14:1 by form count and 1\.24:1 by key–value pair count\.

Table 2:BaFCo dataset statistics for the DLA and KIE tasks, broken down by difficulty level\. Under our difficulty rubric, easy forms do not contain tables or selection elements \(checkboxes/tick marks\); consequently, these fields are absent from the Easy category and are denoted by ‘–’\.EasyMediumHardTotalDataset SizeDLA Forms746660200DLA Pages90108118316DLA Tables–59143202KIE Forms864228156KIE Pages955734186StructureDLA Avg\. entities/form55\.7677\.76118\.7381\.91DLA Avg\. relations/form30\.0137\.7467\.6543\.85KIE Avg\. key\-value pairs/form11\.213\.515\.012\.3

EasyMediumHardTotalAnnotation TypesDLA Form values1,4181,1791,0793,676DLA Inline values155198216569DLA Signatures187189144520DLA Checkboxes–245166411DLA Tick marks–6551116DLA Table rows–313622935DLA Table columns–2629631,225DLA Total annotations1,7602,4513,2417,452KIE Filled key\-value pairs9555683911,926

### 3\.3Semantic Form Entity Definition

Based on our analysis of Bangladeshi government forms, we defined a fine\-grained and comprehensive taxonomy of semantic form entities\. The dataset contains 26 entity types covering structural components \(e\.g\., headers, sections, and tables\), functional fields \(e\.g\., key–value pairs, checkboxes, and signatures\), and auxiliary elements \(e\.g\., instructions, seals, and stamps\) across both single\-page and multi\-page forms \(see[Tab\.˜3](https://arxiv.org/html/2607.05614#S3.T3)\)\. To examine the effect of entity granularity on model performance, we additionally group these 26 entity types into five coarse\-grained categories\.

In addition to entity labels, we annotate relationships between semantically linked fields, enabling the use of BaFCo for downstream document understanding tasks\.[Tab\.˜2](https://arxiv.org/html/2607.05614#S3.T2)summarizes the total number of annotated entities and relationships across the different difficulty levels\. The taxonomy for each entity type was iteratively refined through pilot annotations and expert feedback to ensure coverage, consistency, and applicability across diverse form layouts\. This detailed taxonomy distinguishes our dataset from existing Bangla document resources, which typically employ generic or task\-specific entity label sets\.

Table 3:Semantic Form Entities\. Both coarse and granular form entities are enlisted\.CoarseGranularHeadersHeader, Title, Footer, Section TitleTableTable, Table Caption, Table Section, Table Index, Row Primary Key, Row Value, Column Primary Key, Column ValueImageDiagram / Logo / FigureFieldsForm Key, Form Value, Inline Key, Inline Value, Signature Key, Signature Value, Tick Mark, Checkbox, Image FieldOthersText Block, Page Number, Gibberish, Others

### 3\.4Annotation Guidelines

#### 3\.4\.1DLA:

To ensure high annotation quality and reproducibility, we developed a detailed annotation guideline document that specifies entity definitions, bounding box rules, relationship constraints, and edge cases\. Particular emphasis is placed on resolving ambiguities common in Bangla forms, such as overlapping text regions, visually implicit field boundaries, and mixed printed\-handwritten content\. The guideline is refined through multiple rounds of annotator feedback and validation, resulting in a consistent annotation protocol that minimizes subjective interpretation while retaining flexibility for complex layouts\.

#### 3\.4\.2KIE:

For KIE, we annotate bounding boxes and define explicit relationships between regions\. Two labels —keyandvalue— were used\. Annotators filled a text field for each region: key fields copied the annotated text, while value fields recorded the corresponding entry\. These fields served as ground truth\. To evaluate models, values were overlaid on form images, and LLMs were prompted to extract them given the associated keys\. To assess model performance, the values were superimposed onto the form images, and the LLMs were prompted to extract the values given the corresponding keys\.

### 3\.5Annotation Procedure

17 annotators underwent two days of training using guideline documents, practice forms, and tutorial videos covering dataset nuances\. All were proficient in Bangla and experienced in document labeling\. Each document was independently annotated inLabel Studio††[https://labelstud\.io/](https://labelstud.io/)by a trained annotator and reviewed by an expert reviewer\. Disagreements or uncertainties in labels or bounding boxes were resolved via group discussion and majority voting\. To quantify reliability, we measured agreement between the trained annotator and the expert reviewer on the pre\-majority\-vote annotations, obtaining a Cohen’sκ\\kappaof0\.9740\.974\.

### 3\.6Difficulties Faced During Annotation:

Most annotation errors stemmed from layout complexity and fell into three categories\.Semantic confusionsarose when distinguishing closely related entity types: in fill\-in\-the\-blank forms, standard key–value pairs were frequently mislabeled as inline or signature pairs, and slashes \(“/”\) were occasionally mistaken as separators between Tick Mark entities\.Structural ambiguitieswere most common in guided layouts, where form keys were hard to separate from table cells, and where a few multi\-cell tables were annotated cell\-by\-cell rather than as a single structure\.Technical inconsistenciesincluded misclassification, duplicate boxes, and overly broad inline annotations, along with occasional omissions of essential elements \(e\.g\., header logos, signature links\) and erroneous inclusion of irrelevant footer text\. All these issues were identified and corrected by expert reviewers prior to finalization\.

## 4Experiments

### 4\.1Experimental Setup

#### 4\.1\.1Models:

We evaluate five flagship MLLMs spanning both proprietary and open\-source ecosystems:GPT\-5\.2,Gemini 3 Pro, andClaude Opus 4\.6on the proprietary side, andQwen 3\.6 PlusandKimi K2\.5among open\-source models\. We focus exclusively on MLLMs because they represent the most promising paradigm for general\-purpose document understanding\. Unlike OCR\-based pipelines and encoder architectures, which are typically designed for predefined tasks with fixed output schemas, MLLMs operate in an open\-ended, instruction\-driven manner that better aligns with emerging document AI applications\. Consequently, direct comparisons with OCR or encoder\-based systems are less informative for our objectives\. We therefore center our evaluation on MLLMs, which are also widely accessible through API\-based deployment\. For inference, we use native batch endpoints for proprietary models to improve cost efficiency andOpenRouter††[https://openrouter\.ai/](https://openrouter.ai/)endpoints for open\-source models\.

#### 4\.1\.2Prompts:

To utilize MLLMs’ inference\-time task\-completion capabilities, we used zero\-shot\[zero\_shot\]and Chain\-of\-Thought \(CoT\) prompting\[cot\]\. We included task descriptions, form entity specifications, and output\-structure instructions in the prompts\. For KIE, we followed the prompt structure of GenKIE\[genkie\], where KIE is formulated as a visual question\-answering task\.

#### 4\.1\.3Reasoning Effort:

Besides CoT prompting, for DLA experiments we also use the built\-in reasoning of the flagship models\. We run every setup at two reasoning levels,*low*and*high*, set via the*reasoning\_effort*API parameter\. We use this as our main control because it is the only setting shared across providers for adjusting models’ reasoning level\. Since black\-box APIs do not reveal each model’s internals, it is the fairest basis for comparison\. To cap runaway generation, we limit*max\_output\_tokens*at16,00016\{,\}000for low reasoning and64,00064\{,\}000for high reasoning\. For consistency, we use only the*low*and*high*levels, as these settings were available across all providers \(Gemini 3 Pro’s API did not offer a*medium*reasoning effort level at the time of writing\)\.

#### 4\.1\.4DLA Evaluation Metrics:

We evaluate model predictions using standard detection and classification metrics following prior form understanding and object detection benchmarks\[formnlu,omnidocbench,coco\]\. As in\[formnlu,omnidocbench\], we perform greedy matching between predicted and ground\-truth bounding boxes with two Intersection\-over\-Union \(IoU\) thresholdsτ∈0\.3,0\.5\\tau\\in\{0\.3,0\.5\}to ensure order\-invariant assignment\.

A predicted box is counted as a true positive \(TP\) if matched to a ground\-truth instance withIoU≥τ\\mathrm\{IoU\}\\geq\\tau\. Unmatched predictions are false positives \(FP\), and ground\-truth instances without corresponding predictions are false negatives \(FN\)\. Using these counts, we compute precision, recall, F1 score, and mean average precision \(mAP\)\. For a fixed IoU thresholdτ\\tau, mAP is obtained by averaging class\-wise Average Precision \(AP\) across all form field categories\.

#### 4\.1\.5KIE Evaluation Metrics:

Following prior work\[formnlu,omnidocbench,genkie\], we use precision, recall, F1, and Normalized Edit Similarity \(N​E​SNES\) for KIE evaluation\.

N​E​SNESis defined asN​E​S=1−N​E​DNES=1\-NED, whereN​E​D=d​\(s,t\)max⁡\(\|s\|,\|t\|\)NED=\\frac\{d\(s,t\)\}\{\\max\(\|s\|,\|t\|\)\}andd​\(s,t\)d\(s,t\)is the Levenshtein distance between the predicted stringssand ground\-truthtt\.

### 4\.2Evaluation Setup

To evaluate MLLMs’ performance for each taskt∈Tt\\in\\mathit\{T\}whereT=\{D​L​A,K​I​E\}\\mathit\{T\}=\\left\\\{DLA,\\;KIE\\right\\\}, image of pagejjof formii,𝐛i,j\\mathbf\{b\}\_\{i,j\}is selected from BaFCo datasetB\\mathit\{B\}\. Then along with promptpp\(p∈Ptp\\in\\mathit\{P\_\{t\}\}\) the page image𝐛i,j\\mathbf\{b\}\_\{i,j\}is passed to each MLLM,MθM\_\{\\theta\}from a pool of models \(see[Sec\.˜4\.1\.1](https://arxiv.org/html/2607.05614#S4.SS1.SSS1)\),to generate raw predictiony~i,jp\\tilde\{y\}\_\{i,j\}^\{\\,p\}\.

y~i,jp=Mθ​\(𝐛i,j,p\)\\tilde\{y\}\_\{i,j\}^\{\\,p\}=M\_\{\\theta\}\\\!\\left\(\\mathbf\{b\}\_\{i,j\},\\,p\\right\)\(1\)
The raw predictiony~i,jpt\\tilde\{y\}\_\{i,j\}^\{\\,p\_\{t\}\}is subsequently processed by a validator functionV​\(⋅\)\\mathit\{V\}\(\\cdot\)that checks for adherence of predictions to a predefined output schema customized for DLA and KIE\. Invalid inferences are excluded from evaluation\.

y^i,jpt=V​\(y~i,jpt\)\\hat\{y\}\_\{i,j\}^\{\\,p\_\{t\}\}=\\mathit\{V\}\\\!\\left\(\\tilde\{y\}\_\{i,j\}^\{\\,p\_\{t\}\}\\right\)\(2\)
The validated prediction is then compared against the ground\-truth annotationyi,jy\_\{i,j\}to obtain page\-level performance measurements aggregated across pages and forms\. For each predicted form entity, the output also contains a brief label justification and a confidence score for the bounding box coordinates\.

For DLA, we define a prompt set asP𝐷𝐿𝐴=\{pzs,pcot\}\\mathit\{P\_\{DLA\}\}=\\left\\\{p\_\{\\text\{zs\}\},\\;p\_\{\\text\{cot\}\}\\right\\\}, wherepzsp\_\{\\text\{zs\}\}andpcotp\_\{\\text\{cot\}\}denote DLA\-specific zero\-shot and chain\-of\-thought prompts, respectively\. For KIE the prompt setP𝐾𝐼𝐸\\mathit\{P\_\{KIE\}\}contains a zero\-shot evaluation prompt\. To observe effects of entity set size, in addition to the granular form entity set containing 26 entity categories, we developed a minimal set containing 5 coarse entity categories and mapped all original 26 categories to coarse categories \(See[Tab\.˜3](https://arxiv.org/html/2607.05614#S3.T3)\)\.

## 5Results

### 5\.1Document Layout Analysis \(DLA\)

Table 4:Layout analysis performance for both Granular and Coarse label sets\.Boldindicates the best overall result per column within each entity set, andBlueindicates the best result within each combination of prompt variant \(Zero\-Shot: ZS, Chain\-of\-Thoughts: CoT\) and reasoning effort \(Low, High\)\.μ\\muIoU is used to show overall average IoU regions for all predicted bounding boxes\.μ\\muIoU@@\[0\.3, 0\.5\] is used to show average of bounding boxes that are over the thresholds 0\.3 and 0\.5, the thresholds are selected based on works from Form\-NLU\[formnlu\]and OmniDocBench\[omnidocbench\]\. Across all prompt, reasoning effort, and entity set granularity level combination,Gemini 3 ProoutperformsGPT\-5\.2andClaude Opus 4\.6\. For each metric higher \(↑\\uparrow\) is better\.ReasoningEffortPromptModelIoU@@0\.3IoU@@0\.5mAPF1μ\\muIoUμ\\muIoU@@0\.3mAPF1μ\\muIoUμ\\muIoU@@0\.5Granular Entity SetLowZSGPT\-5\.20\.05300\.11580\.12420\.45560\.01550\.04440\.12590\.6175Claude Opus 4\.60\.00410\.01610\.03160\.43340\.00100\.00490\.03230\.5998Gemini\-3 Pro0\.09000\.19530\.24300\.56470\.04830\.12220\.24490\.6787Kimi K2\.50\.01560\.05660\.06760\.47870\.00480\.01960\.06950\.6087Qwen 3\.6\-Plus0\.00970\.04220\.06910\.41840\.00080\.00880\.07050\.6048CoTGPT\-5\.20\.06430\.12730\.13430\.47820\.02660\.06180\.13490\.5991Claude Opus 4\.60\.00370\.01740\.03550\.42060\.00070\.00510\.03630\.5907Gemini\-3 Pro0\.08530\.19430\.24200\.53900\.04520\.12070\.24330\.6594Kimi K2\.50\.01050\.04800\.06900\.45300\.00270\.01820\.07070\.5989Qwen 3\.6\-Plus0\.00830\.03710\.06500\.41010\.00100\.00720\.07170\.6025HighZSGPT\-5\.20\.05110\.11920\.11480\.46580\.01570\.04810\.11570\.6134Claude Opus 4\.60\.00740\.03210\.04600\.44080\.00120\.00750\.04650\.5771Gemini\-3 Pro0\.11770\.22820\.22430\.53120\.06410\.14110\.22570\.6419Kimi K2\.50\.04530\.11170\.11320\.46170\.01500\.04410\.11400\.6123Qwen 3\.6\-Plus0\.01750\.06060\.06610\.45010\.00400\.02340\.06680\.5910CoTGPT\-5\.20\.06150\.12910\.12340\.47230\.01970\.05810\.12470\.5985Claude Opus 4\.60\.00650\.03100\.05010\.43500\.00160\.01040\.05050\.5683Gemini\-3 Pro0\.11340\.22060\.21930\.54380\.06430\.14350\.22050\.6586Kimi K2\.50\.04760\.11330\.11620\.45590\.01520\.04350\.11750\.6130Qwen 3\.6\-Plus0\.01660\.06090\.06570\.43290\.00520\.02160\.06650\.5959Coarse Entity SetLowZSGPT\-5\.20\.18000\.29740\.27420\.51770\.08520\.16540\.27670\.6495Claude Opus 4\.60\.01680\.05550\.06970\.40810\.00420\.01380\.07220\.5875Gemini\-3 Pro0\.25780\.40200\.36020\.60660\.16060\.28470\.36160\.6986Kimi K2\.50\.04690\.12130\.13060\.45290\.01310\.03920\.13440\.6433Qwen 3\.6\-Plus0\.03300\.12730\.15110\.41110\.00210\.02960\.15840\.5865CoTGPT\-5\.20\.16710\.28610\.26590\.51740\.07810\.15710\.26820\.6409Claude Opus 4\.60\.01310\.05780\.07310\.41640\.00220\.01320\.07510\.5884Gemini\-3 Pro0\.25130\.40510\.36820\.60850\.14400\.28270\.36980\.7002Kimi K2\.50\.04260\.12930\.13420\.43850\.00570\.03590\.13980\.6033Qwen 3\.6\-Plus0\.03770\.11990\.15120\.41230\.00210\.02500\.15770\.5981HighZSGPT\-5\.20\.14340\.28660\.25610\.50460\.05320\.13910\.25920\.6472Claude Opus 4\.60\.04400\.11680\.11810\.42830\.01360\.04540\.11990\.5932Gemini\-3 Pro0\.26460\.41160\.37290\.61250\.14800\.29150\.37450\.7003Kimi K2\.50\.13360\.26380\.22590\.47480\.04660\.11900\.22830\.6205Qwen 3\.6\-Plus0\.05890\.16920\.15830\.47400\.02040\.07140\.16040\.6143CoTGPT\-5\.20\.13540\.27760\.25490\.51580\.06630\.15460\.25750\.6340Claude Opus 4\.60\.03580\.10120\.11120\.45180\.00910\.03850\.11320\.5829Gemini\-3 Pro0\.24440\.39280\.34990\.60710\.14260\.27510\.35110\.6975Kimi K2\.50\.13610\.26020\.22450\.48740\.04720\.12170\.22720\.6281Qwen 3\.6\-Plus0\.06930\.17490\.16150\.47660\.02000\.07330\.16370\.6218

Table 5:Bangla and English Form Comparison for DLA \(IoU@0\.3\) under different experimental setups\. LR and HR stand for Low Reasoning and High Reasoning respectively\. ZS and CoT denote Zero\-Shot and Chain\-of\-Thought prompts\. Shapes indicate higher performance for Bangla \(∙\\bullet\), English \(■\\blacksquare\), or neither \(◆\\blacklozenge\) language\. For granular entity set, the effect of language is minimal \(≤0\.02\\leq 0\.02\)\. For coarse entity set, effect of language is slightly more with maxΔ\\Deltaof0\.12~0\.12\. For each metric higher \(↑\\uparrow\) is better\.VariantModelBanglaEnglishDifference \(Bn\-En\)mAPF1mAPF1Δ\\DeltamAPΔ\\DeltaF1Granular Entity SetLR \+ ZSGPT\-5\.20\.090\.160\.090\.14◆\\blacklozenge0\.00∙\\bullet0\.02Gemini 3 Pro0\.160\.240\.150\.24∙\\bullet0\.01◆\\blacklozenge0\.00LR \+ CoTGPT\-5\.20\.120\.180\.110\.15∙\\bullet0\.01∙\\bullet0\.03Gemini 3 Pro0\.150\.210\.140\.23∙\\bullet0\.01■\\blacksquare0\.02HR \+ ZSGPT\-5\.20\.090\.160\.110\.15■\\blacksquare0\.02∙\\bullet0\.01Gemini 3 Pro0\.160\.240\.160\.23◆\\blacklozenge0\.00∙\\bullet0\.01HR \+ CoTGPT\-5\.20\.120\.180\.130\.17■\\blacksquare0\.01∙\\bullet0\.01Gemini 3 Pro0\.150\.210\.160\.24■\\blacksquare0\.01■\\blacksquare0\.03Coarse Entity SetLR \+ ZSGPT\-5\.20\.340\.440\.220\.32∙\\bullet0\.12∙\\bullet0\.12Gemini 3 Pro0\.380\.510\.350\.42∙\\bullet0\.03∙\\bullet0\.09LR \+ CoTGPT\-5\.20\.290\.440\.280\.40∙\\bullet0\.01∙\\bullet0\.04Gemini 3 Pro0\.350\.470\.250\.40∙\\bullet0\.09∙\\bullet0\.07HR \+ ZSGPT\-5\.20\.340\.440\.270\.35∙\\bullet0\.07∙\\bullet0\.09Gemini 3 Pro0\.380\.510\.360\.47∙\\bullet0\.02∙\\bullet0\.04HR \+ CoTGPT\-5\.20\.290\.440\.270\.38∙\\bullet0\.02∙\\bullet0\.07Gemini 3 Pro0\.350\.470\.420\.52■\\blacksquare0\.07■\\blacksquare0\.05

[Tab\.˜4](https://arxiv.org/html/2607.05614#S5.T4)reports DLA performance across models, prompting strategies, reasoning effort levels, and entity granularities\. Overall,Gemini 3 Proachieves the best results in most configurations, followed byGPT\-5\.2, whileClaude Opus 4\.6is consistently the weakest\. Performance improves noticeably from the granular to the coarse entity set\. For instance, under high reasoning effort with zero\-shot prompting,Gemini 3 Proincreases from 0\.1177 mAP on the granular entity set to 0\.2646 on the coarse set at IoU@0\.3, with similar trends observed across other models\. Among the open\-source models,Kimi K2\.5andQwen 3\.6\-Plustrail the proprietary leaders under low reasoning effort\. However,Kimi K2\.5benefits substantially from increased reasoning effort: its coarse mAP@0\.3 under zero\-shot prompting rises from 0\.0469 to 0\.1336, approachingGPT\-5\.2\(0\.1434\), whereasQwen 3\.6\-Plusimproves only modestly, from 0\.0330 to 0\.0589\.

Increasing reasoning effort has mixed effects, yielding small gains in some settings but slight decreases in Avg\. IoU in others\. CoT prompting likewise provides limited benefit, often performing comparably to zero\-shot prompting\.

The effect of language on MLLM performance is minimal\.[Tab\.˜5](https://arxiv.org/html/2607.05614#S5.T5)compares Bangla and English performance for the two strongest models,Gemini 3 ProandGPT\-5\.2\. For both models, the two languages yield similar results across most configurations, withΔ\\DeltamAP typically within 0\.02 on the granular entity set\. Overall, DLA performance remains largely consistent across both languages and relatively low for granular entities\.

### 5\.2Key Information Extraction \(KIE\)

[Tab\.˜6](https://arxiv.org/html/2607.05614#S5.T6)reports KIE performance using Precision, Recall, F1, and Normalized Edit Similarity \(NES\)\. All models score substantially higher here than on DLA\. Among proprietary models,Gemini 3 Proleads on Bangla \(F1 0\.848, NES 0\.866\), whileGPT\-5\.2leads on English \(F1 0\.847, NES 0\.851\)\.Claude Opus 4\.6trails both but remains competitive, unlike in DLA, where it was the weakest performer\. Among open\-source models,Qwen 3\.6\-Plusperforms best on both languages \(Bangla F1 0\.794, English F1 0\.835\), ahead ofKimi K2\.5\. It ranks second overall on Bangla, surpassingGPT\-5\.2\(F1 0\.781\) andClaude Opus 4\.6\(F1 0\.678\), and matchesClaude Opus 4\.6on English \(F1 0\.835\), narrowing the gap between open\-source and proprietary models\.

Unlike DLA, where language differences were minimal, KIE varies more clearly by language\. Most models score higher on English\-the gap is largest forClaude Opus 4\.6\(0\.678 vs\. 0\.835 F1\) andKimi K2\.5\(0\.681 vs\. 0\.828\), while onlyGemini 3 Profavors Bangla \(0\.848 vs\. 0\.828\)\. Consequently, the best model is language\-dependent:Gemini 3 Proon Bangla andGPT\-5\.2on English\. So cross\-lingual differences are stronger in KIE compared to layout detection\.

Table 6:KIE evaluation results on Bangla \(BN\) and English \(EN\) forms\. Normalized Edit Similarity \(NES\) is reported along with Precision, Recall, and Macro\-F1\. English forms are selected from the same domain as Bangla\.Gemini 3 Properforms the best for Bangla forms, whereas for EnglishGPT\-5\.2performs better\.PrecisionRecallMacro\-F1NESModelBNENBNENBNENBNENProprietaryGPT\-5\.20\.7840\.8520\.7800\.8440\.7810\.8470\.7960\.851Claude Opus 4\.60\.6750\.8390\.6850\.8320\.6780\.8350\.7040\.840Gemini 3 Pro0\.8440\.8320\.8530\.8270\.8480\.8280\.8660\.833Open\-SourceKimi K2\.50\.6840\.8350\.6810\.8250\.6810\.8280\.7000\.832Qwen 3\.6\-Plus0\.7980\.8390\.7930\.8340\.7940\.8350\.8040\.844

## 6Discussion

Granularity of form entities strongly affects DLA performance\. Across all models, performance improves markedly when coarse entity set is used\. Reducing the number of entity types substantially increases both mAP and F1 across most configurations\. For example, mAP score forGemini\-3 Proimproves from 0\.1177 to 0\.2646, for granular to coarse entity set shift for IoU@0\.3, more than doubling performance\. This indicates that much of the challenge in DLA lies in distinguishing fine\-grained entity types; when grouped into broader categories, models localize layout regions more reliably\.

Increasing reasoning effort does not consistently improve geometric precision\. Across both entity sets and prompting strategies, higher reasoning effort yields only small and inconsistent performance changes\. In several cases, it slightly reduces localization quality, particularly Avg\. IoU\. For example,Gemini\-3 Proshows lower Avg\. IoU when moving from low to high reasoning effort in the granular setting, whileGPT\-5\.2shows small improvements in some configurations\. Overall, this mixed behavior suggests that additional reasoning does not reliably improve geometric alignment; layout prediction appears to depend more on visual\-spatial pattern recognition than extended reasoning\.

Chain\-of\-thought prompting provides limited gains for DLA\. Across most models and settings, CoT prompting does not clearly outperform zero\-shot prompting; performance often remains similar or slightly declines in mAP and F1\. For example,Gemini\-3 Proachieves slightly lower mAP with CoT than with zero\-shot in multiple settings across both entity sets\. This suggests that encouraging multi\-step reasoning offers limited benefit for DLA, which appears to rely more on visual\-spatial understanding than explicit reasoning chains\.

Language has limited impact on DLA performance\. Across most experimental variants, performance differences between Bangla and English forms are small, with negligible gaps in mAP and F1 \(see[Tab\.˜5](https://arxiv.org/html/2607.05614#S5.T5)\)\. In the granular entity setting, differences are near zero \(e\.g\.,Δ\\DeltamAP≤0\.02\\leq 0\.02across most setups\)\. Even in the coarse setting, improvements are inconsistent and modest relative to overall performance\. This suggests that DLA difficulty for MLLMs stems mainly from geometric localization and structural understanding rather than language\.

MLLMs perform substantially better on KIE than on DLA\. Across all models, KIE results are considerably stronger than document layout analysis performance\. The best\-performing model achieves F1 scores above 0\.84 on Bangla forms and 0\.80 on English forms \(see[Tab\.˜6](https://arxiv.org/html/2607.05614#S5.T6)\), while DLA results remain much lower across comparable configurations\. This indicates that MLLMs are more effective at textual understanding and semantic extraction than at precise geometric localization of layout regions\.

Language differences are more pronounced in KIE than in DLA\. Unlike DLA experiments\-where Bangla and English forms yield nearly identical performance\. KIE results show clearer language\-dependent variation across models\. For example,Gemini 3 Proachieves the best performance on Bangla forms \(F1 = 0\.848\), whileGPT\-5\.2performs best on English forms \(F1 = 0\.800\) \(see[Tab\.˜6](https://arxiv.org/html/2607.05614#S5.T6)\)\. This suggests that language\-specific factors affect text\-heavy extraction tasks, while layout detection remains largely language\-agnostic\.

## 7Conclusion

We proposeBaFCo, the first Bangla document understanding dataset featuring well\-curated multi\-domain Bangladeshi government forms\. It includes an expanded, fine\-grained set of 26 form entities designed for better Bangla document comprehension\. We evaluated flagship MLLMs on DLA and KIE using both granular and coarse entity sets, revealing significant performance gaps\. DLA performance is inconsistent across models and sensitive to reasoning effort and prompting strategies, whereas KIE results are generally stronger, though some lexical mismatches remain\.

## 8Limitations and Future Works

The scope of generative model\-based Bangla form comprehension can be expanded along several directions:

\(a\) Our experiments focus on DLA and KIE, key building blocks for downstream tasks like summarization and ontology generation\. Future work will explore frontier models on broader document comprehension tasks\.

\(b\) Low\-resource languages face greater challenges in curating high\-quality domain\-specific documents due to limited data access, lack of experts, and other constraints\. Although BaFCo contains fewer annotated forms than high\-resource datasets like\[omnidocbench,publaynet,xfund\], it aims to stimulate research on Bangla form comprehension\. Future work will expand both the quantity and diversity of forms\.

\(c\) To address model limitations and scarcity of large\-scale Bangla data, lightweight post\-training methods and agentic AI approaches can be explored\.

## Acknowledgement

This work was partially funded by Independent University, Bangladesh \(IUB\)\.

## References

## Appendix

## Appendix 0\.AForm Entity Distribution

[Fig\.˜2](https://arxiv.org/html/2607.05614#Pt0.A1.F2)visualizes the ground\-truth entity distribution under the coarse \(5\-class\) and granular \(26\-class\) label sets for DLA, with exact counts and percentages listed in[Tab\.˜7](https://arxiv.org/html/2607.05614#Pt0.A1.T7)\. The distribution is strongly long\-tailed, reflecting the natural composition of real\-world Bangladeshi forms\. At the coarse level,*Fields*dominate \(64\.6%\), followed by*Table*\(27\.3%\), with*Others*\(3\.9%\),*Headings*\(3\.8%\), and*Image*\(0\.4%\) forming the tail\. A similar pattern persists at the granular level:*Form Key*and*Form Value*together account for roughly half of all entities, while table\-cell classes comprise most of the remainder\. Rare classes such as*Table Section Title*and*Others*, by contrast, occur only a handful of times\.

![Refer to caption](https://arxiv.org/html/2607.05614v1/x1.png)\(a\)Granular entity set
![Refer to caption](https://arxiv.org/html/2607.05614v1/x2.png)\(b\)Coarse entity set

Figure 2:DLA entity distributions over the annotated forms, shown for the \(a\) granular \(26\-label\) and \(b\) coarse \(5\-label\) entity sets\. Both follow a long\-tailed pattern dominated by form key/value fields\. Exact counts are given in[Tab\.˜7](https://arxiv.org/html/2607.05614#Pt0.A1.T7)\.Table 7:DLA entity distribution over the annotated forms\. Granular labels are grouped under their coarse category;boldrows give the coarse subtotals\.EntityCount%EntityCount%Fields10,79064\.6Table4,56327\.3Form Key4,38926\.3Table Column Primary Key1,3858\.3Form Value4,02624\.1Table Column Value1,3087\.8Inline Value5733\.4Table Row Value1,0736\.4Signature Value4963\.0Table Row Primary Key3932\.4Signature Key4902\.9Table1901\.1Checkbox3572\.1Table Caption1220\.7Inline Key2701\.6Table Index840\.5Tick Mark1520\.9Table Section Title80\.05Photo Field370\.2Headings6263\.8Others6533\.9Title of Form2351\.4Text Block5483\.3Header2031\.2Gibberish, Mark for removal600\.4Section Title1601\.0Page Number390\.2Footer280\.2Others60\.04Image610\.4Figure/Diagram/Logo610\.4Total: 16,693 entities \(100\.0%\)

## Appendix 0\.BForm Difficulty Examples

[Fig\.˜3](https://arxiv.org/html/2607.05614#Pt0.A2.F3)shows one representative form per difficulty tier\. The Easy example contains only key–value fields and headings, with no tables or checkboxes\. The Medium example contains a columns\-only table alongside key–value fields\. The Hard example contains a full row–column table with multiple pre\-defined rows, characteristic of the densest forms in the dataset\.

![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/form_examples/easy_1.jpg)\(a\)Easy
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/form_examples/medium_1.jpg)\(b\)Medium
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/form_examples/hard_1.jpg)\(c\)Hard

Figure 3:Representative forms for each difficulty tier\. \(a\) Easy forms contain only key–value fields and headings; \(b\) Medium forms add a columns\-only table \(here, the list of heirs\); \(c\) Hard forms contain a full row–column table with pre\-defined rows\.
## Appendix 0\.CQualitative Examples

We manually inspect predictions from all evaluated models to identify common failure modes\. This section summarizes recurring patterns and presents representative examples\.

### 0\.C\.1Document Layout Analysis \(DLA\)

[Fig\.˜4](https://arxiv.org/html/2607.05614#Pt0.A3.F4)illustrates representative success and failure cases for MLLM predictions on*granular*form entity categories\.

Spatial misalignment:A common failure occurs when models correctly classify entities but localize them inaccurately\. As shown in[Fig\.˜4\(a\)](https://arxiv.org/html/2607.05614#Pt0.A3.F4.sf1), predicted bounding boxes are systematically shifted toward the upper portion of the form despite correct category assignments\.

Entity miscategorization:Another frequent failure is semantic rather than spatial\. In[Fig\.˜4\(b\)](https://arxiv.org/html/2607.05614#Pt0.A3.F4.sf2), bounding box localization is largely correct, but key–value entities are assigned incorrect labels, suggesting that spatial localization and semantic classification are only weakly coupled\.

Hallucination:The most severe failure mode is shown in[Fig\.˜4\(c\)](https://arxiv.org/html/2607.05614#Pt0.A3.F4.sf3), where the model confidently predicts entities in visually empty regions\. In particular, it hallucinates inline keys in the blank top\-left and top\-right areas, a tendency that becomes more pronounced in denser forms\.

Successful prediction:In contrast,[Fig\.˜4\(d\)](https://arxiv.org/html/2607.05614#Pt0.A3.F4.sf4)shows a successful prediction\. The sparse layout reduces ambiguity, allowing accurate localization and categorization across the form\.

[Fig\.˜5](https://arxiv.org/html/2607.05614#Pt0.A3.F5)presents analogous examples for the*coarse*entity taxonomy\. Although reducing the label space improves overall localization, the same failure modes remain persist in more subtle forms\.

Positional bias:As shown in[Fig\.˜5\(a\)](https://arxiv.org/html/2607.05614#Pt0.A3.F5.sf1), predictions are often concentrated in the upper portion of the form, with much of the lower half omitted\. This indicates a positional bias, potentially arising from how MLLMs process long visual documents\.

Failure of hierarchical decomposition:[Fig\.˜5\(b\)](https://arxiv.org/html/2607.05614#Pt0.A3.F5.sf2)illustrates a complementary limitation: even when a large enclosing region is detected, nested sub\-fields are frequently missed entirely\. This suggests that current MLLMs reason at a coarse spatial granularity and struggle with hierarchical region decomposition\.

Successful predictions:[Fig\.˜5\(c\)](https://arxiv.org/html/2607.05614#Pt0.A3.F5.sf3)and[Fig\.˜5\(d\)](https://arxiv.org/html/2607.05614#Pt0.A3.F5.sf4)show that structured, low\-density forms and visually distinctive layouts are more tractable\.GPT\-5\.2performs well on the simpler form in[Fig\.˜5\(c\)](https://arxiv.org/html/2607.05614#Pt0.A3.F5.sf3), whileGemini 3 Proaccurately localizes entities even in the more complex example shown in[Fig\.˜5\(d\)](https://arxiv.org/html/2607.05614#Pt0.A3.F5.sf4)\.

![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/granular_examples/granular_shifted_form_gpt_5.2_2082.png)\(a\)Shifted Predictions \(GPT\-5\.2\)
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/granular_examples/granular_double_prediction_gemini_form_2180.png)\(b\)Miscategorized Form Entities \(Gemini 3 Pro\)
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/granular_examples/granular_hallucination_claude_form_1731.png)\(c\)Hallucination \(Claude Opus 4\.6\)
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/granular_examples/granular_relatively_better_performance_gemini_form_1925.png)\(d\)A Well Predicted Form \(Gemini 3 Pro\)

Figure 4:Failure and success modes of MLLM predictions for granular form entities\.![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/coarse_examples/coarse_missing_fields_claude_form_1923.png)\(a\)Partially Correct Predictions \(Claude Opus 4\.6\)
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/coarse_examples/coarse_granular_field_missing_form_1536.png)\(b\)Sub\-Field Prediction Failure \(Claude Opus 4\.6\)
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/coarse_examples/form_2170.png)\(c\)Prediction on a Easy Form \(GPT\-5\.2\)
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/coarse_examples/coarse_good_example_gemini_form_1881.png)\(d\)Prediction on a Complex Form \(Gemini 3 Pro\)

Figure 5:Failure and success modes of MLLM predictions for coarse form entities\.
### 0\.C\.2Key Information Extraction \(KIE\)

[Fig\.˜6](https://arxiv.org/html/2607.05614#Pt0.A3.F6)presents representative success and failure cases for key information extraction on Bangla and English forms\.

Date extraction errors:A recurring failure occurs when extracting dates from Bangla forms\. As shown in[Fig\.˜6\(a\)](https://arxiv.org/html/2607.05614#Pt0.A3.F6.sf1), all evaluated MLLMs misread the date\-of\-birth field\. The ground\-truth value is15\-06\-2008, whereasGemini 3 ProandClaudepredict16\-06\-2008, andGPT\-5\.2predicts18\-06\-2008\. Although structurally plausible, these predictions differ by one or more digits, suggesting that precise character\-level recognition remains challenging when extracting Bangla date expressions\.

Numerical value hallucination:Another failure mode arises when extracting numerical values\. In[Fig\.˜6\(c\)](https://arxiv.org/html/2607.05614#Pt0.A3.F6.sf3), the ground\-truth value is25000, whereasGemini 3 Pro,GPT\-5\.2, andClaudepredict28000,26000, and24000, respectively\. None of these values appears anywhere in the document, indicating that the models hallucinate plausible numbers rather than grounding their predictions in the visual evidence\.

Successful extraction despite layout irregularities:[Fig\.˜6\(b\)](https://arxiv.org/html/2607.05614#Pt0.A3.F6.sf2)shows a representative success case\. TheApplication for the periodfield is visually irregular: the complete date range is written in the region corresponding to theFromfield, while theTofield is empty\. Despite this atypical layout, all evaluated MLLMs correctly extract the intended date range, demonstrating that they can exploit contextual cues even when the spatial organization deviates from the expected form structure\.

Minor spelling variations in Bangla text extraction:Another observed issue involves subtle character\-level spelling errors in Bangla text fields\. As shown in[Fig\.˜6\(d\)](https://arxiv.org/html/2607.05614#Pt0.A3.F6.sf4),Gemini 3 Proreplaces one Bangla consonant with another that has a similar pronunciation\. Although the two characters are phonetically similar and often correspond to a similar*sh*sound in English transliteration, the substitution results in an incorrect spelling of the extracted text\. This suggests that MLLMs may rely on approximate phonetic representations when processing Bangla script, leading to minor but semantically meaningful transcription errors in extracted values\.

![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/kie_examples/20_0_dob.png)\(a\)Wrong predictions of date for Bangla forms
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/kie_examples/493_0_application_for_the_period.png)\(b\)Correct predictions of date for English forms, even in inconsistent structure
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/kie_examples/107_0_bank_bal.png)\(c\)Numerical value hallucination in Bangla forms
![Refer to caption](https://arxiv.org/html/2607.05614v1/res/img/appendix/kie_examples/260_0_father_name.png)\(d\)Minor spelling variations in Bangla text extraction

Figure 6:Failure and success modes of MLLM predictions for KIE Task\.

## Appendix 0\.DPrompts

pk=stask⊕sinst⊕skp\_\{k\}=s\_\{\\text\{task\}\}\\;\\oplus\\;s\_\{\\text\{inst\}\}\\;\\oplus\\;s\_\{k\}\(3\)
where

- •stasks\_\{\\text\{task\}\}denotes the input, task, and output specification,
- •sinsts\_\{\\text\{inst\}\}denotes the form\-entity\-specific instructions,
- •sks\_\{k\}denotes the prompt\-specific instructions for prompt variantkk\.

The operator⊕\\oplusdenotes string concatenation\. In the zero\-shot setting, the prompt\-specific component \(sk=zss\_\{k=\\mathrm\{zs\}\}\) is empty\. In the chain\-of\-thought \(CoT\) setting,sk=cots\_\{k=\\mathrm\{cot\}\}consists of the additional instruction:

> Before producing the final output, internally reason step by step about the layout structure and label assignments, but do not include any intermediate reasoning in the output\.

### 0\.D\.1Prompt Template Details

In the prompt templates below, boldfaced placeholders \(e\.g\.,\{form\_field\_categories\}\) denote variables, while all other braces are literal JSON syntax\.

`Core DLA Prompt \(stasks\_\{\\mathrm\{task\}\}\) Coarse Entity Rules \(sinsts\_\{\\mathrm\{inst\}\}, 5 classes\) Granular Entity Rules \(sinsts\_\{\\mathrm\{inst\}\}, 26 classes\) Prompt\-Specific Instruction \(sk=cots\_\{k=\\mathrm\{cot\}\}\) Zero\-Shot KIE Prompt`