BengaliMCQ: Automatic Generation and Answer Prediction of Academic Multiple-Choice Questions in a Low-Resource Language

# BengaliMCQ: Automatic Generation and Answer Prediction of Academic Multiple-Choice Questions in a Low-Resource Language
Source: [https://arxiv.org/html/2608.15547](https://arxiv.org/html/2608.15547)
\[orcid=https://orcid\.org/0009\-0002\-1099\-7337\]

\[orcid=https://orcid\.org/0009\-0002\-9939\-6553\]

\[orcid=https://orcid\.org/0009\-0003\-1966\-7729\]

\[orcid=https://orcid\.org/0000\-0001\-6797\-7826\]

A\.K\.M\. Nihalul Kabirakm\.nihalul\.kabir@g\.bracu\.ac\.bdSm Azmain Faysalsm\.azmain\.faysal@g\.bracu\.ac\.bdAriana Haque Amiariana\.haque\.ami@g\.bracu\.ac\.bdLawrence Amlan Gomeslawrence\.amlan\.gomes@g\.bracu\.ac\.bdFarig Sadequefarig\.sadeque@bracu\.ac\.bdorganization=Department of Computer Science and Engineering, BRAC University, addressline=Kha 224 Pragati Sarani, Merul Badda, city=Dhaka, postcode=1212, country=Bangladesh

###### Abstract

Traditional retrieval\-augmented generation \(RAG\) frameworks process documents without attending to their hierarchical structure, leading to poor performance, especially in low\-resource languages such as Bengali\. To address this, we propose a structure\-aware RAG framework that models Bengali textbooks as hierarchical graphs and uses a contrastively trained graph neural network to retrieve a small set of relevant passages\. These passages provide focused context for a large language model, enabling topic\-specific multiple\-choice question \(MCQ\) generation and in\-domain answer prediction\. Experimental results demonstrate that our framework outperforms strong dense retrieval baselines across retrieval metrics, produces more relevant MCQs, and achieves superior answer prediction accuracy\.

###### keywords

MCQ generation ,Graph\-guided RAG ,Answer prediction ,Hierarchical graph

††credit:Conceptualization, Methodology, Writing – original draft††credit:Methodology, Writing – original draft, Writing – review and editing††credit:Methodology, Formal analysis, Visualization††credit:Data curation, Formal analysis††credit:Data curation, Investigation††credit:Conceptualization, Supervision††corresponding:Corresponding author## 1Introduction

Automatic MCQ generation has become a popular downstream application of large language models \(LLMs\), especially with recent developments in their capabilities\. However, their effectiveness depends heavily on the relevance of the input context provided to them\. For topic\-specific question generation, it is impractical to feed entire textbook chapters as input to LLMs due to their context\-window limits\. Even when an entire chapter is provided, the language models may not consistently attend to the most relevant passages\. Retrieval‑augmented generation \(RAG\) addresses this by providing the LLMs with a comparatively smaller, targeted set of context for generation\. Standard dense retrieval often ignores the hierarchical structure of documents, resulting in a negative impact on performance, especially if the text is in a low\-resource language like Bengali\.

Graph representations can address this limitation by capturing various relationships within documents\. Most existing work, however, focuses on high\-resource languages\. In this paper, we introduce BengaliMCQ, a document\-graph\-guided automated system that generates highly topic\-relevant academic MCQs in Bengali and predicts answers to in\-domain MCQs with high accuracy\. To summarize, our contributions are:

- •We implement a GNN\-based retrieval framework that learns to rank query\-relevant passages by modeling a textbook’s hierarchical structure, allowing us to reduce context length without sacrificing coverage\.
- •We evaluate our approach through automatic metrics as well as expert validation, showing that our framework outperforms baselines in retrieval quality, question quality, and answer accuracy\.
- •We provide an ablation study for analyzing the contribution of our design choices and demonstrating their value\.

## 2Related work

Though early work on automatic MCQ generation depended on probabilistic and rule\-based approaches like TF\-IDF, n\-grams, and part\-of\-speech tagging\[[1](https://arxiv.org/html/2608.15547#bib.bib1)\], this field soon progressed to neural\-network and transformer\-based frameworks\. For instance, fine\-tuning T5\- and BERT\-based models\[[2](https://arxiv.org/html/2608.15547#bib.bib2),[3](https://arxiv.org/html/2608.15547#bib.bib3)\]has become the de facto standard for downstream NLP tasks like question generation, named\-entity recognition, and question answering\. A notable hybrid method incorporating transformers with a rule\-based approach was proposed by[Mehta et al\. 2021](https://arxiv.org/html/2608.15547#bib.bib4), which utilized the BERTSUM model to summarize text and the Rapid Automatic Keyword Extraction algorithm to extract keywords for creating fill\-in\-the\-blank MCQs\. Distractors \(incorrect options\) were generated using WordNet based on hypernym and hyponym relationships\. However, distractors of higher quality can be generated by LLMs through structured multi\-stage prompting, as shown by[Maity et al\. 2024](https://arxiv.org/html/2608.15547#bib.bib5)\. Currently, this field is being dominated by LLMs\. Recent works have tried to leverage LLMs for automatic MCQ generation and evaluation by utilizing different prompt engineering techniques combined with human\-guided reviews\[[6](https://arxiv.org/html/2608.15547#bib.bib6)\]\. However, these systems typically suffer from performance degradation on long\-context documents\[[7](https://arxiv.org/html/2608.15547#bib.bib7)\]\. To the best of our knowledge, no prior work exploits the document structure of textbooks for generating topic\-relevant MCQs\.

Recent research has begun to fuse graphs with RAG for mapping and modeling these structural relationships\.[Iyer et al\. 2023](https://arxiv.org/html/2608.15547#bib.bib8)trained a GNN on a temporary graph \(during offline learning\) created by modeling relationships between question–answer sentence pairs for answer sentence selection \(AS2\) tasks\. Another recent GNN\-based framework closely related to ours is AutoRev\[[9](https://arxiv.org/html/2608.15547#bib.bib9)\], which models the structure of research papers as hierarchical documents\. The authors used a GNN to retrieve the most salient passages to automatically generate academic peer reviews\. Inspired by these approaches, our work employs GNN\-based contextual learning to retrieve the most topic\-relevant passages for MCQ generation and answer prediction\.

## 3Methodology

### 3\.1Document processing

As target books, we chose four secondary\-school\-level Bengali textbooks: ”Bangla Shahitto” and ”Shohopath” from the literature domain, ”Biology” from the STEM domain, and ”Bangladesh and Global Studies” \(BGS\) from the social science domain\. UsingTesseract\-4optical character recognition\[[10](https://arxiv.org/html/2608.15547#bib.bib10)\]and Google Lens, we extracted and parsed the text to detect chapters, headings, subheadings, passages, and sentences for the node creation in Section[3\.3](https://arxiv.org/html/2608.15547#S3.SS3)\. The original passage structure of the books was preserved by splitting on newline characters to maintain the inherent cohesion of the academic content\.

### 3\.2Query\-Passage dataset creation

As our goal was to train a GNN for ranking relevant passages, we required a labeled dataset where queries \(comprising topic sentences and short questions\) would be mapped to their relevant passages\. However, due to the lack of such a dataset tailored to our books, we created a synthetic one utilizing state\-of\-the\-art LLMs such asGemini 3\.1 ProandGPT\-4\. We prompted one model to generate and another to curate diverse queries from individual passages or small groups of passages\. Our final dataset contains around 22,000 query\-passage pairs with roughly 5,000\-6,000 from each book\. For the training\-validation split, we use an 80:20 ratio\.

![Refer to caption](https://arxiv.org/html/2608.15547v1/figs/Graph.png)Figure 1:A simple schematic illustration of our textbook’s hierarchical graph\. Seq = Sequential edge, Hier = Hierarchical edge
### 3\.3Graph construction

We borrow and enrich the graph representation suggested by[Chitale et al\. 2025](https://arxiv.org/html/2608.15547#bib.bib9)by adding weighted semantic edges for better contextualization\. Let each book be represented as a hierarchical directed multigraph \(𝒢=\(𝒱,ℰ\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\)\), where nodes𝒱\\mathcal\{V\}correspond to the parsed chapters, headings, subheadings, passages, and sentences\. Subsequently, we add bidirectional hierarchical \(ℰhier\\mathcal\{E\_\{\\text\{hier\}\}\}\) and sequential edges \(ℰseq\\mathcal\{E\_\{\\text\{seq\}\}\}\) between the nodes to allow message\-passing both throughout the document hierarchy and along the sequential flow of the book\.

The representation is further enhanced by adding weighted semantic edges between sentence nodes to capture long\-range relationships between related content\. For each sentencess, we add bidirectional semantic edges \(ℰsem\\mathcal\{E\_\{\\text\{sem\}\}\}\) to its top\-KsK\_\{s\}\(Ks∈\{3,5\}K\_\{s\}\\in\\\{3,5\\\}\) most similar neighbors whose cosine similarity exceeds a thresholdTT\(T∈\{0\.5,0\.6,0\.7\}T\\in\\\{0\.5,0\.6,0\.7\\\}\)\. The similarity scores are set as the edge weights\. For obtaining the embeddings, we use a multilingual bi\-encoder, namelyBGE\-M3\[[11](https://arxiv.org/html/2608.15547#bib.bib11)\], fine\-tuned on our query\-passage dataset\. A simple illustration of our document graph for a book is shown in Figure[1](https://arxiv.org/html/2608.15547#S3.F1)\.

### 3\.4GNN architecture and training

For cohesively attending to several types of edges in our graphs, we perform supervised training of a graph attention network \(GAT\) using the dataset created in Section[3\.2](https://arxiv.org/html/2608.15547#S3.SS2)\. Node features are initialized with embeddings \(d​i​m=1024dim=1024\) generated by our fine\-tuned bi\-encoder\. For the passage nodes, we also evaluated initialization with zero vectors and with the mean\-pooled embedding of child nodes\.

#### 3\.4\.1Weighted graph attention layer

Our graphs are heterogeneous, with a combination of weighted and unweighted edges; we therefore design a custom multi\-head attention layer\. We integrate the semantic edge\-weightwi​jw\_\{ij\}with the attention score through a learnable scalarλ\\lambdaand add an edge\-type\-specific learnable bias parameterbt​y​p​eb^\{type\}\. Thus, for an edge from nodeiito nodejjthe base unnormalized attention score becomes:

ei​j=\[xi′∥xj′\]⋅a\+λ⋅wi​j\+bt​y​p​e,e\_\{ij\}=\\bigl\[x^\{\\prime\}\_\{i\}\\parallel x^\{\\prime\}\_\{j\}\\bigr\]\\cdot a\+\\lambda\\cdot w\_\{ij\}\+b^\{type\},\(1\)
wherea∈ℝℋ×𝒟a\\in\\mathbb\{R\}^\{\\mathcal\{\{H\\times D\}\}\}\(ℋ=\\mathcal\{H\}=Attention heads,D=D=Output dimension per head\) is a learnable attention vector;xi′\{\{x\}\_\{i\}^\{\\prime\}\}andxj′\{\{x\}\_\{j\}^\{\\prime\}\}\(xi′,xj′∈ℝℋ×𝒟\{\{x\}\_\{i\}^\{\\prime\}\},\{\{x\}\_\{j\}^\{\\prime\}\}\\in\\mathbb\{R\}^\{\\mathcal\{\{H\\times D\}\}\}\) denote the linear projections of the initial node embeddingsxi\{\{x\}\_\{i\}\}andxj\{\{x\}\_\{j\}\}\(xi,xj∈ℝd​i​m\{\{x\}\_\{i\}\},\{\{x\}\_\{j\}\}\\in\\mathbb\{R\}^\{dim\}\)\. The score in Eqn\.[1](https://arxiv.org/html/2608.15547#S3.E1)is then fed through a standard LeakyReLU and softmax normalization for computing the final attention coefficientαi​j\\alpha\_\{ij\}\.

Aggregation is performed by summation, weighted by the attention coefficients\. Our pipeline contains three such attention layers stacked sequentially\. Their outputs are concatenated with the original node embedding and then projected back down to the initial 1024 dimensions via a lightweight fusion multi\-layer perceptron \(MLP\) to obtain the learned node embeddinghi\\mathit\{h\}\_\{i\}\. This residual design mitigates the typical oversmoothing problem of GNNs\[[12](https://arxiv.org/html/2608.15547#bib.bib12)\]and preserves the initial signal\.

#### 3\.4\.2Training objective

We use a contrastive learning objective, specifically the multi\-positive InfoNCE loss to train the GAT as a passage ranking model\. The loss function we use is:

ℒ=−log⁡\(∑i∈P\+exp⁡\(τ⋅cossim⁡\(q,hip​a​s​s​a​g​e\)\)∑x=1Nbexp⁡\(τ⋅cossim⁡\(q,hxp​a​s​s​a​g​e\)\)\)\\mathcal\{L\}=\-\\log\\left\(\\frac\{\\displaystyle\\sum\_\{i\\in P^\{\+\}\}\\exp\\left\(\\tau\\cdot\\operatorname\{cossim\}\(q,h^\{passage\}\_\{i\}\)\\right\)\}\{\\displaystyle\\sum\_\{x=1\}^\{N\_\{b\}\}\\exp\\left\(\\tau\\cdot\\operatorname\{cossim\}\(q,h^\{passage\}\_\{x\}\)\\right\)\}\\right\)\(2\)
whereqqdenotes the query embedding,P\+P^\{\+\}denotes its ground\-truth positive passages,τ\>0\\tau\>0is a learnable temperature parameter,hip​a​s​s​a​g​e\{h\}^\{passage\}\_\{i\}is the GAT\-learned embedding of passage nodeii, andNbN\_\{\{b\}\}is the total number of passages in the book\. The loss in Eqn\.[2](https://arxiv.org/html/2608.15547#S3.E2)trains the model to assign high scores to all positive \(relevant\) passages while suppressing the scores of all other passages of that book\. The GAT is trained using various configurations based on the values ofKsK\_\{s\}andTT, as described in the semantic edge construction process in Section[3\.3](https://arxiv.org/html/2608.15547#S3.SS3)\. These trained GATs are denoted withG​A​T​\(Ks,T\)\{GAT\}\(K\_\{s\},T\)\.

![Refer to caption](https://arxiv.org/html/2608.15547v1/figs/Framework.png)Figure 2:End\-to\-end BengaliMCQ framework![Refer to caption](https://arxiv.org/html/2608.15547v1/figs/Final_Empirical_Examples.png)Figure 3:Excerpt of two test samples where passages highlighted in yellow color refer to query\-relevant passages identified GNN but missed by DPR

### 3\.5Passage retrieval and grounding

During inference, a user inputs a topic for MCQ generation or a question for answer prediction\. The system then uses the trained GAT’s ranking to retrieve the top\-KcK\_\{c\}\(Kc=10K\_\{c\}=10\) candidate passages that are relevant to the input topic or question\. We perform a further re\-ranking on these with a cross\-encoder, namelyBGE\-Reranker\-V2\-M3\[[11](https://arxiv.org/html/2608.15547#bib.bib11)\], for improved performance and smaller context\. Finally, the top\-KfK\_\{f\}\(withKf=5K\_\{f\}=5\) most relevant passages after the two\-stage ranking are taken as the grounded context\. Augmented with a structured prompt, these passages are sent to the LLM as input for topic\-focused question generation or answer prediction of MCQs\. Selecting the small set of pertinent passages for a given topic or MCQ helps reduce the context window and computational cost\. An illustration of each step of the entire framework is shown in Figure[2](https://arxiv.org/html/2608.15547#S3.F2)\.

## 4Results and analysis

This section presents a comprehensive evaluation of our framework along three aspects: retrieval performance, downstream task performance, and ablation studies of our design choices\.

Table 1:Retrieval performance comparison across different methods\. Bold highlights the best scores, underlined values indicate second\-best\.BaselineHR@5Recall@5nDCG@5MRR@5HR@10Recall@10nDCG@10MRR@10BM250\.5390\.5060\.5060\.5060\.6000\.6000\.4530\.434Translation Baseline \(DPR\)0\.7530\.7100\.6180\.6080\.8210\.7820\.6420\.617Sparse Retrieval \(BGE\-M3\)0\.7950\.7600\.6760\.6660\.8520\.8200\.6960\.674DPR \(Harrier\-OSS\-v1\)0\.5190\.4870\.4180\.4110\.5750\.5450\.4380\.419DPR \(Harrier\-OSSFine\-tuned\)0\.8360\.8020\.7030\.6890\.9010\.8730\.7280\.698DPR \(BGE\-M3Fine\-tuned\)0\.8420\.8030\.7070\.6960\.8990\.8660\.7290\.703GAT\(Ks=5K\_\{s\}=5,T=0\.5T=0\.5\)0\.8820\.8510\.7400\.7200\.9340\.9110\.7610\.726BGE\-M3\+RERANKER0\.9030\.8710\.8150\.8140\.9290\.9010\.8260\.818GAT\+RERANKER0\.9220\.8860\.8280\.8170\.9540\.9220\.8420\.831### 4\.1Retrieval performance

We evaluate the retrieval quality of the proposed framework by comparing against relevant baselines on the validation split \(approximately 4400 samples\) of the query\-passage dataset\. The baselines include: BM25, lexical\-weight\-based sparse retrieval, several dense passage retrieval \(DPR\) pipelines, DPR \+ cross\-encoder re\-ranking hybrids, and a translation\-based DPR pipeline\. Following prior retrieval work, we report normalized discounted cumulative gain \(nDCG@k\), mean reciprocal rank \(MRR\), recall@k and hit rate \(HR@k\) per method\. Higher values indicate better retrieval performance\.

As shown in Table[1](https://arxiv.org/html/2608.15547#S4.T1), our graph attention network consistently outperforms all baselines that do not use a re\-ranker across the retrieval metrics\. Notably, when comparing re\-ranker integrated baselines, the nDCG@5 obtained with only 5 passages by our ”GAT \+ BGE\-Reranker” pipeline surpasses the nDCG@10 obtained with 10 passages by a fine\-tuned ”BGE\-M3 \+ BGE\-Reranker” baseline, demonstrating that our framework achieves better performance with less context\. Figure[3](https://arxiv.org/html/2608.15547#S3.F3)shows empirical examples where GAT was able to retrieve passages that other baselines missed\.

### 4\.2End\-to\-end evaluation

Table 2:Comparison of MCQ generation and answer prediction performance across different baselines\. All metrics are reported as averages across all books for each baseline\.BaselineIWFQFTRDSPPLDivAnsw\.Accuracy \(%\)Aver\. ContextLength \(Tokens\)Zero\-shot0\.3914\.530\.460\.717\.600\.4457\.2143,591Few\-shot0\.3114\.400\.460\.777\.830\.4568\.3543,591CoT0\.2214\.500\.480\.787\.980\.5871\.8743,591DPR \+ Reranker0\.2713\.960\.590\.757\.880\.5282\.802,825GAT \+ Reranker0\.2414\.810\.670\.787\.570\.5291\.411,651To assess downstream performance, we have evaluated our framework against long\-context zero\-shot, few\-shot, and chain\-of\-thought \(CoT\) prompting baselines, and the hybrid retrieval strategy ofDPR \+ Re\-ranker\. For each method, the same LLM \(Gemini 3\.1\) is used for question generation and answer prediction\.

#### 4\.2\.1MCQ generation evaluation protocol

For MCQ evaluation, 10 topics from each book are selected and curated by our domain experts to ensure that our books contain sufficient material on these topics\. For every topic, each baseline generates 20 multiple\-choice questions, resulting in a total of 800 MCQs per method\. The generated MCQs are then evaluated using two complementary approaches\. First, we have used the ”LLM as a judge” technique by prompting GPT\-5\.2 with a structured Item\-Writing Flaws \(IWF\) rubric\[[13](https://arxiv.org/html/2608.15547#bib.bib13)\]and the QUEST framework \(QF\) rubric\[[14](https://arxiv.org/html/2608.15547#bib.bib14)\]\. The IWF rubric considers the pedagogical value of a question and its options through various criteria\. The QF rubric, on the other hand, evaluates MCQs across five dimensions: Quality, Uniqueness, Effort, Structure, and Transparency\. Second, we report automatic metrics that quantify different aspects of question quality:

##### Distractor similarity \(DS\)

To evaluate the plausibility effect of the distractors, we have used the mean\-pooled embeddings of a fine\-tuned BanglaBERT\[[15](https://arxiv.org/html/2608.15547#bib.bib15)\]model\. For each MCQ, the cosine similarities between the embedding of the correct answer and that of the distractors are calculated\. The model was fine\-tuned on our books using masked language modeling \(MLM\)\.

##### Topic relevance \(TR\)

To measure how relevant our LLM\-generated MCQs are to the topic sentences, we also compute the cosine similarity between the sentence embedding of each question and its corresponding topic sentence using a second multilingual sentence encoder,Harrier\-OSS\-v1\-0\.6b\.

##### Perplexity of questions \(PPL\)

To quantify the sentence formation quality, we report the mean perplexity of the question tokens under a lightweight fine\-tuned generative model for Bengali, shahidul034/BanglaGPT\. Similar to BanglaBERT, this was also fine\-tuned on our textbooks but using causal language modeling \(CLM\)\.

##### Question diversity \(Div\)

Question type diversity was evaluated using Shannon entropy, following[Raina and Gales 2022](https://arxiv.org/html/2608.15547#bib.bib16)\. For finding the question categories, a generative LLM was few\-shot prompted to classify each MCQ into one of the 10 question types: What, Who, When, Where, Why, How, Which, Yes/No, Whose, How much/many\.

Higher values of QF, DS, TR, and Div indicate better performance, whereas lower values of IWF and PPL are preferred\. Table[2](https://arxiv.org/html/2608.15547#S4.T2)summarizes the quantitative results\.

#### 4\.2\.2Answer prediction evaluation

Unlike MCQ generation, answer prediction performance can be measured with a simple metric like accuracy\. The assessment was done on a held\-out dataset of 1,000 gold\-standard MCQs collected from past secondary school public exam archives\. We manually curated this dataset to ensure that the questions correspond to our target textbooks\. The results in Table[2](https://arxiv.org/html/2608.15547#S4.T2)indicate that our system achieves an average accuracy of 91\.41%, which is the highest out of all baselines\. These findings indicate that our model generalizes across the academic text types in our corpus\.

### 4\.3Ablation study

We conduct ablation experiments to investigate the retrieval impact of key design choices in the proposed framework\. All experiments are performed on the same validation split, withKsK\_\{s\}= 5 andTT= 0\.7 unless otherwise stated\.

Table 3:Ablation study of the proposed GAT\-based retrieval framework\. The “All Edges”, “Zero” and “Using Temp” rows report the same reference configuration and are therefore identical\.DimensionBaselineHR@5Recall@5nDCG@5MRR@5HR@10Recall@10nDCG@10MRR@10EdgeTypeNo Hier0\.0130\.0120\.0070\.0060\.0160\.0140\.0080\.007No Seq0\.8650\.8240\.7240\.7120\.9200\.8870\.7450\.719No Sem0\.8650\.8260\.7270\.7160\.9200\.8860\.7480\.723All Edges0\.8680\.8290\.7280\.7160\.9200\.8890\.7490\.723PassageInitializationMoS0\.8570\.8180\.7110\.6970\.9150\.8820\.7340\.704Zero0\.8680\.8290\.7280\.7160\.9200\.8890\.7490\.723Bi\-encoder0\.8790\.8390\.7330\.7190\.9290\.8970\.7530\.725TemperatureParameterNo Temp0\.8190\.7780\.7190\.6960\.8460\.8060\.7290\.703Using Temp0\.8680\.8290\.7280\.7160\.9200\.8890\.7490\.723GAT\(KsK\_\{s\},TT\)SensitivityGAT\(3, 0\.5\)0\.8780\.8390\.7350\.7210\.9320\.9010\.7560\.728GAT\(3, 0\.6\)0\.8780\.8390\.7370\.7240\.9300\.8990\.7580\.731GAT\(3, 0\.7\)0\.8760\.8370\.7330\.7190\.9300\.8980\.7540\.727GAT\(5, 0\.5\)0\.8820\.8510\.7400\.7200\.9340\.9110\.7610\.726GAT\(5, 0\.6\)0\.8800\.8420\.7390\.7240\.9290\.8970\.7580\.731GAT\(5, 0\.7\)0\.8750\.8360\.7320\.7180\.9320\.9000\.7540\.725##### Effect of edge types

We discard each type of edge while keeping the rest of the architecture fixed\. In this case, all passage nodes are initialized with zero vectors \(d​i​m=1024dim=1024\) to make sure that the ranking signal is only provided by the message\-passing on the graph\. As seen in Table[3](https://arxiv.org/html/2608.15547#S4.T3), removing hierarchical edges causes performance to collapse, indicating that hierarchical structure is the most important signal to be learned in the textbook graph\. By contrast, the drop from discarding sequential or semantic edges is measurable but modest relative to the full model\.

##### Effect of passage\-node initialization

We compare the following three initialization methods for passage nodes: \(i\) mean\-pooling of child sentence embeddings, \(ii\) directly encoding with the fine\-tunedBGE\-M3encoder, and \(iii\) initialization with zero vectors\. Interestingly, zero initialization outperforms mean\-pooling, although direct encoding withBGE\-M3achieves the best overall performance\.

##### Effect of temperature parameterτ\>0\\tau\>0

From our experimental observations, training with a learnable temperature produces better ranking metrics than a fixed temperature of 1\.0 \(equivalent to not using temperature\)\. The learned value typically settles between 15 and 20\.

##### Effect of hard negative mining

We tested offline hard negative mining \(TopK\-MarginPos\)\[[17](https://arxiv.org/html/2608.15547#bib.bib17)\]with the same fine\-tuned BGE\-M3 bi\-encoder\. Our experiments show that hard negative mining substantially degraded validation performance \(Recall@5 below 50%\) and prevented our model from learning effectively\.

##### Impact of varyingKsK\_\{s\}andTTvalues

We test varying top\-KsK\_\{s\}and thresholdTTvalues for semantic edge creation and report the impact on retrieval performance\. From Table[3](https://arxiv.org/html/2608.15547#S4.T3), we can see that although the performance impact varies depending on the metric, \(Ks=5,T=0\.5K\_\{s\}=5,\\ T=0\.5\) yields the best retrieval performance\. Therefore, we adopted these values for the full re\-ranker\-integrated pipeline\.

### 4\.4Expert validation

Table 4:Human evaluation of topic relevance \(1 = not relevant, 2 = somewhat relevant, 3 = relevant, and 4 = highly relevant\)\.DomainNNMean%≥3\\geq 3Literature603\.1776\.7Biology403\.5892\.5Social Science∗403\.1380\.0Overall1403\.2782\.1∗Averaged across both experts \(κ=0\.55\\kappa=0\.55\)\.We were aware of the limitations of automatic metrics and therefore conducted a small\-scale expert validation of our generated MCQs’ topic relevance on a 4\-point scale\. Two domain experts \(literature and STEM\) evaluated 140 randomly sampled MCQs distributed as follows: 60 MCQs from the literature textbooks \(”Shohopath” and ”Bangla Shahitto”\), 40 from biology, and 40 from the social science \(BGS\) textbook\. The results \(Table[4](https://arxiv.org/html/2608.15547#S4.T4)\) demonstrate that 82\.1% of all MCQs are scored as relevant \(score≥3\\geq 3\), with biology MCQs performing the best\. For estimating inter\-annotator agreement, both experts independently rated the 40 Social Science MCQs\. Their evaluations yielded moderate agreement \(Cohen’sκ=0\.55\\kappa=0\.55\), with both annotators independently rating 80\.0% of the cross\-domain questions as relevant \(score≥3\\geq 3\)\.

## 5Conclusion

In this work, we develop BengaliMCQ, an automated system for topic\-relevant MCQ generation and answer prediction for a low\-resource language\. By modeling textbooks as hierarchical document graphs and leveraging a GNN to retrieve topic\-relevant content, BengaliMCQ reduces LLM input length while maintaining high coverage\. Our experiments demonstrate strong empirical performance through automatic metrics and expert validation, reducing the average context length from 43,591 to 1,651 tokens while raising answer prediction accuracy to 91\.41%\. The strategies provided in this paper can help close the gap between structure\-aware retrieval and automated educational technology\.

A central limitation of our work is that our GNN training is supervised by a synthetic dataset, which may introduce noise due to hallucination\[[18](https://arxiv.org/html/2608.15547#bib.bib18)\]\. A completely human\-curated dataset would likely provide a more reliable and robust training signal\. Another limitation lies in our automatic evaluation process, as LLM\- and embedding\-based scores are only approximations\. While useful, our human evaluation was limited to a small stratified subset of topics and items, as well as to only two domain experts\. In our future work, we plan to conduct broader, multi\-annotator studies with a larger sample of books as well as a human\-curated dataset for reliable supervision\.

## Acknowledgements

The authors would like to express their sincere gratitude to Abul Kalam Azad and Azmal Hossain from Ideal School and College, Motijheel, Dhaka, for their valuable time, insights, and contributions during the expert validation of the generated multiple\-choice questions \(MCQs\)\.

## References

- Nwafor and Onyenwe \[2021\]C\.A\. Nwafor, I\.E\. Onyenwe, An automated multiple\-choice question generation using natural language processing techniques, Int\. J\. Nat\. Lang\. Comput\. 10 \(2\) \(2021\) 1–10,[https://doi\.org/10\.5121/ijnlc\.2021\.10201](https://doi.org/10.5121/ijnlc.2021.10201)\.
- Roy and Manik \[2024\]S\.C\. Roy, M\.M\.H\. Manik, Question\-answering system for Bangla: fine\-tuning BERT\-Bangla for a closed domain, arXiv preprint arXiv:2410\.03923 \(2024\),[https://doi\.org/10\.48550/arXiv\.2410\.03923](https://doi.org/10.48550/arXiv.2410.03923)\.
- Patil et al\. \[2022\]S\. Patil, et al\., State\-of\-the\-art approach to e\-learning with cutting edge NLP transformers: implementing text summarization, question and distractor generation, question answering, Int\. J\. Adv\. Comput\. Sci\. Appl\. 13 \(1\) \(2022\) 445–453,[https://doi\.org/10\.14569/IJACSA\.2022\.0130155](https://doi.org/10.14569/IJACSA.2022.0130155)\.
- Mehta et al\. \[2021\]P\.K\. Mehta, et al\., Automated MCQ generator using natural language processing, Int\. Res\. J\. Eng\. Technol\. 8 \(5\) \(2021\) 2705–2710,[https://www\.irjet\.net/archives/V8/i5/IRJET\-V8I5497\.pdf](https://www.irjet.net/archives/V8/i5/IRJET-V8I5497.pdf)\.
- Maity et al\. \[2024\]S\. Maity, et al\., A novel multi\-stage prompting approach for language agnostic MCQ generation using GPT, in: Adv\. Inf\. Retr\., ECIR 2024, Lect\. Notes Comput\. Sci\., vol\. 14610, Springer, Cham, 2024, pp\. 268–277,[https://doi\.org/10\.1007/978\-3\-031\-56063\-7\_18](https://doi.org/10.1007/978-3-031-56063-7_18)\.
- Mucciaccia et al\. \[2025\]S\.S\. Mucciaccia, et al\., Automatic multiple\-choice question generation and evaluation systems based on LLM: a study case with university resolutions, in: Proc\. 31st Int\. Conf\. Comput\. Linguist\. \(COLING 2025\), Assoc\. Comput\. Linguist\., Abu Dhabi, UAE, 2025, pp\. 2246–2260,[https://aclanthology\.org/2025\.coling\-main\.154/](https://aclanthology.org/2025.coling-main.154/)\.
- Hsieh et al\. \[2024\]C\.\-P\. Hsieh, et al\., RULER: what’s the real context size of your long\-context language models?, in: 1st Conf\. Lang\. Model\. \(COLM 2024\), 2024,[https://openreview\.net/forum?id=kIoBbc76Sy](https://openreview.net/forum?id=kIoBbc76Sy)\.
- Iyer et al\. \[2023\]R\. Iyer, et al\., Question\-answer sentence graph for joint modeling answer selection, in: Proc\. 17th Conf\. Eur\. Chapter Assoc\. Comput\. Linguist\. \(EACL 2023\), Assoc\. Comput\. Linguist\., Dubrovnik, Croatia, 2023, pp\. 968–979,[https://doi\.org/10\.18653/v1/2023\.eacl\-main\.68](https://doi.org/10.18653/v1/2023.eacl-main.68)\.
- Chitale et al\. \[2025\]M\.P\. Chitale, et al\., Graph\-guided passage retrieval for author\-centric structured feedback, arXiv preprint arXiv:2505\.14376 \(2025\),[https://doi\.org/10\.48550/arXiv\.2505\.14376](https://doi.org/10.48550/arXiv.2505.14376)\.
- Smith \[2007\]R\.W\. Smith, An overview of the Tesseract OCR engine, in: Proc\. 9th Int\. Conf\. Doc\. Anal\. Recognit\. \(ICDAR 2007\), vol\. 2, IEEE, 2007, pp\. 629–633,[https://doi\.org/10\.1109/ICDAR\.2007\.56](https://doi.org/10.1109/ICDAR.2007.56)\.
- Chen et al\. \[2024\]J\. Chen, et al\., M3\-Embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation, in: Find\. Assoc\. Comput\. Linguist\.: ACL 2024, Assoc\. Comput\. Linguist\., Bangkok, Thailand, 2024, pp\. 2318–2335,[https://doi\.org/10\.18653/v1/2024\.findings\-acl\.137](https://doi.org/10.18653/v1/2024.findings-acl.137)\.
- Rusch et al\. \[2023\]T\.K\. Rusch, et al\., A survey on oversmoothing in graph neural networks, arXiv preprint arXiv:2303\.10993 \(2023\),[https://doi\.org/10\.48550/arXiv\.2303\.10993](https://doi.org/10.48550/arXiv.2303.10993)\.
- Moore et al\. \[2023\]S\. Moore, et al\., Assessing the quality of multiple\-choice questions using GPT\-4 and rule\-based methods, in: O\. Viberg, I\. Jivet, P\.J\. Muñoz\-Merino, M\. Perifanou, T\. Papathoma \(Eds\.\), Responsive and Sustainable Educational Futures, Lect\. Notes Comput\. Sci\., vol\. 14200, Springer Nature Switzerland, Cham, 2023, pp\. 229–245,[https://doi\.org/10\.1007/978\-3\-031\-42682\-7\_16](https://doi.org/10.1007/978-3-031-42682-7_16)\.
- Ebner et al\. \[2025\]M\. Ebner, et al\., Ensuring quality in AI\-generated multiple\-choice questions for higher education with the QUEST framework, in: Ł\. Tomczyk \(Ed\.\), New Media Pedagogy: Research Trends, Methodological Challenges, and Successful Implementations, Commun\. Comput\. Inf\. Sci\., vol\. 2537, Springer Nature Switzerland, Cham, 2025, pp\. 293–303,[https://doi\.org/10\.1007/978\-3\-031\-95627\-0\_20](https://doi.org/10.1007/978-3-031-95627-0_20)\.
- Bhattacharjee et al\. \[2022\]A\. Bhattacharjee, et al\., BanglaBERT: language model pretraining and benchmarks for low\-resource language understanding evaluation in Bangla, in: Find\. Assoc\. Comput\. Linguist\.: NAACL 2022, Assoc\. Comput\. Linguist\., Seattle, United States, 2022, pp\. 1318–1327,[https://doi\.org/10\.18653/v1/2022\.findings\-naacl\.98](https://doi.org/10.18653/v1/2022.findings-naacl.98)\.
- Raina and Gales \[2022\]V\. Raina, M\. Gales, Multiple\-choice question generation: towards an automated assessment framework, arXiv preprint arXiv:2209\.11830 \(2022\),[https://doi\.org/10\.48550/arXiv\.2209\.11830](https://doi.org/10.48550/arXiv.2209.11830)\.
- Moreira et al\. \[2024\]G\. de Souza P\. Moreira, et al\., NV\-Retriever: improving text embedding models with effective hard\-negative mining, arXiv preprint arXiv:2407\.15831 \(2024\),[https://doi\.org/10\.48550/arXiv\.2407\.15831](https://doi.org/10.48550/arXiv.2407.15831)\.
- Tan et al\. \[2024\]Z\. Tan, et al\., Large language models for data annotation and synthesis: a survey, in: Proc\. 2024 Conf\. Empir\. Methods Nat\. Lang\. Process\. \(EMNLP 2024\), Assoc\. Comput\. Linguist\., Miami, Florida, USA, 2024, pp\. 930–957,[https://doi\.org/10\.18653/v1/2024\.emnlp\-main\.54](https://doi.org/10.18653/v1/2024.emnlp-main.54)\.