L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

# A Large-Scale Multilingual Benchmark forEvaluating Factual Knowledge of Large Language Models Across Indic Languages
Source: [https://arxiv.org/html/2608.15535](https://arxiv.org/html/2608.15535)
## L3Cube\-IndicQuest v2: A Large\-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

Rinit JainTirthraj MahajanAffiliation:\[4pt\] Pune Institute of Computer Technology, PuneAffiliation:L3Cube Labs, PuneAdvait JoshiAffiliation:\[4pt\] Pune Institute of Computer Technology, PuneAffiliation:L3Cube Labs, PuneRaviraj JoshiAffiliation:L3Cube Labs, PuneAffiliation:Indian Institute of Technology Madras\[4pt\]\{rinitjain9, tirthraj2004, advaitkjoshi, ravirajoshi\}@gmail\.com

###### Abstract

We present L3Cube\-IndicQuest v2, a large\-scale gold\-standard multilingual question\-answering benchmark for evaluating the India\-specific factual knowledge of Large Language Models \(LLMs\)\. The benchmark comprises 3,471 curriculum\-grounded English question–answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain\-specific reference books\. We introduce a practical hybrid construction strategy that combines context\-grounded LLM\-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality\. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question–answer pairs across 20 languages\. We evaluate six LLMs under three protocols: LLM\-as\-a\-judge and two deterministic lexical criteria, exact\-substring and word\-overlap matching\. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge\. The frontier commercial model leads by a wide margin, and among open\-weight models Gemma4 31B outperforms the Indic\-specialised Sarvam 30B in every evaluated Indic language\.

## 1Introduction

Large Language Models have advanced rapidly, yet their representation of India\-specific knowledge remains weak\. Models that do well on general English benchmarks often answer incorrectly when asked about Indian history, law, regional culture, or curriculum\-level facts[22](https://arxiv.org/html/2608.15535#bib.bib3)\. Indic languages are under\-represented in the pre\-training corpora of most multilingual models[15](https://arxiv.org/html/2608.15535#bib.bib2);[13](https://arxiv.org/html/2608.15535#bib.bib13), and widely used knowledge benchmarks remain English\-centric[12](https://arxiv.org/html/2608.15535#bib.bib12)\.

00101020203030404050506060707080809090100100Gemini 2\.5 FlashGemma4 31BGPT\-5\.4 MiniSarvam 30BGemma2 9BLlama 3\.1 8B86\.986\.962\.262\.260\.160\.148\.748\.741\.141\.135\.435\.4626237\.337\.337\.837\.827\.727\.721\.921\.918\.218\.2Accuracy on English subset \(%\)LLM\-as\-a\-judgeExact substringFigure 1:Accuracy on the English subset under the LLM judge \(Gemma 3 12B\) and under exact\-substring matching\. Gemma4 31B and GPT\-5\.4 Mini swap places between the two protocols\.Short\-form factual question answering is the standard way of measuring such knowledge\. SimpleQA[27](https://arxiv.org/html/2608.15535#bib.bib24)set the format: short questions with exactly one correct answer, which keeps grading tractable\. SimpleQA Verified[11](https://arxiv.org/html/2608.15535#bib.bib25)later showed that such benchmarks still need deduplication and label checking\. Both are English\-only, and neither covers regional knowledge, so neither can measure how well a model knows facts specific to a country or culture\. Benchmarks for Indic languages mostly test reading comprehension or translation rather than open\-domain recall[4](https://arxiv.org/html/2608.15535#bib.bib4);[23](https://arxiv.org/html/2608.15535#bib.bib5);[6](https://arxiv.org/html/2608.15535#bib.bib26)\. L3Cube\-IndicQuest v1[20](https://arxiv.org/html/2608.15535#bib.bib1)was the first to target India\-specific factual knowledge across 20 languages, with 200 manually curated pairs per language over five domains, but its scale and reliance on Wikipedia limited what it could measure\.

Scaling that work by hand was not practical\. IndicQuest v2 therefore splits the task: LLM agents do the volume work and human annotators do the judgement work\. One agent generates candidate pairs from chunked curricular text, a second scores them and cross\-checks the answers, a deduplication engine collapses near\-identical questions, and three annotators review everything that survives\. Figure[1](https://arxiv.org/html/2608.15535#S1.F1)summarises the benchmark and the headline results\. Our contributions are:

- •A hybrid agent–human pipeline for building curriculum\-grounded QA data at scale\. We combine agentic generation and validation, semantic deduplication, and human verification in a scalable framework for producing high\-quality benchmark data\.
- •IndicQuest\-v2111[l3cube\-pune/IndicQuest\-v2](https://huggingface.co/datasets/l3cube-pune/IndicQuest-v2), a publicly released benchmark for India\-specific factual knowledge and hallucination evaluation spanning 20 languages, comprising 3,471 English question–answer pairs across nine domains and 69,420 multilingual pairs in total\.
- •Evidence on the limits of automated validation: human review removed 20–25% of pairs that had already passed both automated gates\.
- •Evaluation of six LLMs spanning commercial, general\-purpose open\-weight, and Indic\-specialized models under three protocols: an LLM judge \(Gemma 3 12B\), exact\-substring matching, and word\-overlap matching\. The first accepts paraphrases; the other two are deterministic and need no judge model\. The ranking holds across all three\.

## 2Related Work

#### IndicQuest v1\.

L3Cube\-IndicQuest v1[20](https://arxiv.org/html/2608.15535#bib.bib1)is the direct predecessor of this work: 4,000 pairs across English and 19 Indic languages, covering Literature, History, Geography, Politics, and Economics\. With Llama\-3\.1\-405B\-Instruct as judge, English outperformed every Indic language, and Manipuri, Odia, and Urdu were weakest\. Table[1](https://arxiv.org/html/2608.15535#S2.T1)summarises how v2 extends it\.

![Refer to caption](https://arxiv.org/html/2608.15535v1/dataset_construction_pipeline.png)

Figure 2:IndicQuest v2 dataset construction pipeline: PDF extraction and chunking, Agent 1 generation, Agent 2 validation, semantic deduplication, manual expert review, and translation into 19 Indic languages\.
#### Multilingual QA Benchmarks\.

TyDi QA[3](https://arxiv.org/html/2608.15535#bib.bib8)covers eleven typologically diverse languages\. XQuAD[2](https://arxiv.org/html/2608.15535#bib.bib9)and MLQA[16](https://arxiv.org/html/2608.15535#bib.bib10)provide cross\-lingual reading comprehension sets derived from SQuAD[18](https://arxiv.org/html/2608.15535#bib.bib27), and MEGA[1](https://arxiv.org/html/2608.15535#bib.bib14)spans 16 datasets and 70 languages\. All supply a context passage or transfer an existing task, whereas IndicQuest targets open\-domain recall with no passage\.

Table 1:IndicQuest v1 versus v2: key attributes\.
#### Short\-Form Factuality Benchmarks\.

SimpleQA Verified[11](https://arxiv.org/html/2608.15535#bib.bib25)refines SimpleQA[27](https://arxiv.org/html/2608.15535#bib.bib24)through deduplication and topic balancing, finding that label noise and redundancy shift measured accuracy\. IndicQuest v2 uses the same short\-answer design for India\-specific knowledge and releases it in 20 languages\.

#### Culture\- and Region\-Specific Evaluation\.

BLEnD[17](https://arxiv.org/html/2608.15535#bib.bib16)and INCLUDE[21](https://arxiv.org/html/2608.15535#bib.bib17)test everyday cultural and regional knowledge\. For India, MILU[25](https://arxiv.org/html/2608.15535#bib.bib18)covers exam\-style understanding, IndicGenBench[24](https://arxiv.org/html/2608.15535#bib.bib19)covers generation, and PARIKSHA[26](https://arxiv.org/html/2608.15535#bib.bib20)studies human–LLM evaluator agreement\. IndicQuest v2 instead grounds open\-domain factual QA in Indian curricular texts\.

#### Indic NLP Datasets and Models\.

IndicQA[4](https://arxiv.org/html/2608.15535#bib.bib4)covers eleven Indic languages using Wikipedia context paragraphs,[23](https://arxiv.org/html/2608.15535#bib.bib5)propose a closed\-form QA benchmark, and IndicSQuAD[6](https://arxiv.org/html/2608.15535#bib.bib26)builds an extractive set for nine Indic languages from SQuAD\. All three evaluate comprehension over a supplied passage, whereas IndicQuest v2 measures closed\-book recall\. IndicNLPSuite[15](https://arxiv.org/html/2608.15535#bib.bib2)provides foundational monolingual corpora, and Airavata[7](https://arxiv.org/html/2608.15535#bib.bib11)showed gains from Hindi instruction\-tuning, which Sarvam’s 30B model extends to broader Indic pre\-training\.

#### LLM\-as\-a\-Judge\.

[28](https://arxiv.org/html/2608.15535#bib.bib6)established the paradigm with MT\-Bench and Chatbot Arena, AlpacaEval[5](https://arxiv.org/html/2608.15535#bib.bib7)added length\-controlled scoring to reduce verbosity bias, and[10](https://arxiv.org/html/2608.15535#bib.bib21)survey its failure modes\. Since judge quirks can carry into reported scores, we report two deterministic criteria alongside the judge \(Section[5\.2](https://arxiv.org/html/2608.15535#S5.SS2)\)\.

## 3Dataset Construction

Table 2:Representative pairs from the IndicQuest v2 English subset, one per domain\. Every gold answer is one to five words and appears verbatim in the source text\.### 3\.1Source Material

IndicQuest v2 draws on formal Indian educational and reference texts, a deliberate shift from the web sources used in v1:

- •School curricula:NCERT \(Classes 6–12\), SSC Maharashtra Board \(Classes 4–10\), CBSE, ICSE, and HSC textbooks in History, Geography, Science, and Social Studies\.
- •Competitive examination material:UPSC and MPSC preparation books covering Indian polity, history, economics, and general knowledge\.
- •Domain references:books on Indian law and the Constitution, and specialised texts on the history of Indian science and sport\.

Indian sports history and Indian contributions to science are largely absent from school curricula, making them the most novel domains in v2\. Construction

### 3\.2Question Design Principles

Three constraints govern every question, and both agents enforce them\.

#### Short answers\.

A short answer is one to five words long and appears verbatim in the source text: a name, a date, a place, or a term, never a sentence or an explanation\. This makes evaluation simpler\. A short phrase either matches the gold reference or it does not, so there is little of the ambiguity that comes with comparing long free\-form answers, and the responses can also be scored by plain string comparison with no judge model \(Section[4\.3](https://arxiv.org/html/2608.15535#S4.SS3)\)\.

#### Non\-ambiguity\.

Each question is written so that exactly one answer is correct\. The two pipeline agents described in Section[3\.3](https://arxiv.org/html/2608.15535#S3.SS3)enforce this from both ends: the generation agent avoids vague or directional phrasings and uses complete proper names, and the validation agent rejects any pair admitting more than one defensible answer\. Ambiguous pairs are discarded rather than repaired\.

#### Self\-containment and India specificity\.

Questions must be answerable without the source passage and must not refer to it, since no context is supplied at evaluation time\. Every question must concern Indian history, geography, polity, law, culture, science, or sport\. Table[2](https://arxiv.org/html/2608.15535#S3.T2)gives one example per domain\.

### 3\.3Construction Pipeline

The principles above are enforced by a six\-stage pipeline \(Figure[2](https://arxiv.org/html/2608.15535#S2.F2)\); prompts are in Appendix[C](https://arxiv.org/html/2608.15535#A3)\.

- •Extraction:source books converted to text, split into passage\-sized chunks\.
- •Agent 1:generates candidate pairs from each chunk\.
- •Agent 2:scores each pair and cross\-checks the answer\.
- •Deduplication:collapses near\-identical questions from overlapping books\.
- •Verification:three annotators review every surviving pair\.
- •Translation:the verified English set goes into 19 Indic languages\.

### 3\.4Agent 1: Question–Answer Generation

Agent 1 runs on Gemini 2\.5 Flash, taking chunked text from the source PDFs and generating factual, self\-contained pairs\. Its prompt \(Appendix[C\.1](https://arxiv.org/html/2608.15535#A3.SS1)\) enforces the constraints of Section[3\.2](https://arxiv.org/html/2608.15535#S3.SS2), plus enough difficulty to separate models\.

### 3\.5Agent 2: Quality Validation

Agent 2, also on Gemini 2\.5 Flash \(Appendix[C\.2](https://arxiv.org/html/2608.15535#A3.SS2)\), scores every candidate on nine dimensions \(0–10 each, 90 total\): Correctness, Relevance, Difficulty, Overall Quality, India Specificity, Context Richness, Answer Quality, Grammar and Structure, and Specialised Knowledge\. It also sets a binarycorrectness\_flagand agoogle\_search\_confidencerating from external sources\. Pairs flagged FALSE are dropped; the rest are ranked by score and passed on\.

### 3\.6Semantic Deduplication

Books covering the same topic yield equivalent questions in different words\. The engine clusters ranked pairs by sentence\-embedding similarity[19](https://arxiv.org/html/2608.15535#bib.bib15)and keeps the highest\-scoring member of each cluster\. Table[3](https://arxiv.org/html/2608.15535#S3.T3)shows a real case: four questions about Qutbuddin Aibak, from four books, cluster together and only the top one survives\. Most clusters are singletons and pass through unchanged\.

StatusQuestionAnswerScoreSourceCluster 19 — Brahmi script \(4 variants\)RetainedWhat type of script was used for the twelve Ashokan inscription findspots that appeared in South India?Brahmi script90NCERT Class 7 HistoryDiscardedIn which script were the majority of Ashokan inscriptions composed across the greater part of the Indian subcontinent?Brahmi script88SSC History Class 9DiscardedWhich ancient Indian script was used for the short inscriptions found in Sri Lanka dating to the second and first centuries BC?Brahmi script85UPSCDiscardedWhich script did Ashoka primarily use for writing the majority of his royal messages and inscriptions?Brahmi script81MPSC HistoryCluster 20 — single representative, no duplicates foundRetainedWhich Sultan of Delhi politely refused asylum to Jalaluddin when Changez Khan was near the Indus?Iltutmish90NCERT Class 7Table 3:Semantic deduplication in practice\. Green rows are retained as cluster representatives; red rows are discarded duplicates\. Cluster 19 collapses four equivalent questions drawn from four source books; Cluster 20 shows the common singleton case, which passes unchanged to manual verification\.
### 3\.7Domain Split and Manual Verification

The English dataset then contains 3,471 pairs across nine domains: Culture \(534\), History \(510\), Geography \(499\), Law \(494\), Political Science \(450\), Science \(325\), Sports \(273\), Art \(233\), and Commercial Studies \(153\)\. Sizes reflect the source material available\. Three annotators next reviewed every pair independently, flagging factual errors, ambiguous phrasing, trivially answerable content, and weak India specificity\. Any pair flagged by one reviewer was removed after discussion\. This cut 20–25% of the set: a share of Agent 1’s output passed every automated check but failed expert review\.

### 3\.8Translation and Release

The verified pairs were translated into 19 Indic languages: Assamese, Bengali, Dogri, Gujarati, Hindi, Kannada, Konkani, Maithili, Malayalam, Marathi, Meitei \(Manipuri\), Nepali, Odia, Punjabi, Sanskrit, Sindhi, Tamil, Telugu, and Urdu, using GPT\-5\.4 Mini via batched API calls\. Google Cloud and Gemini Translation were tried first but handled Law and Political Science terminology inconsistently\. Questions and answers went in separate batches with no pairing information, so the model never saw factual associations and translation stayed purely linguistic\. The sets were merged afterwards\.

## 4Evaluation Methodology

Table 4:Domain\-wise model accuracy on the English subset \(%\) under the LLM judge, with domain sizes in parentheses\. Shading: green\>\>85%, light green 70–85%, yellow 55–70%, orange 40–55%, red<<40%\. Corresponding lexical\-criterion scores are given in Table[7](https://arxiv.org/html/2608.15535#A2.T7)\.### 4\.1Models Evaluated

Six LLMs were selected to span a range of scales, architectures, and training philosophies:

- •Gemini 2\.5 Flash\(Google\): frontier reference model\. It is also the Agent 1 and Agent 2 backbone, which makes its score a data point on generator–evaluator overlap\.
- •GPT\-5\.4 Mini\(OpenAI\): a general\-purpose model testing whether broad multilingual pre\-training transfers to India\-specific factual knowledge\.
- •Gemma4 31B\(Google\): an open\-weight model, included to test whether open\-weight systems at the 30B scale can challenge proprietary ones\.
- •Sarvam 30B: an Indic\-native model pre\-trained on Indian\-language corpora, included to test whether Indic\-specific pre\-training yields an advantage\.
- •Gemma2 9B[8](https://arxiv.org/html/2608.15535#bib.bib23): a mid\-sized model representative of deployments in resource\-constrained settings\.
- •Llama 3\.1 8B[9](https://arxiv.org/html/2608.15535#bib.bib22): a community baseline establishing the performance floor\.

### 4\.2LLM\-as\-a\-Judge Evaluation

Each model was prompted with each question without retrieval or context augmentation and instructed to answer in the language of the question \(Appendix[C\.3](https://arxiv.org/html/2608.15535#A3.SS3)\)\. Responses were evaluated by Gemma 3 12B \(Appendix[C\.4](https://arxiv.org/html/2608.15535#A3.SS4)\), which receives the question, the gold answer and the model response, and returns a binary verdict with a natural\-language justification\. The judge accepts paraphrases and alternate transliterations: “Chhatrapati Shivaji Maharaj” and “Shivaji Raje Bhosle” are both correct answers to*What was the name of the king who founded the Maratha Empire?*Responses in a language other than that of the question are marked incorrect, penalising language inconsistency\. The pipeline is shown in Figure[4](https://arxiv.org/html/2608.15535#A1.F4)\.

### 4\.3Deterministic Lexical Evaluation

Because every gold answer is one to five words, the benchmark can also be scored without a judge\. We report two deterministic criteria alongside the judge verdicts, applied to the same 3,471 responses per model:

- •Exact substring:the complete gold answer must occur in the response as one contiguous, case\-insensitive substring\.
- •Word overlap:every word of the gold answer must occur somewhere in the response, in any order, case\-insensitively and after stripping edge punctuation\.

Neither criterion subsumes the other: substring matching ignores word boundaries, while word overlap ignores order but demands exact tokens\. Both are stricter than the judge and fully reproducible from the released responses\.

### 4\.4Metrics

Domain accuracy is the fraction of correct verdicts among the\|Dd\|\|D\_\{d\}\|questions of domaindd, under whichever protocol is in use\. Weighted accuracy aggregates across domains in proportion to domain size:

Weighted Accuracy=∑d\|Dd\|⋅Accuracyd∑d\|Dd\|\\text\{Weighted Accuracy\}=\\frac\{\\displaystyle\\sum\_\{d\}\|D\_\{d\}\|\\cdot\\text\{Accuracy\}\_\{d\}\}\{\\displaystyle\\sum\_\{d\}\|D\_\{d\}\|\}\(1\)

## 5Results and Analysis

### 5\.1English Results

Table[4](https://arxiv.org/html/2608.15535#S4.T4)gives domain\-wise judge accuracy for all six models on the English subset\. The overall ordering is:

- •Gemini 2\.5 Flash: 86\.9% \(3,018 of 3,471 correct\)
- •Gemma4 31B: 62\.2% \(2,158\)
- •GPT\-5\.4 Mini: 60\.1% \(2,086\)
- •Sarvam 30B: 48\.7% \(1,692\)
- •Gemma2 9B: 41\.1% \(1,427\)
- •Llama 3\.1 8B: 35\.4% \(1,228\)

This ordering is stable but not uniform: GPT\-5\.4 Mini overtakes Gemma4 31B in Geography, Political Science, and Sports, and Llama 3\.1 8B overtakes Gemma2 9B in Law\. Gemini’s lead should be read alongside its role as the Agent 1 and Agent 2 backbone \(see Limitations\)\.

Table 5:Accuracy \(%\) on the English subset under all three evaluation protocols\.Table 6:Judge accuracy \(%\) of Gemma4 31B and Sarvam 30B across domains and languages\. Eng = English, Guj = Gujarati, Hin = Hindi, Ass = Assamese, Tel = Telugu, Odi = Odia, San = Sanskrit, Mar = Marathi, Ben = Bengali, Tam = Tamil\.
### 5\.2Agreement Between Evaluation Protocols

Table[5](https://arxiv.org/html/2608.15535#S5.T5)compares the judge against the two deterministic criteria\. Three results follow\.

- •The ranking does not depend on the protocol\.The three protocols give almost the same ordering, and the two lexical criteria differ by at most 0\.5 pp on any model\. The ranking is a property of the models, not of the judge we chose\.
- •Gemma4 31B matches GPT\-5\.4 Mini\.The judge puts Gemma4 ahead by 2\.10 pp, while exact substring puts GPT\-5\.4 Mini ahead by 0\.43 pp and word overlap by 0\.09 pp\. No protocol separates them, so an open\-weight model here almost matches a commercial one\.
- •Lexical scores are a strict floor\.Every model scores 17–25 pp lower under string matching, and the shortfall is uneven\. It is smallest in Law and Sports, whose answers are dates and fixed English names such as*Durand Cup*, and largest where answers are Indic words such as*Waghnakh*and*Toranas*, which have more than one accepted spelling \(Ex:*Vagh Nakh*,*Toran*\)\. String matching therefore under\-reports the knowledge the benchmark measures, so we report the LLM judge as the primary protocol and the lexical scores as a reproducible floor\.

### 5\.3Multilingual Results

Table[6](https://arxiv.org/html/2608.15535#S5.T6)reports Gemma4 31B and Sarvam 30B, the two same\-scale models, across all nine domains in ten languages; Figure[3](https://arxiv.org/html/2608.15535#S5.F3)plots their weighted accuracy side by side\. Gemma4 31B leads in every language, and its spread across languages \(56\.9–66\.9%\) is far narrower than Sarvam’s \(33\.2–48\.7%\)\.

### 5\.4Key Observations

ENHIGUMRTATEASBNODSA002020404060608080Weighted accuracy \(%\)Gemma4 31BSarvam 30BFigure 3:Weighted accuracy of Gemma4 31B and Sarvam 30B across ten languages, under the LLM\-as\-a\-judge Gemma3 12B\.1. 1\.Indic\-native pre\-training does not guarantee an advantage \(Gemma4 31B vs Sarvam 30B\)\.The comparison shows a substantial performance difference between a dense model \(Gemma4 31B\) and a mixture\-of\-experts model \(Sarvam 30B\), despite their similar nominal parameter scales: - •Gemma4 31B reaches 62\.2% against Sarvam 30B’s 48\.7%: a gap of 13\.5 pp, or 466 questions\. - •The ordering holds under both lexical criteria \(37\.3% versus 27\.7% on exact substring\)\. - •Gemma4’s largest margins are Art \(21\.5 pp\), Culture \(20\.8 pp\), Law \(19\.6 pp\), and History \(16\.5 pp\)\. - •Sarvam 30B comes closest in Science \(3\.7 pp\), Sports \(4\.0 pp\), and Political Science \(4\.9 pp\), but leads in no domain\. - •Gemma4 31B leads in every language, and the gap widens to 17\.4–28\.3 pp across Indic languages, peaking in Hindi\. This result indicates that Indic\-specific training does not by itself guarantee an advantage, although the architectural and training differences between the two models prevent attributing the gap to pre\-training strategy alone\.
2. 2\.Parameter count no longer predicts rank cleanly\.Gemini 2\.5 Flash leads every domain and the English ranking by 24\.8 pp, matching v1’s finding that frontier models dominate\. Below it, larger is not always better: the open\-weight Gemma4 31B almost ties with GPT\-5\.4 Mini \(Section[5\.2](https://arxiv.org/html/2608.15535#S5.SS2)\), and Llama 3\.1 8B beats the larger Gemma2 9B in Law\. Architecture and training data matter as much as parameter count\.
3. 3\.English supremacy is no longer universal\.In v1, English beat every Indic language for every model\. In v2 that holds for Sarvam 30B but not Gemma4 31B, where Hindi \(66\.9%\) and Assamese \(62\.3%\) edge out English \(62\.2%\)\. English is no longer a guaranteed ceiling, though most Indic languages still trail\.
4. 4\.Domain hierarchy: Sports and Law are the weak points\.Averaged over all six models, Commercial Studies is strongest \(∼\\sim70%\), then Political Science \(∼\\sim68%\) and Science \(∼\\sim64%\), while Sports \(∼\\sim41%\) and Law \(∼\\sim46%\) are weakest\. In v1 Geography was weakest\. Both stay lowest in every language for both models \(Table[6](https://arxiv.org/html/2608.15535#S5.T6)\), so procedural and fast\-changing knowledge appear under\-represented in pre\-training corpora\.
5. 5\.Language\-level performance is model\-dependent\.Tamil is weakest for both Sarvam 30B \(33\.2%\) and Gemma4 31B \(56\.9%\), but the second\-weakest differs: Marathi and Bengali for Sarvam, Gujarati for Gemma4\. Performance depends on each model’s pre\-training corpus, not the language alone\.
6. 6\.Hallucination in curriculum\-grounded generation\.The 20–25% manual removal rate shows how often a frontier model, given source text and checked by a second automated pass, still produces pairs that fail expert review\. It sets a lower bound on the oversight such pipelines need\.

## 6Future Work

Two extensions are planned\. First, a region\-specific structure: v2 translates one English pool into all 19 languages, whereas an 80–20 split, with 20% of questions curated natively per region, would test regional depth too\. Example, the 20% of Marathi dataset would cover Maharashtra\-specific sources to gain more cultural context\. Second, extending the evaluation to all 19 languages and all six models under all three protocols\. Since string matching penalises spelling variants \(Section[5\.2](https://arxiv.org/html/2608.15535#S5.SS2)\), a spelling\-normalised criterion would give a fairer judge\-free floor for Indic\-script responses\.

## 7Conclusion

We present L3Cube\-IndicQuest v2, a benchmark for evaluating LLM knowledge in the Indian context\. Built through a two\-agent Gemini 2\.5 Flash pipeline, semantic deduplication, and human verification, it contains 3,471 English pairs across nine domains and 69,420 parallel pairs across 20 languages\. All three protocols agree on the ranking, led by Gemini 2\.5 Flash\. The 13\.5 pp gap between Gemma4 31B and Sarvam 30B, widening to 17\.4–28\.3 pp across Indic languages, challenges the assumption that Indic\-native pre\-training is enough to beat a capable general\-purpose model\. However, a fair comparison requires closely matched model size, architecture, and training setup\. Law and Sports remain hard everywhere\. Two methodological findings follow: the 20–25% removal rate shows that human oversight stays essential, and the 17–25 pp shortfall of string matching against the judge, largest where answers are Indic words with more than one accepted spelling, shows that lexical metrics under\-report the very knowledge such benchmarks measure\.

## Limitations

The multilingual evaluation covers 9 of the 19 Indic languages, 2 of the 6 models, and the judge protocol only; conclusions about the rest are limited to the English subset\. Judge verdicts come from a single model \(Gemma 3 12B\)\. The lexical criteria reproduce the ranking, but cannot confirm absolute judge accuracy, since both are stricter criteria rather than independent ground truth; a human agreement study is still needed\. Gemini 2\.5 Flash is both the generation and evaluation model, introducing a familiarity bias we cannot fully control for\. GPT\-5\.4 Mini is both the translation model and an evaluated model; its English\-only scores are unaffected, but this matters once all six are evaluated\. The manual review is reported as an aggregate removal rate, with no inter\-annotator agreement statistic\. Finally, the 19 non\-English subsets are machine translations of one English pool, so translation errors, especially in low\-resource languages, may affect scores independently of model knowledge\.

## Acknowledgements

This work was carried out under the L3Cube Labs, Pune mentorship programme\. We thank our mentor for the continuous guidance and support that shaped this work\. This work is a part of the L3Cube\-IndicNLP project222[L3Cube\-IndicNLP](https://github.com/l3cube-pune/indic-nlp)[14](https://arxiv.org/html/2608.15535#bib.bib28)\.

## References

- Ahujaet al\.\(2023\)K\. Ahuja, H\. Diddee, R\. Hada, M\. Ochieng, K\. Ramesh, P\. Jain, A\. Nambi, T\. Ganu, S\. Segal, M\. Ahmed, K\. Bali, and S\. SitaramMEGA: multilingual evaluation of generative AI\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px2.p1.1)\.
- Artetxeet al\.\(2020\)M\. Artetxe, S\. Ruder, and D\. YogatamaOn the cross\-lingual transferability of monolingual representations\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px2.p1.1)\.
- Clarket al\.\(2020\)J\. H\. Clark, E\. Choi, M\. Collins, D\. Garrette, T\. Kwiatkowski, V\. Nikolaev, and J\. PalomakiTyDi QA: a benchmark for information\-seeking question answering in typologically diverse languages\.Transactions of the Association for Computational Linguistics8,pp\. 454–470\.Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px2.p1.1)\.
- Doddapaneniet al\.\(2022\)S\. Doddapaneni, R\. Aralikatte, G\. Ramesh, S\. Goyal, M\. M\. Khapra, A\. Kunchukuttan, and P\. KumarTowards leaving no Indic language behind: building monolingual corpora, benchmark and models for Indic languages\.Note:arXiv:2212\.05409Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p2.1),[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px5.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled AlpacaEval: a simple way to debias automatic evaluators\.Note:arXiv:2404\.04475Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px6.p1.1)\.
- Endaitet al\.\(2025\)S\. Endait, R\. Ghatage, A\. Kulkarni, R\. Patil, and R\. JoshiIndicSQuAD: a comprehensive multilingual question answering dataset for Indic languages\.Note:arXiv:2505\.03688Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p2.1),[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px5.p1.1)\.
- Galaet al\.\(2024\)J\. Gala, T\. Jayakumar, J\. A\. Husain, M\. S\. U\. R\. Khan, D\. Kanojia, R\. Puduppully, M\. M\. Khapra, R\. Dabre, R\. Murthy, A\. Kunchukuttan,et al\.Airavata: introducing Hindi instruction\-tuned LLM\.Note:arXiv:2401\.15006Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px5.p1.1)\.
- Gemma Teamet al\.\(2024\)Gemma Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin,et al\.Gemma 2: improving open language models at a practical size\.Note:arXiv:2408\.00118Cited by:[5th item](https://arxiv.org/html/2608.15535#S4.I1.i5.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian,et al\.The Llama 3 herd of models\.Note:arXiv:2407\.21783Cited by:[6th item](https://arxiv.org/html/2608.15535#S4.I1.i6.p1.1)\.
- Guet al\.\(2024\)J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu,et al\.A survey on LLM\-as\-a\-judge\.Note:arXiv:2411\.15594Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px6.p1.1)\.
- Haaset al\.\(2025\)L\. Haas, G\. Yona, G\. D’Antonio, S\. Goldshtein, and D\. DasSimpleQA verified: a reliable factuality benchmark to measure parametric knowledge\.Note:arXiv:2509\.07968Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p2.1),[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px3.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p1.1)\.
- Joshiet al\.\(2020\)P\. Joshi, S\. Santy, A\. Budhiraja, K\. Bali, and M\. ChoudhuryThe state and fate of linguistic diversity and inclusion in the NLP world\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 6282–6293\.Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p1.1)\.
- Joshi \(2022\)R\. JoshiL3cube\-mahanlp: marathi natural language processing datasets, models, and library\.arXiv preprint arXiv:2205\.14728\.Cited by:[Acknowledgements](https://arxiv.org/html/2608.15535#Sx2.p1.1)\.
- Kakwaniet al\.\(2020\)D\. Kakwani, A\. Kunchukuttan, S\. Golla, G\. N\.C\., A\. Bhattacharyya, M\. M\. Khapra, and P\. KumarIndicNLPSuite: monolingual corpora, evaluation benchmarks and pre\-trained multilingual language models for Indian languages\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 4948–4961\.Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p1.1),[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px5.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, B\. Oguz, R\. Rinott, S\. Riedel, and H\. SchwenkMLQA: evaluating cross\-lingual extractive question answering\.Note:arXiv:1910\.07475Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px2.p1.1)\.
- Myunget al\.\(2024\)J\. Myung, N\. Lee, Y\. Zhou, J\. Jin, R\. A\. Putri, D\. Antypas,et al\.BLEnD: a benchmark for LLMs on everyday knowledge in diverse cultures and languages\.Note:arXiv:2406\.09948Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px4.p1.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSQuAD: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 2383–2392\.Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px2.p1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 3982–3992\.Cited by:[§3\.6](https://arxiv.org/html/2608.15535#S3.SS6.p1.1)\.
- Roheraet al\.\(2024\)P\. Rohera, C\. Ginimav, A\. Salunke, G\. Sawant, and R\. JoshiL3Cube\-IndicQuest: a benchmark question answering dataset for evaluating knowledge of LLMs in Indic context\.Note:arXiv:2409\.08706Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p2.1),[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px1.p1.1)\.
- Romanouet al\.\(2024\)A\. Romanou, N\. Foroutan, A\. Sotnikova, Z\. Chen, S\. H\. Nelaturu,et al\.INCLUDE: evaluating multilingual language understanding with regional knowledge\.Note:arXiv:2411\.19799Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px4.p1.1)\.
- Shafayatet al\.\(2024\)S\. Shafayat, E\. Kim, J\. Oh, and A\. OhMulti\-fact: assessing multilingual LLMs’ multi\-regional knowledge using FActScore\.Note:arXiv:2402\.18045Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p1.1)\.
- Singhet al\.\(2024a\)A\. K\. Singh, R\. Murthy, V\. Kumar, J\. Sen, and G\. RamakrishnanIndic QA benchmark: a multilingual benchmark to evaluate question answering capability of LLMs for Indic languages\.Note:arXiv:2407\.13522Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p2.1),[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px5.p1.1)\.
- Singhet al\.\(2024b\)H\. Singh, N\. Gupta, S\. Bharadwaj, D\. Tewari, and P\. TalukdarIndicGenBench: a multilingual benchmark to evaluate generation capabilities of LLMs on Indic languages\.Note:arXiv:2404\.16816Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px4.p1.1)\.
- Vermaet al\.\(2024\)S\. Verma, M\. S\. U\. R\. Khan, V\. Kumar, R\. Murthy, and J\. SenMILU: a multi\-task Indic language understanding benchmark\.Note:arXiv:2411\.02538Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px4.p1.1)\.
- Wattset al\.\(2024\)I\. Watts, V\. Gumma, A\. Yadavalli, V\. Seshadri, M\. Swaminathan, and S\. SitaramPARIKSHA: a large\-scale investigation of human\-LLM evaluator agreement on multilingual and multi\-cultural data\.Note:arXiv:2406\.15053Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px4.p1.1)\.
- Weiet al\.\(2024\)J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. FedusMeasuring short\-form factuality in large language models\.Note:arXiv:2411\.04368Cited by:[§1](https://arxiv.org/html/2608.15535#S1.p2.1),[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px3.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.Cited by:[§2](https://arxiv.org/html/2608.15535#S2.SS0.SSS0.Px6.p1.1)\.

## Appendix AEvaluation Pipeline

![Refer to caption](https://arxiv.org/html/2608.15535v1/llm_judge_eval_pipeline_excalidraw.png)Figure 4:LLM\-as\-a\-judge evaluation pipeline\. Each of the six models responds to the same question; Gemma 3 12B evaluates each response against the gold\-standard answer and returns a TRUE/FALSE verdict\.
## Appendix BDomain\-wise Lexical Results

Table[7](https://arxiv.org/html/2608.15535#A2.T7)gives the full domain\-wise scores for both deterministic criteria defined in Section[4\.3](https://arxiv.org/html/2608.15535#S4.SS3), on the same 3,471\-question English subset used for Table[4](https://arxiv.org/html/2608.15535#S4.T4)\.

ModelArtComm\.Cult\.Geo\.Hist\.LawPol\.Sci\.Sci\.SportsCombined\(a\) Exact substringGemini 2\.5 Flash49\.868\.043\.853\.766\.585\.668\.955\.764\.561\.97GPT\-5\.4 Mini37\.349\.735\.232\.333\.141\.348\.036\.333\.737\.77Gemma4 31B39\.945\.136\.330\.732\.843\.546\.434\.830\.437\.34Sarvam 30B25\.339\.923\.422\.024\.522\.940\.934\.826\.427\.72Gemma2 9B19\.730\.119\.120\.615\.119\.234\.224\.321\.621\.92Llama 3\.1 8B16\.329\.415\.218\.012\.016\.028\.917\.918\.718\.24\(b\) Word overlapGemini 2\.5 Flash48\.567\.343\.354\.565\.985\.471\.357\.564\.162\.23GPT\-5\.4 Mini36\.048\.434\.632\.132\.441\.348\.737\.533\.037\.54Gemma4 31B38\.645\.136\.030\.932\.043\.349\.335\.130\.037\.45Sarvam 30B24\.039\.923\.021\.224\.322\.544\.435\.426\.027\.86Gemma2 9B18\.429\.418\.520\.014\.518\.234\.424\.321\.621\.43Llama 3\.1 8B15\.928\.814\.617\.411\.415\.030\.017\.918\.317\.89Table 7:Domain\-wise accuracy \(%\) under the two deterministic criteria, English subset\. Combined is the micro\-average over all 3,471 questions\. Rows are ordered by combined exact\-substring score\.
## Appendix CAgent and Evaluation Prompts

Only the structure of each prompt is shown; rule bodies, worked examples, and JSON schemas are abridged \(marked\[…\]\)\. Template variables in double braces \(e\.g\.,\{\{ context \}\}\) are substituted at runtime\. Prompts are reproduced verbatim from the implementation, and their spelling follows the original\.

### C\.1Agent 1: QA Generation Prompt

YouareanexpertquizmasterAIspecializedingeneratinghigh\-quality,India\-specifictriviaquestionsfortrainingdatasets\.Yoursolepurposeistocreatecontext\-rich,unambiguousQApairsthattestdeepknowledgeaboutIndia\.

CRITICALREQUIREMENTS\-FollowtheserulesSTRICTLY:

1\.INDIA\-SPECIFICFOCUS

2\.CONTEXT\-RICHQUESTIONS

3\.ABSOLUTEANSWERS

4\.COMPLETENAMES

5\.HIGHDIFFICULTY

6\.SHORTANSWERS\(1\-5words,verbatimfromtext\)

7\.NOTEXTREFERENCES

8\.PERFECTGRAMMAR

9\.GOLDENDATASETQUALITY

10\.JSONOUTPUTONLY

11\.SPECIFICDETAILS

12\.UNIQUEKNOWLEDGE

13\.CLEARQUESTIONSTRUCTURE\(What/Which/Who/When/Where/How\)

14\.VERBALCONSISTENCY

15\.PRECISETERMINOLOGY

16\.QUALITYOVERQUANTITY\(returnerrorJSONoverlow\-qualitypairs\)

17\.AVOIDTECHNICALTRIVIA

18\.AVOIDVAGUEDIRECTIONALQUESTIONS

19\.AVOIDINCOMPLETEANSWERS

20\.AVOIDAWKWARDPHRASING

21\.FOCUSONSIGNIFICANTKNOWLEDGE

EXAMPLESOFGOODQAPAIRS:

EXCELLENTEXAMPLES:\[\.\.\.\]

BADEXAMPLES\(avoidthese\):\[\.\.\.\]

QUALITYCHECKLIST\-Beforegenerating,ensureeachquestion:\[\.\.\.\]

JSONFORMATTINGRULES:

Ifyoucangeneratequestions:

\[\{"question":"\.\.\.","answer":"\.\.\."\}\]

IfyouCANNOTgenerateanyquestionsthatmeetallcriteria:

\{"error":"NoIndia\-specificdifficultquestionswithshortanswerscouldbegenerated\."\}

GENERATIONINSTRUCTIONS:\[\.\.\.\]

TEXT:

\{\{context\}\}

### C\.2Agent 2: Quality Validation Prompt

YouareanULTRA\-STRICTvalidatorforIndia\-specificQAtripletsthatenforcesGOLDENTRUTHstandards\.YourroleistoensureeveryQApairmeetsthehighestqualitybenchmarksfortestingLLMknowledgeaboutIndia\.

Inputvariables:

\-source\_text:JSONarrayof\{"chunk\_id","chunk\_text"\}

\-qa\_triplets:JSONarrayof\{"chunk\_id","question","answer"\}

TASK

1\.Findthematchingchunk\_textusingchunk\_id\.

2\.ApplyULTRA\-STRICTvalidationagainstgoldentruthstandards\.

3\.Assignqualityscores\(correctness,relevance,difficulty\)\.

4\.Checkduplicatesacrossalltriplets\.

5\.VerifyGOLDENTRUTHviaGoogleSearchwithmultiplesources\.

ULTRA\-STRICTVALIDATIONRULES\(ZEROTOLERANCEFORVIOLATIONS\)

1\.INDIA\-SPECIFICFOCUS\(CRITICAL\)

2\.CONTEXT\-RICH&SELF\-CONTAINED\(CRITICAL\)

3\.ABSOLUTEANSWERREQUIREMENTS\(CRITICAL;answerverbatiminsource,1\-5words\)

4\.PERFECTGRAMMAR&STRUCTURE\(CRITICAL\)

5\.HIGHDIFFICULTY&SPECIALIZEDKNOWLEDGE\(CRITICAL\)

6\.NOTEXTREFERENCES\(CRITICAL\)

7\.GOLDENTRUTHVERIFICATION\(CRITICAL;sets"correctness\_flag"\)

8\.QUALITYSTANDARDS\(CRITICAL\)

9\.DUPLICATEDETECTION\(BATCH\-LEVEL;sameanswerAND\>=80%questionoverlap\)

10\.COMPREHENSIVESCORINGSYSTEM\(Correctness/Relevance/Difficulty/OverallQuality,each0\-10\)

11\.DETAILEDSCORINGGUIDELINES\(0\-10SCALE\)\[\.\.\.\]

\-OverallQuality=Correctness\(40%\)\+Relevance\(30%\)\+Difficulty\(30%\)

12\.EXAMPLESOFVALIDATION\[\.\.\.\]

13\.ULTRA\-STRICTVALIDITY

\-"valid":trueONLYifALLcriteriapassAND"correctness\_flag":true

\-Minimumoverallqualityscoreof7requiredforvalidity

Outputschema\(exactJSON\):

\{

"results":\[\{"chunk\_id","question","answer","valid","correctness\_flag","errors","scores":\{\.\.\.\},"detailed\_assessment":\{\.\.\.\},"duplicate\_indices"\}\],

"batch\_summary":\{"total\_triplets","valid\_triplets","invalid\_triplets","duplicate\_groups","average\_scores":\{\.\.\.\}\}

\}

Input:

source\_text:\{\{source\_text\|tojson\(indent=2\)\}\}

qa\_triplets:\{\{qa\_triplets\|tojson\(indent=2\)\}\}

### C\.3Model Evaluation Prompt

Youareaquestion\-answeringassistant\.Yourtaskistoanswerquestionsaccuratelyandconcisely\.

\#\#InputFormat

\[\{"QID":"string","Question":"string"\},\.\.\.\]

\#\#OutputFormat

\[\{"QID":"string","Answer":"string","Reasoning":"string\(optional\)"\},\.\.\.\]

\#\#Instructions

1\.AnswerAccuracy:basedonyourknowledge\.

2\.AnswerLength:1to5wordsmaximum\.

3\.UnknownAnswers:set"Answer"tonull\.Donotmakeupanswers\.

4\.Reasoning:optional\.

5\.OutputFormat:returnONLYtheJSONarray\.

6\.OutputLanguage:answerONLYinthelanguageoftheQUESTION\.

\#\#Example\[\.\.\.\]

Now,answerthefollowingquestions:

\{\{questions\_json\}\}

### C\.4LLM\-as\-a\-Judge Prompt

Youareanevaluationjudge\.Yourtaskistodetermineifthegivenanswermatchesthegroundtruthanswerforthegivenquestion\.

\#\#Input

Question:\{\{question\}\}

GroundTruthAnswer:\{\{ground\_truth\_answer\}\}

GivenAnswer:\{\{llm\_answer\}\}

\#\#Instructions

1\.Comparethe"GivenAnswer"againstthe"GroundTruthAnswer"inthecontextofthequestion\.

2\.Correctifsemanticallyequivalent\(samemeaning/sameentityorfact\)\.

3\.Minorphrasing,formatting,orextradetailisacceptableifthecoreansweriscorrect\.

4\.Clearlywrong,meaning\-changing,orunrelatedanswersareincorrect\.

5\.Answersinadifferentlanguagethanthequestionandgroundtruthareincorrect\.

\#\#OutputFormat

ReturnONLYaJSONobject\(noadditionaltext,nomarkdownfences\):

\{"is\_correct":true,"reasoning":"briefexplanation"\}