Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain

# Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain
Source: [https://arxiv.org/html/2607.20510](https://arxiv.org/html/2607.20510)
Dmitrii Khizbullin1,\*,Zaid Alyafeai1,Abdelrahman Eldesokey1,Nourah AlSultan2, Raghad Alshalan2,David R\. Pugh1,Bernard Ghanem1

1King Abdullah University of Science and Technology \(KAUST\),2stc \*Corresponding author:[dmitrii\.khizbullin@kaust\.edu\.sa](https://arxiv.org/html/2607.20510v1/mailto:[email protected])

###### Abstract

We introduce Telco\-GAIA, a bilingual, multi\-modal benchmark for evaluating tool\-using agents on the data of a real\-world telecommunications operator\. Telco\-GAIA comprises 100 human\-verified question\-answering tasks, in English and Arabic, that each demand multi\-hop reasoning \(4\.2 hops on average\) over three heterogeneous sources: a static website snapshot \(HTML, images, and linked PDFs\), a synthetic relational SQL database, and external web archives, spanning text, image, and tabular modalities\. The benchmark is delivered as a sandboxed Docker environment and scored by normalized exact string matching, making evaluation objective, deterministic, and reproducible over time without any LLM\-as\-a\-Judge\. Evaluating a purpose\-built reference agent across twelve commercial and open LLMs, we find Telco\-GAIA challenging: even the strongest model solves only 71% of tasks; under a moderate cost budget, this falls to about 40%, and the visually grounded categories remain the weakest, where the average backend scores below 30%, leaving substantial headroom in document and image understanding\. Telco\-GAIA offers a rigorous, reproducible testbed for enterprise agents and a template for constructing closed\-domain benchmarks\.

Telco\-GAIA: Bilingual Benchmark for Agents in Telecom Domain

Dmitrii Khizbullin1,\*, Zaid Alyafeai1, Abdelrahman Eldesokey1, Nourah AlSultan2,Raghad Alshalan2,David R\. Pugh1,Bernard Ghanem11King Abdullah University of Science and Technology \(KAUST\),2stc\*Corresponding author:[dmitrii\.khizbullin@kaust\.edu\.sa](https://arxiv.org/html/2607.20510v1/mailto:[email protected])

## 1Introduction

Large language model \(LLM\) agents are increasingly deployed in customer\-facing enterprise settings, where they must navigate proprietary websites, parse internal documents, and query operational databases to resolve customer requests\. Telecommunications is a prototypical example of such a setting: a single operator exposes hundreds of plans, devices, roaming options, and promotions on a large bilingual website, backed by relational billing and usage data\. Determining whether an agent can reliably retrieve and reason over these heterogeneous sources is, therefore, a practical prerequisite for deployment\.

Rigorous evaluation of such agents, however, remains difficult\. A common practice generates question\-answer pairs semi\-automatically from text chunks and scores responses with an LLM\-as\-a\-Judge \(LLMaJ\)\. This is convenient but neither objective nor reproducible: LLMaJ scores are sensitive to the judge model and to prompt wordingZhenget al\.\([2023](https://arxiv.org/html/2607.20510#bib.bib41)\); Wanget al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib42)\)\. Moreover, automatically generated questions are not guaranteed to have a unique, verifiable answer: over half of open\-domain questions admit multiple valid answersMinet al\.\([2020](https://arxiv.org/html/2607.20510#bib.bib43)\), and audits find 13–17% of responses scored incorrect to be actually correct, penalised by gold\-standard reference answers that are themselves incomplete or wrongMinet al\.\([2021](https://arxiv.org/html/2607.20510#bib.bib44)\)\. Academic benchmarks address parts of the problem but leave a gap for this setting\. Open\-domain agent benchmarks such as GAIAMialonet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib1)\)rely on the live internet and are thus not reproducible over time; sandboxed web environments such as WebArenaZhouet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib5)\)are general\-purpose and do not combine relational\-database, PDF, and image retrieval within a single domain; and enterprise RAG benchmarks such as WixQACohenet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib9)\)are text\-only, monolingual, and provide no live agent environment or structured data\. None targets a bilingual, multi\-modal, telecom\-specific corpus with deterministic scoring\.

We introduceTelco\-GAIA111Questions and resources:[https://huggingface\.co/datasets/kaust\-generative\-ai/telco\-gaia](https://huggingface.co/datasets/kaust-generative-ai/telco-gaia)\. The ground\-truth answers are released as a separate gated dataset to limit contamination:[https://huggingface\.co/datasets/kaust\-generative\-ai/telco\-gaia\-groundtruth](https://huggingface.co/datasets/kaust-generative-ai/telco-gaia-groundtruth)\., a benchmark for evaluating tool\-using agents on the public data of a real telecommunications operator serving tens of millions of subscribers\. Telco\-GAIA draws design inspiration from GAIAMialonet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib1)\)but targets a closed enterprise domain in a sandboxed, reproducible environment\. It comprises 100 multi\-hop tasks \(mean 4\.2 hops\) spanning seven categories \(Pricing, Images, Web Archives, PDF, PDF Visual, Database, and Miscellaneous\) in English and Arabic \(65 and 35 tasks, respectively\)\. Each task has a single, human\-verified correct answer scored by normalized exact string matching, removing any dependence on an LLM judge\. Solving a task requires retrieving and reasoning over three heterogeneous sources: a static snapshot of the operator’s website \(HTML pages, images, and linked PDFs\), a synthetic relational SQLite customer database exposed via a REST API, and external web archives \(Wikipedia and ArXiv\)\. The website and database are served locally via Docker, while the web archives are virtually immutable; the environment is therefore semi\-closed, and runs are reproducible across time\. Because the operator’s site is real and public, a model might answer some tasks from parametric memory rather than retrieving from the served snapshot; this shortcut only weakens over time, as the live site drifts from our frozen snapshot, forcing genuine retrieval and hardening the benchmark rather than decaying it\.[Appendix˜A](https://arxiv.org/html/2607.20510#A1)details the harness, evaluator, and architecture\.

To establish baseline difficulty, we evaluate a purpose\-built tool\-calling reference agent across twelve commercial and open models\. The benchmark proves challenging: the strongest model reaches 71% overall, and accuracy is lowest on the visually grounded categories \(Images and PDF Visual\), exposing a concrete weakness of current agents in document and image understanding\.[Table˜1](https://arxiv.org/html/2607.20510#S1.T1)situates Telco\-GAIA against prior benchmarks\.

BenchmarkMulti\-modalTelecomdomainRelationalDBGolden stepsannotationMulti\-/bi\-lingualGAIAMialonet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib1)\)✓––✓–WebArenaZhouet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib5)\)–––––VisualWebArenaKohet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib26)\)✓––––TheAgentCompanyXuet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib25)\)–––✓‡–WorkArenaDrouinet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib6)\)–––––AgentBenchLiuet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib7)\)––✓––τ\\tau\-benchYaoet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib27)\)–––––WixQACohenet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib9)\)–––––LiveRAGCarmelet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib29)\)–––––FinanceBenchIslamet al\.\([2023](https://arxiv.org/html/2607.20510#bib.bib20)\)–––––MMLongBench\-DocMaet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib31)\)✓––––TelAgentBenchLeeet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib22)\)–✓–✓‡–†TeleQnAMaatouket al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib32)\)–✓–––ToolHopYeet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib34)\)–––✓–Telco\-GAIA \(ours\)✓✓✓✓✓Table 1:Comparison of key benchmark properties\. Telco\-GAIA is the only benchmark combining all five properties simultaneously\.‡partial golden\-steps annotation \(only a subset of tasks, or milestone checkpoints rather than reasoning steps\)\.†TelAgentBench is monolingual Korean; all other prior benchmarks are monolingual English\.In summary, the principal contributions of Telco\-GAIA are as follows\.

1. 1\.A bilingual, multi\-modal, multi\-hop telecom agent benchmark\.Telco\-GAIA spans three modalities \(text, images, and tabular data\) drawn from heterogeneous sources within one harness: a single operator’s website \(HTML text and on\-page images\), linked PDF documents \(text and embedded images\), a synthetic relational database \(tabular data\), and archival web resources \(Wikipedia and ArXiv\), in both Arabic and English\. No prior benchmark couples these properties \([Table˜1](https://arxiv.org/html/2607.20510#S1.T1)\)\.
2. 2\.A reproducible, contamination\-resistant environment\.The website snapshot and customer database are frozen and served locally via Docker, so every run, whenever it is conducted, operates on an identical corpus, unlike open\-internet benchmarks whose answers drift as live web content changes\.
3. 3\.Objective, deterministic scoring\.Each of the 100 tasks has a single human\-verified answer scored by normalized exact string matching, eliminating the judge\-model, prompt, and scale sensitivity of LLM\-as\-a\-Judge metrics used in RAG frameworks such as RAGASEset al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib13)\)and ARESSaad\-Falconet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib12)\)\.
4. 4\.A relational customer database as a first\-class modality\.A synthetic SQLite database of 53 customers with twelve months of billing and usage history, queried via SQL over a REST API, requires agents to interleave structured queries with website and PDF retrieval, a structured/unstructured combination absent from prior RAG benchmarks\.

## 2Related Work

Telco\-GAIA sits at the intersection of several benchmark families \([Table˜1](https://arxiv.org/html/2607.20510#S1.T1)\)\. General agent benchmarks such as GAIAMialonet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib1)\)establish exact\-match, tool\-using evaluation but run on the open internet, sacrificing reproducibility\. Sandboxed web environments such as WebArenaZhouet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib5)\)and VisualWebArenaKohet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib26)\)are reproducible and general\-purpose but lack a relational database and domain grounding\. Enterprise RAG benchmarks such as WixQACohenet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib9)\)offer a single\-company corpus yet are text\-only, monolingual, and without a live agent environment\. Closest in spirit, TelAgentBenchLeeet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib22)\)targets telecom agents but is Korean\-only, exposes its data through abstract APIs rather than a served website, and has no image or PDF modality\. Telco\-GAIA is unique in unifying all of these properties;[Appendix˜D](https://arxiv.org/html/2607.20510#A4)provides a detailed description of related work\.

## 3Dataset Creation Principles

This section describes the design principles and per\-category specifications used to construct the 100 tasks in the Telco\-GAIA Benchmark\. The dataset comprises 65 English tasks and 35 Arabic tasks, distributed across seven categories: Pricing, Miscellaneous, Images, Web Archives, PDF, PDF Visual, and Database\. The English subset was constructed first; the Arabic subset followed, built on the same principles through a combination of translation and fresh authoring\.

### 3\.1Common principles

The following principles apply to every task regardless of category or language\.

#### Bootstrapping by a human annotator\.

A human annotator authors a seed set of tasks, an LLM agent expands them in the same spirit, and every task, manual or synthetic, is independently validated by one or more humans\.

#### Strict causal chains\.

A*hop*is one retrieval\-and\-reasoning step, such as navigating the website to a target page, finding a fact in a PDF, querying the database, or reading a value off an image\. The reasoning process must form a single linear chain where the output of hopNNis a required input for hopN\+1N\{\+\}1\. Before finalisation, every task is verified against three criteria: \(i\) every retrieved fact is a necessary input to a subsequent step \(no dead ends\); \(ii\) strict causal ordering holds \(hopN\+1N\{\+\}1is impossible without hopNN\); \(iii\) no dangling or redundant facts remain\. A common failure mode is*spoiling*: revealing an intermediate or final answer in an earlier step, which lets the agent skip a sub\-chain of fact discovery\.

#### Verification asymmetry\.

Every task must be*easy to verify*\(a human can confirm the answer in under 2 minutes using thestepsfield\) yet*hard to solve*\(the agent must perform genuine search, disambiguation, and multi\-tool coordination\)\. Difficulty comes from discovery, disambiguation, the effort to factor out potential alternative solutions, and non\-obvious cross\-source bridges\.

#### Arabic subset\.

The Arabic subset was created by native Arabic speakers, combining human translations of existing English tasks with newly authored ones, and following the distribution of the English categories except Database\.

### 3\.2Category\-specific principles

#### Pricing\.

Pricing tasks retrieve exact monetary values \(plan costs, device prices, add\-on fees, and VAT\-inclusive totals\) from the operator’s website\. Questions imply or state the navigation path when a plan name is generic, and disambiguate Prepaid from Postpaid and Consumer from Business, whose prices differ; answers are exact numbers\.

#### Miscellaneous\.

Miscellaneous tasks cover non\-price reasoning: coverage checks, help\-centre FAQs, device comparisons, and plan features, drawn from a deliberately diverse mix of pages\. We forbid Yes/No answers\. The*result*of each check must drive the next hop \(e\.g\., if a city is covered, look up its specific roaming code\), keeping every step causal and the final answer a concrete value\.

#### Images\.

Image tasks require the agent to locate a specific on\-page visual \(a banner or promotional image, identified by page and section\) and extract a*deterministic*fact from it: a count of people or tiers, a brand logo, or a colour\. This visual fact is the first link of a 4–5\-hop, multi\-page chain, so a model that cannot read the image cannot proceed\.

#### Web Archives\.

Web Archives tasks bridge the operator’s site to external knowledge, requiring at least one fact retrieved from Wikipedia or ArXiv that is not reliably held in an LLM’s parametric memory\. The target entity is never named in the question \(anti\-spoiling\): the agent must identify it from context clues and only then look it up\. We prefer update\-resilient targets such as infobox fields and ISO codes, and use bidirectional Wiki↔\\leftrightarrowSite patterns\.

#### PDF\.

PDF tasks require downloading a PDF linked from the website and extracting a specific textual value that then drives the chain \(e\.g\., a clause number mapped to a plan ranking, then to a price\)\. Each task draws on a PDF from a distinct product group \(postpaid, prepaid, business, vacation packages\) so that no document is over\-exploited\.

#### PDF Visual\.

PDF Visual tasks target facts that exist only in a PDF’s*visual layout*and not its text layer: counting coloured boxes, table rows, icons, or floor\-plan elements\. Every such fact is verified to be non\-extractable as plain text, isolating genuine document\-image understanding from mere OCR\.

#### Database\.

Database tasks pair the website with a synthetic relational SQL database and demand 4–5 mostly\-database hops along creative multi\-source routes \(DB→\\toWeb→\\toDB, Web→\\toDB→\\toDB, DB→\\toWeb→\\toPDF→\\toMath\), with SQL ranging from multi\-table joins to temporal logic\. Because SQL is the agent’s strongest tool, we enforce*website\-bridge integrity*: every cross\-source bridge fact must live only on the website or in a PDF, never in a database column; otherwise the agent could answer it with SQL and skip the browser entirely \(“plans\-table leakage”\)\. To keep these tasks from being trivially solvable, we further inject controlled data\-quality artefacts \(duplicate invoices, credit notes, internal test accounts,NULL\-versus\-zero usage\) so that a carelessSELECT SUM/COUNT\(\*\)returns the wrong number and only an agent that inspects and filters the data succeeds;[Appendix˜B](https://arxiv.org/html/2607.20510#A2)details these traps and their empirically measured catch rates\.

## 4Dataset Statistics

![Refer to caption](https://arxiv.org/html/2607.20510v1/x1.png)Figure 1:Distribution of the tasks across categories and languages\.![Refer to caption](https://arxiv.org/html/2607.20510v1/x2.png)Figure 2:Distribution of reasoning hops per task\.[Figure˜1](https://arxiv.org/html/2607.20510#S4.F1)shows the joint distribution of the 100 tasks across the seven categories and two languages: the English subset \(65 tasks\) covers all seven categories, while the Arabic subset \(35 tasks\) mirrors six, omitting the English\-only Database category\. Tasks require 4\.2 reasoning hops on average \([Figure˜2](https://arxiv.org/html/2607.20510#S4.F2)\)\. Answers are 83 numeric \(counts, prices, durations, codes, calculations\) and 17 textual \(plan names, payment methods, country names, image\-derived attributes\)\. On tooling, the local browser is universal, as every task navigates the served operator website at least once, whilepdf\_reader\(32 tasks\),internet\_browser\(16, Web Archives only\), anddatabase\_query\(10, the entire Database category\) are added as needed; 48 tasks rely on the local browser alone \([Figure˜6](https://arxiv.org/html/2607.20510#A3.F6)\)\.

## 5Reference Agent Performance

ModelAccuracyCost \($\)Duration \(minutes\)TurnsMeanMaxMeanMaxclaude\-opus\-4\-8†71\.0%176\.932\.037\.213\.9100gpt\-5\.568\.0%227\.7134\.266\.717\.684Kimi\-K2\.6†68\.0%163\.42155\.0302\.520\.2100gemini\-3\.5\-flash67\.0%115\.292\.725\.119\.4100claude\-sonnet\-4\-6†67\.0%183\.763\.638\.118\.4100DeepSeek\-V4\-Pro48\.0%231\.2775\.6190\.115\.7100gpt\-5\.442\.0%58\.218\.515\.111\.226DeepSeek\-V4\-Flash40\.0%27\.9828\.8105\.219\.3100gpt\-5\.238\.0%33\.656\.213\.714\.053gemini\-3\.1\-flash\-lite37\.0%28\.862\.612\.339\.8100gpt\-5\.4\-mini15\.0%13\.078\.520\.111\.431gpt\-5\.4\-nano13\.0%3\.826\.412\.511\.434Table 2:Full\-sweep results across twelve models on all 100 tasks \(single run\), sorted by accuracy\. Duration and turns are per\-task\.†run under rate limits, so durations may be slightly inflated\.CategoryTotalAccuracyStd DevDatabase1066\.0%8\.4%Pricing1662\.5%9\.3%Web Archives1647\.5%4\.4%Miscellaneous1641\.2%8\.8%PDF1026\.0%11\.4%Images1625\.0%7\.7%PDF Visual163\.8%‡4\.4%Overall10038\.0%2\.9%Table 3:Reference agent accuracy by category \(GPT\-5\.2 backend, mean±\\pmsample std over 5 runs\)\.‡See discussion in §[5](https://arxiv.org/html/2607.20510#S5)\.CategoryArabicEnglishGapPricing69\.4%66\.7%−2\.8\-2\.8Miscellaneous45\.8%55\.0%\+9\.2\+9\.2Images23\.6%28\.3%\+4\.7\+4\.7Web Archives61\.1%60\.0%−1\.1\-1\.1PDF48\.3%63\.3%\+15\.0\+15\.0PDF Visual18\.1%19\.2%\+1\.1\+1\.1All \(excl\. DB\)44\.3%47\.4%\+3\.1\+3\.1Table 4:English vs\. Arabic accuracy by category, averaged across the twelve models of[Table˜2](https://arxiv.org/html/2607.20510#S5.T2)\(the Database category is English\-only and excluded\)\. Gap is English minus Arabic, in percentage points\.[Table˜2](https://arxiv.org/html/2607.20510#S5.T2)reports a single\-run sweep of the reference agent across twelve commercial and open models, comparing overall accuracy against API cost and per\-task latency and turn counts\. Accuracy spreads cleanly from 13% to 71%, while cost varies by nearly two orders of magnitude \($3\.82–$231\)\.

The resulting cost–accuracy frontier is far from monotonic: the top models \(claude\-opus\-4\-8 71%, gpt\-5\.5 68%\) sit among the more expensive, yet near\-frontier accuracy is reachable far more cheaply: gemini\-3\.5\-flash attains 67%, within one point of gpt\-5\.5, at $115: roughly half its cost and an order of magnitude lower latency\. A cluster of mid\-tier models \(DeepSeek\-V4\-Flash, gpt\-5\.2, gemini\-3\.1\-flash\-lite\) delivers 37–40% for under $35, while the smallest backends \(gpt\-5\.4\-mini/nano\) remain at or below 15% despite being the cheapest to run\. Accuracy, cost, and latency thus vary along largely independent axes, so the appropriate backend depends on the deployment budget rather than on a single quality ranking\. We do not analyse absolute durations, as they depend on each model’s unknown rate limits and inference provider\.

The Turns columns of[Table˜2](https://arxiv.org/html/2607.20510#S5.T2)expose two opposite failure modes on unsolved tasks\. The agent loop is capped at 100 turns \([Appendix˜C](https://arxiv.org/html/2607.20510#A3)\): every non\-GPT backend reaches this ceiling on at least one task \(maximum = 100\), with gemini\-3\.1\-flash\-lite exhausting the budget on roughly a third of its tasks \(mean 39\.8 turns\) yet still scoring only 37%, so these models fail by an inability to converge rather than by giving up\. The GPT backends never approach the cap \(maxima of 26–84 turns\) and instead stop on their own; for the strongest GPT this is decisiveness \(gpt\-5\.5 reaches 68% in at most 84 turns\), but for the weaker ones it is premature surrender, as gpt\-5\.2, gpt\-5\.4\-mini, and gpt\-5\.4\-nano halt after at most 53, 31, and 34 turns while scoring just 38%, 15%, and 13%\. Either way, turn count tracks effort rather than competence: the most turn\-hungry backend \(gemini\-3\.1\-flash\-lite, 39\.8 mean\) scores only 37%, whereas the most accurate \(claude\-opus\-4\-8, 71%\) uses just 13\.9, so additional deliberation does not buy additional accuracy\. In the agentic setting, model capability outweighs sheer effort: extra turns cannot compensate for a weaker backend, so investing in a stronger model is more effective than granting a weaker one a larger budget\.

To locate where this difficulty concentrates,[Table˜3](https://arxiv.org/html/2607.20510#S5.T3)breaks down performance by category for gpt\-5\.2, a medium\-capacity, inexpensive backend \(cf\.[Table˜2](https://arxiv.org/html/2607.20510#S5.T2)\) representative of a moderate cost budget, averaged over five independent runs to account for non\-determinism; per\-category standard deviations are propagated from per\-task pass rates \([Section˜C\.1](https://arxiv.org/html/2607.20510#A3.SS1)\)\.

The most difficult categories are those requiring visual understanding\. Images \(25\.0%\) and PDF \(26\.0%\) both suffer from OCR\-like errors: off\-by\-one\-digit plan codes, miscounted visual elements, and confusion between marketing labels and numeric identifiers; for instance, collapsing a textual plan label into a plan\-shaped token \(e\.g\. “P2\+”\) and emitting it in place of the numeric subscription code rather than abstaining\. A related failure is grounding on the wrong visual element: on a task asking for the colour of a specific catalogue card, every model in the sweep instead reports the colour of the page’s dominant banner, defaulting to the most salient region rather than the queried target\.‡At 3\.8%, PDF Visual is hardest for gpt\-5\.2: the GPT models keep invoking the browser’s unavailable PDF\-download instead of the PDF reader tool, so the failure is one of tool choice rather than reading the pages; stronger vision backends do much better\.

Not all category gaps reflect reasoning, however: roughly half of the Database misses by the DeepSeek backends are not wrong queries but failures to emit a parseable final answer, with a placeholder or boilerplate returned in its place\. As the other eleven backends comply with the same answer\-format instruction, this points to weaker instruction\-following in those models rather than to the scoring\.

[Table˜4](https://arxiv.org/html/2607.20510#S5.T4)reports English versus Arabic accuracy per category, averaged across the twelve models\. The two languages are well balanced overall \(47\.4% vs\. 44\.3%, a 3\.1\-point English edge\), and most categories agree within a few points, with Pricing and Web Archives even favouring Arabic slightly; the Arabic subset is therefore comparably difficult rather than systematically easier or harder\. A controlled comparison excludes the English\-only Database category\. The clear exceptions, both favouring English, are PDF \(a 15\-point advantage\) and Miscellaneous \(9 points\), the only categories with a sizeable cross\-lingual gap\. PDF Visual is comparable across both languages \(a 1\.1\-point gap\)\.

The details of the reference agent implementation can be found in[Appendix˜C](https://arxiv.org/html/2607.20510#A3)\.

## 6Conclusion

We presented Telco\-GAIA, a bilingual, multi\-modal benchmark that evaluates tool\-using agents over the public data of a real telecommunications operator\. Its 100 human\-verified, strictly causal multi\-hop tasks draw on a served website, linked PDFs, a synthetic relational database, and external web archives, and are scored by normalized exact string matching, keeping evaluation objective, deterministic, and reproducible\. Across twelve commercial and open backends the benchmark proves hard: the strongest model reaches only 71%, a moderate\-budget backend about 38%, and the visually grounded categories lag furthest behind, exposing document and image understanding as the dominant bottleneck\. We further observe that model capability, rather than expended effort, drives success, and that the English and Arabic subsets are closely matched\. Beyond the resource itself, Telco\-GAIA offers a reusable template for constructing reproducible, closed\-domain enterprise benchmarks for the agentic era\.

## Limitations

#### Compact dataset size\.

At 100 tasks Telco\-GAIA is deliberately compact: the small size limits statistical power and fine\-grained per\-category comparison, but each task is a densely annotated, shortcut\-free multi\-hop chain \(mean 4\.2, up to 7 hops\) with a human\-verified solution path, echoing other depth\-over\-scale benchmarks \(TheAgentCompanyXuet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib25)\), 175 tasks; WorkArenaDrouinet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib6)\), 33 templates\)\. The same logic applies to language: rather than scaling the Arabic subset by machine translation, which injects noise even under careful controlAlrashedet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib3)\), every Arabic task is human\-translated or natively authored\. Evaluation stays cheap regardless: a full run costs from a few dollars to under $300 per model \([Table˜2](https://arxiv.org/html/2607.20510#S5.T2)\)\.

#### Single operator and domain\.

Built from the corpus of a single operator in one market, Telco\-GAIA is realistic and internally consistent but may not transfer to other operators, industries, or regulatory contexts; we thus present it as a difficulty probe and a template for closed enterprise benchmarks rather than a measure of general enterprise competence\.

#### Synthetic customer database\.

The relational customer database is fully synthetic, a deliberate choice that avoids exposing personal or commercially sensitive data and keeps the benchmark freely shareable\. Consequently, although we deliberately inject controlled data\-quality artefacts \([Appendix˜B](https://arxiv.org/html/2607.20510#A2)\), the Database category does not capture the full scale or organic noise of production telecom data\.

## Acknowledgments

The research was funded by Sponsored Research Agreement ORA \#6323 between King Abdullah University of Science and Technology \(KAUST\) and Saudi Telecommunications Company \(stc\)\.

## References

- Fineweb\-Edu\-Ar: machine\-translated corpus to support Arabic small language models\.External Links:[Link](https://arxiv.org/abs/2411.06402)Cited by:[Compact dataset size\.](https://arxiv.org/html/2607.20510#Sx1.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2026a\)Claude Sonnet 4\.6 System Card\.Note:[https://www\.anthropic\.com/claude\-sonnet\-4\-6\-system\-card](https://www.anthropic.com/claude-sonnet-4-6-system-card)Cited by:[Appendix C](https://arxiv.org/html/2607.20510#A3.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2026b\)Introducing Claude Opus 4\.8\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-8](https://www.anthropic.com/news/claude-opus-4-8)Cited by:[Appendix C](https://arxiv.org/html/2607.20510#A3.SS0.SSS0.Px1.p1.1)\.
- V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan \(2025\)τ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.External Links:2506\.07982,[Link](https://arxiv.org/abs/2506.07982)Cited by:[§D\.2](https://arxiv.org/html/2607.20510#A4.SS2.p6.8)\.
- D\. Carmel, S\. Filice, G\. Horowitz, Y\. Maarek, A\. Shtoff, O\. Somekh, and R\. Tavory \(2025\)LiveRAG: a diverse Q&A dataset with varying difficulty level for RAG evaluation\.External Links:2511\.14531,[Link](https://arxiv.org/abs/2511.14531)Cited by:[§D\.3](https://arxiv.org/html/2607.20510#A4.SS3.p5.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.12.1.1.1)\.
- D\. Cohen, L\. Burg, S\. Pykhnivskyi, H\. Gur, S\. Kovynov, O\. Atzmon, and G\. Barkan \(2025\)WixQA: a multi\-dataset benchmark for enterprise retrieval\-augmented generation\.External Links:2505\.08643,[Link](https://arxiv.org/abs/2505.08643)Cited by:[§D\.3](https://arxiv.org/html/2607.20510#A4.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.11.1.1.1),[§1](https://arxiv.org/html/2607.20510#S1.p2.1),[§2](https://arxiv.org/html/2607.20510#S2.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-V4\.Note:[https://api\-docs\.deepseek\.com/news/news260424](https://api-docs.deepseek.com/news/news260424)Preview release; V4\-Pro and V4\-Flash\. Model card:[https://huggingface\.co/deepseek\-ai/DeepSeek\-V4\-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)Cited by:[Appendix C](https://arxiv.org/html/2607.20510#A3.SS0.SSS0.Px1.p1.1)\.
- A\. Drouin, M\. Gasse, M\. Caccia, I\. H\. Laradji, M\. Del Verme, T\. Marty, L\. Boisvert, M\. Thakkar, Q\. Cappart, D\. Zügner, N\. Chapados, and A\. Lacoste \(2024\)WorkArena: how capable are web agents at solving common knowledge work tasks?\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),External Links:[Link](https://proceedings.mlr.press/v235/drouin24a.html)Cited by:[§D\.2](https://arxiv.org/html/2607.20510#A4.SS2.p2.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.9.1.1.1),[Compact dataset size\.](https://arxiv.org/html/2607.20510#Sx1.SS0.SSS0.Px1.p1.1)\.
- S\. Es, J\. James, L\. Espinosa\-Anke, and S\. Schockaert \(2024\)RAGAS: automated evaluation of retrieval augmented generation\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations \(EACL\),pp\. 150–158\.External Links:[Link](https://aclanthology.org/2024.eacl-demo.16)Cited by:[item 3](https://arxiv.org/html/2607.20510#S1.I1.i3.p1.1)\.
- A\. Ezzakri, N\. Piovesan, M\. Sana, A\. De Domenico, F\. Ayed, and H\. Zhang \(2026\)TeleTables: a benchmark for large language models in telecom table interpretation\.External Links:2601\.04202,[Link](https://arxiv.org/abs/2601.04202)Cited by:[§D\.5](https://arxiv.org/html/2607.20510#A4.SS5.p6.1)\.
- R\. Friel, M\. Belyi, and A\. Sanyal \(2024\)RAGBench: explainable benchmark for retrieval\-augmented generation systems\.External Links:2407\.11005,[Link](https://arxiv.org/abs/2407.11005)Cited by:[§D\.3](https://arxiv.org/html/2607.20510#A4.SS3.p2.1)\.
- Google DeepMind \(2025\)Gemini 3\.Note:[https://blog\.google/products\-and\-platforms/products/gemini/gemini\-3/](https://blog.google/products-and-platforms/products/gemini/gemini-3/)Model cards: Gemini 3\.5 \([https://deepmind\.google/models/gemini/](https://deepmind.google/models/gemini/)\), Gemini 3\.1 Flash\-Lite \([https://deepmind\.google/models/model\-cards/gemini\-3\-1\-flash\-lite/](https://deepmind.google/models/model-cards/gemini-3-1-flash-lite/)\)Cited by:[Appendix C](https://arxiv.org/html/2607.20510#A3.SS0.SSS0.Px1.p1.1)\.
- J\. He, N\. Hu, W\. Long, J\. Chen, and J\. Z\. Pan \(2024\)MINTQA: a multi\-hop question answering benchmark for evaluating LLMs on new and tail knowledge\.External Links:2412\.17032,[Link](https://arxiv.org/abs/2412.17032)Cited by:[§D\.4](https://arxiv.org/html/2607.20510#A4.SS4.p7.1)\.
- P\. Islam, A\. Kannappan, D\. Kiela, R\. Qian, N\. Scherrer, and B\. Vidgen \(2023\)FinanceBench: a new benchmark for financial question answering\.External Links:2311\.11944,[Link](https://arxiv.org/abs/2311.11944)Cited by:[§D\.5](https://arxiv.org/html/2607.20510#A4.SS5.p1.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.13.1.1.1)\.
- Kimi Team \(2025\)Kimi K2: Open Agentic Intelligence\.Note:Technical report; the table evaluates the Kimi\-K2\.6 checkpoint \([https://huggingface\.co/moonshotai/Kimi\-K2\.6](https://huggingface.co/moonshotai/Kimi-K2.6)\)External Links:2507\.20534,[Link](https://arxiv.org/abs/2507.20534)Cited by:[Appendix C](https://arxiv.org/html/2607.20510#A3.SS0.SSS0.Px1.p1.1)\.
- J\. Y\. Koh, R\. Lo, L\. Jang, V\. Duvvur, M\. C\. Lim, P\. Huang, G\. Neubig, S\. Zhou, R\. Salakhutdinov, and D\. Fried \(2024\)VisualWebArena: evaluating multimodal agents on realistic visual web tasks\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://aclanthology.org/2024.acl-long.50)Cited by:[§D\.2](https://arxiv.org/html/2607.20510#A4.SS2.p4.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.8.1.1.1),[§2](https://arxiv.org/html/2607.20510#S2.p1.1)\.
- S\. Krishna, K\. Krishna, A\. Mohananey, S\. Schwarcz, A\. Stambler, S\. Upadhyay, and M\. Faruqui \(2025\)Fact, fetch, and reason: a unified evaluation of retrieval\-augmented generation\.InProceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 4745–4759\.External Links:[Link](https://aclanthology.org/2025.naacl-long.243)Cited by:[§D\.4](https://arxiv.org/html/2607.20510#A4.SS4.p3.1)\.
- V\. D\. Lai, M\. Krumdick, C\. Lovering, V\. Reddy, C\. Schmidt, and C\. Tanner \(2025\)SEC\-QA: a systematic evaluation corpus for financial QA\.InProceedings of the 10th Workshop on Financial Technology and Natural Language Processing \(FinNLP\),pp\. 221–236\.External Links:[Link](https://aclanthology.org/2025.finnlp-2.15)Cited by:[§D\.5](https://arxiv.org/html/2607.20510#A4.SS5.p2.1)\.
- S\. Lee, D\. Arya, S\. Cho, G\. Han, S\. Hong, W\. Jang, S\. Lee, S\. Park, S\. Sek, I\. Song, S\. Yoon, and E\. Davis \(2024\)TelBench: a benchmark for evaluating telco\-specific large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track \(EMNLP\),External Links:[Link](https://aclanthology.org/2024.emnlp-industry.45)Cited by:[§D\.5](https://arxiv.org/html/2607.20510#A4.SS5.p7.1)\.
- S\. Lee, D\. Jang, D\. Arya, G\. Han, I\. Song, S\. Kim, S\. Kim, S\. Lee, S\. Hong, S\. Sek, S\. Cho, S\. Park, S\. Yoon, W\. Jang, and E\. Davis \(2025\)TelAgentBench: a multi\-faceted benchmark for evaluating LLM\-based agents in telecommunications\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track \(EMNLP\),External Links:[Link](https://aclanthology.org/2025.emnlp-industry.83)Cited by:[§D\.5](https://arxiv.org/html/2607.20510#A4.SS5.p4.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.4.3.1.1),[§2](https://arxiv.org/html/2607.20510#S2.p1.1)\.
- F\. Li, Y\. Wang, Y\. Xu, S\. Wang, J\. Liang, Z\. Chen, W\. Liu, Q\. Feng, T\. Duan, Y\. Huang, Q\. Song, and X\. Li \(2024\)Performance evaluations of large language models for customer service\.International Journal of Machine Learning and Cybernetics\.External Links:[Document](https://dx.doi.org/10.1007/s13042-024-02432-9),[Link](https://link.springer.com/article/10.1007/s13042-024-02432-9)Cited by:[§D\.5](https://arxiv.org/html/2607.20510#A4.SS5.p8.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang \(2024\)AgentBench: evaluating LLMs as agents\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=zAdUB0aCTQ)Cited by:[§D\.2](https://arxiv.org/html/2607.20510#A4.SS2.p3.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.10.1.1.1)\.
- Y\. Ma, Y\. Zang, L\. Chen, M\. Chen, Y\. Jiao, X\. Li, X\. Lu, Z\. Liu, Y\. Ma, X\. Dong, P\. Zhang, L\. Pan, Y\. Jiang, J\. Wang, Y\. Cao, and A\. Sun \(2024\)MMLongBench\-Doc: benchmarking long\-context document understanding with visualizations\.InAdvances in Neural Information Processing Systems: Datasets and Benchmarks Track \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/ae0e43289bffea0c1fa34633fc608e92-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§D\.5](https://arxiv.org/html/2607.20510#A4.SS5.p3.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.14.1.1.1)\.
- A\. Maatouk, F\. Ayed, N\. Piovesan, A\. De Domenico, M\. Debbah, and Z\. Luo \(2024\)TeleQnA: a benchmark dataset to assess large language models telecommunications knowledge\.IEEE Network\.External Links:[Link](https://ieeexplore.ieee.org/document/11021474)Cited by:[§D\.5](https://arxiv.org/html/2607.20510#A4.SS5.p5.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.15.1.1.1)\.
- G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom \(2024\)GAIA: a benchmark for general AI assistants\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=fibxvahvs3)Cited by:[§D\.1](https://arxiv.org/html/2607.20510#A4.SS1.p1.2),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.6.1.1.1),[§1](https://arxiv.org/html/2607.20510#S1.p2.1),[§1](https://arxiv.org/html/2607.20510#S1.p3.1),[§2](https://arxiv.org/html/2607.20510#S2.p1.1)\.
- S\. Min, J\. Boyd\-Graber, C\. Alberti, D\. Chen, E\. Choi, M\. Collins, K\. Guu, H\. Hajishirzi, K\. Lee, J\. Palomaki,et al\.\(2021\)NeurIPS 2020 EfficientQA competition: systems, analyses and lessons learned\.InProceedings of the NeurIPS 2020 Competition and Demonstration Track, PMLR,Vol\.133,pp\. 86–111\.External Links:[Link](https://arxiv.org/abs/2101.00133)Cited by:[§1](https://arxiv.org/html/2607.20510#S1.p2.1)\.
- S\. Min, J\. Michael, H\. Hajishirzi, and L\. Zettlemoyer \(2020\)AmbigQA: answering ambiguous open\-domain questions\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 5783–5797\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.466)Cited by:[§1](https://arxiv.org/html/2607.20510#S1.p2.1)\.
- OpenAI \(2025\)OpenAI GPT\-5 System Card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[Appendix C](https://arxiv.org/html/2607.20510#A3.SS0.SSS0.Px1.p1.1)\.
- J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. Zaharia \(2024\)ARES: an automated evaluation framework for retrieval\-augmented generation systems\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),External Links:[Link](https://aclanthology.org/2024.naacl-long.20)Cited by:[§D\.3](https://arxiv.org/html/2607.20510#A4.SS3.p4.1),[item 3](https://arxiv.org/html/2607.20510#S1.I1.i3.p1.1)\.
- H\. Su, H\. Yen, M\. Xia, W\. Shi, N\. Muennighoff, H\. Wang, H\. Liu, Q\. Shi, Z\. S\. Siegel, M\. Tang, R\. Sun, J\. Yoon, S\. O\. Arik, D\. Chen, and T\. Yu \(2025\)BRIGHT: a realistic and challenging benchmark for reasoning\-intensive retrieval\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=7irEr2vqvk)Cited by:[§D\.4](https://arxiv.org/html/2607.20510#A4.SS4.p6.1)\.
- Y\. Tang and Y\. Yang \(2024\)MultiHop\-RAG: benchmarking retrieval\-augmented generation for multi\-hop queries\.InProceedings of the First Conference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=t4eB3zYWBK)Cited by:[§D\.4](https://arxiv.org/html/2607.20510#A4.SS4.p5.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics \(TACL\)10,pp\. 539–554\.External Links:[Link](https://aclanthology.org/2022.tacl-1.31)Cited by:[§D\.4](https://arxiv.org/html/2607.20510#A4.SS4.p2.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. Sui \(2024\)Large language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://aclanthology.org/2024.acl-long.511)Cited by:[§1](https://arxiv.org/html/2607.20510#S1.p2.1)\.
- J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. Glaese \(2025\)BrowseComp: a simple yet challenging benchmark for browsing agents\.External Links:[Link](https://arxiv.org/abs/2504.12516)Cited by:[§D\.1](https://arxiv.org/html/2607.20510#A4.SS1.p3.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. Yu \(2024\)OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://neurips.cc/virtual/2024/poster/97468)Cited by:[§D\.2](https://arxiv.org/html/2607.20510#A4.SS2.p7.1)\.
- F\. F\. Xu, Y\. Song, B\. Li, Y\. Tang, K\. Jain, M\. Bao, Z\. Z\. Wang, X\. Zhou, Z\. Guo, M\. Cao, M\. Yang, H\. Y\. Lu, A\. Martin, Z\. Su, L\. M\. Maben, R\. Mehta, W\. Chi, L\. Jang, Y\. Xie, S\. Zhou, and G\. Neubig \(2025\)TheAgentCompany: benchmarking LLM agents on consequential real world tasks\.InAdvances in Neural Information Processing Systems: Datasets and Benchmarks Track \(NeurIPS\),External Links:[Link](https://openreview.net/forum?id=LZnKNApvhG)Cited by:[§D\.2](https://arxiv.org/html/2607.20510#A4.SS2.p5.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.1.1.2.1.1),[Compact dataset size\.](https://arxiv.org/html/2607.20510#Sx1.SS0.SSS0.Px1.p1.1)\.
- X\. Yang, K\. Sun, H\. Xin, Y\. Sun, N\. Bhalla, X\. Chen, S\. Choudhary, R\. D\. Gui, Z\. W\. Jiang, Z\. Jiang, L\. Kong, B\. Moran, J\. Wang, Y\. E\. Xu, A\. Yan, C\. Yang, E\. Yuan, H\. Zha, N\. Tang, L\. Chen, N\. Scheffer, Y\. Liu, N\. Shah, R\. Wanga, A\. Kumar, W\. Yih, and X\. L\. Dong \(2024\)CRAG – comprehensive RAG benchmark\.InAdvances in Neural Information Processing Systems: Datasets and Benchmarks Track \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/1435d2d0fca85a84d83ddcb754f58c29-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§D\.3](https://arxiv.org/html/2607.20510#A4.SS3.p3.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2369–2380\.External Links:[Link](https://aclanthology.org/D18-1259)Cited by:[§D\.4](https://arxiv.org/html/2607.20510#A4.SS4.p1.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2025\)τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=roNSXZpUDN)Cited by:[§D\.2](https://arxiv.org/html/2607.20510#A4.SS2.p6.8),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.2.2.1.1.1)\.
- J\. Ye, Z\. Du, X\. Yao, W\. Lin, Y\. Xu, Z\. Chen, Z\. Wang, S\. Zhu, Z\. Xi, S\. Yuan, T\. Gui, Q\. Zhang, X\. Huang, and J\. Chen \(2025\)ToolHop: a query\-driven benchmark for evaluating large language models in multi\-hop tool use\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://aclanthology.org/2025.acl-long.150)Cited by:[§D\.4](https://arxiv.org/html/2607.20510#A4.SS4.p8.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.16.1.1.1)\.
- O\. Yoran, S\. J\. Amouyal, C\. Malaviya, B\. Bogin, O\. Press, and J\. Berant \(2024\)AssistantBench: can web agents solve realistic and time\-consuming tasks?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 8938–8968\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.505)Cited by:[§D\.1](https://arxiv.org/html/2607.20510#A4.SS1.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2306.05685)Cited by:[§1](https://arxiv.org/html/2607.20510#S1.p2.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2024\)WebArena: a realistic web environment for building autonomous agents\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=rmiwIL98uQ)Cited by:[§D\.2](https://arxiv.org/html/2607.20510#A4.SS2.p1.1),[Table 1](https://arxiv.org/html/2607.20510#S1.T1.4.7.1.1.1),[§1](https://arxiv.org/html/2607.20510#S1.p2.1),[§2](https://arxiv.org/html/2607.20510#S2.p1.1)\.
- A\. Zhu, A\. Hwang, L\. Dugan, and C\. Callison\-Burch \(2024\)FanOutQA: a multi\-hop, multi\-document question answering benchmark for large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 18–37\.External Links:[Link](https://aclanthology.org/2024.acl-short.2)Cited by:[§D\.4](https://arxiv.org/html/2607.20510#A4.SS4.p4.1)\.

## Appendix ABenchmark Harness

The benchmark is packaged as a self\-contained Docker environment that exposes the local website and the SQL database to the agent via local HTTP endpoints, while Wikipedia and ArXiv are accessed directly on the internet\. Answers are evaluated byexact string matchingafter minimal normalization, with no LLM\-as\-a\-Judge at any stage\.[Figure˜3](https://arxiv.org/html/2607.20510#A1.F3)illustrates the evaluation pipeline\.

![Refer to caption](https://arxiv.org/html/2607.20510v1/x3.png)Figure 3:Benchmark architecture\. The agent reads task inputs and queries the Docker container; the evaluator scores the submission against ground truth\.### A\.1Docker Container

The container \(bench\_compose\) bundles two services managed bysupervisord:

- •Port 8080 — nginx: Serves the static website snapshot \(HTML pages, images, PDFs\) from the February 2026 crawl\. Both English and Arabic site trees are available\.
- •Port 8081 — Datasette: Exposes the synthetic SQLite customer database as a read\-only REST API with JSON output, supporting read\-only SQL queries\.

### A\.2Agent Inputs

The agent under test receives two files \(never the ground truth\):

- •questions\.json— a JSON array of 100 tasks, each containing onlytask\_idandquestion\.
- •environment\.md— a Markdown document describing the available tools and endpoints\.

A*task*is the unit of evaluation: itsquestionfield is the natural\-language prompt shown to the agent, while its full serialization across all fields below is the ground\-truth*record*\. Each full ground\-truth record carries the fieldstask\_id,category,language,question,final\_answer,answer\_type,steps,num\_steps,tools, andnum\_tools; onlytask\_idandquestionare exposed to the agent\. The harness additionally restricts the reference agent to each task’s permitted tool subset \(itstoolsfield; see[Appendix˜C](https://arxiv.org/html/2607.20510#A3)\)\. This is task metadata used only to gate which tools are exposed; it never reveals the answer\.

### A\.3Submission Format

The agent must produce asubmission\.jsonin GAIA\-compatible format:

```
[
  {
    "task_id": "task-001",
    "model_answer": "50",
    "reasoning_trace": "I navigated to..."
  }
]
```

Each entry must includetask\_idandmodel\_answer; thereasoning\_tracefield is optional but recommended for diagnostic purposes\.

### A\.4Scoring

Evaluation is performed byevaluate\.py, a pure\-Python script with no external dependencies:

```
python evaluate.py \
    --submission path/to/submission.json \
    --ground-truth ground_truth.json \
    --output results.json
```

The scorer applies GAIA\-convention normalization \(lowercasing, whitespace stripping, removal of currency symbols and commas\) and performs exact string matching\. Each task receives a binary score \(1 if correct, 0 otherwise\)\. The outputresults\.jsoncontains the overall accuracy, per\-task scores, and a category breakdown reporting accuracy for each of the seven categories independently\. Missing submissions are scored as 0; extra task IDs trigger a warning but do not affect scoring\.

### A\.5Local Services in Action

Figures[4](https://arxiv.org/html/2607.20510#A1.F4)and[5](https://arxiv.org/html/2607.20510#A1.F5)show the two containerized services as they appear to the agent at runtime\.

![Refer to caption](https://arxiv.org/html/2607.20510v1/x4.png)Figure 4:The operator website served locally via nginx on port 8080\. The agent navigates this static snapshot exactly as it would the live site\.![Refer to caption](https://arxiv.org/html/2607.20510v1/x5.png)Figure 5:The Datasette interface on port 8081 exposing the synthetic SQLite database \(int\_db\) with 2,015 rows across 8 tables\.

## Appendix BAdversarial Database Design

A clean database is trivially solvable: once an agent discovers the schema, a singleSELECTreturns the answer\. To make database reasoning genuinely hard, we deliberately inject controlled data\-quality artefacts \(the kind that accumulate in any production billing system\) so that a naiveSELECT SUM/COUNT\(\*\)returns the*wrong*number while the gold answer is unchanged\. Only an agent that inspects the data, detects the anomaly, and applies the right deduplication or filter can recover the correct value\. The artefacts span ten families, including duplicate \(double\-fired\) invoices, credit\-note rows with negative amounts, internal test accounts, andNULL\-versus\-zero usage; every database task triggers at least one\.

#### Wording is the control knob\.

The difficulty of a trap is governed not by the data alone but by the question’s wording\. Over\-explicit phrasing \(“excluding duplicates and credit notes”\) hands the agent the exact filter and neutralises the trap, while over\-implicit phrasing risks ambiguity for human reviewers\. The sweet spot is a*subtle*qualifier \(“the legitimate invoiced amount”, “all real active subscribers”, “since the subscription started”\) that a careful human acts on by inspecting the data, but that an agent cannot mechanically compile into a SQL clause\. The wording must keep the gold answer the only defensible reading while leaving the trap live\.

#### Not all traps are equal\.

A five\-fold evaluation of the reference agent lets us grade traps by empirical catch rate, which spans 100% down to 0%\. The decisive factor is whether a trap poisons the agent’s*inevitable*query path: an internal test account that out\-ranks the true customer \(or a subtle qualifier that demands data inspection\) is caught on essentially every run, whereas a trap that merely hopes the agent forgets a status filter is probabilistic \(40–60%\), and explicit wording that spells out the filter neutralises it entirely\.

## Appendix CReference Agent Implementation

The reference agent is a tool\-calling LLM loop built on LiteLLM\. At each iteration the model receives a task\-specific system prompt, selects a tool, observes the result, and decides whether to call another tool or emit a final answer\. Four tools are available, but for each task the reference agent is restricted to the subset listed in that task’stoolsfield \(e\.g\., a Web Archives task exposeslocal\_browserandinternet\_browseronly\)\. This gating prevents shortcut solutions that bypass the intended reasoning chain, such as reading a plan price directly from the database when the task requires retrieving it from the website\. The model receives schemas only for these permitted tools\. The four tools are:

- •local\_browser— a Playwright\-based headless browser restricted to the served operator website \(localhost:8080\)\.
- •internet\_browser— a second Playwright instance restricted to\*\.wikipedia\.organd\*\.arxiv\.orgfor Web Archives tasks\.
- •pdf\_reader— downloads PDFs and extracts content viapdfplumber\(text layer\) with aPyMuPDFvision fallback for visual PDFs\.
- •database\_query— executes read\-only SQL against the Datasette API onlocalhost:8081\.

[Figure˜6](https://arxiv.org/html/2607.20510#A3.F6)shows how these four tools are distributed across the 100 tasks\.

![Refer to caption](https://arxiv.org/html/2607.20510v1/x6.png)Figure 6:Tool usage across the 100 tasks\. The greylocal\_browserellipse is a superset \(used by every task\)\.The agent outputs results in the GAIA\-compatible submission format described in[Appendix˜A](https://arxiv.org/html/2607.20510#A1)\.

#### Evaluated backends\.

The full sweep \([Table˜2](https://arxiv.org/html/2607.20510#S5.T2)\) covers twelve commercial and open backends: gpt\-5\.5, gpt\-5\.4, gpt\-5\.2, gpt\-5\.4\-mini, and gpt\-5\.4\-nanoOpenAI \([2025](https://arxiv.org/html/2607.20510#bib.bib35)\); gemini\-3\.5\-flash and gemini\-3\.1\-flash\-liteGoogle DeepMind \([2025](https://arxiv.org/html/2607.20510#bib.bib37)\); claude\-opus\-4\-8Anthropic \([2026b](https://arxiv.org/html/2607.20510#bib.bib38)\)and claude\-sonnet\-4\-6Anthropic \([2026a](https://arxiv.org/html/2607.20510#bib.bib39)\); Kimi\-K2\.6Kimi Team \([2025](https://arxiv.org/html/2607.20510#bib.bib36)\); and DeepSeek\-V4\-Pro and DeepSeek\-V4\-FlashDeepSeek\-AI \([2026](https://arxiv.org/html/2607.20510#bib.bib40)\)\.

### C\.1Score variance estimation

For each taskii, we estimate its solve probabilityp^i\\hat\{p\}\_\{i\}as its pass rate across theN=5N\{=\}5runs, yielding sample varianceσ^i2=p^i​\(1−p^i\)​N/\(N−1\)\\hat\{\\sigma\}\_\{i\}^\{2\}=\\hat\{p\}\_\{i\}\(1\-\\hat\{p\}\_\{i\}\)\\,N/\(N\{\-\}1\)\. The category mean accuracy isp¯=1T​∑ip^i\\bar\{p\}=\\frac\{1\}\{T\}\\sum\_\{i\}\\hat\{p\}\_\{i\}, and since task outcomes are independent across runs, the standard deviation of the category score isσ=1T​∑iσ^i2\\sigma=\\frac\{1\}\{T\}\\sqrt\{\\sum\_\{i\}\\hat\{\\sigma\}\_\{i\}^\{2\}\}, whereTTis the number of tasks in the category\. This task\-level variance propagation avoids under\-estimating variability when different tasks fluctuate but their totals coincidentally cancel\.

## Appendix DLiterature Review

Evaluation benchmarks for AI agents and retrieval\-augmented generation \(RAG\) systems have proliferated rapidly since 2023\. We organize the relevant prior work into five clusters, each illuminating a different dimension of the design space\.

### D\.1General AI Agent Benchmarks

GAIAMialonet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib1)\)\(Meta/HuggingFace, ICLR 2024\) is the direct inspiration for Telco\-GAIA\. It comprises 466 real\-world questions organized into three difficulty levels, requiring general\-purpose tool use including open internet browsing, file parsing \(PDF, spreadsheet, audio\), and multi\-step reasoning\. Questions are hand\-crafted and non\-synthetic; answers are exact strings verified by human annotators\. Human performance is 92%, while GPT\-4 equipped with plugins achieved only∼\\sim15% at launch, with top agents since reaching∼\\sim75%\. The key distinction from Telco\-GAIA is that GAIA is fully open\-domain and relies on the live internet, making it non\-reproducible across time and inapplicable to closed enterprise corpora\.

AssistantBenchYoranet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib2)\)\(2024\) contains 214 realistic web tasks \(e\.g\., monitoring real\-estate listings, discovering local businesses\) evaluated on the open internet\. Like GAIA, it uses automatic evaluation and reveals that no model exceeded 25 accuracy points at publication time\. It has no domain focus, no database modality, and no sandboxed environment\.

BrowseCompWeiet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib4)\)\(OpenAI, 2025\) consists of 1,266 questions requiring persistent internet navigation to locate hard\-to\-find, multi\-hop facts\. It is an open\-web benchmark measuring search persistence and creativity rather than domain\-specific knowledge\.

### D\.2Sandboxed Web Agent Environments

WebArenaZhouet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib5)\)\(ICLR 2024\) is the closest architectural ancestor of Telco\-GAIA\. It constructs fully sandboxed, Docker\-hosted realistic websites across four domains: e\-commerce, social forum, code hosting, and content management\. Agents interact via browser automation \(UI clicks and form submission\); 812 tasks are evaluated\. The best GPT\-4\-based agent achieved only 14\.4% success rate versus 78% for humans\. Unlike Telco\-GAIA, WebArena requires browser\-level UI interaction rather than HTTP/API tool use, has no dedicated relational database modality, no PDF or image retrieval tasks, and is general\-purpose rather than domain\-specific\.

WorkArenaDrouinet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib6)\)\(ServiceNow/ICML 2024\) evaluates LLM agents on 33 enterprise knowledge\-work tasks inside a live ServiceNow instance \(e\.g\., filling forms, ordering from service catalogs, reading dashboards\)\. It introduces BrowserGym, an environment with multimodal observations\. GPT\-4 achieves 42\.7%\. Its task instances are programmatically generated from parameterized templates rather than hand\-authored\. WorkArena is limited to UI interaction inside a single SaaS platform and has no external retrieval corpus or structured database modality\.

AgentBenchLiuet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib7)\)\(ICLR 2024\) is a comprehensive multi\-environment suite covering 8 agent tasks: operating system shell, database \(SQL\), knowledge graph, card games, lateral thinking puzzles, web shopping, web browsing, and household navigation\. It evaluates 29 LLMs and finds a large gap between commercial and open\-source models\. While AgentBench includes a SQL environment, it does not combine structured and unstructured retrieval in a single coherent domain scenario\.

VisualWebArenaKohet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib26)\)\(ACL 2024\) extends WebArena with 910 visually grounded web tasks requiring multimodal agents to interpret images, screenshots, and visual page layouts\. It introduces execution\-based visual evaluation metrics but retains WebArena’s general\-purpose multi\-domain design and lacks PDF, database, or domain\-specific retrieval\.

TheAgentCompanyXuet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib25)\)\(NeurIPS 2025\) creates a self\-contained simulated software company with internal websites, code repositories, and communication tools\. It evaluates agents on 175 consequential professional tasks \(writing code, browsing internal docs, communicating with simulated coworkers\); the best agent completes only 30% autonomously\. Unlike Telco\-GAIA, it targets general office work rather than domain\-specific QA and has no structured database modality or PDF retrieval\.

τ\\tau\-benchYaoet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib27)\)\(Sierra/Princeton, ICLR 2025\) evaluates tool\-calling agents on realistic customer\-service dialogues in retail and airline domains, with a telecom domain added in 2025 \(τ2\\tau^\{2\}\-benchBarreset al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib28)\)\)\. GPT\-4o achieves only∼\\sim35% pass@1 in the airline domain \(∼\\sim61% in retail\) and exhibits poor consistency \(pass@8<<25% in retail\); the harder telecom domain introduced inτ2\\tau^\{2\}\-bench drops even GPT\-4\.1 to∼\\sim34%\. Its domain databases and user tasks are synthetically generated with GPT\-4\. Unlike Telco\-GAIA,τ\\tau\-bench focuses on conversational tool use rather than multi\-source retrieval, has no website or PDF modality, and tests single\-turn tool calls rather than multi\-hop reasoning chains\.

OSWorldXieet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib8)\)\(NeurIPS 2024\) evaluates multimodal agents on 369 real\-world computer tasks spanning Ubuntu, Windows, and macOS using virtual machines\. It targets full desktop control rather than retrieval\-focused evaluation\.

### D\.3Enterprise RAG Benchmarks

WixQACohenet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib9)\)\(Wix\.com, May 2025\) is the most structurally similar RAG benchmark to Telco\-GAIA\. It provides three datasets grounded in a snapshot of the Wix Help Center knowledge base \(6,220 articles\): 200 expert\-written pairs, 200 expert\-validated simulated pairs, and 6,222 LLM\-generated synthetic pairs\. The KB snapshot is released alongside the dataset under MIT licence\. WixQA differs from Telco\-GAIA in that it is a pure text RAG benchmark: there is no live agent environment, no relational database, no image or PDF modality, and no bilingual \(Arabic/English\) content\.

RAGBenchFrielet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib10)\)\(2024\) provides 100k evaluation examples across five industry\-specific corpora \(user manuals, biomedical texts, finance, legal, conversational AI\)\. It introduces the TRACe evaluation framework with explainability metrics\. RAGBench is large\-scale but is a static corpus dump without an agent environment or structured data sources\.

CRAGYanget al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib11)\)\(2024\) is a comprehensive RAG benchmark covering multiple domains and task types, introduced as a KDD Cup competition\. It assesses retrieval quality, factual accuracy, and reasoning, but targets open\-domain web retrieval rather than a closed enterprise corpus\.

ARESSaad\-Falconet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib12)\)\(2023\) is an automated evaluation framework for RAG systems that uses synthetic data generation and fine\-tuned classifiers as judges\. It is a meta\-framework rather than a fixed\-corpus benchmark\.

LiveRAGCarmelet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib29)\)\(TII/SIGIR 2025\) is a competition\-derived benchmark of 895 synthetic questions over a fixed 10\-billion\-token web corpus \(FineWeb\-10BT\), with difficulty scores calibrated from 70 competing teams\. It enforces a fixed retrieval corpus and fixed LLM \(Falcon3\-10B\), isolating RAG pipeline quality\. Unlike Telco\-GAIA, LiveRAG is open\-domain, text\-only, and evaluates retrieval pipeline components rather than multi\-modal agent reasoning\.

### D\.4Multi\-Hop Retrieval Benchmarks

HotpotQAYanget al\.\([2018](https://arxiv.org/html/2607.20510#bib.bib14)\)\(2018\) contains 112,779 Wikipedia\-based questions requiring 2\-hop reasoning with supporting sentence annotations\. It is the canonical multi\-hop QA baseline but has no agent environment and no closed or domain\-specific corpus\.

MuSiQueTrivediet al\.\([2022](https://arxiv.org/html/2607.20510#bib.bib15)\)\(2022\) systematically composes single\-hop questions into 25k 2–4\-hop questions to prevent reasoning shortcut exploitation\. It exhibits a 3×\\timeslarger human\-machine gap than HotpotQA\. Like HotpotQA, it is Wikipedia\-only with no agent environment\.

FRAMESKrishnaet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib16)\)\(Google DeepMind, NAACL 2025\) contains 824 challenging multi\-hop questions each requiring information from 2–15 Wikipedia articles\. It provides a unified evaluation of factual accuracy, retrieval, and reasoning\. Multi\-step retrieval improves accuracy from 47% \(single step\) to 66%\.

FanOutQAZhuet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib17)\)\(2024\) comprises 1,034 multi\-document Wikipedia questions with human\-written decompositions\. It evaluates closed\-book, open\-book, and evidence\-provided settings\.

MultiHop\-RAGTang and Yang \([2024](https://arxiv.org/html/2607.20510#bib.bib18)\)\(2024\) focuses on multi\-hop RAG evaluation using English news articles, evaluating both the retriever and the reader components\.

BRIGHTSuet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib19)\)\(ICLR 2025\) contains 1,385 reasoning\-intensive retrieval queries drawn from StackExchange, LeetCode, and math competitions\. State\-of\-the\-art retrievers score only 18 nDCG@10, demonstrating the difficulty of reasoning\-intensive retrieval\.

MINTQAHeet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib30)\)\(2024\) evaluates LLMs on 28k multi\-hop questions specifically targeting*new*knowledge \(post\-training\-cutoff facts\) and*tail*knowledge \(rare entities\)\. It tests four QA strategies including iterative decomposition with RAG\. Unlike Telco\-GAIA, MINTQA is Wikipedia\-based with no agent environment, no tool use, and no domain\-specific corpus\.

ToolHopYeet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib34)\)\(ByteDance, ACL 2025\) is the benchmark most directly comparable to Telco\-GAIA’s multi\-hop design philosophy\. It comprises 995 user queries paired with 3,912 tools, where each query requires sequential tool invocations with meaningful interdependencies between hops, in which the output of one tool call serves as input to the next\. GPT\-4o, the best\-performing model, achieves only 49% accuracy\. ToolHop validates that multi\-hop tool chaining remains a frontier challenge\. However, it uses synthetic tools rather than real\-world retrieval sources, has no domain grounding, no website navigation, and no structured database or PDF modality\.

### D\.5Domain\-Specific and Structured Data Benchmarks

FinanceBenchIslamet al\.\([2023](https://arxiv.org/html/2607.20510#bib.bib20)\)\(Patronus AI, 2023\) contains 10,231 questions grounded in SEC filings \(PDFs\) about publicly traded companies\. GPT\-4 with a retrieval system incorrectly answered or refused 81% of questions\. FinanceBench is a single\-domain, text\-based, closed\-corpus benchmark with no agent environment or database modality; evidence is the extracted page text of filings \(not their visual layout\), and a substantial share of its questions are programmatically generated from financial metrics rather than hand\-written\.

SEC\-QALaiet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib21)\)\(2024\) extends the financial QA paradigm to multi\-document questions spanning multiple long\-context filings\. It introduces a continuous dataset refresh capability to prevent data contamination\.

MMLongBench\-DocMaet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib31)\)\(2024\) provides 1,082 expert\-annotated questions across 135 long PDF documents \(avg\. 47\.5 pages\) requiring reasoning over text, tables, figures, and layout structure\. 33% of questions require cross\-page evidence; even GPT\-4o achieves only 44\.9% F1\. MMLongBench\-Doc is the closest benchmark to Telco\-GAIA’s PDF Visual category, but it has no agent environment, no website navigation, no database, and is English\-only\.

TelAgentBenchLeeet al\.\([2025](https://arxiv.org/html/2607.20510#bib.bib22)\)\(SK Telecom, EMNLP 2025 Industry\) is the telecom\-domain benchmark most closely related to Telco\-GAIA\. It is a Korean\-language synthetic benchmark evaluating LLM agents on five capabilities: Reasoning, Planning, Action \(tool use\), RAG, and Instruction Following, with over 1,700 instances\. It provides a sandbox environment of 23 telecom business\-support\-system \(BSS\) APIs backed by a simulated database, all grounded in SK Telecom’s own service data\. Unlike Telco\-GAIA, however, it exposes this environment through abstract function\-call APIs rather than a scraped served website, does not expose a relational database that the agent queries directly, has no image or PDF modality, and is Korean\-only\.

TeleQnAMaatouket al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib32)\)\(2023\) is the foundational telecom knowledge benchmark, comprising 10,000 multiple\-choice questions across five categories: lexicon, research overview, research publications, standards overview, and standards specifications\. GPT\-4 achieved 64% on the hardest standards category\. Its question\-answer pairs are produced by an automated LLM\-based generation pipeline with human quality review\. TeleQnA tests static domain knowledge rather than agent capabilities; it has no tool use, no retrieval environment, and no multi\-hop reasoning\.

TeleTablesEzzakriet al\.\([2026](https://arxiv.org/html/2607.20510#bib.bib33)\)\(Jan 2026\) provides 500 human\-verified question\-answer pairs for evaluating LLMs on table interpretation in 3GPP technical specifications\. It targets a specific modality \(structured tables in standards documents\) and demonstrates that smaller models \(<<10B parameters\) fail on complex telecom table reasoning\. Unlike Telco\-GAIA, it is a static QA benchmark with no agent environment\.

TelBenchLeeet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib23)\)\(EMNLP 2024 Industry\) evaluates telco\-specific LLMs on customer service expertise across 34 models, but is a text\-only evaluation without an agent environment\.

TeleEval\-CSLiet al\.\([2024](https://arxiv.org/html/2607.20510#bib.bib24)\)\(2024\) provides 8\.1k customer\-service examples across 15 subtasks and three call stages, also without an agent environment\.