Paper page - RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios

Source: https://huggingface.co/papers/2610.19698 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

RuleTrek is a multi-domain benchmark that evaluates large language model agents on identifying, deducing, and applying domain-specific rules across complex, multi-file scenarios, revealing substantial performance gaps.

Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introducesRuleTrek, a multi-domain benchmark for agentic rule reasoning in complex scenarios. Itsrule-grounded constructionpipeline extractsstructured rule graphs, generates and incrementally composesrule-specific sub-scenarioswhile ensuring consistency and solvability. To systematically evaluate agentic rule reasoning in complex scenarios,RuleTrekassesses three capabilities in interactive workspaces:Rule Identification(RI),Rule Deduction(RD), andEnd-to-End Rule Reasoning(E2E). Experiments on 11 frontier LLMs show that the highest RI and RD scores are only 62.47\% and 60.43\%, respectively, while E2E scores range from 26.97\% to 55.29\%. These results demonstrate that currentLLM agentsface substantial challenges in rule reasoning over complex scenarios. We make our code and dataset publicly available at https://github.com/yuan247/RuleTrek{RuleTrek}.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2610\.19698

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.19698 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.19698 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.19698 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.