OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

Hugging Face Daily Papers Papers
llm-agents operations-research benchmark evaluation feasibility client-simulation

Summary

OR-License 是一个端到端基准，评估 LLM 智能体能否直接从真实客户原始材料中解决运筹优化问题；结果显示部分模型目标质量很高，但没有一个达到专家级的可行性与部署调整后质量。

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.
Original Article
View Cached Full Text

Cached at: 10/20/26, 03:33 AM

Paper page - OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

Source: https://huggingface.co/papers/2610.24087 Published on Oct 16

·

Submitted byhttps://huggingface.co/CedarWang

PSWangon Oct 20

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

OR-License evaluates whether LLM agents can solve real-world operations research problems from original client materials, finding that while some achieve high objective quality, none match expert-level feasibility and deployment-adjusted quality.

Can large language model (LLM) agents deliver expert-leveloperations research(OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduceOR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduceORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checksfeasibilityand compares objective quality against original expert solutions accepted by clients before benchmark construction.Deployment-Adjusted Quality(DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strictfeasibilityand a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifiesformulationandimplementationfailures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problemformulation, and evidence-linked self-improvement for reliable long-horizon agents.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2610\.24087

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.24087 in a model README.md to link it from this page.

Datasets citing this paper1

#### OR-License/OR-License Viewer• Updatedabout 1 hour ago • 92.6M • 385 • 8

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.24087 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

PORA: Learning When More Reasoning Pays Off for LLM Agents

arXiv cs.AI

PORA 提出了一种配对结果引导的推理分配方法，让 LLM 智能体学习在何种状态与剩余预算下使用长推理更划算，在 BFCL memory 等任务上大幅提升性能并显著节省 token 与模型请求。

COOPA: A Modular LLM Agent Architecture for Operations Research Problems

arXiv cs.LG

This paper introduces COOPA, a modular LLM agent architecture for operations research problems that combines iterative confidence-based modeling, element-level provenance, and multi-solver routing. Evaluated across eight LLM backbones and four baselines, COOPA achieves the best macro-average accuracy on six backbones and improves over the strongest baseline by up to 6.7 percentage points.