DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
Summary
This paper introduces DiagnosticIQ, a benchmark for evaluating LLMs in translating industrial symbolic maintenance rules into actionable steps. It highlights that while frontier models perform well on standard tasks, they exhibit brittleness and pattern-matching behaviors under structural perturbations.
View Cached Full Text
Cached at: 05/12/26, 07:19 AM
# DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
Source: [https://arxiv.org/abs/2605.08614](https://arxiv.org/abs/2605.08614)
[View PDF](https://arxiv.org/pdf/2605.08614)
> Abstract:Monitoring complex industrial assets relies on engineer\-authored symbolic rules that trigger based on sensor conditions and prompt technicians to perform corrective actions\. The bottleneck is not detection but response: translating rules into maintenance steps requires asset\-specific knowledge gained through years of practice\. We investigate whether LLMs can serve as decision support for this rule\-to\-action step and introduce \\ours\{\}, a benchmark of 6\{,\}690 expert\-validated multiple\-choice questions from 118 rule\-action pairs across 16 asset types\. We contribute \(i\) a symbolic\-to\-MCQA pipeline normalizing rules to Disjunctive Normal Form with embedding\-based distractor sampling, \(ii\) five variants probing distinct failure modes \(Pro, Pert, Verbose, Aug, Rationale\), and \(iii\) a benchmark of 29 LLMs and 4 embedding baselines\. A human evaluation \(9 practitioners, mean 45\.0\\%\) confirms \\ours\{\} requires specialist knowledge beyond operational experience\. Three findings stand out\. The frontier has closed: the top three LLMs lie within one Macro point, with Bradley\-Terry Elo placing claude\-opus\-4\-6 30 points above the next model\. Yet \\ours\{\}\\,Pro exposes brittleness, with every model losing 13\-\-60\\% relative accuracy under distractor expansion\. \\ours\{\}\\,Aug exposes pattern\-matching: under condition inversion, frontier models still select the original answer 49\-\-63\\% of the time\. The deployment bottleneck is not capability but calibration: frontier models handle template\-style fault detection but break under structural perturbation\.
## Submission history
From: Dhaval Patel Dr \[[view email](https://arxiv.org/show-email/03eaa11f/2605.08614)\] **\[v1\]**Sat, 9 May 2026 02:17:39 UTC \(6,633 KB\)Similar Articles
FactoryLLM: A Safe and Open-Source AI Playground for Evaluating LLMs in Smart Factories
FactoryLLM is an open-source AI playground for evaluating LLM-based RAG models in smart factory fault diagnostics, supporting local LLMs and dual evaluation metrics. A case study with three LLMs showed groundedness scores above 0.88 across 30 maintenance queries from 600 pages of cross-machine documentation.
IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs
The paper introduces IndustryBench, a benchmark evaluating LLMs on industrial procurement QA in Chinese against national standards, highlighting safety compliance gaps. It reveals that extended reasoning often lowers safety-adjusted scores and reshuffles model rankings when safety violations are considered.
Plan First, Judge Later, Run Better: A DMAIC-Inspired Agentic System for Industrial Anomaly Detection
DMAIC-IAD is a multi-agent LLM system inspired by the DMAIC quality-management framework for industrial anomaly detection, using a 'Plan First, Judge Later' approach that formulates strategies via standardized operating procedures and ranks them with an execution-free judge model, achieving 37.76% improvement over agentic baselines across four data modalities.
RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learning
RimRule proposes a neuro-symbolic method that distills compact, interpretable rules from failure traces using the Minimum Description Length principle, improving LLM tool-use performance without modifying weights, and demonstrating rule portability across models.
MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation
This paper introduces MTDiag, a multi-turn diagnostic dialogue dataset for evaluating Large Language Models in realistic clinical diagnostic scenarios, addressing limitations of static QA benchmarks.