EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

arXiv cs.AI Papers
multi-agent llm-agents engineering-design benchmark rag hpc langgraph

Summary

EngiAI introduces a multi-agent framework and benchmark suite for LLM-driven engineering design, evaluating workflow, RAG, and HPC dimensions. Proprietary models achieve 96-97% task completion on Beams2D, while conditional branching remains challenging with 20-53% for Photonics2D.

arXiv:2605.19743v1 Announce Type: new Abstract: Large Language Model (LLM) agents are increasingly applied to engineering design tasks, yet existing evaluation frameworks do not adequately address multi-agent systems that combine simulation, retrieval, and manufacturing preparation. We introduce a benchmark suite with three evaluation dimensions: (1) a workflow benchmark with seven prompt styles targeting distinct cognitive demands-including direct tool use, semantic disambiguation, conditional branching, and working-memory tasks; (2) a Retrieval-Augmented Generation (RAG) benchmark with gated scoring isolating retrieval contributions to parameter selection; and (3) an High Performance Computing (HPC) benchmark evaluating end-to-end ML training orchestration on a SLURM cluster. Alongside the benchmark we present EngiAI, a Multi-Agent System (MAS) reference implementation built on LangGraph that operationalizes the benchmark by coordinating seven specialized agents through a supervisor architecture, unifying topology optimization, document retrieval, HPC job orchestration, and 3D printer control. Across four LLM backends and two EngiBench problems, proprietary models achieve 96-97% average task completion on Beams2D, while open-source 4B-parameter models reach 55-78%, with clear generational improvement. Conditional branching proves most challenging, with task completion dropping to 20-53% for the conditional style on Photonics2D. RAG gating confirms near-perfect retrieval-augmented scores ($\approx 1.0$) versus near-zero without retrieval, validating the evaluation design. On HPC orchestration, one model completes all pipeline steps in 100% of runs while another drops to 50%, revealing that multi-step instruction following degrades over long-running workflows.
Original Article

Similar Articles

EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?

Hugging Face Daily Papers

EngiWorld introduces a comprehensive benchmark for AI agents in professional engineering, evaluating their performance across multiple domains and software with a focus on geometric and physical validity. The evaluation reveals that current frontier models have limited success, indicating a need for improvement.

BEAMS: Benchmarking and Evaluating AI for Modeling and Simulation

arXiv cs.AI

The BEAMS Initiative presents a benchmark suite for evaluating AI tools in modeling and simulation, focusing on human-centered and responsible AI practices. Tests reveal variability across LLM-based engines, with better performance in qualitative tasks than causal reasoning.

AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

Hugging Face Daily Papers

AgentWorld introduces a benchmark for evaluating long-horizon collaboration in multi-agent LLM systems with 100 tasks and a new causal collaboration effectiveness metric. Experiments show that even top models achieve only 52% task success, highlighting failure modes like communication breakdowns.