MARQ: A Modular, LLM-Agnostic Requirement Quality Assessor Infused with Systems Engineering Knowledge

arXiv cs.AI Papers
requirements-engineering systems-engineering llm knowledge-infusion quality-assessment evaluation

Summary

MARQ is a modular, LLM-agnostic prototype that infuses expert-derived systems engineering knowledge into LLM-based requirement quality assessment, showing significant F1 improvements over off-the-shelf baselines on an expert-labeled NASA requirement set across Claude Opus 5, GPT-5.6 Sol, and Gemma 3 12B, though gains are model-dependent.

arXiv:2610.36234v1 Announce Type: new Abstract: Requirements engineering (RE) is foundational to systems engineering (SE), as requirement defects can propagate downstream and adversely affect development outcomes. Although large language models (LLMs) are promising for requirement quality assessment, prior work shows that off-the-shelf models miss a large share of expert-identified defects, perform unevenly across quality criteria, and vary across executions. This study investigates whether infusing expert-derived SE knowledge can improve LLM-based requirement quality evaluation beyond off-the-shelf capability. We present MARQ, a modular, LLM-agnostic prototype that supplements LLMs with a criterion-specific knowledge stack operationalizing INCOSE requirement quality criteria and evaluated against off-the-shelf baselines on an expert-labeled NASA requirement set using Claude Opus 5, GPT-5.6 Sol, and the open-weight Gemma 3 12B, with 100 executions per model. MARQ achieves median F1 scores of 0.80 for Claude Opus 5, 0.62 for GPT-5.6 Sol, and 0.53 for Gemma 3 12B, compared with off-the-shelf baselines of 0.55, 0.52, and 0.18, respectively. However, the nature of these gains is model-dependent. Claude Opus 5 improves defect detection and false-alarm control; GPT-5.6 Sol gains detection and better execution-level consistency at the cost of more false alarms; and Gemma 3 12B reaches the performance of off-the-shelf proprietary models, although it remains highly variable. Criterion-level analysis shows gains concentrated in Correctness and Conformance, trade-offs between misses and false alarms within individual criteria, and model-dependent responses across several criteria. These findings position SE knowledge infusion as a promising model-agnostic pathway for strengthening trustworthy LLM-based RE support while underscoring the need for criterion-specific interventions and human-in-the-loop review.
Original Article

Similar Articles

QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits

arXiv cs.LG

Introduces QASM-Eval, the first comprehensive dataset for training and evaluating LLMs on OpenQASM 3 programs with hardware-oriented features, including expert-verified test and training sets. Evaluation shows that fine-tuning on QASM-Eval significantly improves LLM performance, with Llama-3-70B achieving 85% pass@1, outperforming few-shot GPT-5.2.

ModelEquivBench: Certifying Multi-Relational Evaluation of LLM-Generated Optimization Models

arXiv cs.AI

ModelEquivBench is a certifying multi-relational evaluation system for LLM-generated optimization models, reporting per-pair semantic profiles across seven equivalence relations instead of a single accuracy score. It evaluates GPT-5.4, Claude Sonnet 4.6, and Qwen3.5-397B-A17B on a fixed benchmark, revealing stage-wise failures that coarse baselines miss.

LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

Hugging Face Daily Papers

LiveMACE is an open-source benchmark that uses live cryptocurrency and US-equity markets to evaluate LLM agent capabilities in evolving conditions. Across five frontier LLMs over 30 days, it finds an outcome–capability gap, where higher returns do not consistently reflect stronger tool use, memory, rule-following, or collaboration.