Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

# A Black-Box Evaluation Framework for LLM-Based DSM Generation
Source: [https://arxiv.org/html/2607.05985](https://arxiv.org/html/2607.05985)
\\UseRawInputEncoding

N\. Potters, T\. HofmanN\. Potters and T\. Hofman \(e\-mail: t\.hofman@tue\.nl\) are with the Eindhoven University of Technology \(TU/e\), Dept\. of Mechanical Engineering,[Control Systems Technology](https://www.tue.nl/en/research/research-groups/control-systems-technology)section,[Engineering Systems Design](https://www.tue.nl/en/research/research-groups/group-hofman)group, P\.O\.Box 513, 5600 MB Eindhoven, The Netherlands\.

###### Abstract

This paper presents a black\-box evaluation framework to systematically assess the ability of Large Language Models \(LLMs\) to generate Design Structure Matrices \(DSMs\) from structured technical documentation\. Motivated by the closed\-source nature of current Auto\-DSM pipelines, the framework introduces a reproducible methodology that benchmarks generated DSMs \(GEN\-DSMs\) against manually validated ground\-truth matrices \(GT\-DSMs\)\. The evaluation integrates both single\-run and multi\-run perspectives, combining structural metrics \(Completeness, Correctness, Coupling Density\), classification metrics \(Selective Accuracy, Abstention Coverage\), and stability measures \(Entropy, Fleiss’κ\\kappa\)\. To synthesize these aspects, a Composite Quality Score \(Q\) is proposed\. Controlled experiments are conducted on two datasets: a fictive abstract system and a real\-world refrigerator decomposition, covering variations in phrasing, parameter–dataset alignment, and system complexity\. Results show that LLMs can produce structurally plausible DSMs and achieve high reproducibility under well\-structured inputs, but remain sensitive to ambiguity, inconsistent dependency definitions, and prompt formulation\. The findings highlight systematic sources of hallucination and abstention failure, demonstrating both the potential and current limitations of LLM\-driven DSM automation\. The proposed framework provides a transparent benchmark for auditing Auto\-DSM pipelines and establishes foundations for integrating LLM\-based decomposition methods into model\-based systems engineering \(MBSE\) workflows\.

## IIntroduction

The growing complexity of engineered systems has driven demand for tools and methods that support system\-level analysis and decision\-making processes\. Among these, the Design Structure Matrix \(DSM\), introduced by Steward\[[46](https://arxiv.org/html/2607.05985#bib.bib38)\], remains a foundational method for visualizing and analyzing interdependencies between system elements\. Its compactN×NN\\times Nformat supports modularity analysis, design optimization, and process structuring across domains such as product development and systems engineering\[[22](https://arxiv.org/html/2607.05985#bib.bib39)\]\. Over decades, its use has matured through improvements in visualization and analysis tools—e\.g\., hierarchical clustering and dependency algorithms—forming a robust framework for downstream decision support\[[22](https://arxiv.org/html/2607.05985#bib.bib39),[8](https://arxiv.org/html/2607.05985#bib.bib40)\]\.

The DSM\-building phase—identifying components and interactions—remains manual, resource\-intensive, and error\-prone\[[8](https://arxiv.org/html/2607.05985#bib.bib40),[22](https://arxiv.org/html/2607.05985#bib.bib39),[13](https://arxiv.org/html/2607.05985#bib.bib8),[52](https://arxiv.org/html/2607.05985#bib.bib7)\]\. System decompositions typically rely on expert interviews and informal documentation, introducing variability, inconsistent terminology, and interpretation errors\[[22](https://arxiv.org/html/2607.05985#bib.bib39),[39](https://arxiv.org/html/2607.05985#bib.bib41),[44](https://arxiv.org/html/2607.05985#bib.bib42)\]\. Poor documentation\[[43](https://arxiv.org/html/2607.05985#bib.bib43)\]and tacit organizational knowledge\[[44](https://arxiv.org/html/2607.05985#bib.bib42)\]further undermine reproducibility\. As noted in\[[17](https://arxiv.org/html/2607.05985#bib.bib20)\], interviews are “just as important as reading design documents,” but they trade speed for accuracy, making DSM system decomposition costly and inconsistent\.

Model\-Based Systems Engineering \(MBSE\) provides a structured approach to managing system complexity and mitigating these issues\[[38](https://arxiv.org/html/2607.05985#bib.bib53)\]\. By formalizing specifications in digital models, MBSE enables scalable and consistent dependency mapping\[[16](https://arxiv.org/html/2607.05985#bib.bib31),[10](https://arxiv.org/html/2607.05985#bib.bib32),[51](https://arxiv.org/html/2607.05985#bib.bib33)\]\. A recent extension, the Elephant Specification Language \(ESL\)\[[53](https://arxiv.org/html/2607.05985#bib.bib4)\], extracts dependencies from structured natural language, improving traceability and reducing ambiguity\. ESL supports four of the five DSM\-based analysis steps defined by Eppinger and Browning\[[22](https://arxiv.org/html/2607.05985#bib.bib39)\], but the critical first step—system decomposition—remains manual, time\-intensive, and inconsistent\.

Advances in natural language processing \(NLP\), particularly Large Language Models \(LLMs\) such as OpenAI’s GPT series, offer a potential breakthrough\. These models extract dependencies and generate structured outputs from unstructured data sources\[[3](https://arxiv.org/html/2607.05985#bib.bib37),[9](https://arxiv.org/html/2607.05985#bib.bib44)\], producing possibilities in dependency mapping for MBSE workflows\. Automating the initial DSM draft reduces reliance on expert elicitation while retaining human oversight, shifting engineers from full decomposition to targeted validation\.

LLMs mitigate longstanding challenges in automated DSM construction by handling terminology variability\[[3](https://arxiv.org/html/2607.05985#bib.bib37),[9](https://arxiv.org/html/2607.05985#bib.bib44)\], extracting dependencies from unstructured sources\[[22](https://arxiv.org/html/2607.05985#bib.bib39),[32](https://arxiv.org/html/2607.05985#bib.bib2)\], flagging uncertain dependencies through abstention responses \(“I don’t know”\) for expert validation\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\], enhancing consistency in line with MBSE principles\[[38](https://arxiv.org/html/2607.05985#bib.bib53)\]\. Although poor documentation quality remains an obstacle\[[43](https://arxiv.org/html/2607.05985#bib.bib43)\], contextual inference, explicit uncertainty marking, and pooling information across multiple documents help mitigate its impact\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\]\. These advances rebalance the speed–quality trade\-off identified in\[[17](https://arxiv.org/html/2607.05985#bib.bib20)\], reducing weeks of preparation\[[52](https://arxiv.org/html/2607.05985#bib.bib7)\]to minutes\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\], while enabling aggregation across sources for more robust analysis\.

Despite these advances, research on LLM\-based DSM generation remains in its early stages\. To date, Auto\-DSM\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\]is the only academic attempt to apply LLMs for automated system decomposition within the DSM domain, demonstrating the feasibility of this radical approach\. However, its evaluation was case\-specific, relying on a diesel engine handbook compared against an expert\-derived DSM, where unequal information sources undermine reported accuracy\. Reported results focused on completeness, with no validation of correctness, and only point metrics were provided\. Without distributional analysis, stability and reproducibility across runs cannot be assessed—an essential requirement for dependable MBSE workflows\. Moreover, the absence of controlled datasets prevents systematic variation of phrasing, complexity, and scope, making it impossible to isolate failure modes or establish benchmarks\. Consequently, fundamental questions of accuracy, reproducibility, and reliability remain open, limiting trust and hindering the integration of LLM\-based DSM tools into engineering practice\. To address these gaps, this paper makes the following contributions:

1. 1\.Black\-box evaluation framework:A systematic methodology to assess Auto\-DSM by comparing LLM\-generated DSMs against validated ground\-truth DSMs under controlled and reproducible experimental conditions\.
2. 2\.Extended evaluation metrics:Introduction of new single\-run and multi\-run metrics that move beyond accuracy, capturing abstention behavior, incorrectness penalties, composite quality, and reliability across runs\.
3. 3\.Systematic identification of limitations:Application of the framework to both synthetic and real\-world datasets provides the first reproducible evidence of Auto\-DSM’s constraints, including reproducibility gaps and performance variability, thereby guiding future LLM\-based DSM automation\.

The evaluation framework and datasets are made publicly available at[https://github\.com/nielspotters/Auto\-DSM\_Evaluation\_framework](https://github.com/nielspotters/Auto-DSM_Evaluation_framework)\.

The remainder of this paper is organized as follows\. Section II presents the preliminaries\. Section III introduces the black\-box evaluation methodology\. Section IV details the experimental setup\. Section V reports results\. Section VI discusses limitations and future work\. Section VII concludes the paper\.

## IIPreliminaries

This section reviews the key challenges in applying LLMs for DSM construction, presents preliminary experiments on Auto\-DSM, and formulates the research problem\.

### II\-AHallucinations

A key challenge in this context is hallucination—when the LLM generates confident but unsupported or incorrect dependencies\. Such errors may arise from intrinsic factors \(e\.g\., model bias, incomplete training data\) or extrinsic ones \(e\.g\., ambiguous or noisy input\)\[[30](https://arxiv.org/html/2607.05985#bib.bib47),[31](https://arxiv.org/html/2607.05985#bib.bib48)\]\. In systems engineering tasks, hallucinations often manifest as fabricated or omitted DSM links\. These outputs are typically expressed with high linguistic confidence, making them difficult to detect without reference to ground truth\. Koh\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\]attempted to mitigate this by configuring the model with temperature = 0 and enabling abstention responses \(“I don’t know”\)\. However, empirical results show that this option is unreliable, as the model may assert a dependency in one direction while rejecting its symmetric counterpart—an inconsistency also observed in Koh’s example cases, where correctness never reaches 100% across symmetric relations\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\]\.

Existing literature emphasizes that LLMs exhibit a systematic non\-abstention preference: by default, models are more likely to produce an answer than to abstain, even when uncertain\[[49](https://arxiv.org/html/2607.05985#bib.bib56),[54](https://arxiv.org/html/2607.05985#bib.bib5)\]\. This tendency introduces risks in engineering applications, where an unsupported answer may appear as a valid system dependency\. Several strategies have been proposed to counter this bias, including conformal abstention with theoretical guarantees\[[54](https://arxiv.org/html/2607.05985#bib.bib5)\], strict prompting, verbal confidence thresholding, and Chain\-of\-Thought reasoning\[[37](https://arxiv.org/html/2607.05985#bib.bib55)\]\. These approaches demonstrate that abstention behavior can be influenced through prompting or post\-processing\. Yet, no current Auto\-DSM work explicitly evaluates abstention and confidence calibration under controlled test cases\. As a result, claims about the reliability of “I don’t know” responses \(e\.g\., via temperature tuning\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\]\) remain unevaluated\. Establishing a baseline assessment of abstention behavior is therefore essential, both to validate current prompt instructions and to determine whether advanced abstention strategies can improve the consistency and accuracy of LLM\-generated DSMs\.

Koh reports only point performance metrics without any distributional analysis—leaving the stability and reproducibility of outputs unexamined\. This omission is problematic in engineering contexts, where design decisions require repeatable and verifiable outputs\. Similar concerns have been raised in adjacent fields: Colas et al\.\[[14](https://arxiv.org/html/2607.05985#bib.bib25)\]demonstrated that, in reinforcement learning, statistically reliable conclusions only emerged after 15 to 50 independent runs, depending on effect size and variance\. Likewise, Liem and Panichella\[[36](https://arxiv.org/html/2607.05985#bib.bib30)\]argued that randomization without proper control undermines reproducibility in predictive software engineering\. Without explicitly quantifying output variability, hallucinations may remain undetected and the robustness of LLM\-generated DSMs cannot be assured\.

### II\-BCurrent State Experiments

While the Auto\-DSM approach shows promise, the method remains immature and requires refinement before adoption in industrial workflows\. Initial tests using the available no\-code implementation revealed inconsistencies, as the code failed to reproduce the results reported in Koh’s study\.

The first experiment assessed the sensitivity of Auto\-DSM to terminology\. Using the Heavy Duty Diesel Engines dataset\[[33](https://arxiv.org/html/2607.05985#bib.bib36)\]from Koh’s example, three system parameter terms—“Diesel engine,” “Combustion engine,” and “Engine”—were compared, with dependency type set to “Mechanical\.” The system identified only six elements across these definitions, whereas Koh reported eleven elements for an unspecified system parameter[Table I](https://arxiv.org/html/2607.05985#S2.T1)\. Defining no dependency type resulted in ten elements for “Engine,” but the remaining definitions did not change\. This indicates both terminology sensitivity and internal inconsistency, as the first pipeline prompt does not incorporate dependency type\. These findings underscore the need to better understand LLM behavior in system decomposition before integration into automated workflows\.

TABLE I:Comparison of system definitions and elements between Koh\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\]and empirical resultsA second analysis evaluated whether the workflow reduced hallucinations\. The Auto\-DSM approach attempted to enforce consistency by setting the model’s temperature to 0 and allowing “I don’t know” responses\. However, this did not yield stable outputs\. Across ten runs, the lowest reproducibility for a DSM entry was only 60% \([Table X](https://arxiv.org/html/2607.05985#A1.T10)\)\. While some variability reflected the model’s uncertainty, this was true for only two entries \(highlighted in[Table XI](https://arxiv.org/html/2607.05985#A1.T11)and[Table XII](https://arxiv.org/html/2607.05985#A1.T12)\)\. In many cases, the model alternated between “yes” and “no,” indicating hallucinations \(see[Appendix B](https://arxiv.org/html/2607.05985#A2)\)\. Notably, the model was 100% confident about the dependency from camshaft to crankcase, yet rejected the symmetric relation in 90% of runs, again revealing inconsistencies\.

### II\-CProblem Statement

These preliminary tests highlight the need for deeper analysis of system decomposition in the proposed LLM pipeline and for assessing its potential in engineering practice\. In particular, experiments with the Heavy Duty Engines dataset\[[33](https://arxiv.org/html/2607.05985#bib.bib36)\]demonstrated that Auto\-DSM is sensitive to terminology, exhibits inconsistent abstention behavior, and fails to reproduce results across repeated runs\. Yet, due to the extensiveness of this dataset, such analysis is analytically intensive and difficult to interpret, as the reasoning of the LLM spans large volumes of information\.

Moreover, as in Koh’s evaluation, comparing Auto\-DSM outputs with expert\-derived DSMs is methodologically unsound, since the LLM was provided with a handbook while the expert DSM was produced independently\. Koh implicitly assumes that both sources capture the same system knowledge, but this is neither stated nor verified\. To properly assess Auto\-DSM performance, a controlled dataset with an explicit ground\-truth DSM is required, ensuring that both the model and the reference rely on the same knowledge base\.

To address these limitations, this research adopts a controlled experimental design, constraining both data input and parameters to enable systematic evaluation of performance and limitations\. While advances in DSM visualization and analysis have streamlined their use, the building process itself remains a bottleneck, constrained by manual effort and inconsistencies in data and communication\. Existing model\-based approaches, such as ESL, alleviate some of these challenges but do not automate the decomposition of system elements and dependencies\. Similarly, while LLMs show potential in this direction, current methods lack the maturity and validation required for industrial adoption\.

This research therefore seeks to bridge this gap by designing and applying a black\-box evaluation framework for LLM\-based DSM system decomposition\. The framework is tested on an existing Auto\-DSM pipeline to provide a baseline assessment of current abstention and reliability, while establishing the foundations for future improvements\. These objectives lead to the following research question:

How can a black\-box evaluation framework be designed to systematically assess LLM\-based system decomposition for DSM generation, specifically in terms of identifying components and their dependencies?

To address this research question the following sub\-questions are formulated:

- •What evaluation criteria and metrics, grounded in comparison with validated ground\-truth DSMs, can be used to assess the quality, coverage, and inaccuracies of LLM\-based system decompositions, including the reliability of abstention behavior?
- •How can controlled experimental setups—varying input phrasing, dataset scope, and system complexity—be designed to evaluate the accuracy, reproducibility, and hallucination behavior of current Auto\-DSM methods, thereby establishing a baseline assessment of their performance?

## IIIMethodology

This section outlines the methodological framework designed to evaluate the ability of large language models \(LLMs\) to generate Design Structure Matrices \(DSMs\) from structured textual input\. Building on prior work in design automation and language model evaluation, the approach is grounded in black\-box testing principles, where only the input and output of the DSM generation pipeline are observable and used for assessment\.

### III\-ABlack\-Box Evaluation of the Auto\-DSM Pipeline

The Auto\-DSM pipeline is evaluated using a black\-box testing approach, motivated by the closed nature of its implementation\. The publicly released no\-code version by Koh\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\]provides no access to internal logic, including prompt structure, vector storage, model embeddings or intermediate reasoning steps\. As such, evaluation must be based solely on the observable relationship between structured input and DSM output\.

Black\-box testing is well suited for this context, as it focuses on validating functional behavior without reference to internal architecture or processing logic\[[1](https://arxiv.org/html/2607.05985#bib.bib26)\],\[[40](https://arxiv.org/html/2607.05985#bib.bib27)\]\. It is widely applied when system internals are inaccessible or abstracted, and is commonly used to detect incorrect functionalities, interface inconsistencies, and structural errors\. Pressman\[[42](https://arxiv.org/html/2607.05985#bib.bib29)\]identifies five key categories of errors that black\-box testing is intended to uncover: \(1\) incorrect or missing functions, \(2\) interface errors, \(3\) data structure or external database access issues, \(4\) behavioral or performance errors, and \(5\) initialization and termination faults\. Among these, the first and fourth are particularly relevant for evaluating the Auto\-DSM pipeline, where the core objective is to verify whether the system correctly infers component dependencies \(i\.e\., functional correctness\) and does so consistently across multiple executions \(i\.e\., behavioral stability\) from structured textual input\.

In the proposed setup, Auto\-DSM receives structured system descriptions as input and outputs a Design Structure Matrix \(DSM\)\. The evaluation framework compares the generated DSM \(GEN\-DSM\) to a manually derived ground truth \(GT\-DSM\), focusing on accuracy, consistency, and variability\. To account for reproducibility, the system is executed N times per input, enabling measurement of output variability and supports quantification of stochastic behavior in the Auto\-DSM pipeline\.

### III\-BEvaluation Framework

The evaluation framework consists of two parts, first calculating the metrics for a single run based on the DSM characteristics and comparing the single run GEN\-DSM with the GT\-DSM\. Secondly calculating the aggregated metrics based onNNruns\. For both the single and aggregated comparison metrics visualization graphs are used and build into the evaluation framework to improve analysis utilizing different type of heatmaps\.

#### III\-B1GT\-DSM vs\. GEN\-DSM: Single\-Run Evaluation

The single\-run evaluation framework consists of four sequential stages: DSM alignment, metric computation, visualization, and structured output generation\. Its purpose is to quantitatively assess the agreement between a generated DSM \(GEN\-DSM\) and a manually curated ground truth DSM \(GT\-DSM\), establishing a reproducible benchmark for model performance\.

A key step is reconciling discrepancies in component labels between the GT and GEN matrices, which may result from formatting inconsistencies or semantic misinterpretations by the LLM\. Initial alignment is performed via fuzzy string matching \(token sort ratio\) using theRapidFuzzlibrary, with 100% matches accepted automatically\. For ambiguous cases, semantic similarity is computed using theall\-MiniLM\-L6\-v2transformer\-based encoder, pretrained on over one billion sentence pairs and widely applied in semantic retrieval111[https://huggingface\.co/sentence\-transformers/all\-MiniLM\-L6\-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)\. The highest semantic match from the GEN\-DSM is proposed for user confirmation; if rejected, the user removes one of the conflicting components\. The resulting matched set forms the basis for all subsequent computations, with evaluation restricted to components common to both matrices\.

Each DSM entry is drawn from−1,0,1\{\-1,0,1\}, representing absent, uncertain, and present dependencies, respectively\. Three DSM\-specific metrics—completeness,correctness, andNon Zero Fraction \(NZF\)—provide macro\-level structural assessment, following\[[32](https://arxiv.org/html/2607.05985#bib.bib2),[29](https://arxiv.org/html/2607.05985#bib.bib6)\]\. Completeness is the fraction of entries classified as active or inactive, excluding uncertain values\. Correctness is the proportion of active dependencies that are reciprocated, indicating structural symmetry\. NZF, equivalent to coupling density, is the proportion of active dependencies relative to all possible off\-diagonal entries, normalized for system size\. These metrics are formally defined using Iverson bracket notation, where\[P\]=1\[P\]=1if predicatePPis true and0otherwise:

Completeness=∑i=1N∑j=1,j≠iN\[DSMi​j∈\{1,−1\}\]N​\(N−1\)\\text\{Completeness\}=\\frac\{\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1,\\,j\\neq i\}^\{N\}\[\\text\{DSM\}\_\{ij\}\\in\\\{1,\-1\\\}\\bigr\]\}\{N\(N\-1\)\}\(1\)
Correctness=∑i=1N∑j=i\+1N\[DSMi​j=1∧DSMj​i=1\]∑i=1N∑j=1,j≠iN\[DSMi​j=1\]\\text\{Correctness\}=\\frac\{\\sum\_\{i=1\}^\{N\}\\sum\_\{j=i\+1\}^\{N\}\[\\text\{DSM\}\_\{ij\}=1\\land\\text\{DSM\}\_\{ji\}=1\]\}\{\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1,\\,j\\neq i\}^\{N\}\[\\text\{DSM\}\_\{ij\}=1\]\}\(2\)
NZF=∑i=1N∑j=1,j≠iN\[DSMi​j=1\]N​\(N−1\)\\text\{NZF\}=\\frac\{\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1,\\,j\\neq i\}^\{N\}\[\\text\{DSM\}\_\{ij\}=1\]\}\{N\(N\-1\)\}\(3\)
Here,NNis the total number of system elements, andi,ji,jindex the row and column positions within theN×NN\\times Nmatrix\. Diagonal entries \(i=ji=j\) represent self\-dependencies and are excluded\.

While these metrics capture overall structural consistency, they do not measure prediction accuracy at the individual cell level\. To address this, we incorporateSelective Accuracy, a classification\-based metric grounded in confusion matrix theory\. This approach, common in supervised learning and black\-box model auditing\[[24](https://arxiv.org/html/2607.05985#bib.bib15),[41](https://arxiv.org/html/2607.05985#bib.bib61)\], treats each DSM cell as an independent classification into one of three states: presence\(1\)\(1\), absence\(−1\)\(\-1\), or uncertainty\(0\)\(0\)\. The inclusion of the uncertainty state introduces a rejection option, which is evaluated throughCertainty Coverage\. In this context, Certainty Coverage is equivalent to the previously definedcompletenessmetric\. Predictions are compared element\-wise to the GT\-DSM\.

LetTP,FP,TN,FN, andIDKdenote true positives, false positives, true negatives, false negatives, and uncertain predictions, respectively, withTotal=N​\(N−1\)\\text\{Total\}=N\(N\-1\)\.Selective Accuracyis then defined as:

Selective Accuracy \(SA\)=TP\+TNTotal−IDK\\text\{Selective Accuracy \(SA\)\}=\\frac\{\\text\{TP\}\+\\text\{TN\}\}\{\\text\{Total\}\-\\text\{IDK\}\}\(4\)
This follows the paradigm of selective classification, or classification with a reject option, where accuracy is conditioned on the model choosing to respond\[[20](https://arxiv.org/html/2607.05985#bib.bib10),[26](https://arxiv.org/html/2607.05985#bib.bib11)\]\. In this framework, Selective Accuracy measures correctness among committed predictions, whereas Certainty Coverage quantifies the proportion of the DSM for which the model abstains from prediction\.

In combination,Selective Accuracyand the three Product Architect DSM metrics—Completeness,Correctness, andNZF—constitute the single\-run evaluation framework\. This framework enables transparent, reproducible, and interpretable comparison between the GEN\-DSM and GT\-DSM\. It evaluates a single deterministic output and does not capture stochastic variability across multiple runs, which is addressed in the multi\-run evaluation \(Section[III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2)\)\.

For interpretability, the framework generates three visual diagnostics: \(i\) a matched DSM heatmap showing correct predictions, \(ii\) a mismatch DSM heatmap highlighting errors, and \(iii\) an uncertainty DSM heatmap marking the rejected response entries\. All are exported in rasterized \(PNG\) and tabular \(Excel\) formats for both qualitative review and quantitative analysis\.

#### III\-B2Multi\-Run Evaluation and Composite Quality Aggregation

While the single\-run evaluation framework offers insights into the correctness of a single LLM\-generated DSM instance, it does not capture the reproducibility or behavioral boundaries of the model\. A core objective of this research is to assess not only the classification performance, but also the consistency and variability of the DSM generation pipeline\. In line with best practices in empirical NLP evaluation\[[18](https://arxiv.org/html/2607.05985#bib.bib19),[5](https://arxiv.org/html/2607.05985#bib.bib12)\], we systematically analyze the outputs of repeated pipeline executions under identical input conditions to quantify variability and robustness\. Despite setting the decoding temperature to zero, non\-determinism in LLM inference can arise from parallel computation artifacts, floating\-point instability, or token alignment ambiguities\[[2](https://arxiv.org/html/2607.05985#bib.bib18)\]\. Therefore, we execute the Auto\-DSM pipelineNNtimes for a fixed input and evaluate the resultingNNDSMs collectively\. Before any performance metrics are computed, the number of repeated Auto\-DSM executions \(denoted asNN\) must be determined in a statistically grounded manner\. Rather than reporting agreement with the Ground Truth DSM \(GT\-DSM\) based on a single instance—thus risking results influenced by chance—it is essential to estimate a confidence interval for the aggregated agreement acrossNNindependent runs\. This aligns the evaluation with established principles of statistical reliability\. To generate an aggregated agreement, size differences between generated DSMs have to be adressed due to inconsistent component identification\.

##### Label Harmonization and Alignment

Each generated DSM may contain a distinct set of identified system components\. To compare across runs, all DSMs are realigned to a common label setℒ=⋃i=1NLi\\mathcal\{L\}=\\bigcup\_\{i=1\}^\{N\}L\_\{i\}, whereLiL\_\{i\}is the label set extracted in runii\. This union ensures structural comparability without discarding model output\. Each matrix is reindexed usingℒ\\mathcal\{L\}, with missing rows/columns filled usingNaNto indicate absence of the dependency, rather than a semantic zero \(which carries the specific meaning of uncertainty in the Auto\-DSM context\)\. Let𝒱r​c=\{Dr​c\(1\),Dr​c\(2\),…,Dr​c\(N\)\}\\mathcal\{V\}\_\{rc\}=\\\{D^\{\(1\)\}\_\{rc\},D^\{\(2\)\}\_\{rc\},\\ldots,D^\{\(N\)\}\_\{rc\}\\\}denote the non\-NaNvalues at cell\(r,c\)\(r,c\)acrossNNaligned DSMs\.

##### Agreement Criterion and Sample\-Size Policy

To evaluate the reproducibility and consistency of model predictions across multiple executions of the Auto\-DSM pipeline, we adopt a two\-pronged agreement analysis based on \(1\) the raw agreement rate and \(2\) Fleiss’κ\\kappastatistic\.

Theraw agreement ratemeasures the proportion of identical predictions across multiple runs for each cell in the DSM\. For every off\-diagonal cell\(i,j\)\(i,j\), we collect a prediction vector overNNruns asvi​j=\[x1,x2,…,xN\]v\_\{ij\}=\[x\_\{1\},x\_\{2\},\\ldots,x\_\{N\}\], wherexk∈\{−1,0,1\}x\_\{k\}\\in\\\{\-1,0,1\\\}denotes confidently absent, uncertain, or confidently present dependencies, respectively\. The agreement rate for a given cell is defined as the proportion of predictions belonging to the majority class\. The DSM\-wide agreement is then summarized by themeanandstandard deviationof all valid cell\-level agreement scores\.

To account for agreement by chance and better capture categorical reliability, we computeFleiss’κ\\kappaon a per\-cell basis\[[25](https://arxiv.org/html/2607.05985#bib.bib16)\]\. Each off\-diagonal cell\(i,j\)\(i,j\)is treated as havingNNindependent categorical labels\. Fleiss’κ\\kappafor each cell is given by:

κi​j=Pi​j−Pe1−Pe,Pi​j=∑cnc​\(nc−1\)N​\(N−1\),Pe=∑cpc2,\\kappa\_\{ij\}=\\frac\{P\_\{ij\}\-P\_\{e\}\}\{1\-P\_\{e\}\},\\quad P\_\{ij\}=\\frac\{\\sum\_\{c\}n\_\{c\}\(n\_\{c\}\-1\)\}\{N\(N\-1\)\},\\quad P\_\{e\}=\\sum\_\{c\}p\_\{c\}^\{2\},\(5\)
wherencn\_\{c\}is the count of labelc∈\{−1,0,1\}c\\in\\\{\-1,0,1\\\}in the cell\(i,j\)\(i,j\), andpcp\_\{c\}is the marginal probability of classccacross all predictions and cells\. Only off\-diagonal cells with at least five valid predictions are included to ensure reliable estimation\. The overall agreement across the DSM is then computed as the average over all such valid cells:

κ¯=1\|𝒞\|​∑\(i,j\)∈𝒞κi​j,\\bar\{\\kappa\}=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{\(i,j\)\\in\\mathcal\{C\}\}\\kappa\_\{ij\},\(6\)
where𝒞\\mathcal\{C\}denotes the set of all off\-diagonal cells with sufficient observations\.

To establish engineering\-grade reproducibility, we adopt a conservative threshold ofκ¯≥0\.60\\bar\{\\kappa\}\\geq 0\.60, corresponding to “substantial agreement” on the Landis–Koch scale\[[34](https://arxiv.org/html/2607.05985#bib.bib17)\]\. This threshold is motivated by empirical findings in DSM literature, where inter\-rater disagreement among human experts is nontrivial\[[47](https://arxiv.org/html/2607.05985#bib.bib3),[23](https://arxiv.org/html/2607.05985#bib.bib35)\]\.

An initial sample size ofN=30N=30pipeline runs is used\. If the resulting average agreementκ¯<0\.60\\bar\{\\kappa\}<0\.60, we incrementally increase the sample size by 10 runs until either the threshold is met or a maximum ofN=60N=60is reached\. This adaptive sampling approach is guided by empirical findings from stochastic system evaluations, which show that reproducible agreement signals typically emerge within 15–50 independent executions\[[14](https://arxiv.org/html/2607.05985#bib.bib25)\]\.

##### Match and Mismatch Frequencies

For each cell, we compute the match frequencyMr​cM\_\{rc\}and mismatch frequencyM¯r​c\\bar\{M\}\_\{rc\}:

Mr​c\\displaystyle M\_\{rc\}=1\|𝒱r​c\|​∑x∈𝒱r​c\[x=GTr​c\],\\displaystyle=\\frac\{1\}\{\|\\mathcal\{V\}\_\{rc\}\|\}\\sum\_\{x\\in\\mathcal\{V\}\_\{rc\}\}\[x=\\text\{GT\}\_\{rc\}\],\(7\)M¯r​c\\displaystyle\\bar\{M\}\_\{rc\}=1−Mr​c\.\\displaystyle=1\-M\_\{rc\}\.\(8\)These are derived from the matched and mismatched heatmaps, which encode the cell\-wise alignment of the generated DSM against a reference ground\-truth DSM based on the results from the single\-run evaluation\.

##### Uncertainty Frequency

The frequency of uncertain responses \(i\.e\., model output0\) is given by:

Ur​c=1\|𝒱r​c\|​∑x∈𝒱r​c\[x=0\],U\_\{rc\}=\\frac\{1\}\{\|\\mathcal\{V\}\_\{rc\}\|\}\\sum\_\{x\\in\\mathcal\{V\}\_\{rc\}\}\[x=0\],\(9\)which measures the likelihood that the model abstains from a confident prediction for dependency\(r,c\)\(r,c\)\.

##### Per\-Cell Aggregation and Entropy

In the context of information theory, the entropy per cell for a variable taking values in\{−1,0,1\}\\\{\-1,0,1\\\}is calculated based on the discrete probability mass function over these three outcomes\. Letp−1p\_\{\-1\},p0p\_\{0\}, andp1p\_\{1\}denote the probabilities of observing the values−1\-1,0, and11, respectively, such thatp−1\+p0\+p1=1p\_\{\-1\}\+p\_\{0\}\+p\_\{1\}=1\. The variability of predictions per cell is quantified using Shannon entropy:

Hr​c=−∑v∈\{−1,0,1\}pv⋅log2⁡pv,H\_\{rc\}=\-\\sum\_\{v\\in\\\{\-1,0,1\\\}\}p\_\{v\}\\cdot\\log\_\{2\}p\_\{v\},\(10\)wherepv=1\|𝒱r​c\|​∑x∈𝒱r​c\[x=v\]p\_\{v\}=\\frac\{1\}\{\|\\mathcal\{V\}\_\{rc\}\|\}\\sum\_\{x\\in\\mathcal\{V\}\_\{rc\}\}\[x=v\]and\[P\]=1\[P\]=1ifPPis true, 0 otherwise\. As in standard entropy definitions, terms with zero probability do not contribute to the sum due to the convention that0​log⁡0:=00\\log 0:=0, justified by the limiting behaviorlimp→0\+p​log⁡p=0\\lim\_\{p\\to 0^\{\+\}\}p\\log p=0\. This yields values in the range\[0,log2⁡3\]\[0,\\log\_\{2\}3\], since the maximum entropy occurs whenp−1=p0=p1=13p\_\{\-1\}=p\_\{0\}=p\_\{1\}=\\frac\{1\}\{3\}\.

To make the stability term in the Composite Quality ScoreQQdirectly comparable across systems, we normalize the entropy to a\[0,1\]\[0,1\]range:

Hr​cnorm=Hr​clog2⁡3\.H^\{\\text\{norm\}\}\_\{rc\}=\\frac\{H\_\{rc\}\}\{\\log\_\{2\}3\}\.\(11\)Where,Hr​cnorm=0H^\{\\text\{norm\}\}\_\{rc\}=0indicates perfect stability \(all runs gave the same value for the cell\) andHr​cnorm=1H^\{\\text\{norm\}\}\_\{rc\}=1indicates maximum instability \(the three possible values are equally likely\)\.

This ensures mathematical consistency and avoids undefined expressions\. The entropyHr​cH\_\{rc\}thus captures the uncertainty of the model per matrix entry, with higher values indicating greater prediction instability\. For matrix cells with high entropy but low participation \(i\.e\., highNaNfrequency\), interpretability is enhanced by computing the fill ratiofr​c=\|𝒱r​c\|/Nf\_\{rc\}=\|\\mathcal\{V\}\_\{rc\}\|/N\.

##### Selective accuracy per cell

The selective accuracy per call is given by:

S​Ar​c=Mr​c1−Ur​cSA\_\{rc\}=\\frac\{M\_\{rc\}\}\{1\-U\_\{rc\}\}\(12\)which defines the correctness of the confident responses of the model\. SinceMr​c¯\\bar\{M\_\{rc\}\}contains both the confident as uncertainty responses,Ur​cU\_\{rc\}has to be subtracted getting the selective accuracy \(i\.e\. the accuracy without uncertainty, GEN =0responses\)\.

##### Confidence Penalty

While these metrics capture aspects of correctness and robustness, they assume uniform cost for different types of errors\. This assumption is problematic in engineering design contexts, where the impact of errors varies significantly\. Specifically, false positives may lead to unnecessary design coupling and overengineering, while false negatives can result in undetected dependencies that compromise system integration\[[7](https://arxiv.org/html/2607.05985#bib.bib57)\]\. Uncertain predictions \(IDK\) offer limited engineering value, as they leave ambiguity in critical design decisions\. Uncertain predictions are preferable to incorrect ones, as they signal ambiguity and allow for focused investigation by engineers since they are clearly represented in the responses\. In contrast, confidently incorrect outputs are harder to detect in the absence of a ground truth DSM—a gap the Auto\-DSM pipeline aims to address by enabling DSM generation from unstructured textual descriptions\. To address these concerns, we adopt a cost\-sensitive error model inspired by\[[21](https://arxiv.org/html/2607.05985#bib.bib58)\], in which distinct penalties are assigned to different prediction types:

Pr​c=1\|𝒱r​c\|​\(∑x∈𝒱r​cwi​n​c⋅\[x≠0∧x≠GTr​c\]\+wi​d​k⋅\[x=0\]\)\.P\_\{rc\}=\\frac\{1\}\{\|\\mathcal\{V\}\_\{rc\}\|\}\\left\(\\sum\_\{x\\in\\mathcal\{V\}\_\{rc\}\}w\_\{inc\}\\cdot\[x\\neq 0\\land x\\neq\\text\{GT\}\_\{rc\}\]\+w\_\{idk\}\\cdot\[x=0\]\\right\)\.\(13\)Wherewi​n​c=1w\_\{inc\}=1andwi​d​k=0\.5w\_\{idk\}=0\.5, this formulation penalizes false predictions most heavily, reflecting their disruptive impact on system architecture\. Abstentions incur a smaller penalty, consistent with selective classification frameworks\[[12](https://arxiv.org/html/2607.05985#bib.bib59),[20](https://arxiv.org/html/2607.05985#bib.bib10)\], where cautious behavior is preferable to confidently incorrect outputs\. Weight factors can be adjusted dependent on domain or case\.

##### Composite Quality Score \(Q\)

To synthesize these indicators into a single quality measure, we define a Composite Quality Score \(Q\) that aggregates accuracy, stability, and penalization into a single metric, we define:

Qr​c=wacc⋅S​Ar​c\+wstab⋅\(1−Hr​c\)−wpen⋅Pr​c,Q\_\{rc\}=w\_\{\\text\{acc\}\}\\cdot SA\_\{rc\}\+w\_\{\\text\{stab\}\}\\cdot\(1\-H\_\{rc\}\)\-w\_\{\\text\{pen\}\}\\cdot P\_\{rc\},\(14\)The weightswacc=0\.5w\_\{\\text\{acc\}\}=0\.5,wstab=0\.3w\_\{\\text\{stab\}\}=0\.3, andwpen=0\.2w\_\{\\text\{pen\}\}=0\.2reflect engineering priorities where correctness is paramount, stability is desirable, and error severity must be penalized but not overly dominant\. This weighted composite score serves as a principled starting point, consistent with established practices in multi\-objective evaluation\[[27](https://arxiv.org/html/2607.05985#bib.bib60)\], and addresses known limitations of single\-metric benchmarks in ML\[[4](https://arxiv.org/html/2607.05985#bib.bib13)\]\.

##### Normalized Composite Quality Score

Since the Q is a linear combination with a subtractive penalty term, its raw score is not inherently bounded between 0 and 1\. To facilitate interpretability and cross\-system comparison, we normalize the score:

Qnorm,r​c=Qr​c−QminQmax−Qmin,Q\_\{\\text\{norm\},rc\}=\\frac\{Q\_\{rc\}\-Q\_\{\\min\}\}\{Q\_\{\\max\}\-Q\_\{\\min\}\},\(15\)whereS​Ar​c∈\[0,1\]SA\_\{rc\}\\in\[0,1\],Hr​c∈\[0,1\]H\_\{rc\}\\in\[0,1\], andPr​c∈\[0,1\]P\_\{rc\}\\in\[0,1\]\. The theoretical bounds are given by:

Qmax\\displaystyle Q\_\{\\max\}=wacc\+wstab,\\displaystyle=w\_\{\\text\{acc\}\}\+w\_\{\\text\{stab\}\},\(16\)Qmin\\displaystyle Q\_\{\\min\}=−wpen\.\\displaystyle=\-w\_\{\\text\{pen\}\}\.\(17\)This normalization ensures that the best\-performing model achievesQnorm=1Q\_\{\\text\{norm\}\}=1\(i\.e\., perfect accuracy, full stability, no penalty\), while the worst case yieldsQnorm=0Q\_\{\\text\{norm\}\}=0\(i\.e\., total failure across all metrics\)\. Such linear normalization is commonly applied in composite metric design to support inter\-model comparability, especially when integrating partially competing objectives\[[11](https://arxiv.org/html/2607.05985#bib.bib14),[24](https://arxiv.org/html/2607.05985#bib.bib15)\]\. The use ofQnormQ\_\{\\text\{norm\}\}provides a dimensionless and interpretable index of DSM generation quality, facilitating fair benchmarking of LLM performance across datasets\.

##### Heatmap Visualization and Export

To enable rapid interpretation of the evaluation results, aggregated metric matrices are visualized as heatmaps\. Each heatmap represents pairwise component relationships, allowing both global patterns and localized anomalies to be identified at a glance\. Metrics such as match frequency, mismatch frequency, uncertainty frequency, normalized entropy, performance quality, average DSM values, fill ratio, and raw agreement are each generated, and exported in tabular \(Excel\) and visual \(annotated heatmap\) formats with indexed component labels\. These visual diagnostics facilitate both macro\-level system interpretation and component\-level inspection of LLM prediction behavior\.

This multi\-run framework enables empirical quantification of variability in model behavior during LLM\-based DSM generation\. By integrating structural, classification, and entropy\-based metrics, it identifies regions of epistemic uncertainty and unstable dependency predictions—insights that are critical in engineering contexts where overlooked interactions may compromise system integrity\. Beyond correctness, the framework captures consistency across executions, supporting rigorous validation of the Auto\-DSM pipeline\. It also lays the groundwork for iterative improvements across multiple fronts: prompt engineering, to refine input clarity and task interpretation; decoding control, to reduce stochasticity and stabilize output; ensemble\-based generation, to aggregate multiple outputs for enhanced robustness; and data architecture, to ensure consistent formatting, labeling, and preprocessing of structured input across evaluations\. Crucially, this approach avoids the common methodological pitfall of treating a single model output as representative—an assumption particularly hazardous in high\-stakes design environments where system dependencies must be interpreted reliably\.

## IVExperimental Setup

The experimental setup for black\-box evaluation of the Auto\-DSM comprises three parts: \(1\) input variables, defining variability in parameter and dataset inputs; \(2\) executed experiments, which instantiate test cases to quantify how specific inputs affect LLM behavior; and \(3\) ground truth and technical\-document dataset design, producing controlled, symmetric, and sound datasets for a given system\.

### IV\-AVariable Parameters

The pipeline supports variability in both input parameters and dataset types\. To design the experiments, this variability is first defined and structured\.

#### IV\-A1System input definition

The Auto\-DSM pipeline uses two parameters to define the target system and the dependency type\. These parameters are injected into the prompts for component and dependency identification, enabling a scoped analysis for a specified system decomposition\. Because the parameters alter the prompt and thus the task posed to the LLM, different settings may yield different responses; their influence therefore warrants analysis\. The parameters are open\-ended to allow broad customization\. In this study, we assume the engineer targets a specific system decomposition; at minimum, the system must be defined\. Both systems and subsystems can be specified, enabling hierarchical DSM analysis\. The dependency type may be omitted to generate a DSM from general interactions, or specified to analyze particular dependency categories; omission still permits topological analysis of the system via unspecified interactions\.

#### IV\-A2Dataset variation

Whereas the system definition is straightforward, the dataset can vary along multiple dimensions\. Technical documentation underlying the system decomposition may differ in phrasing of interactions as well as in the complexity of the decomposition itself\. Both aspects are investigated to assess the effects of linguistic variation and decomposition complexity on accuracy and scalability\.

##### Interaction Definition

Component\-to\-component DSM dependencies can be classified into four possible definition types:

1. 1\.Direct link: A is linked to B
2. 2\.Indirect link: A is linked to C via B
3. 3\.No link: C is not linked to D
4. 4\.Undefined link: Link between A or B with D \(based on combining previous three types\)\.

These definitions evaluate how the pipeline handles different phrasings of stated dependencies and help identify semantic reasoning issues arising from interaction formulation\.

Beyond distinguishing interaction types, ambiguity and inaccuracies must also be considered\. Traditional document\-centric approaches have been shown to introduce inconsistencies that can lead to errors\[[15](https://arxiv.org/html/2607.05985#bib.bib51)\]; the experiments therefore also evaluate robustness to such noise\.

##### System complexity:

In addition to interaction types, this study examines how variation in system decomposition affects Auto\-DSM performance\. Specifically, we evaluate how increasing data complexity influences the LLM’s ability to generate accurate DSMs\. Following established benchmarks in the DSM literature, system complexity is assessed along three dimensions: system size \(number of components\), interconnectivity \(number and density of interfaces\), and architectural structure \(e\.g\., degree of modularity or hierarchy\), as visualized by Hennig et al\. in the experimental design space in[Figure 1](https://arxiv.org/html/2607.05985#S4.F1)\[[28](https://arxiv.org/html/2607.05985#bib.bib62)\]\.

![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/Complexity_Design_space_Hennig2021.png)Figure 1:Experimental design space by Hennig et al\.\[[28](https://arxiv.org/html/2607.05985#bib.bib62)\]By systematically varying these decomposition characteristics together with the interaction definitions and system\-input parameters, the study assesses the robustness, accuracy, and scalability of Auto\-DSM under increasingly complex and noisy or mismatched input conditions\.

### IV\-BExperiments

Given the variability options, the experiments are designed to incorporate these variations and test pipeline behavior under each case\. Specifically, we assess how selected input parameters and dataset properties influence Auto\-DSM, with attention to inaccuracies, hallucinations, and known limitations\. For every experiment, black\-box testing is performed: a dataset and GT\-DSM are generated, validated for correctness, refined if required for the specific case, input parameters are set, Auto\-DSM is executedN−N\-times, results are evaluated, and outcomes are integrated into the evaluation framework\. A visual representation is illustrated in[Figure 6](https://arxiv.org/html/2607.05985#S4.F6)\.

The experiments are grouped into three categories: 1\) Interaction Definition, 2\) System Parameters vs\. Dataset, and 3\) Complexity levels within the system decomposition\. These categories isolate the effects of natural\-language descriptions, inaccuracies in system and dataset design, and decomposition complexity on Auto\-DSM performance\.

#### IV\-B1Interaction Definition

These experiments examine whether alternative phrasings of interactions affect pipeline responses\. Datasets are based on a synthetic system in which interaction\-type distinctions are omitted to isolate phrasing effects, resulting in the system parameters Abstract system with no defined dependency type\. Three experiments are conducted:

##### Phrasing of interaction definitions

The first experiment evaluates how the four interaction types—Direct, Indirect, No link, Undefined—affect pipeline accuracy\. An abstract system containing all four definitions enables a structured comparison\. Performance for each interaction type is quantified using the evaluation framework\.

##### Symmetry within interaction definitions

The second experiment extends the first by mirroring all interactions so that every relation is bidirectional \(i\.e\.,Component A is linked to B; Component B is linked to A\)\. The goal is to assess whether bidirectional definitions influence DSM symmetry\.

##### Inaccuracies within interaction definitions

The third experiment measures the impact of inaccuracies by refining the abstract\-system dataset with duplicated statements for direct interactions—either reinforcing the same relation \(redundancy\) or contradicting it \(ambiguity\)\. The evaluation framework is used, and performance is compared against the Uni\-directional dataset to quantify the effect of inaccuracies\. To assess redundancy, a component pair with high uncertainty is selected, whereas a component pair with high performance is chosen to evaluate ambiguity\.

Based on these results, the most reliable interaction definitions are selected for subsequent dataset design\. This choice focuses later experiments on variables of interest while maintaining higher accuracy and stability; lower\-performing definitions may introduce excessive variance and confound analysis near the limits of Auto\-DSM\.

#### IV\-B2System Parameters vs\. Dataset

The Auto\-DSM pipeline accepts two input parameters \(target system and dependency type\) alongside any defined system decomposition\. Because datasets and parameters are independently specified, they may match or mismatch\. Both conditions are tested to quantify the influence of parameter–dataset alignment\. Given the earlier assumption that engineers define the system, the cases in[Table II](https://arxiv.org/html/2607.05985#S4.T2)are considered\.

TABLE II:Comparison of system and interaction type inclusion across parameters and datasetsIn[Table II](https://arxiv.org/html/2607.05985#S4.T2),Includedindicates that the defined parameter appears in the dataset;Excludedindicates it does not\. Experiments 4 and 8 warrant clarification: although seemingly counterintuitive, these use datasets for the defined \(4\) or another system \(8\) that contain only components and no interactions\. These cases explore model responses when inputs are mismatched\. To minimize interference from decomposition complexity, a low\-complexity system is used\. As before, GEN\-DSM and GT\-DSM are compared using the evaluation framework\.

#### IV\-B3Complexity within System Decomposition

Complexity experiments are defined with the experimental design space in mind\. Three cases are considered: a\) increasing complexity within a subsystem, b\) introducing hierarchy \(two subsystems\), and c\) adding noise by including other systems alongside the defined system parameter\.

##### Complexity within Subsystems

Two subsystems with different complexity levels are selected based on the experimental design space\. The highest\- and lowest\-complexity subsystems are characterized by size, inter\-connectivity \(computed via NZF,[Equation 3](https://arxiv.org/html/2607.05985#S3.E3)\), and the modularity/integral structure of the GT\-DSMs\. The evaluation framework is then used to quantify the effect of subsystem complexity\.

##### Complexity within hierarchy

Because a system decomposition often spans multiple subsystems and components, hierarchical effects must also be evaluated\. This experiment assesses not only intra\-subsystem interactions but also how accurately the pipeline identifies components across both subsystems and the inter\-subsystem interactions\. Two subsystems \(those used above\) are provided while the system is defined as the pipeline input parameter, enabling comparison against the GT\-DSM and assessment of whether hierarchy alters system\-level performance relative to subsystem\-level analysis\.

##### Complexity by Noise

This experiment evaluates the extent to which irrelevant information—relative to the defined input variables—affects the generated DSM\. Using the same dataset as the hierarchy experiment, the input variable is set to both the low\- and high\-complexity subsystems to measure how dataset noise influences pipeline performance\.

### IV\-CDataset Design

A Ground Truth \(GT\) dataset is required to evaluate pipeline responses with the evaluation framework\. Using controlled datasets—where variables can be systematically manipulated—enables rigorous assessment of Auto\-DSM, and repeated runs quantify deviations from the expected response\.

Accordingly, system decompositions are specified in plain text, from which both the GT\-DSM and experimental datasets are derived\. To focus on the pipeline’s natural\-language processing, only textual technical documents are used; images, audio, and model\-based inputs are excluded\. To keep the analysis compact and manually verifiable, two GT datasets are employed:1\) a fictive abstract systemand2\) a Refrigerator\.

#### IV\-C1Fictive Abstract System Decomposition Dataset

The fictive abstract system comprises 5 components, minimally constructed to cover the four interaction definitions described earlier\. The decomposition is:

> ”An abstract system is composed of components A, B, C, D, and E\. The components interact to ensure system functionality, as detailed below: Component A is linked to Component B\. Component B is linked to Component C\. Component C is not linked to Component D\. Component D is linked to component E\.”

Because the focus is phrasing, no dependency types are included—only components and interactions\. The corresponding fictive abstract GT\-DSM was manually constructed, resulting in[Figure 2](https://arxiv.org/html/2607.05985#S4.F2)\.

![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/Fictive_Abastract_GT_dsm.png)Figure 2:Ground Truth DSM of Fictive Abstract SystemThis dataset is also used for the Bi\-directional dataset, incorporating symmetry by stating every interaction twice in different sequence, i\.e\.:

> ”… Component A is linked to Component B\. Component B is linked to Component A\. …”

With minor refinements, the same dataset supports the duplication and contradiction experiments as well\. The text is modified as follows:

- •Duplication:”… Component D is linked to component E\. Component D is linked to component E\. …”
- •Contra\-dictionary:”… Component D is linked to component E\. Component D is not linked to component E\. …”

Lastly, the Bi\-directional dataset is refined to incorporate different dependencies for direct defined dependency pairs\. This dataset is used for the Input Parameter vs Dataset experiments and is defined as:

> ”… Component A is\{dep\-type\}linked to Component B\. …”

These interaction types are incorporated to investigate the accuracy of predicting specific interaction types as to left unspecified\. The abstract system isolates the experiment to minimize effects of pretrained knowledge while employing two interaction types from the HDDSM framework described by Tilstra et al\.\[[47](https://arxiv.org/html/2607.05985#bib.bib3)\]\.

#### IV\-C2Refrigerator Decomposition Dataset

To evaluate LLM\-based system decomposition and DSM generation, a system with analytical control, input validation, and a clearly bounded scope is required\. While the abstract system is controllable and well\-bounded, it lacks scalability, which is necessary for the input\-parameter–vs–dataset and complexity experiments\. A real system is therefore needed to assess decomposition performance on real\-world data\. Koh\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\]considered two candidates: a domestic refrigerator and a diesel engine\. We select the domestic refrigerator as the primary ground truth because it has compact scale, well\-defined subsystem boundaries, and manageable complexity\. By contrast, Koh’s diesel engine represents a subsystem within a larger vehicle platform and lacks a clear system boundary, complicating component/interface selection, hierarchy definition, sizing relationships, and interconnectivity with surrounding subsystems\. The refrigerator can be decomposed comprehensively—from product level to key components and interactions—without exceeding the intended scope\.

##### Refrigerator Ground Truth Dataset

To generate the ground truth from the main components, we use access to ChatGPT \(powered by OpenAI’s Language Model, GPT\-4o,[http://openai\.com](http://openai.com/)\) to efficiently propose generally accepted refrigerator decompositions based on its broad knowledge\.

The ground truth decomposition was created using a structured prompt sequence, consisting of three sequential prompts where the prompts first extracts major subsystems, then main components per subsystem, and finally interactions\. The interactions cover both intra\- \(within a subsystem\) and inter\-subsystem connectivity\.[Figure 3](https://arxiv.org/html/2607.05985#S4.F3)visualizes the Chain of Thought for the GT\-decomposition, the actual used prompts can be found in[Figure 9](https://arxiv.org/html/2607.05985#A2.F9),[Figure 10](https://arxiv.org/html/2607.05985#A2.F10),[Figure 11](https://arxiv.org/html/2607.05985#A2.F11)of Appendix[B](https://arxiv.org/html/2607.05985#A2)\.

![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/GT-pipeline2.png)Figure 3:Ground Truth Dataset Generating Chain of ThoughtAll interactions are classified according to the High\-Dimensional Design Structure Matrix \(HDDSM\) framework by Tilstra\[[47](https://arxiv.org/html/2607.05985#bib.bib3)\], widely adopted in engineering design research\. This framework partitions interactions into five categories—Information, Material, Energy, Spatial, and Movement—with finer subtypes \(e\.g\., 1\.2 Control \[CI\], 3\.5 Electrical \[EE\]\)\. The same framework is used in recent model\-based systems engineering for DSM generation, such as the ESL framework by Wilschut et al\.\[[53](https://arxiv.org/html/2607.05985#bib.bib4)\]\.

To ensure accuracy, reproducibility, and consistency across executions, the prompt structure incorporates the following strategies:

- •Task Instruction Prompting:Each prompt clearly defines the model’s role, task, and scope\. For instance, Prompt 1 begins with:“You are a systems engineer specialized in technical system decomposition and dependency analysis\.”This technique helps align model outputs with domain expectations\[[50](https://arxiv.org/html/2607.05985#bib.bib21),[19](https://arxiv.org/html/2607.05985#bib.bib22)\]\.
- •Constrained Generation:Prompts include specific limitations such as:“Do not invent components or interaction types\.”This restricts the LLM to predefined, verifiable terms—particularly in Prompt 2 and 3, which limit output to HDDSM interactions—thus reducing hallucination and improving consistency\[[50](https://arxiv.org/html/2607.05985#bib.bib21)\]\.
- •Chain\-of\-Thought Structuring:The decomposition task is split into sequential prompts: structure \(Prompt 1\), intra\-subsystem interactions \(Prompt 2\), and inter\-subsystem dependencies \(Prompt 3\)\. This step\-wise breakdown simulates reasoning chains, improving output reliability\[[48](https://arxiv.org/html/2607.05985#bib.bib24)\]\.
- •Format\-forcing Templates:All prompts include strict output formats, such as:\[Subsystem; From \(Component\); To \(Component\); Interaction Type\(s\); \.\.\.; Target Subsystem\]\. This reduces formatting ambiguity and enables direct parsing into DSMs\[[19](https://arxiv.org/html/2607.05985#bib.bib22)\]\.
- •In\-context learning:In\-context learning relies on examples closely related to the task, which the model uses to generalize to new tasks\. I\.e\.\# Example Template: Subsystem 1; \.\.\.; Subsystem 1\[[6](https://arxiv.org/html/2607.05985#bib.bib63)\]\.

![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/Drainage_and_Moisture_Number_Interactions_GT-DSM.png)Figure 4:GT‑DSM of the Drainage & Moisture Control System
![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/Refrigeration_Number_Interactions_GT-DSM.png)Figure 5:GT‑DSM of the Refrigeration System

Applying these strategies yields a structured and technically robust refrigerator decomposition\. The ground truth dataset is manually validated for correctness and symmetry to ensure consistency\. The system comprises 7 subsystems—Refrigeration System, Control System, User Interface & Access System, Structural & Insulation System, Defrost System, Lighting System, Drainage & Moisture Control System—each with, on average, more than 4 components, totaling 31 components\. The dataset is stored in a spreadsheet with filterable columns and index numbers to support referencing when generating textual inputs\. A partly preview is provided in[Table XIII](https://arxiv.org/html/2607.05985#A3.T13)\. The main statistics are summarized in Table[Table III](https://arxiv.org/html/2607.05985#S4.T3), confirming the decomposition’s extensiveness and its suitability for comprehensive analysis by experimental customization\.

TABLE III:System decomposition and DSM statisticsTo evaluate metrics based on DSM characteristics and classifications, the dataset is manually converted into a GT\-DSM and checked for correctness and symmetry to ensure consistency between the DSM and dataset\. The resulting dataset and DSM forms a validated ground truth reference for assessing the quality and correctness of LLM\-generated DSMs\.

A more detailed overview of the refrigerator decomposition is provided in[Table XIV](https://arxiv.org/html/2607.05985#A3.T14)of[Appendix C](https://arxiv.org/html/2607.05985#A3)in the Appendix\. Based on this overview, theDrainage & Moisture Controlsubsystem is selected as the low\-complexity case due to its small size and low NZF for both inter\- and intra\-subsystem interactions;[Figure 5](https://arxiv.org/html/2607.05985#S4.F5)also shows a modular structure supporting this choice\. Among the remaining subsystems, theRefrigeration Systemexhibits the highest complexity: it has the largest size and the highest inter\- and intra\-subsystem interconnectivity, and, as seen in[Figure 5](https://arxiv.org/html/2607.05985#S4.F5), greater structural complexity than the Drainage & Moisture Control system\. A combined decomposition capturing both intra\-subsystem and inter\-subsystem interactions between the Refrigeration and Drainage & Moisture Control subsystems is visualized in[Figure 13](https://arxiv.org/html/2607.05985#A3.F13)of[Appendix B](https://arxiv.org/html/2607.05985#A2)\.

##### Refrigerator input dataset

For each experiment, technical documents for the relevant subsystem decompositions are generated from the ground truth\. For the input parameter vs dataset experiments a controlled technical document is manually generated stating direct dependencies based on the generated GT\-database to isolate semantic reasoning by matching the subsystem decomposition as much as possible to the fictive abstract technical document\. However for complexity also more natural flowing documentation should be tested to see the semantic reasoning of the model\. Using prompt engineering, the GT dataset is converted into textual technical documentation; the same strategies—Task Instruction Prompting, Constrained Generation, Chain\-of\-Thought Structuring, and format\-forcing templates—are applied \(see[Appendix B](https://arxiv.org/html/2607.05985#A2)\)\.

![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/Dataflow_Experiment2.png)Figure 6:Schematic Overview of the Refrigerator System Decomposition Experimental SetupTo construct experimental datasets, the corresponding subsystems and interactions are first filtered from the GT dataset\. This selection is inserted into the prompt and executed in ChatGPT \(GPT\-4o\)\. The prompt in[Figure 12](https://arxiv.org/html/2607.05985#A2.F12)focuses on phrasing the GT data for a provided list of inter\-subsystem interactions\. To guarantee alignment between the GT dataset and the technical document, each statement includes an index number for reference\. This facilitates manual verification and correction of any hallucinations or inconsistencies, yielding a well\-defined, GT\-consistent dataset\.

The overall data\-generation and execution process for the refrigerator experiments is shown in[Figure 6](https://arxiv.org/html/2607.05985#S4.F6)\. This flow is also representable for manual generated technical documents, however then the process step between ”Structured GT\-dataset” and ”Experimental dataset” should be ”Human”Manually generatedinstead of ”ChatGPT”Prompt Engineering\.

## VResults & Analysis

TABLE IV:Baseline Agreement and Reproducibility Metrics for Different Input Parameter Sets[Table IV](https://arxiv.org/html/2607.05985#S5.T4)summarizes the reproducibility statistics for four representative baseline datasets evaluated using the Auto\-DSM pipeline\. Each dataset was tested using a consistent input parameter configuration overN=30N=30independent executions\. The raw agreement rate \(mean and standard deviation\) and Fleiss’κ\\kappaare reported to characterize both consistency and chance\-corrected reproducibility\.

All four cases exhibit strong reproducibility signals, with mean agreement rates exceeding 0\.90 andκ¯\\bar\{\\kappa\}values well above the conservative threshold of 0\.60\. These results empirically validate the sufficiency of the initial sample sizeN=30N=30in all tested configurations\.

### V\-AInteraction Definition

The datasets containing different interaction definitions are compared to the GT\-DSM as well as to each other to gain insight into the classification behavior of the Auto\-DSM\. While this will provide a first estimation on the influence of semantic reasoning within the technical documentation by the pipeline, the results are not comprehensive enough and will be supported by other experiments to come to a conclusion on the classification limits, behavior and performance\.

#### V\-A1Uni\-directional Dependency Definition

The first experiment contains the baseline experiment with the dataset of the non\-symmetric abstract system decomposition between components A\-E\. The notation for every cell in the DSM is stated asR​CRC, whereR=R​o​w,C=C​o​l​u​m​nR=Row,C=Column\. The per cell level Quality performance is visualized in[Figure 7](https://arxiv.org/html/2607.05985#S5.F7), indicating 6 problematic cells for the dependency definitions:A​B,B​C,C​B,D​E,E​B,E​DAB,BC,CB,DE,EB,ED\. A first observation of[Figure 7](https://arxiv.org/html/2607.05985#S5.F7)indicates that the performance of the bottom left triangle is better than the top right triangle\. Indicating that directional asymmetry is systematically present\. Since Auto\-DSM executes the dependency type identification based per column, it first searches for the column to row dependency and afterwards to the row to column dependency\. Looking at the dependency between A and B this is noted as: ”Component A is linked to Component B”, filling in column A row B first and Column B row A second with the defined prompt resulting in the prompt question directionA→BA\\rightarrow BwhereQB​A=1Q\_\{BA\}=1whileB→AB\\rightarrow Aresults inQA​B=0\.81Q\_\{AB\}=0\.81\. This is also the case for the dependency between component B and component C\. While this seems like a reasonable conclusion, the other two pairs do not support this hypothesis sinceQE​D=QD​EQ\_\{ED\}=Q\_\{DE\}andQB​E\>QE​CQ\_\{BE\}\>Q\_\{EC\}\. In order to make a conclusion the interaction definitions should be investigated to see if this influences the Auto\-DSM responses\.

![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/Koh_GT_ID_performance_quality_heatmap.png)Figure 7:Performance Quality Heatmap of the Uni\-directional Dependency Definition for an Abstract SystemTABLE V:Non\-zero entropy values, the probabilities forNNresponses and the matched frequency based on the Ground Truth value \(GT\)\.Looking into the entropy matrix the exact same 6 cells indicate different levels of variability by being non\-zero for entropy\. In order to understand the behavior of these cells and the dependencies, the normalized entropy is divided into its response probabilities, and compared to the GT\-DSM\. I\.e\. looking into the matched, uncertainty and mismatched frequencies, allowing for a structured overview presented in[Table V](https://arxiv.org/html/2607.05985#S5.T5)\.

A first observation is that Auto\-DSM has uncertainty in its confidence to assign a dependency or rejects the dependency classification\. For cell AB, CB, DE and ED Auto\-DSM is inconsistent in its confidence to identify a dependency\. However, if Auto\-DSM responds with confidence the dependency identification is correct\. Looking into the dependency definition for these cases, these are all direct defined dependencies and direct defined dependencies only\. Indicating that it could be correlated to the interaction definition\. However, cell BC is also a direct defined dependencies but this cell has no problems with confidence but repeatedly classifies the dependency incorrectly as an absent dependency \(77%\), contradicting the confidence uncertainty pattern\. While all direct dependencies are stated the same, the classification performance variates largely from matching 100%, to 40% with an uncertainty rate of 60%, to 100% confident response with 77% incorrect responses indicating variability introduced by other factors\. Cell EB is particularly interesting, since it is one of the 10 undefined dependency definitions, where every other undefined dependency definition is identified correctly\. EB not only shows a variability in confidence but also highlighting contradictions within the confident classifications\. This stochastic behavior would be explainable to the dependency not being defined, however one would expect to also see this behavior for the other 9 undefined dependencies, indicating again high but unexplainable variability based on interaction definition\.

Looking into the dependency definitions over the complete DSM and not only the non\-perfect quality scores, the following insights are gathered: Auto\-DSM predicts the direct dependencies for 62,7% correct, 12,8% incorrect and 24,5% of the times rejecting the classification\. Indirect dependencies as well as absent dependencies are 100% predicted correctly by as absent\. Undefined dependencies are predicted 90,3% of the times correct, 3,4% of the times incorrect and 6,3% of the time rejected\. Where the technical document gives clear direct evidence of different type of dependency definitions, we still see splits between correct, uncertain or even incorrect responses\. These splits could be coming from definition ambiguity in the prompt by leaving the linkage\-type empty or the direction not being specified allowing for hallucinations and freedom of interpretation of the model\. So while the dependency definition may have somewhat of an influence on the dependency identification, there is not enough evidence to make a clear and direct correlation between performance and different types of uni\-directional interaction definitions\.

![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/Koh_GT_ID_BI_performance_quality_heatmap.png)Figure 8:Performance Quality Heatmap of the Bi\-directional Dependency Definition for an Abstract System
#### V\-A2Bi\-directional Dependency Definition

When incorporating a bidirectional statements of the system decomposition for the same sample size and dataset, there is a remarkable difference for the pattern in the performance metric Q heatmap visualized in[Figure 8](https://arxiv.org/html/2607.05985#S5.F8)compared to Uni\-directional\. Introducing Bi\-directional dependency definitions improved the performance for direct, indirect and absent defined dependencies\. This pattern is expected and indicates correct reasoning of the Auto\-DSM method, since defined dependencies should be confidently predicted while undefined dependencies introduce ambiguity in data input where uncertainty in responses is expected to increase\. Comparing the bottom left triangle to the top right triangle of the performance metric highlights the increase in symmetry performance, where this was pattern was less visible for the uni\-directional abstract system\. This is backed by the average Correctness parameter for both cases, where the correctness for Uni\-directional is on average 46,16% with aσ=30%\\sigma=30\\%, the Bi\-directional clearly indicates a increase with an average of 95,93% and aσ=4,63%\\sigma=4,63\\%\. Cell AC and ED are scoring the lowest on the performance indicator with a score of 0\.886 and 0\.957, looking into the entropy and its probabilities compared to the GT\-DSM cell AC was predicted incorrectly less than 7% of the runs, while ED was predicted as uncertain even less only 3%\. This error margin is lower than the 90% agreement rate found by\[[23](https://arxiv.org/html/2607.05985#bib.bib35)\]but the complexity of the system decomposition is not comparable between cases, Auto\-DSM has to be incorporated in order to assess if the classification performance stays within the range for more complex tasks\.

Secondly, it is remarkable that with the uni\-directional it seemed that Auto\-DSM predicted confidently undefined dependencies as an absence, the Bi\-directional however has the same confidence with a highest uncertainty percentage not exceeding 3% but the predictions made are largely incorrect ranging within the incorrect prediction percentages between 50% and even 100% for non\-perfect quality scores\. Both the cells EB and BE predicted even 100% incorrectness, which is remarkable since this was also a problematic cell for the uni\-directional system decomposition\. This is unrelated to the dependency definition since these findings only apply to this specific pair, but indicate confident repeatable incorrect prediction behavior\. This could be due to the lack of defining a dependency type in the input parameter setting in combination with the presence of undefined dependency definitions in the input dataset\. This combination introduces a lot of ambiguity, where a model could fall back to its trainings data recognizing patterns or knowledge instead of rejecting the classification as also discussed by Li et al\.\[[35](https://arxiv.org/html/2607.05985#bib.bib50)\]\. To test this experiments between the input parameter and the dataset should be preformed\.

#### V\-A3Inconsistent Dependency Definitions

Both the documents for the duplication statement and Contra\-dictionary statement are executed with the sample size defined by the baseline execution\. Since the non\-symmetric DSM provided insights in high uncertainty for the dependency pair DE and ED, this pair is selected to see if introducing duplications has influence on the classification predictions\. The duplication performance Q heatmap has the same pattern for the direct, indirect and absence dependencies between components A, B, C and D as for the single stated Uni\-directional experiment, indicating the same prediction behavior but with some small deviations due to the uncertainty behavior in confidence\. Undefined dependency definitions highlight again stochastic behavior ranging from a perfect quality score of 1 to aQE​B=0\.165Q\_\{EB\}=0\.165, again for cell EB\. However, for the dependency pair DE/ED the performance went up to a perfect classification quality score of 1 visualized in[Table VI](https://arxiv.org/html/2607.05985#S5.T6)\. This proves the statement Koh made on pooling is indeed true for Auto\-DSM\[[32](https://arxiv.org/html/2607.05985#bib.bib2)\], where more of the same data contributes to a more confident response\.

TABLE VI:Probabilitiesp​\(i\)p\(i\), normalized metricsQnQ\_\{n\}andHnH\_\{n\}, and matched rateMr​cM\_\{rc\}values based on Ground Truth \(GT\) for inconsistency dependency definition experiments\.Next to the duplication experiment, also contradicting dependency definitions were tested to identify the ability of Auto\-DSM to detect problematic dependencies and classifying them as an uncertainty by rejection\. Contradicting dependency definitions are tested on the best performing dependency pair between A and B of the single statement Uni\-directional experiment\. In order to eliminate the effect of statement sequence two experiments are tested, direct dependency followed by an absence dependency and vice versa\.[Table VI](https://arxiv.org/html/2607.05985#S5.T6)shows that the performance Q, the entropy and the probabilities compared to the GT\-DSM are heavily influenced by not only the contradiction but also by the sequence of the contradiction\. Expected would be that at the uncertainty would increase, however the results in[Table VI](https://arxiv.org/html/2607.05985#S5.T6)show that the predictions are inclined to confident but incorrect predictions to the last stated dependency definition in the sequence instead of rejecting due to ambiguity in description\.

The experiments demonstrate that interaction definition exerts a measurable yet inconsistent influence on Auto\-DSM’s classification performance\. Uni\-directional dependency definitions reveal directional asymmetry and high variability in confidence and correctness, with problematic cells \(e\.g\., EB, BC\) exhibiting stochastic misclassification patterns that cannot be fully attributed to the interaction type alone\. Bi\-directional definitions substantially improve correctness, reduce variance, and enhance symmetry across the DSM, yet also introduce instances of confidently incorrect predictions, particularly for undefined dependencies\. Inconsistent definitions highlight Auto\-DSM’s sensitivity to redundancy and sequence effects, where duplication increases confidence and correctness, but contradictory inputs often result in confident misclassifications rather than uncertainty\. Collectively, these findings indicate that while clear and consistent interaction definitions improve performance, Auto\-DSM remains susceptible to ambiguity\-induced bias and systematic errors, necessitating further controlled experiments to isolate the contribution of interaction semantics from other influencing factors\.

TABLE VII:Overview of experiments with component modes, occurrence, normalized quality and entropy metrics \(QnormQ\_\{\\text\{norm\}\},HnormH\_\{\\text\{norm\}\}\), Selective Accuracy \(SA\), and Completeness and the metrics mean and standard deviations\.

### V\-BSystem Parameters vs Dataset

TheSystem Parameters vs Datasetexperiments were executed using \(i\) the refined bi\-directional abstract system decomposition with interaction types and \(ii\) the refrigerator subsystemDrainage & Moisture Control\(D&M\)\. For this analysis, both manually generated technical documents based on the abstract decomposition and the refrigerator GT\-dataset were created to state only direct, symmetric dependencies\. This isolates the influence of system\-parameter selection from the dataset by excluding added complexity and leveraging results from the dependency\-definition experiments\. Building on the interaction\-definition experiments, DSM\-wide aggregated performance metrics were used for analysis\.

As explained in the Experimental Setup, eight experiments were executed\. These are divided into two categories by objective: Experiments 1–4 analyze the abstention behavior of Auto\-DSM in dependency identification, while Experiments 5–8 focus on abstention in system component identification\. Observed inaccuracies and inconsistencies motivated follow\-up experiments to diagnose problematic responses\.[Table VII](https://arxiv.org/html/2607.05985#S5.T7)reports the results forN=30N=30, listing the input parameter, input dataset, the most frequently classified number of components \(Component Mode\), the occurrence of this mode, and the mean and standard deviation of performance quality, entropy, selective accuracy, and completeness, to characterize accuracy, variability, and uncertainty DSM\-wide relative to the GT\.

#### V\-B1Dependency Type Parameter vs Dataset

The first four experiments assess the influence of input parameters versus dataset for dependency\-type interactions, examining behavior when uncertainty is introduced by increased strictness or by mismatches between input parameters and dataset\.

##### Abstract system

For the abstract system, direct dependencies were refined by adding a dependency type\. Utilizing insights from the interaction\-definition experiments, we evaluated overall DSM performance and the effect of dependency types when pattern and definition influences are known\. Experiment 1\.1 produced an almost perfect score, except for cell AB \(direct mechanical statement\), which showed an uncertainty frequency of 0\.17\. Experiment 2\.1 yielded a lower performance score and very low completeness: the model preferred abstention over asserting absence, even though the document clearly contained no electrical dependency types; when confident, classifications were perfect\. Experiment 3\.1 mirrored the dependency\-definition results: high performance for direct and absent links, with lowest match frequency 0\.9, and all mismatches due to uncertainty responses\. Undefined dependencies exhibited substantial uncertainty and inaccuracy when the model was confident, increasing entropy and lowering selective accuracy\. Experiment 4\.1 produced a consistent all\-zeros DSM, correctly acknowledging knowledge boundaries\.

##### Real system

Guided by these findings, a real system was evaluated using a manually generated dataset for the D&M subsystem, represented in[1](https://arxiv.org/html/2607.05985#LST1), which contains only bi\-directional direct statements\. As expected, Experiments 1\.2 and 3\.2 provided roughly the same performance because every dependency pair included theMixture Materialinteraction type\. However, relative to the abstract findings, near\-perfect scores were expected \(as in Experiment 1\.1\), since dependencies were directly stated and included in the dataset\. Three explanations are plausible: \(i\) real\-system effects, \(ii\) poor interpretation of the dependency type by the LLM, and \(iii\) overgeneralization because all direct dependencies in the technical document share the same input dependency type\.

To verify whether the dependency\-type parameter caused hallucinations—potentially because Auto\-DSM recognizes the system and defaults to pretrained knowledge—Experiment 9 swapped the abstract system’s “mechanically” dependencies for “Mixture Material,” yielding an identical dataset but with D&M dependency types present\. This resulted in a high performance score,Qn=0\.91Q\_\{n\}=0\.91, 0\.08 lower than Experiment 1\.1 but higher than Experiment 1\.2\. Entropy and completeness increased relative to Experiment 1\.1, suggesting that the LLM has more difficulty with this HDDSM dependency type, although it is not the sole factor in the unexpected behavior\. The model exhibited increased uncertainty by abstaining on some classifications, yet the results indicate the hallucination was not introduced by the dependency type and that the HDDSM framework can be classified\[[47](https://arxiv.org/html/2607.05985#bib.bib3)\]\. Note thatMixture Materialis defined for every direct dependency in the D&M dataset, which may cause overgeneralization\.

To investigate overgeneralization further, Experiments 10 and 11 changed the input parameter to other dependency types\. Experiment 10 showed worse performance,Qn=0\.74Q\_\{n\}=0\.74, despiteHn=0H\_\{n\}=0and 100% completeness: it confidently misclassified the Drain Pan–Drain hole pair as Proximity and classified the Drain pan–evaporation tray pair—expected to be Proximity—as having no Proximity dependency\. These results contradict the direct definitions in[1](https://arxiv.org/html/2607.05985#LST1)and reveal a previously unobserved hallucination under direct definitions, potentially caused by the model defaulting to pretrained knowledge when prompts are weakly constrained\. Experiment 11 checked the third dependency type; as shown in[Table VII](https://arxiv.org/html/2607.05985#S5.T7), the dependency\-type identification was completely correct and deterministic, aligning with the abstract\-system experiments\. This fluctuation in system\-type parameter identification may arise because the Tilstra framework has not been benchmarked for LLM\-based dependency classification; semantic reasoning may allow the LLM to identify some dependency types more reliably than others\. Overgeneralization is also plausible: when all defined dependencies share the same type, entropy and completeness appear strong while selective accuracy drops, yielding an almost all\-ones GEN\-DSM\. Exploring other or larger systems may clarify whether, when unsure, the LLM recognizes systems and falls back to training data rather than acknowledging knowledge boundaries\.

Concluding for these dependency\-type experiments: in real\-world systems, dependency types combined with real components tend to trigger learned patterns or pretrained knowledge \(Experiments 2\.2 and 4\.2\), whereas abstract systems \(Experiments 2\.1 and 4\.1\) better acknowledge knowledge boundaries\. In 2\.1 and 4\.1, high uncertainty responses drove completeness toward zero; in 2\.2 and 4\.2, the model was confident but incorrect\. Experiment 4\.2 notably returned an all\-present GEN\-DSM where a full\-uncertainty matrix \(as in 4\.1\) was expected\. Abstract\-system experiments provided consistently higher performance, and the mechanical interaction category scored slightly better thanMixture Material, indicating that the LLM can reason with the Tilstra HDDSM framework but recognizes some interaction categories more reliably than others\.

#### V\-B2System Definition Mismatch \(Component Identification\)

Where Experiments 1–4 \(and follow\-ups 9–11\) exposed inconsistencies for the dependency input parameter, the system\-definition mismatch reveals issues in component identification by Auto\-DSM\. In Experiments 5–8, the concise D&M decomposition served as the dataset and the input system parameter wasBicycle\. ForN=30N=30, more than 75% of responses identified real bicycle components such asPedalsorChain\. ForMixture Material, the dependency analysis for the non\-rejected component sets produced an all\-zeros matrix, correctly indicating knowledge boundaries for inter\-component dependencies\. In contrast, forMechanically, the model confidently classified some bicycle components as having no dependency, revealing a chain of hallucinations and a tendency to respond confidently rather than abstain\.

To gain more insight into component\-identification hallucinations, Experiments 13 and 14 tested whether Auto\-DSM prefers pretrained knowledge over factually correct but real\-world\-incongruent decompositions\. The input parameter remainedBicyclewith the DSM decomposition, and the document opened with:

> ”A bicycle is developed, containing the following components: The drain hole, drain pan, evaporation tray, and moisture control filters, which interact through the below stated dependencies to ensure functionality\.”

Both cases produced results similar to Experiments 1\.2 and 3\.2, indicating that component identification prioritized the technical documentation rather than defaulting to training data or rejecting due to conflict\. Experiment 15 explored whether removing the initial component list affects identification; as shown in[Table VII](https://arxiv.org/html/2607.05985#S5.T7), the 4/4 components occurred with 100% frequency forN=30N=30, indicating no effect\. Experiments 16–19 examined whether pretrained knowledge fills uncertainty in real\-system decompositions\. First, when the document stated that D&M is a refrigerator subsystem \(input parameter:refrigerator; Experiments 16 and 17\), results matched those of 1\.2 and 3\.2, confirming correct hierarchical identification\. For Experiments 18 and 19, this statement was omitted, introducing ambiguity resulting in the abstention frequency 57\-87%\. When the subsystem appeared in the dataset, Auto\-DSM correctly identified all components\. However, once the hierarchical structure was excluded from the technical document, the LLM again showed inconsistent recognition of knowledge boundaries \(as with the bicycle example\), with more frequent abstention and high variance in confidence, indicating instability\.

This highlights that Auto\-DSM prioritizes information in the technical documentation when it matches the input parameter\. In the absence of such alignment, the LLM defaults to pretrained knowledge and hallucinates by introducing random components instead of issuing uncertain rejections\.

To conclude the findings for the input\-parameters–dataset experiments, responses exhibit inconsistency and inaccuracy\. While these experiments set the stage, they already underscore the importance of evaluating such pipelines across multiple categories\. The findings relate to\[[35](https://arxiv.org/html/2607.05985#bib.bib50)\], which suggests that an LLM struggles to acknowledge its boundaries\. Once the technical document provides a well\-defined system decomposition—whether aligned with its training data or not—Auto\-DSM can identify components\.

TABLE VIII:Results for Complexity Experiments with Component Modes, Occurrence, Normalized Quality and Entropy Metrics \(Qn​o​r​mQ\_\{norm\},Hn​o​r​mH\_\{norm\}\), Selective Accuracy \(SA\), and Completeness and the metrics mean and standard deviationsThe overall results for Input Parameter vs Dataset imply that the model has preferences among the Tilstra HDDSM interaction categories, fails to consistently acknowledge classification decision boundaries embedded in the prompt, and tends to incorporate patterns or system knowledge from pretrained data when systems likely appear in the training set—or even fully defaults to training\-data priors when parameter–dataset alignment is imperfect\. The resulting behavior ranges from perfect dependency identification, to confident yet deterministically incorrect full\-present matrix classifications, to high\-variance, low\-accuracy outputs, underscoring susceptibility to hallucination\-like behavior when semantic constraints are weak or when dependency\-type definitions permit broad interpretation\.

### V\-CComplexity

The Complexity experiments compare system decompositions for real systems\. We vary \(i\) subsystem complexity, \(ii\) writing complexity, and \(iii\) multi\-subsystem hierarchical complexity to evaluate ambiguity/noise, scalability, and hierarchical identification\. These experiments provide insight into both component and dependency identification\.

#### V\-C1Subsystem comparison

The drainage & moisture control subsystem input\-parameter–vs\-dataset experiments showed inconsistent subsystem dependency identification\. Despite these results, the subsystem\-complexity experiment was performed to test whether the behavior relates to pretrained knowledge and to semantic reasoning specific to the Drainage & Moisture Control \(D&M\) subsystem\. The refrigeration system was selected for its higher decomposition complexity and because it is likely the most common refrigerator subsystem, potentially better represented in LLM training data; this selection enables assessment of whether the model accurately identifies interactions\. Based on the ground\-truth dataset, a technical document was generated with a manually constructed system decomposition containing symmetric, directly defined dependencies, visualized in[2](https://arxiv.org/html/2607.05985#LST2)\. The best\-performing input\-parameter–vs\-dataset settings from earlier experiments \(defined existing system and dependency type\) were reused\. For the refrigeration system, the input parameterLiquid Materialwas chosen to investigate over\-generalization, as it is the most frequent dependency type in the decomposition \(22/37, 59% of direct dependencies\)\.[Table VIII](https://arxiv.org/html/2607.05985#S5.T8)presents results and compares them to Experiment 1\.2; performance is notably similar, and the metrics align more closely with Experiment 1\.2 than with any of the other experiments\. In combination with the Experiment 1\.2 findings in[Table VII](https://arxiv.org/html/2607.05985#S5.T7), this pattern suggests possible over\-generalization: in both systems, the most prevalent dependency type performs the worst when present in the dataset\. Notably, both dependency types belong to the same HDDSM “Material” category, so LLM semantic reasoning may also influence identification\.

To assess component and dependency identification under increased decomposition complexity, the Gas Material dependency type was selected because it occurs in both the drainage & moisture control and refrigeration subsystems, thereby eliminating variability in dependency\-type behavior\.[Table VIII](https://arxiv.org/html/2607.05985#S5.T8)summarizes overall performance metrics for both subsystems\. Although refrigeration appears to perform better overall, examining quality for the directly stated dependencies described in[1](https://arxiv.org/html/2607.05985#LST1)and[2](https://arxiv.org/html/2607.05985#LST2)reveals a different picture:QD&M=1Q\_\{D\\&M\}=1withN​Z​FD&M=0\.17NZF\_\{D\\&M\}=0\.17, versusQR​E​F=0\.3Q\_\{REF\}=0\.3withN​Z​FD&M=0\.09NZF\_\{D\\&M\}=0\.09\. Cell\-wise comparison indicates perfect classification for the D&M subsystem and consistent but incorrect classification for the refrigeration system, which falsely marks the dependencies as absent\. Considering the NZF of theGas Materialdependency for both cases indicates that the Refrigeration subsystem struggles to identify this dependency type, whereas the D&M subsystem achieves a perfect score\. These findings suggest that complexity may affect the identification of the system decomposition: dependency types with lower NZF values may be more difficult for Auto\-DSM to identify and should be examined closely when evaluating system\-identification performance\.

TABLE IX:Refrigeration System \(REF\) and Drainage & Moisture Control System \(D&M\) coverage with Generated DSM \(GEN\-DSM\) component counts
#### V\-C2Writing complexity

Across experiments, a relatively simple technical document was used as input; however, LLM semantic reasoning must be considered\. To test the effect of writing complexity on LLM behavior, the synthetically generated technical report in[4](https://arxiv.org/html/2607.05985#LST4)was compared to the manually generated document in[1](https://arxiv.org/html/2607.05985#LST1)\. The synthetic report produces a formal system\-decomposition narrative in fluent technical English\. To isolate differences, the same input parameters and dependency type were used\. Given inconsistency across dependency types, two cases were compared \(see[Table VIII](https://arxiv.org/html/2607.05985#S5.T8)\): leaving the dependency type empty to expose over\-generalization, and using the high\-performingAlignmenttype\. Whereas Experiment 3\.2 was already close to an all\-present dependency response, Experiment 22 returned, forN=30N=30, thirty fully dependent matrices\. Likewise, although Experiment 11 achieved a perfect score, the extensive dataset yielded an almost all\-present dependency matrix in which only the dependencies between the moisture control filters and the drain hole toggled between present and absent\.[Table VIII](https://arxiv.org/html/2607.05985#S5.T8)shows that—even when the exact dependency type exists in the dataset—sentence\-level semantics can re\-introduce over\-generalization, which must be handled with care\. With more extensive datasets, it becomes harder to trace the origin of dependency identification; large documents can obscure incorrect or ambiguous interactions\. Future work should add referencing/traceability so engineers can verify whether identifications are based on correct or incorrect text, and incorporate prompt\-level structuring constraints to narrow the LLM’s decision space and improve both identification and traceability\.

#### V\-C3Multiple system analysis

Despite limitations revealed by the highest\-complexity refrigeration decomposition and by the Input Parameter vs Dataset findings, follow\-up experiments with a dual\-subsystem decomposition were conducted\. A combined dataset was constructed that included both the Refrigeration System and the Drainage & Moisture Control System, incorporating intra\-subsystem interactions and inter\-subsystem dependencies \([3](https://arxiv.org/html/2607.05985#LST3)\)\.

With the system input parameterRefrigerator, only the Refrigeration subsystem was returned\. Although linguistic similarity could explain this, Experiments 16–19 showed that D&M components are valid responses even when both subsystems and a dual system definition are present\. Consequently, combinations of input parameters were tested\. The inputs in[Table IX](https://arxiv.org/html/2607.05985#S5.T9)examine alternative phrasings for dual\-subsystem definitions\. The resulting GEN\-DSMs consistently included only Refrigeration components; identification stabilized on a single subsystem depending on wording\.

Sensitivity to system\-parameter phrasing was evaluated across six structured variations \([Table IX](https://arxiv.org/html/2607.05985#S5.T9)\)\. In all but one case, outputs contained complete Refrigeration sets \(7/7\) and at most one drainage\-related component\. Output logs indicated that Auto\-DSM extracted components for both subsystems but stored them as two lists, after which only the first list was used for dependency\-type identification\. A notable exception was the phrasing“Refrigerator decomposition including both the Drainage & Moisture Control and Refrigeration Subsystem”, which yielded a single list; in 21/30 runs, all 11 components—7 Refrigeration and 4 D&M—were included in the GEN\-DSM\.

While this phrasing demonstrated that the LLM is capable of jointly recognizing both subsystems, it shows the effect of poorly constrained prompts Where just only defining the input parameter differently allows for such different outcomes\. Further analysis revealed that the order in which subsystems are mentioned in the input parameter directly impacts the outcome\. This asymmetry provides strong evidence that the LLM processes composite noun phrases in a sequential manner, frequently prioritizing the first identified entity\. It appears that only one system listing is actively used to construct the DSM, even if there are two acknowledged during textual reasoning\. In conclusion, although semantically refined input phrasing improves component identification, the model remains unable to robustly construct structured DSMs involving multiple interacting subsystems\. The results suggest that further intervention—either in the form of prompt decomposition, architectural support for subsystem segmentation, or DSM post\-processing—is necessary to ensure consistency, modularity, and dependency accuracy in multiple system decomposition tasks\.

These results are first\-order observations: Auto\-DSM degrades with larger decompositions and multi\-subsystem scope, exhibiting over\-generalization, first\-entity bias, and weak boundary acknowledgment, while aggregate metrics may mask cell\-level errors rooted in pretrained patterns\. Differences across complexity factors—including writing style, subsystem structure, and multi\-system hierarchy—should be carefully examined to clarify overall behavior and to guide strategies for reducing errors through stricter prompt design, subsystem\-aware processing, and text\-to\-dependency traceability\.

## VIDiscussion

### VI\-AEvaluation Boundaries and Dataset Circularity

The use of ChatGPT to construct both the ground\-truth dataset and the input documentation introduces a circular evaluation risk\. By evaluating Auto\-DSM on documents structurally aligned with its own pretraining distribution, performance may reflect pattern recognition rather than genuine system reasoning\. While this approach offers control for a first\-order evaluation, future research must extend the framework to domain\-specific, out\-of\-distribution systems\. Testing with proprietary industrial data would allow clearer distinction between memorization and generalizable decomposition capabilities\.

### VI\-BReasoning Under Complexity and Context Scarcity

The complexity experiments highlight Auto\-DSM’s limitations in handling ambiguous inputs, multi\-subsystem hierarchies, and low\-context prompts\. While component identification remained robust, dependency identification degraded as document length and structural depth increased\. These results align with findings from Shi et al\.\[[45](https://arxiv.org/html/2607.05985#bib.bib52)\], suggesting that in the absence of strong context, LLMs revert to prior training distributions\. This behavior is especially evident in overgeneralization, hallucinated links, and first\-entity bias, indicating that increased prompt clarity alone is insufficient without architectural support for contextual integration\.

### VI\-CPrompt Sensitivity and Interaction Definitions

Auto\-DSM’s outputs are highly sensitive to both the formulation and consistency of input definitions\. Bi\-directional phrasing improved symmetry and reduced entropy, but confidently incorrect classifications persisted, especially for undefined or contradictorily defined interactions\. These effects reveal a lack of internal checks for semantic consistency, and highlight the necessity of standardized interaction schemas within the pipeline\. Furthermore, when definitions are redundant or conflicting, the model’s outputs vary depending on the order and phrasing of input, revealing its sensitivity to sequence effects and susceptibility to prompt\-based bias\. This highlights the need for stricter control over input parameter formatting\.

### VI\-DCurrent Pipeline Limitations

Despite promising initial results, the Auto\-DSM pipeline is not yet robust for deployment in complex or safety\-critical engineering contexts\. Major limitations include:

- •Limited presence of hallucination detection or abstention mechanisms\.
- •No support for multi\-layered or hierarchical system structures\.
- •High sensitivity to prompt wording and insufficient robustness to low\-context scenarios\.
- •No internal error handling or validation of malformed inputs\.
- •Lack of dependency traceability prevents verification of identified interactions, making it difficult to trace hallucinated or incorrect links back to their source in the input document\.

The evaluation focused primarily on dependency identification\. Although component identification remained stable across experiments, future use cases may demand extended metrics to assess component recognition, especially in larger systems or those with non\-standard naming conventions\.

### VI\-EBeyond DSM: Toward Flexible Representations

Current DSM representations, while structured and interpretable, may not fully capture the goal of the decomposition\. Future work could explore integrating Auto\-DSM outputs into executable system representations, such as the Elephant Specification Language \(ESL\) or constraint\-based models\. This could support richer downstream tasks, including simulation, validation, and automated design\-space exploration\.

### VI\-FPathways for Improvement

Refinements to the current framework should include:

- •Schema\-constrained prompt engineering with fixed dependency categories\.
- •Subsystem\-aware architecture to support multi\-layer decomposition\.
- •Retrieval\-augmented generation or agent\-based verification to reduce hallucination\.
- •Evaluation on unseen, real\-world datasets to measure generalization performance\.
- •Investigation of alternative linguistic formulations for dependencies \(e\.g\., “depends on”, “interacts with”\) to assess model robustness to phrasing variations\.

These directions will contribute to a more interpretable, accurate, and scalable system decomposition methodology that bridges data\-driven reasoning with engineering traceability\.

## VIIConclusion

This paper presented a black\-box evaluation framework for assessing LLM\-based system decomposition in the context of Design Structure Matrices \(DSM\)\. The framework was designed to be model\-agnostic, robust to evolving system complexity, and capable of capturing both structural accuracy and behavioral patterns in LLM outputs\. It introduced metrics for correctness, consistency, abstention, and entropy—providing a comprehensive lens for evaluating performance beyond raw accuracy\.

To address the central research question, the framework enables evaluation of decomposition quality, dependency classification, and abstention behavior using controlled, ground\-truth datasets\. Repeated experiments across prompt formulations, dependency definitions, and complexity levels exposed several behavioral patterns: overconfidence in hallucinated links, sensitivity to input phrasing, first\-entity bias in composite system descriptions, and semantic inconsistency under complexity\. These outcomes confirm that even within known systems, Auto\-DSM struggles to generalize robustly, especially when semantic clarity or contextual cues are weak\.

While the current evaluation leverages training\-aligned documents for initial validation, future extensions must target systems unfamiliar to the model to test true generalization\. Moreover, the findings motivate several architectural and methodological refinements, including improved prompt schema design, multi\-system input handling, and integration with more expressive representations such as ESL or constraint models\.

Ultimately, this paper provides a foundational evaluation strategy for automated system decomposition and a roadmap for advancing LLM\-enabled engineering tools within the DSM domain\. As system complexity increases and models evolve, such frameworks will be essential for ensuring traceable, verifiable, and reliable integration of AI into engineering design processes\.

## Acknowledgement

The authors acknowledges that ChatGPT \(powered by OpenAI’s language model, GPT\-4o; http://openai\.com\) was used for the purpose of grammatical check and content rephrasing\. The editing was performed by the human author\.

## References

- \[1\]\(2003\)Software testing techniques\.Dreamtech\.Cited by:[§III\-A](https://arxiv.org/html/2607.05985#S3.SS1.p2.1)\.
- \[2\]Y\. Belinkov and Y\. Bisk\(2017\)Synthetic and natural noise both break neural machine translation\.arXiv preprint arXiv:1711\.02173\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.p1.4)\.
- \[3\]Y\. Bengio, Y\. Lecun, and G\. Hinton\(2021\-06\)Deep learning for ai\.Commun\. ACM64\(7\),pp\. 58–65\.External Links:ISSN 0001\-0782,[Document](https://dx.doi.org/10.1145/3448250)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p4.1),[§I](https://arxiv.org/html/2607.05985#S1.p5.1)\.
- \[4\]X\. Bouthillier, P\. Delaunay, M\. Bronzi, A\. Trofimov, B\. Nichyporuk, J\. Szeto, N\. Mohammadi Sepahvand, E\. Raff, K\. Madan, V\. Voleti,et al\.\(2021\)Accounting for variance in machine learning benchmarks\.Proceedings of Machine Learning and Systems3,pp\. 747–769\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px8.p1.3)\.
- \[5\]X\. Bouthillier, C\. Laurent, and P\. Vincent\(2019\)Unreproducible research is reproducible\.InInternational Conference on Machine Learning,pp\. 725–734\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.p1.4)\.
- \[6\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[5th item](https://arxiv.org/html/2607.05985#S4.I3.i5.p1.1)\.
- \[7\]T\. R\. Browning\(2002\)Applying the design structure matrix to system decomposition and integration problems: a review and new directions\.IEEE Transactions on Engineering management48\(3\),pp\. 292–306\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px7.p1.1)\.
- \[8\]T\. Browning\(2015\-11\)Design structure matrix extensions and innovations: a survey and new opportunities\.IEEE Transactions on Engineering Management63,pp\. 1–26\.External Links:[Document](https://dx.doi.org/10.1109/TEM.2015.2491283)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p1.1),[§I](https://arxiv.org/html/2607.05985#S1.p2.1)\.
- \[9\]S\. Bubeck, V\. Chandrasekaran, R\. Eldan, J\. Gehrke, E\. Horvitz, E\. Kamar, P\. Lee, Y\. T\. Lee, Y\. Li, S\. Lundberg,et al\.\(2023\)Sparks of artificial general intelligence: early experiments with gpt\-4\.arXiv preprint arXiv:2303\.12712\.Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p4.1),[§I](https://arxiv.org/html/2607.05985#S1.p5.1)\.
- \[10\]K\. X\. Campo, T\. Teper, C\. E\. Eaton, A\. M\. Shipman, G\. Bhatia, and B\. Mesmer\(2023\)Model\-based systems engineering: evaluating perceived value, metrics, and evidence through literature\.Systems Engineering26\(1\),pp\. 104–129\.Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p3.1)\.
- \[11\]N\. Chinchor and B\. M\. Sundheim\(1993\)MUC\-5 evaluation metrics\.InFifth Message Understanding Conference \(MUC\-5\): Proceedings of a Conference Held in Baltimore, Maryland, August 25\-27, 1993,Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px9.p1.6)\.
- \[12\]C\. Chow\(2003\)On optimum recognition error and reject tradeoff\.IEEE Transactions on information theory16\(1\),pp\. 41–46\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px7.p2.2)\.
- \[13\]P\. J\. Clarkson, C\. Simons, and C\. Eckert\(2004\-10\)Predicting change propagation in complex design\.Journal of Mechanical Design126\(5\),pp\. 788–797\.External Links:[Document](https://dx.doi.org/10.1115/1.1765117)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p2.1)\.
- \[14\]C\. Colas, O\. Sigaud, and P\. Oudeyer\(2018\)How many random seeds? statistical power analysis in deep reinforcement learning experiments\.External Links:1806\.08295,[Link](https://arxiv.org/abs/1806.08295)Cited by:[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p3.1),[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px2.p9.3)\.
- \[15\]F\. M\. Dambietz, C\. Rennpferdt, M\. Hanna, and D\. Krause\(2021\)Using mbse for the enhancement of consistency and continuity in modular product\-service\-system architectures\.Systems9\(3\)\.External Links:[Document](https://dx.doi.org/10.3390/systems9030063)Cited by:[§IV\-A2](https://arxiv.org/html/2607.05985#S4.SS1.SSS2.Px1.p2.1)\.
- \[16\]P\. De Saqui\-Sannes, R\. A\. Vingerhoeds, C\. Garion, and X\. Thirioux\(2022\)A taxonomy of mbse approaches by languages, tools and methods\.IEEE Access10,pp\. 120936–120950\.Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p3.1)\.
- \[17\]Q\. Dong\(2002\)Building and creating a dsm\(Website\)DSMweb\.org\.Note:Accessed: Aug\. 20, 2025External Links:[Link](https://dsmweb.org/building-and-creating-a-dsm/)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p2.1),[§I](https://arxiv.org/html/2607.05985#S1.p5.1)\.
- \[18\]R\. Dror, G\. Baumer, S\. Shlomov, and R\. Reichart\(2018\)The hitchhiker’s guide to testing statistical significance in natural language processing\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1383–1392\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.p1.4)\.
- \[19\]S\. Ekin\(2023\)Prompt Engineering For ChatGPT: A Quick Guide To Techniques, Tips, And Best Practices\.TechRxiv Preprint\.Note:DOI: 10\.36227/techrxiv\.22683919\.v2Cited by:[1st item](https://arxiv.org/html/2607.05985#S4.I3.i1.p1.1),[4th item](https://arxiv.org/html/2607.05985#S4.I3.i4.p1.1)\.
- \[20\]R\. El\-Yanivet al\.\(2010\)On the foundations of noise\-free selective classification\.\.Journal of Machine Learning Research11\(5\)\.Cited by:[§III\-B1](https://arxiv.org/html/2607.05985#S3.SS2.SSS1.p11.1),[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px7.p2.2)\.
- \[21\]C\. Elkan\(2001\)The foundations of cost\-sensitive learning\.InInternational joint conference on artificial intelligence,Vol\.17,pp\. 973–978\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px7.p1.1)\.
- \[22\]S\. Eppinger and T\. Browning\(2012\-05\)Design structure matrix methods and applications\.External Links:ISBN 9780262301428,[Document](https://dx.doi.org/10.7551/mitpress/8896.001.0001)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p1.1),[§I](https://arxiv.org/html/2607.05985#S1.p2.1),[§I](https://arxiv.org/html/2607.05985#S1.p3.1),[§I](https://arxiv.org/html/2607.05985#S1.p5.1)\.
- \[23\]S\. F\-Schmitz, D\. C\. Wynn, W\. Biedermann, P\. J\. Clarkson, and U\. Lindemann\(2011\)Improving data quality in dsm modelling: a structural comparison approach\.InICED 11 \- 18th International Conference on Engineering Design \- Impacting Society Through Engineering Design,ICED 11 \- 18th International Conference on Engineering Design \- Impacting Society Through Engineering Design,pp\. 369–380\(English\)\.External Links:ISBN 9781904670247Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px2.p8.1),[§V\-A2](https://arxiv.org/html/2607.05985#S5.SS1.SSS2.p1.2)\.
- \[24\]T\. Fawcett\(2006\)An introduction to roc analysis\.Pattern recognition letters27\(8\),pp\. 861–874\.Cited by:[§III\-B1](https://arxiv.org/html/2607.05985#S3.SS2.SSS1.p8.3),[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px9.p1.6)\.
- \[25\]J\. L\. Fleiss, J\. C\. Nee, and J\. R\. Landis\(1979\)Large sample variance of kappa in the case of different sets of raters\.\.Psychological bulletin86\(5\),pp\. 974\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px2.p3.4)\.
- \[26\]Y\. Geifman and R\. El\-Yaniv\(2019\)Selectivenet: a deep neural network with an integrated reject option\.InInternational conference on machine learning,pp\. 2151–2159\.Cited by:[§III\-B1](https://arxiv.org/html/2607.05985#S3.SS2.SSS1.p11.1)\.
- \[27\]D\. J\. Hand\(2009\)Measuring classifier performance: a coherent alternative to the area under the roc curve\.Machine learning77\(1\),pp\. 103–123\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px8.p1.3)\.
- \[28\]A\. Hennig, T\. G\. Topcu, and Z\. Szajnfarber\(2021\)Complexity should not be in the eye of the beholder: how representative complexity measures respond to the commonly\-held beliefs of the literature\.InInternational Design Engineering Technical Conferences and Computers and Information in Engineering Conference,Vol\.85420,pp\. V006T06A047\.Cited by:[Figure 1](https://arxiv.org/html/2607.05985#S4.F1),[§IV\-A2](https://arxiv.org/html/2607.05985#S4.SS1.SSS2.Px2.p1.1)\.
- \[29\]K\. Hölttä\-Otto and O\. De Weck\(2007\)Degree of modularity in engineering systems and products with technical and business constraints\.Concurrent Engineering15\(2\),pp\. 113–126\.Cited by:[§III\-B1](https://arxiv.org/html/2607.05985#S3.SS2.SSS1.p3.4)\.
- \[30\]L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin, and T\. Liu\(2025\-01\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Trans\. Inf\. Syst\.43\(2\)\.External Links:ISSN 1046\-8188,[Document](https://dx.doi.org/10.1145/3703155)Cited by:[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p1.1)\.
- \[31\]Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. Fung\(2023\-03\)Survey of hallucination in natural language generation\.ACM Comput\. Surv\.55\(12\)\.External Links:[Document](https://dx.doi.org/10.1145/3571730)Cited by:[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p1.1)\.
- \[32\]E\. C\.Y\. Koh\(2024\)Auto\-DSM: Using a Large Language Model to Generate a Design Structure Matrix\.Natural Language Processing Journal9,pp\. 100103\.External Links:ISSN 2949\-7191,[Document](https://dx.doi.org/10.1016/j.nlp.2024.100103)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p5.1),[§I](https://arxiv.org/html/2607.05985#S1.p6.1),[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p1.1),[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p2.1),[TABLE I](https://arxiv.org/html/2607.05985#S2.T1),[TABLE I](https://arxiv.org/html/2607.05985#S2.T1.1.1.1.2.1),[§III\-A](https://arxiv.org/html/2607.05985#S3.SS1.p1.1),[§III\-B1](https://arxiv.org/html/2607.05985#S3.SS2.SSS1.p3.4),[§IV\-C2](https://arxiv.org/html/2607.05985#S4.SS3.SSS2.p1.1),[§V\-A3](https://arxiv.org/html/2607.05985#S5.SS1.SSS3.p1.1)\.
- \[33\]P\. A\. Lakshminarayanan and A\. Agarwal\(2020\-01\)Design and development of heavy duty diesel engines a handbook: a handbook\.External Links:ISBN 978\-981\-15\-0969\-8,[Document](https://dx.doi.org/10.1007/978-981-15-0970-4)Cited by:[TABLE X](https://arxiv.org/html/2607.05985#A1.T10),[TABLE XI](https://arxiv.org/html/2607.05985#A1.T11),[TABLE XII](https://arxiv.org/html/2607.05985#A1.T12),[§II\-B](https://arxiv.org/html/2607.05985#S2.SS2.p2.1),[§II\-C](https://arxiv.org/html/2607.05985#S2.SS3.p1.1)\.
- \[34\]J\. R\. Landis and G\. G\. Koch\(1977\)An application of hierarchical kappa\-type statistics in the assessment of majority agreement among multiple observers\.Biometrics,pp\. 363–374\.Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px2.p8.1)\.
- \[35\]J\. Li, Y\. Tang, and Y\. Yang\(2025\)Know the unknown: an uncertainty\-sensitive method for llm instruction tuning\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 2972–2989\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.153)Cited by:[§V\-A2](https://arxiv.org/html/2607.05985#S5.SS1.SSS2.p2.1),[§V\-B2](https://arxiv.org/html/2607.05985#S5.SS2.SSS2.p4.1)\.
- \[36\]C\. C\. S\. Liem and A\. Panichella\(2020\)Run, forest, run? on randomization and reproducibility in predictive software engineering\.CoRRabs/2012\.08387\.External Links:[Link](https://arxiv.org/abs/2012.08387),2012\.08387Cited by:[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p3.1)\.
- \[37\]N\. Madhusudhan, S\. T\. Madhusudhan, V\. Yadav, and M\. Hashemi\(2024\)Do llms know when to not answer? investigating abstention abilities of large language models\.External Links:2407\.16221,[Link](https://arxiv.org/abs/2407.16221)Cited by:[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p2.1)\.
- \[38\]A\. M\. Madni and M\. Sievers\(2018\)Model\-based systems engineering: motivation, current status, and research opportunities\.Systems Engineering21\(3\),pp\. 172–190\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/sys.21438)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p3.1),[§I](https://arxiv.org/html/2607.05985#S1.p5.1)\.
- \[39\]A\. D\. Marwick\(2001\-10\)Knowledge management technology\.IBM Syst\. J\.40\(4\),pp\. 814–830\.External Links:[Document](https://dx.doi.org/10.1147/sj.404.0814)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p2.1)\.
- \[40\]G\.J\. Myers, C\. Sandler, and T\. Badgett\(2011\)The art of software testing\.ITPro collection,Wiley\.Cited by:[§III\-A](https://arxiv.org/html/2607.05985#S3.SS1.p2.1)\.
- \[41\]F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg,et al\.\(2011\)Scikit\-learn: machine learning in python\.the Journal of machine Learning research12,pp\. 2825–2830\.Cited by:[§III\-B1](https://arxiv.org/html/2607.05985#S3.SS2.SSS1.p8.3)\.
- \[42\]R\. S\. Pressman\(2005\)Software engineering: a practitioner’s approach\.Palgrave macmillan\.Cited by:[§III\-A](https://arxiv.org/html/2607.05985#S3.SS1.p2.1)\.
- \[43\]P\. Rubens\(2002\)Science and technical writing: a manual of style\.Routledge\.Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p2.1),[§I](https://arxiv.org/html/2607.05985#S1.p5.1)\.
- \[44\]I\. Rus and M\. Lindvall\(2002\-05\)Guest Editors’ Introduction: Knowledge Management in Software Engineering\.IEEE Software19\(03\),pp\. 26–38\.External Links:ISSN 1937\-4194,[Document](https://dx.doi.org/10.1109/MS.2002.1003450)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p2.1)\.
- \[45\]W\. Shi, X\. Han, M\. Lewis, Y\. Tsvetkov, L\. Zettlemoyer, and W\. Yih\(2024\)Trusting your evidence: hallucinate less with context\-aware decoding\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 783–791\.Cited by:[§VI\-B](https://arxiv.org/html/2607.05985#S6.SS2.p1.1)\.
- \[46\]D\. V\. Steward\(1981\)The design structure system: a method for managing the design of complex systems\.IEEE Transactions on Engineering ManagementEM\-28\(3\),pp\. 71–74\.External Links:[Document](https://dx.doi.org/10.1109/TEM.1981.6448589)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p1.1)\.
- \[47\]A\. Tilstra, C\. Seepersad, and K\. Wood\(2012\-11\)A High\-definition Design Structure Matrix \(HDDSM\) for the Quantitative Assessment of Product Architecture\.Journal of Engineering Design23,pp\. 1–23\.External Links:[Document](https://dx.doi.org/10.1080/09544828.2012.706748)Cited by:[§III\-B2](https://arxiv.org/html/2607.05985#S3.SS2.SSS2.Px2.p8.1),[§IV\-C1](https://arxiv.org/html/2607.05985#S4.SS3.SSS1.p4.3),[§IV\-C2](https://arxiv.org/html/2607.05985#S4.SS3.SSS2.Px1.p3.1),[§V\-B1](https://arxiv.org/html/2607.05985#S5.SS2.SSS1.Px2.p2.1)\.
- \[48\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou\(2023\)Chain\-of\-Thought Prompting Elicits Reasoning in Large Language Models\.arXiv preprint arXiv:2201\.11903\.Cited by:[3rd item](https://arxiv.org/html/2607.05985#S4.I3.i3.p1.1)\.
- \[49\]B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. Wang\(2025\-06\)Know your limits: a survey of abstention in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 529–556\.External Links:ISSN 2307\-387X,[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00754)Cited by:[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p2.1)\.
- \[50\]J\. White, P\. Walkinshaw, B\. Lamb, C\. Zhang, A\. Shrestha, M\. Gerndt, H\. Eskandari, G\. Karsai, and A\. Dubey\(2023\)A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT\.InProc\. 1st International Workshop on Engineering with Chatbots \(BotSE@ICSE\),pp\. 1–9\.Cited by:[1st item](https://arxiv.org/html/2607.05985#S4.I3.i1.p1.1),[2nd item](https://arxiv.org/html/2607.05985#S4.I3.i2.p1.1)\.
- \[51\]F\. Wilking, D\. Horber, S\. Goetz, and S\. Wartzack\(2024\)Utilization of system models in model\-based systems engineering: definition, classes and research directions based on a systematic literature review\.Design Science10,pp\. e6\.Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p3.1)\.
- \[52\]T\. Wilschut, L\. Etman, J\. Rooda, and J\. Vogel\(2018\)Generation of a function\-component\-parameter multi\-domain matrix from structured textual function specifications\.Research in Engineering Design29\(4\),pp\. 531–546\.Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p2.1),[§I](https://arxiv.org/html/2607.05985#S1.p5.1)\.
- \[53\]T\. Wilschut, A\. T\. Hofkamp, T\. J\. L\. Schuijbroek, L\. F\. P\. Etman, and J\. E\. Rooda\(2024\)Bridging the Gap Between Requirements Engineering and Systems Architecting: The Elephant Specification Language\.Design Science10,pp\. e25\.External Links:[Document](https://dx.doi.org/10.1017/dsj.2024.30)Cited by:[§I](https://arxiv.org/html/2607.05985#S1.p3.1),[§IV\-C2](https://arxiv.org/html/2607.05985#S4.SS3.SSS2.Px1.p3.1)\.
- \[54\]Y\. A\. Yadkori, I\. Kuzborskij, D\. Stutz, A\. György, A\. Fisch, A\. Doucet, I\. Beloshapka, W\. Weng, Y\. Yang, C\. Szepesvári,et al\.\(2024\)Mitigating llm hallucinations via conformal abstention\.arXiv preprint arXiv:2405\.01563\.Cited by:[§II\-A](https://arxiv.org/html/2607.05985#S2.SS1.p2.1)\.

## Appendix ACurrent situation Auto\-DSM

TABLE X:Occurrence Frequency \(%\) Over 10 Runs of Auto‑DSM \(Engine & Mechanical; Dataset\[[33](https://arxiv.org/html/2607.05985#bib.bib36)\]\)TABLE XI:Most Occurring Response Over 10 Runs of Auto‑DSM \(Engine & Mechanical; Dataset\[[33](https://arxiv.org/html/2607.05985#bib.bib36)\]\)Most occurring valueCylinder headCrankcaseConnecting rodPistonsCamshaftCrankshaftCylinder head110110Crankcase111051Connecting rod011101Pistons111111Camshaft100011Crankshaft111111TABLE XII:Number of Rejection Responses Over 10 Runs of Auto‑DSM \(Engine & Mechanical; Dataset\[[33](https://arxiv.org/html/2607.05985#bib.bib36)\]\)
## Appendix BPrompt Engineering

Prompt 1 \- Define System Structure\# Role: You are a systems engineer specialized in technical system decomposition and dependency analysis\. \# Task: Generate a complete system decomposition of a standard household refrigerator, existing of subsystems and components\. \# Instructions: \- Identify all main subsystems\. \- For each subsystem, list its key components\. \- Use only realistic and commonly found components\. \# Constraints: \- Do not invent components or subsystems\. \- Use consistent formatting across all subsystems\. \# Output Format: \[Subsystem Name 1\] • \[Component A\] • \[Component B\] \.\.\. \[Subsystem Name 2\] • \[Component F\] • \[Component G\] \.\.\.

Figure 9:Prompt 1 used to define the refrigerator subsystem decomposition and its components\.Prompt 2 \- Intra\-Subsystem Dependency Mapping\# Role: You are a systems engineer specialized in technical system decomposition and dependency analysis\. \# Task: Identify all intra\-subsystem dependencies between components in a single subsystem\. \# Instructions: \- For each component pair within the same subsystem, determine whether a dependency exists\. \- If yes, classify it using only Tilstra’s \(2012\) HDDSM interaction types\. \# Constraints: \- Do not invent components or interaction types\. \- Complete one subsystem at a time\. \- Output must follow the structured tabular format\. \# Interaction Types \(Tilstra, 2012\): 1\. Information: 1\.1 Status \[SI\], 1\.2 Control \[CI\] 2\. Material: 2\.1 Human \[HM\], 2\.2 Gas \[GM\], 2\.3 Liquid \[LM\], 2\.4 Solid \[SM\], 2\.5 Plasma \[PM\], 2\.6 Mixture \[MM\] 3\. Energy: 3\.1 Human \[HE\], 3\.2 Acoustic \[AE\], 3\.3 Biological \[BE\], 3\.4 Chemical \[CE\], 3\.5 Electrical \[EE\], 3\.6 Electromagnetic \[EME\], 3\.7 Hydraulic \[HYE\], 3\.8 Mechanical \[ME\], 3\.9 Magnetic \[MAG\], 3\.10 Pneumatic \[PE\], 3\.11 Radioactive \[NE\], 3\.12 Thermal \[TE\], 3\.13 Strain energy \[SE\] 4\. Spatial: 4\.1 Proximity \[P\], 4\.2 Alignment \[A\] 5\. Movement: 5\.1 Translational \[LRM\], 5\.2 Rotational \[RRM\] \# Subsystem and its decomposition: \{Insert Subsystem \+ Components from output prompt 1\} \# Output Format: \[Subsystem; From \(Component\); To \(Component\); Interaction Type\(s\); \.\.\.; Target Subsystem\] \# Example Template: Subsystem 1; Component A; Component B; Interaction Type 1 \[Abbreviation\]; Interaction Type 2 \[Abbreviation\]; \.\.\.; Subsystem 1

Figure 10:Prompt 2 for intra\-subsystem interaction identification using Tilstra \(2012\) HDDSM\.Prompt 3 \- Inter\-Subsystem Dependency Mapping \(Optimized Version\)\# Role: You are a systems engineer specialized in technical system decomposition and dependency analysis\. \# Task: Identify all inter\-subsystem dependencies between components of the specified Subsystem and the remaining Subsystems\. \# Specified Subsystem and its decomposition: Name: \{Insert Subsystem A \+ Components from prompt 1\} \# Remaining Subsystems: \{Insert remaining Subsystems \+ Components from prompt 1\} \(previously defined\) \# Instructions: \- Compare each component from the specified Subsystem to all the components of the remaining Subsystems\. \- Use \*\*only the components defined\*\* in the reamining subsystem list for the dependency analysis\. \- If a valid dependency exists, classify it using the interaction types from Tilstra’s \(2012\) HDDSM framework\. \# Constraints: \- Do \*\*not invent\*\* any components or interaction types\. \- Do \*\*not assume\*\* new relationships\. Only use information available in the subsystem decomposition\. \- Follow the output format precisely\. \# Output Format: \[Subsystem; From \(Component\); To \(Component\); Interaction Type\(s\); \.\.\.; Target Subsystem\] \# Example Template: Subsystem 1; Component A; Component G; Interaction Type 1 \[Abbreviation\]; Interaction Type 2 \[Abbreviation\]; \.\.\.; Subsystem 2

Figure 11:Prompt 3 for inter\-subsystem dependency mapping using explicit injection of a specified Subsystem and controlled reference to the remaining subsystems\.Prompt 1 \- System Decomposition Article Generation \(Excel\-based\)\# Role: You are a mechanical systems engineer and technical writer, specialized in decomposing systems\. Your expertise lies in translating structured technical data into formal, traceable system decomposition reports for engineering documentation\. \# Task: Write a clear, technically accurate, traceable and well\-structured system decomposition article for the subsystem∗⁣∗\*\*SUBSYSTEM\_NAME∗⁣∗\*\*, which is part of the system∗⁣∗\*\*Refrigerator∗⁣∗\*\*\. Use the structured Excel data as source for the system decomposition article\. This Excel file contains one row per interaction, formatted according to the structure defined below\. \# Data Format: Each row contains: 1\. \[INDEX\] — Unique ID for the interaction 2\. \[Subsystem\] — Subsystem of the source component \(Component A\) 3\. \[From \(Component\)\] — Source component \(Component A\) 4\. \[To \(Component\)\] — Target component \(Component B\) 5–7\. \[Interaction Type\(s\)\] — One or more interaction types \(Tilstra classification\) 8\. \[Target Subsystem\] — Subsystem of the target component \(Component B\) Use the following interaction classification when interpreting types: 1\. Information \- \[SI\] Status \- \[CI\] Control 2\. Material \- \[HM\] Human \- \[GM\] Gas \- \[LM\] Liquid \- \[SM\] Solid \- \[PM\] Plasma \- \[MM\] Mixture 3\. Energy \- \[HE\] Human \- \[AE\] Acoustic \- \[BE\] Biological \- \[CE\] Chemical \- \[EE\] Electrical \- \[EME\] Electromagnetic \- \[HYE\] Hydraulic \- \[ME\] Mechanical \- \[MAG\] Magnetic \- \[PE\] Pneumatic \- \[NE\] Radioactive \- \[TE\] Thermal \- \[SE\] Strain energy 4\. Spatial \- \[P\] Proximity \- \[A\] Alignment 5\. Movement \- \[LRM\] Translational \- \[RRM\] Rotational When referring to an interaction, include both its name \+ category and abbreviation, e\.g\., Thermal Energy \[TE\]\. \# Instructions: The article should: 1\. Begin with a concise, high\-level introduction explaining the role and function of ‘SUBSYSTEM\_NAME‘ within the full system ‘SYSTEM‘\. 2\. Present all interactions in a smooth, technical narrative\. 3\. Discuss each interaction once, referring to its unique ‘\[INDEX\]‘ \(e\.g\., "\[5\]"\) for traceability\. 4\. Indicate whether each interaction is intra\-subsystem \(within the same subsystem\) or inter\-subsystem \(across subsystems\)\. 5\. For each interaction: \- Name the source and target components\. \- Describe all interaction types using their \*\*Tilstra classification\*\*\. \- Group related components or functions logically to enhance readability and flow\. 6\. Write in fluent, formal, technical English, consistent with engineering documentation standards\. \# Constraints: \- \*\*No new components, interactions, or interaction types\*\* may be added, inferred, or assumed beyond the input data\. \- Use \*\*every input row exactly once\*\*, no skipping or duplication\. \- Only use interaction types from the \*\*Tilstra \(2012\)\*\* classification\. Do not modify, rename, or combine categories\. \- Always include both the \*\*full name and abbreviation\*\* for interaction types, e\.g\., \*Thermal Energy \[TE\]\*\. \- Avoid using bullet points or lists in the article unless strictly necessary — maintain a \*\*narrative format\*\*\. \- Do not generalize or invent functionality that is not explicitly described in the data\. \# Example: \*\*Excel Input Row:\*\* INDEX: 1; Subsystem: Refrigeration System; From: Compressor; To: Condenser coil; Interaction Types: Liquid \[LM\], Thermal \[TE\], Mechanical \[ME\]; Target Subsystem: Refrigeration System \*\*Correct Response:\*\* The compressor mechanically and thermally interacts with the condenser coil, delivering pressurized refrigerant in liquid form \[1\]\. As both components belong to the Refrigeration System, this is classified as an intra\-subsystem interaction\. \# Input: Here is the input table, filtered for \{SUBSYSTEM\_NAME\}: Chat\_input\.xlsx Use this data to generate the full system decomposition article as described above\.

Figure 12:Prompt for generating a system decomposition article using structured interaction data and Tilstra’s \(2012\) classification framework\.
## Appendix CDatasets

![Refer to caption](https://arxiv.org/html/2607.05985v1/Figures/2subsystemD_and_M_and_REF_No_Interactions_GT_DSM.png)Figure 13:GT\-DSM for the Two Combined Subsystems: Drainage & Moisture Control System and Refrigeration SystemTABLE XIII:Section of Refrigerator system decomposition datasetTABLE XIV:Subsystem metrics for the refrigerator including component sizes, interaction counts, and NZF values per subsystem
## Appendix DTechnical Documentation

TheDrainage&MoistureControlSystemiscomposedofcomponentsdrainhole,drainpan,evaporationtray,andmoisturecontrolfilters\.Thecomponentsinteracttoensuresystemfunctionality,asdetailedbelow:

ThedrainholeisbothMixtureMaterial\[MM\]andAlignment\[A\]linkedtothedrainpan\[77\]\.ThedrainpanisbothMixtureMaterial\[MM\]andAlignment\[A\]linkedtothedrainhole\[83\]\.

ThedrainpanisbothMixtureMaterial\[MM\]andProximity\[P\]linkedtotheevaporationtray\[84\]\.TheevaporationtrayisbothMixtureMaterial\[MM\]andProximity\[P\]linkedtothedrainpan\[89\]\.

TheevaporationtrayisbothMixtureMaterial\[MM\]andGasMaterial\[GM\]linkedtothemoisturecontrolfilters\[90\]\.ThemoisturecontrolfiltersarebothMixtureMaterial\[MM\]andGasMaterial\[GM\]linkedtotheevaporationtray\[189\]\.

Listing 1:Manual Drainage & Moisture Control System DecompositionTheRefrigerationSystemiscomposedofthecompressor,condensercoil,evaporatorcoil,expansiondevice,piping/tubing,refrigerantfluid,andevaporatorfan\.Thecomponentsinteracttoensuresystemfunctionality,asdetailedbelow:

ThecompressorisLiquidMaterial\[LM\],ThermalEnergy\[TE\]andMechanicalEnergy\[ME\]linkedtothecondensercoil\[1\],andisbothLiquidMaterial\[LM\]andMechanicalEnergylinkedtotherefrigerantfluid\[2\]\.Additionally,thecompressorisProximity\[P\]andMechanical

Energy\[ME\]linkedtothepiping/tubing\[3\]\.

ThecondensercoilisLiquidMaterial\[LM\]andThermalEnergy\[TE\]linkedtotheexpansiondevice\[12\]andrefrigerantfluid\[13\]\.ThecondensercoilisalsoProximity\[P\]andLiquid

Material\[LM\]linkedtothepiping/tubing\[14\]\.Nexttothat,thecondensercoilisLiquidMaterial\[LM\],ThermalEnergy\[TE\]andMechanicalEnergy\[ME\]linkedtothecompressor\[16\]\.

TheevaporatorcoilisLiquidMaterial\[LM\]andThermalEnergy\[TE\]linkedtotheexpansiondevice\[94\]andtherefrigerantfluid\[95\]\.TheevaporatorcoilisProximity\[P\]andLiquidMaterial\[LM\]linkedtothepiping/tubing\[96\]\.TheevaporatorcoilisalsoProximity\[P\],GasMaterial\[GM\],andThermalEnergy\[TE\]linkedtotheevaporatorfan\[106\]\.

Simultaneously,theevaporatorfanisProximity\[P\],GasMaterial\[GM\],andThermalEnergy\[TE\]linkedtotheevaporatorcoil\[107\]\.TheevaporatorfanisProximity\[P\]linkedtothepiping/tubing\[108\],whileProximity\[P\]andGasMaterial\[GM\]linkedtotherefrigerantfluid

\[109\]\.

TheexpansiondeviceisLiquidMaterial\[LM\]andThermalEnergy\[TE\]linkedtoboththecondensercoil\[120\]andevaporatorcoil\[121\],andonlyLiquidMaterial\[LM\]linkedtorefrigerantfluid\[122\]\.Italsoislinkedwiththepiping/tubingthroughProximity\[P\]andLiquidMaterial\[LM\]\[123\]\.

Thepiping/tubingisProximity\[P\]andMechanicalEnergy\[ME\]linkedtothecompressor\[202\],andProximity\[P\]totheevaporatorfan\[206\]\.Thepiping/tubingisProximity\[P\]andLiquidMaterial\[LM\]linkedtothecondensercoil\[203\],expansiondevice\[204\],evaporatorcoil\[205\],andrefrigerantfluid\[207\]\.

TherefrigerantfluidisThermalEnergy\[TE\]andLiquidMaterial\[LM\]linkedtothecondensercoil\[220\],andevaporatorcoil\[222\]\.ItisLiquidMaterial\[LM\]linkedtotheexpansiondevicevia\[221\],andLiquidMaterial\[LM\]andProximity\[P\]linkedtothepiping/tubing\[223\]\.TherefrigerantfluidisLiquidMaterial\[LM\]andMechanicalEnergy\[ME\]linkedtothecompressor\[227\],andisProximity\[P\]andGasMaterial\[GM\]linkedtoevaporatorfan\[228\]\.

Listing 2:Manual Refrigeration System System DecompositionTherefrigerationsubsysteminteractswiththeDrainage&MoistureControlSystem,andallinteractionsdescribedbelowoccurbetweenthesetwosubsystems,makingthemintersubsystem\.

ThecompressorisThermalEnergy\[TE\]andProximity\[P\]linkedtothedrainpan\[9\]\.ThedrainpanisalsoThermalEnergy\[TE\]andProximity\[P\]linkedtothecompressor\[86\]\.

TheevaporatorcoilisProximity\[P\],MixtureMaterial\[MM\]andLiquidMaterial\[LM\]linkedtothedrainhole\[101\]\.ThedrainholealsoisProximity\[P\],MixtureMaterial\[MM\]andLiquidMaterial\[LM\]linkedtotheevaporatorcoil\[79\]\.

ThedrainholeisGasMaterial\[GM\]andThermalEnergy\[TE\]linkedtotheevaporatorfan\[78\]\.Theevaporatorfan,inturn,isGasMaterial\[GM\]andThermalEnergy\[TE\]linkedtothedrainhole\[118\]\.

TheevaporatorfanisGasMaterial\[GM\]andMixtureMaterial\[MM\]linkedtothemoisturecontrolfilters\[119\]\.Similarly,theevaporatorcoilisGasMaterial\[GM\]andMixtureMaterial\[MM\]linkedtothemoisturecontrolfilters\[105\]\.

Lastly,themoisturecontrolfiltersareGasMaterial\[GM\]andMixtureMaterial\[MM\]linkedtotheevaporatorfan\[185\]\.TheyarealsoGasMaterial\[GM\]andMixtureMaterial\[MM\]linkedtotheevaporatorcoil\[188\]\.

Listing 3:Manual Inter\-subsystem Refrigeration and Drainage & Moisture Control System DecompositionSystemDecomposition:Drainage&MoistureControlSystem\(Refrigerator\)

TheDrainage&MoistureControlSystemintherefrigeratorensuresthatexcesswaterandmoistureareeffectivelycollected,transferred,evaporated,andmanaged\.Itsprimaryfunctionistokeeptheinteriordryandfreeofcondensation,whichhelpspreventicebuildup,odors,andmoldformation\.Thissubsystemconsistsoffourkeycomponents:thedrainhole,drainpan,evaporationtray,andmoisturecontrolfilters\.

Theprocessbeginswiththedrainhole,whichdirectsmeltedwaterintothedrainpan\.ThisinteractioninvolvesthemovementofaMixture\[MM\]—typicallywaterwithminorimpurities—andisguidedbyAlignment\[A\]toensureproperflowbetweencomponents\[77\]\.Sincebothelementsbelongtothesamesubsystem,thisisanintra\-subsysteminteraction\.

Inreturn,thedrainpanalsoconnectsbacktothedrainhole,reinforcingthesamepathway\.Itreceivesthewatermixture,wherespatialalignmentplaysakeyroleinmaintainingthecorrectflowdirection\.ThisinteractionagaininvolvesMixture\[MM\]andAlignment\[A\]andremainswithinthesamesubsystem\[83\]\.

Next,thedrainpansendscollectedwatertotheevaporationtray\.ThisisastraightforwardtransferofaMixture\[MM\]aidedbyProximity\[P\],allowingthewatertoflowpassivelybetweenthetwocomponents\[84\]\.Thistooisintra\-subsystem\.

Theevaporationtraycanalsoreturnwatertothedrainpan,forinstanceduringredistributionoroverflow\.Aswiththepreviousinteraction,theevaporationtrayanddrainpaninteractioninvolvesaMixture\[MM\]andreliesonProximity\[P\]toallowtheliquidtomovefreelybetweenbothsurfaces\[89\]\.Theevaporationtrayandthemoisturecontrolfiltersarecloselycoupledinmanagingmoisturereleaseandreabsorption\.Aswaterintheevaporationtrayturnstovapor,theresultingmoisture\-richairflowstowardthefilters\.ThisprocessinvolvesbothMixture\[MM\]andGas\[GM\],capturingthephysicalphasechangeanddirectionalvaportransferfromtraytofilters\[90\]\.Conversely,themoisturecontrolfiltersalsoinfluencetheevaporationenvironmentbyreleasingfilteredorredirectedmoisture\-ladenairbacktowardtheevaporationtray\.ThisreturninteractionlikewiseinvolvesGas\[GM\]andMixture\[MM\],reinforcingtheambienthumidityexchangenearthetray\[189\]\.Bothareintra\-subsysteminteractionsandtogetherformabidirectionalmoistureregulationmechanismthatsupportsefficientandconsistentevaporation\.

Listing 4:Generated Technical Narritive Document Drainage & Moisture Control System Decomposition