NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation

# NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation
Source: [https://arxiv.org/html/2608.07530](https://arxiv.org/html/2608.07530)
11institutetext:Technical University of Munich, Munich, Germany
11email:\{yuchen\.zhou, niels\.bobet, maribel\.acosta\}@tum\.de###### Abstract

SHACL is a core technology for validating the conformance of RDF knowledge graphs \(KGs\)\. Yet, authoring SHACL shapes requires technical expertise that most domain experts lack\. Translating natural language requirements into SHACL \(NL2SHACL\) would lower this barrier\. However, there is no dedicated benchmark for NL2SHACL, and evaluating generated shapes requires methods beyond string comparison, as semantically equivalent shapes can differ in serialisation and structure\. To tackle these challenges, we present NL2SHACL\-Bench, a benchmark suite for natural language to SHACL translation\. Using NL2SHACL\-Bench, we evaluate four state\-of\-the\-art large language models \(LLMs\) for this task\. Our results show that current LLMs are highly capable of generating syntactically valid SHACL, but still struggle to produce semantically equivalent constraints for complex logical and structural patterns\. This indicates that NL2SHACL\-Bench provides a meaningful basis for measuring advances in the NL2SHACL state of the art\. Resource Type:Benchmark Source Code:[https://de\-tum\.github\.io/NL2SHACL\-Bench](https://de-tum.github.io/NL2SHACL-Bench) License:MIT license Dataset:[https://zenodo\.org/records/20082565](https://zenodo.org/records/20082565), CC\-BY\-SA 4\.0

## 1Introduction

The Shapes Constraint Language \(SHACL\)\[[21](https://arxiv.org/html/2608.07530#bib.bib7)\]is a W3C recommendation for validating RDF knowledge graphs \(KGs\)\. It is widely used to ensure data quality and consistency in KG\-based systems, supporting tasks such as compliance checking and metadata validation\[[14](https://arxiv.org/html/2608.07530#bib.bib31)\]\. However, creating SHACL shapes for validating real\-world KGs requires expertise in both the application domain and semantic web technologies, which most domain experts lack\.

In practice, constraints are often expressed in natural language by domain experts and then manually translated into SHACL\[[8](https://arxiv.org/html/2608.07530#bib.bib27)\]\. This process is time\-consuming and error\-prone, motivating the need for automated solutions\. Translating natural language requirements into SHACL \(NL2SHACL\), therefore, offers a promising way to lower this barrier\.

Despite this motivation, NL2SHACL has not yet been systematically studied as a dedicated task\. Existing SHACL resources are typically mined from structured data\[[3](https://arxiv.org/html/2608.07530#bib.bib16),[6](https://arxiv.org/html/2608.07530#bib.bib21),[15](https://arxiv.org/html/2608.07530#bib.bib32)\]or designed for specific domains\[[16](https://arxiv.org/html/2608.07530#bib.bib33)\], and therefore do not provide aligned natural language descriptions\. Moreover, evaluating generated SHACL shapes is non\-trivial, since semantically equivalent shapes may differ substantially in serialization and graph structure, making string\-based comparison insufficient\. As a result, there is currently no dedicated benchmark, standardized dataset, or evaluation methodology for systematic NL2SHACL evaluation\.

To address these challenges, we presentNL2SHACL\-Bench, a benchmark suite for natural language to SHACL translation\. Our suite aims at providing datasets and metrics to assess the performance of NL2SHACL solutions\. To this end, we collect cross\-domain and domain\-specific openly available SHACL shapes and create a dataset that includes natural language specifications or descriptions of these shapes and ontology annotations\. To this end, NL2SHACL\-Bench includes a software framework for creating such datasets\. Lastly, we define and implement a set of metrics capturing syntactic, structural, and semantic equivalence\. In summary, our suite comprises: \(i\) NL2SHACL\-Framework, an extensible pipeline for dataset construction and evaluation; \(ii\) NL2SHACL\-Dataset, currently covering six datasets across five domain\-specific settings and one general domain \(DBpedia\) with 240 manually verified NL–SHACL pairs annotated with ontology information; and \(iii\) NL2SHACL\-Metrics, defining eight metrics across syntactic, structural, and semantic dimensions tailored to assess the quality of translated shapes\. The contributions of this work are as follows:

- •We introduceNL2SHACL\-Bench, the first benchmark for NL\-to\-SHACL translation\.
- •We propose an extensible framework for dataset construction and evaluation\.
- •We construct a multi\-domain dataset of NL–SHACL pairs with ontology annotations and human\-validated descriptions\.
- •We design evaluation metrics that capture syntactic, structural, and semantic equivalence\.

Using NL2SHACL\-Bench, we evaluate several state\-of\-the\-art LLMs to demonstrate the usefulness of the benchmark for systematic NL2SHACL evaluation\.

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2608.07530#S2)reviews related work\. Section[3](https://arxiv.org/html/2608.07530#S3)introduces the foundations of SHACL and formally defines the NL2SHACL translation task\. Section[4](https://arxiv.org/html/2608.07530#S4)presents NL2SHACL\-Bench in detail, describing the framework, dataset, and evaluation metrics\. Section[5](https://arxiv.org/html/2608.07530#S5)reports our experimental evaluation of four state\-of\-the\-art LLMs on the benchmark\. Finally, Section[6](https://arxiv.org/html/2608.07530#S6)concludes the paper and outlines directions for future work\.

## 2Related Work

##### Automatic SHACL Generation

Authoring SHACL shapes manually is a complex and time\-consuming task, which has motivated a range of approaches for automatic shape generation\. Existing methods primarily derive SHACL shapes from structured artefacts such as ontologies or RDF data\. Astrea\[[3](https://arxiv.org/html/2608.07530#bib.bib16)\]generates SHACL shapes from OWL ontologies by mapping ontology axioms to SHACL constraint patterns, using a knowledge graph of mappings \(Astrea\-KG\) to cover both value and model restrictions\. sheXer\[[6](https://arxiv.org/html/2608.07530#bib.bib21)\]mines RDF graph topology to infer constraints, supporting both SHACL and ShEx generation\. QSE\[[15](https://arxiv.org/html/2608.07530#bib.bib32)\]targets scalable extraction from very large knowledge graphs, using statistical measures such as support and confidence to filter unreliable constraints\. SCOOP\[[4](https://arxiv.org/html/2608.07530#bib.bib3)\]extracts constraints from schema definitions, RDF data, and query logs within a single pipeline, while XSD2SHACL\[[5](https://arxiv.org/html/2608.07530#bib.bib2)\]derives shapes from XML Schema \(XSD\) rather than RDF or ontology axioms\. Despite these advances, automatically generated shapes exhibit several limitations\. In particular, data\-driven extraction can introduce spurious constraints due to noisy or incorrect data\. Finally, these approaches typically generate a large number of candidate shapes, making manual validation and selection costly in practice\[[14](https://arxiv.org/html/2608.07530#bib.bib31)\]\. In practice, SHACL constraints are often derived from requirements expressed in natural language by domain experts, which are then translated into formal shapes by knowledge engineers\[[8](https://arxiv.org/html/2608.07530#bib.bib27),[9](https://arxiv.org/html/2608.07530#bib.bib28)\], motivating the need for methods that can automate this process\. To the best of our knowledge, this setting has not been systematically studied, which forms the focus of this work\.

##### SHACL Benchmarks and Datasets

Existing SHACL resources are scattered across public repositories and curated collections\. TheShapes of Youindex aggregates SHACL and ShEx shapes from public Git repositories, but many files are incomplete, inaccessible, or lack accompanying ontologies, and numerous repositories contain only a small number of shapes\. As a result, these resources are not well\-suited for systematic benchmarking\. Several works have proposed datasets involving SHACL shapes\[[16](https://arxiv.org/html/2608.07530#bib.bib33),[9](https://arxiv.org/html/2608.07530#bib.bib28),[15](https://arxiv.org/html/2608.07530#bib.bib32)\]\. Schaffenrath et al\.\[[16](https://arxiv.org/html/2608.07530#bib.bib33)\]construct a benchmark with 58 manually designed shapes for validating a tourism knowledge graph\. Mäkelburg et al\.\[[9](https://arxiv.org/html/2608.07530#bib.bib28)\]provide expert\-defined constraints for an electronic invoicing KG\. Rabbani et al\.\[[15](https://arxiv.org/html/2608.07530#bib.bib32)\]introduce a dataset with SHACL shapes extracted automatically from cross\-domain KGs\. However, these datasets do not include corresponding natural language descriptions\. Overall, existing resources either lack natural language annotations, are domain\-specific, or are not designed for generation tasks\. To date,*no benchmark pairs natural language constraint descriptions with gold\-standard SHACL shapes*, nor has a dedicated evaluation methodology been proposed for NL\-to\-SHACL\. This gap motivates the benchmark introduced in this work\.

##### LLMs for Shapes Management

Zhang et al\.\[[23](https://arxiv.org/html/2608.07530#bib.bib38)\]investigate the use of LLMs for generating ShEx schemas for Wikidata and Yago\. Their approach combines local entity information with global graph context to produce syntactically valid and semantically meaningful ShEx expressions\. However, it focuses on ShEx and on schema extraction from structured knowledge graphs, rather than generation from natural language descriptions\. Westermann et al\.\[[22](https://arxiv.org/html/2608.07530#bib.bib35)\]explore the use of LLMs to translate textual constraints into SHACL shapes within an industrial validation pipeline, effectively addressing the same NL\-to\-SHACL task\. Their results show that LLMs can produce near\-complete SHACL constraints with limited post\-editing\. However, their evaluation relies on manual expert inspection over a proprietary, single\-domain dataset, without standardized metrics or a released benchmark\. Despite these advances, the capability of LLMs to generate SHACL shapes from natural language descriptions has not been systematically studied, and no dedicated benchmark exists for evaluating this task\.

## 3Preliminaries and Problem Formulation

ex:PersonShapea sh:NodeShapesh:targetClassex:Personsh:pathex:status sh:in \(“active”, “inactive”\)sh:pathex:address sh:node ex:AddressShapeex:AddressShapea sh:NodeShapesh:targetClassex:Addresssh:pathex:postalCode sh:or \( \[ sh:pattern"ˆ\[0\-9\]\{5\}$"\], \[ sh:pattern"ˆ\[A\-Z\]\{2\}\[0\-9\]\{4\}$"\] \)referencesh:PropertyShapesh:PropertyShapesh:PropertyShape

Figure 1:Schematic representation of a SHACL shapes graph\.SHACL operates over two graphs: data graph and shapes graph\. Thedata graphcorresponds to the RDF graph𝒢\\mathcal\{G\}to be validated\. We assume𝒢\\mathcal\{G\}is described with an ontologyOO\. In general, a shapes graph is an RDF graph containing zero or more shapes, where both node shapes and property shapes may independently declare targets and trigger validation\[[21](https://arxiv.org/html/2608.07530#bib.bib7)\]\. The definitions below focus onnode\-shape\-rootedshapes graphs, where every property shape is either inlined within a node shape viash:propertyor reachable from one\. Under this scope, a shapes graph𝒮\\mathcal\{S\}is a set of node shapes\{s1,⋯,sn\}\\\{s\_\{1\},\\cdots,s\_\{n\}\\\}\.

Anode shapesis\_\{i\}declares constraints that apply to focus nodes, which are selected via atarget, e\.g\.,sh:targetClass, which selects all instances of a given class inOO\. Other targets includesh:targetSubjectsOf, andsh:targetObjectsOf, orsh:targetNode\. Within a node shape,property shapes\(introduced viash:property\) describe constraints on values reachable through a specific path, using predicates defined inOO, declared withsh:path\. The actual restrictions come fromconstraint componentssuch assh:and,sh:or,sh:in,sh:minCount,sh:datatype,sh:pattern,sh:class, orsh:nodeKind, each parameterized by values the shape author supplies\. A node shapesis\_\{i\}can contain references to another node shapesjs\_\{j\}\(allowingi=ji=j\)\. Figure[1](https://arxiv.org/html/2608.07530#S3.F1)illustrates these concepts\. A shapes graphs𝒮\\mathcal\{S\}can be serialized as an RDF graph, and we denote\|𝒮\|\|\\mathcal\{S\}\|as its number of RDF triples\.

The validation of a𝒮\\mathcal\{S\}over a𝒢\\mathcal\{G\}produces a report withconformanceresults\.𝒢⊧𝒮\\mathcal\{G\}\\models\\mathcal\{S\}denotes that the data graph conforms to the shapes graph\[[1](https://arxiv.org/html/2608.07530#bib.bib1)\]\. Two shapes graphs𝒮\\mathcal\{S\}and𝒮′\\mathcal\{S\}^\{\\prime\}are*equivalent*, written𝒮≡𝒮′\\mathcal\{S\}\\equiv\\mathcal\{S\}^\{\\prime\}, iff∀𝒢:𝒢⊧𝒮⇔𝒢⊧𝒮′\\forall\\mathcal\{G\}:\\ \\mathcal\{G\}\\models\\mathcal\{S\}\\ \\Leftrightarrow\\ \\mathcal\{G\}\\models\\mathcal\{S\}^\{\\prime\}\.

Following a task formulation similar to\[[1](https://arxiv.org/html/2608.07530#bib.bib1)\], we define the NL2SHACL translation task as follows\.

###### Definition 1\(NL2SHACL Translation\)

Given a natural language specificationTTdescribing constraints over a domain, and an ontologyOOproviding the vocabulary \(classes and properties\) referenced byTT\. The task is to produce a shapes graph𝒮\\mathcal\{S\}overOOs\.t\.𝒮\\mathcal\{S\}encodes the constraints expressed inTT\. A translator is a mappingf:\(T,O\)↦𝒮f:\(T,O\)\\mapsto\\mathcal\{S\}\. We sayffiscorrecton\(T,O\)\(T,O\)iff𝒮=f​\(T,O\)\\mathcal\{S\}=f\(T,O\)is equivalent to a reference shapes graph𝒮∗\\mathcal\{S\}^\{\*\}that captures the semantics ofTT, i\.e\.,

∀𝒢overO:𝒢⊧f\(T,O\)⇔𝒢⊧𝒮∗\.\\forall\\mathcal\{G\}\\text\{ over \}O:\\quad\\mathcal\{G\}\\models f\(T,O\)\\;\\Leftrightarrow\\;\\mathcal\{G\}\\models\\mathcal\{S\}^\{\*\}\.

## 4NL2SHACL\-Bench

To address the lack of dedicated resources for natural language to SHACL translation and the limitations of existing evaluation approaches, we presentNL2SHACL\-Bench, a benchmark suite for the NL2SHACL task\. This benchmark is designed to provide a standardized basis for evaluating model performance on translating natural language requirements into SHACL constraints\. NL2SHACL\-Bench consists of three main components: an extensible framework for dataset construction and evaluation \(§[4\.1](https://arxiv.org/html/2608.07530#S4.SS1)\); a semi\-automatically curated dataset covering multiple domains \(§[4\.2](https://arxiv.org/html/2608.07530#S4.SS2)\); and a set of tailored evaluation metrics \(§[4\.3](https://arxiv.org/html/2608.07530#S4.SS3)\)\. In the following, we introduce each component in detail\.

### 4\.1NL2SHACL\-Framework

![Refer to caption](https://arxiv.org/html/2608.07530v1/x1.png)Figure 2:Overview of the NL2SHACL framework\. The design is modular and standardized, allowing flexible benchmarking of NL2SHACL systems\.We propose an extensible framework that supports the end\-to\-end benchmarking of NL2SHACL systems\. As shown in Figure[2](https://arxiv.org/html/2608.07530#S4.F2), our framework consists of three modules\. \(i\) dataset construction, \(ii\) NL2SHACL translation, and \(iii\) shapes evaluation\. Our framework is implemented in Python and is publicly available\.

NL2SHACL\-Framework further exhibits several desirable properties\. First, it ismodularized: each module can be used independently, allowing users to focus on dataset construction, model evaluation, or the full pipeline as needed\. Second, it isextensible: users can incorporate custom input files to generate new datasets, replace the translation module with arbitrary models, and extend the evaluation module with additional metrics\. Third, it is alsoreproducible, as it defines a consistent pipeline for dataset construction and evaluation\. Overall, the framework is designed to standardize the benchmarking pipeline for NL2SHACL\. In the following, we describe each module in detail\.

#### 4\.1\.1Dataset Construction Module

This module is a semi\-automated pipeline for constructing a paired dataset for the NL2SHACL task\. In practice, as discussed in Section[2](https://arxiv.org/html/2608.07530#S2), shapes graphs lack natural language descriptions\. Therefore, this module creates these descriptions from a reference shapes graph and an ontology\. To transform these inputs into structured NL–SHACL pairs, the module comprises three components: data pre\-processing, description construction, and description reviewing\.

##### Component 1: Data Pre\-processor\.

For a given shapes graph𝒮∗\\mathcal\{S\}^\{\*\}, we extract individualshape fragments\. Each shape fragment \(or just fragment\) corresponds to a node shape together with all the property shapes and other node shapes it references; this is extracted from the shape\-based constraint components \(e\.g\.,sh:node,sh:property\)\.111In our current implementation, we exclude SPARQL\-based constraints as NL2SPARQL is a separate research problem that requires dedicated methods\.In this way, each fragment represents a complete constraint subgraph without requiring external shape definitions\. With this definition, a fragment is also a shapes graph, which we denote𝒮i∗\\mathcal\{S\}\_\{i\}^\{\*\}\. An example of a fragment with two node shapes is shown in Figure[1](https://arxiv.org/html/2608.07530#S3.F1)\. Then, the component performs a structural completeness check to ensure that each extracted fragment yields a valid SHACL shapes graph\. This check verifies that everysh:propertyblock carries ash:pathdeclaration, and that the fragment contains at least onesh:NodeShape; fragments failing either condition are removed\. Lastly, for each fragment, this component extracts ontology term URIs from all SHACL predicates that can reference domain vocabulary\. This includes target and path declarations \(e\.g\.,sh:path,sh:targetClass,sh:node\) as well as value terms \(e\.g\.,sh:hasValue,sh:class\)\. For each extracted URI, the component queries a local ontology index and, when available, retrieves its type, label, and description; if no metadata is found but the URI exists, the URI itself is still retained\. This produces anontology snippetOiO\_\{i\}: a compact, prompt\-ready extract of exactly the vocabulary referenced by the fragment\. Fragments with fully unresolved domain terms are discarded\.

##### Component 2: Description Constructor\.

This component generates a natural language descriptionTiT\_\{i\}for each shape fragment𝒮i∗\\mathcal\{S\}\_\{i\}^\{\*\}\. First, we check whether a fragment already contains ash:description, which may have been provided by a human during the creation of the SHACL shape\. If present, it is manually reviewed to assess its usability\. Descriptions that are deemed usable are curated and directly retained\. If the description is missing or not directly usable \(e\.g\., incomplete or lacking sufficient detail to reflect the underlying constraints\), we generate a description draft using an LLM\. Specifically, we prompt the model with the reference SHACL shapes in𝒮i∗\\mathcal\{S\}\_\{i\}^\{\*\}and a carefully designed instruction that guides the translation of SHACL constraints and ontology terms into fluent, domain\-appropriate natural language\. The model is assigned the role of a domain expert who is not familiar with RDF or SHACL, and is asked to express the underlying data constraints in natural language as they would in practice, using a clear and sufficiently detailed description\. This design aims to simulate how human domain experts articulate validation requirements, making the generated descriptions more realistic and closer to real\-world usage\. We adopt this approach because many SHACL shapes lack descriptions or contain incomplete or imprecisesh:descriptionfields, and LLM\-based generation helps reduce manual annotation effort\. The model is instructed to produce a single coherent paragraph per fragment, following a consistent register and format within each subset\. The output of this component is a description draft for each fragment, regardless of whether it originates from existing annotations or LLM generation\.

##### Component 3: Description Reviewer\.

To ensure the quality of LLM\-generated descriptionsTiT\_\{i\}and mitigate issues such as hallucinations, we introduce a human validation component\. To support this process, we develop a graphical user interface \(GUI\) to assist annotators in reviewing and editing descriptions222The UI implementation and examples are available at[https://github\.com/DE\-TUM/NL2SHACL\-Framework/tree/main/Dataset\-Construction/Description\-Reviewer](https://github.com/DE-TUM/NL2SHACL-Framework/tree/main/Dataset-Construction/Description-Reviewer)\.\. We conduct a two\-round review process involving two annotators with experience in SHACL\. In the first round, the annotators independently review each description alongside its corresponding shape fragment, identifying issues across six categories: incorrectly described constraints, missing or redundant information, overly technical phrasing, unnatural language, exposure of SHACL\-specific terminology, and content hallucinated by the model beyond what the shape encodes\. The latter two categories specifically target LLM\-friendly phrasing patterns that could otherwise bias downstream evaluation\. In the second round, the annotators discuss the flagged cases and agree on the necessary revisions\. Through this process, the final descriptionTi∗T^\{\*\}\_\{i\}are refined to faithfully and completely reflect the constraints encoded in the corresponding SHACL shapes\.

![Refer to caption](https://arxiv.org/html/2608.07530v1/x2.png)Figure 3:An example record from the Invoice dataset, showing a reference shapes graph𝒮i∗\\mathcal\{S\}^\{\*\}\_\{i\}, the natural language descriptionTi∗T^\{\*\}\_\{i\}, and the relevant ontology snippetOiO\_\{i\}\.
##### Output Dataset\.

The output of this module is a dataset withrecordsin a unified format\. Each record is a triple\(𝒮i∗,Ti∗,Oi\)\(\\mathcal\{S\}^\{\*\}\_\{i\},T^\{\*\}\_\{i\},O\_\{i\}\)of a reference shapes graph𝒮i∗\\mathcal\{S\}\_\{i\}^\{\*\}, a natural language descriptionTi∗T^\{\*\}\_\{i\}, and an ontology snippetOiO\_\{i\}for the referenced terms\. In the remainder of this paper, we userecordanddata pointinterchangeably to refer to this triple\. Figure[3](https://arxiv.org/html/2608.07530#S4.F3)shows an example of a record created by our framework\.

#### 4\.1\.2NL2SHACL Translator Module

This is an off\-the\-shelf module that provides the translation mappingff\(Definition[1](https://arxiv.org/html/2608.07530#S3.Ex1)\)\. Given a natural language descriptionTi∗T\_\{i\}^\{\*\}and the corresponding ontology snippetOiO\_\{i\},ffproduces a translated shapes graph𝒮i=f​\(Ti∗,Oi\)\\mathcal\{S\}\_\{i\}=f\(T\_\{i\}^\{\*\},O\_\{i\}\)\. The module is model\-agnostic\. Users can plug in arbitrary translation systems, including LLM\-based or rule\-based approaches\. In our evaluation, we instantiate this module using a prompting\-based LLM translator, which is described in Section[5](https://arxiv.org/html/2608.07530#S5)\.

#### 4\.1\.3Shapes Evaluation Module

This module evaluates the quality of the translated shapes𝒮i\\mathcal\{S\}\_\{i\}\. The evaluation is performed using a set of predefined metrics, which produce quantitative scores reflecting different aspects of quality\. The module is extensible and allows additional metrics to be incorporated\. The metrics used in this work are described in Section[4\.3](https://arxiv.org/html/2608.07530#S4.SS3)\.

### 4\.2NL2SHACL\-Dataset

#### 4\.2\.1Dataset Overview

The NL2SHACL dataset comprises multiple sub\-datasets covering both cross\-domain \(DBpedia\) and domain\-specific settings, including business invoices, chemical data, healthcare information management, public procurement, and open government data\. This dataset has been created with the Dataset Construction Module; therefore, each data point follows a unified structure consisting of a natural language descriptionTi∗T\_\{i\}^\{\*\}, a reference shapes graph𝒮i∗\\mathcal\{S\}\_\{i\}^\{\*\}, and an ontology snippetOiO\_\{i\}capturing the semantics of the terms referenced in the shapes graph\. In total, the dataset contains 296 data points across 6 sub\-datasets\. Table[1](https://arxiv.org/html/2608.07530#S4.T1)summarizes the statistics of each subset\. The dataset covers 21 of the 33 SHACL Core constraint components \(excluding SHACL\-SPARQL\)\. The absent components reflect the characteristics of the selected source repositories rather than limitations of the framework, which can process any of these components when present in source data\.333A full per\-subset coverage breakdown, including the absent components, is available at[https://github\.com/DE\-TUM/NL2SHACL\-Dataset\#constraint\-component\-coverage](https://github.com/DE-TUM/NL2SHACL-Dataset#constraint-component-coverage)\.

Table 1:NL2SHACL benchmark statistics and filtering summary\.Dataset StatisticsShape FilteringSubset\# DataRecords\# NodeShapes\# Prop\.ShapesAvg\. NLLengthAvg\. Shapesper Record\# RawRecords\# FilteredRecordsCHEMROF\[[17](https://arxiv.org/html/2608.07530#bib.bib8)\]7474441154\.56\.9612817DCAT\[[18](https://arxiv.org/html/2608.07530#bib.bib11)\]2035103116\.26\.90211ePO\[[19](https://arxiv.org/html/2608.07530#bib.bib9)\]†5050139116\.53\.78378235Invoice\[[10](https://arxiv.org/html/2608.07530#bib.bib12)\]787811375\.12\.45846SNIK\[[20](https://arxiv.org/html/2608.07530#bib.bib10)\]8114952\.07\.50270DBpedia\[[2](https://arxiv.org/html/2608.07530#bib.bib14)\]1010119\.52\.10––Overall240258856108\.14\.64––
- †\\daggerSampled from 143 records\.

#### 4\.2\.2Data Sources and Construction

The dataset of reference shapes𝒮∗\\mathcal\{S\}^\{\*\}is constructed from multiple data sources\. Four subsets are selected from theShapes of Youindex\. In addition, the Invoice subset is derived from our previous work, and the DBpedia subset is curated by the authors using the DBpedia ontology\.

We select the four repositories from the index444From over 120 SHACL repositories initially collected, only a small number were suitable for inclusion due to issues such as inaccessibility, missing ontologies, or insufficient numbers of usable shapes\.due to the following desirable properties: \(1\) each represents a well\-defined application domain; \(2\) an associated ontology is available to support ontology\-grounded description and evaluation; and \(3\) each contains a sufficient number of shapes \(more than 10555Determined empirically: repositories below this size do not represent a domain’s constraint patterns meaningfully, and annotation cost is fixed regardless of repository size\.\)\. The selected shapes encode meaningful constraints that are neither trivially shallow nor excessively complex\. Table[1](https://arxiv.org/html/2608.07530#S4.T1)summarizes the filtering process and the number of remaining records for each subset\.

#### 4\.2\.3Sub\-dataset Description

Below, we briefly describe each subset\.

CHEMROF\.The CHEMROF data source\[[17](https://arxiv.org/html/2608.07530#bib.bib8)\]models chemical entities and their relationships, ranging from atoms to complex mixtures\. The shapes specify constraints such as closed\-world restrictions, mandatory identifiers, enumerated values, and hierarchical classifications\.

DCAT\.The DCAT data source\[[18](https://arxiv.org/html/2608.07530#bib.bib11)\]is based on the DCAT\-AP vocabulary for describing datasets and metadata in European data portals\. The shapes define constraints on metadata properties such as license, language, and publisher, often requiring values to link to controlled vocabularies\. Notably, this subset includes nested constraints viash:node, introducing multi\-level validation dependencies\.

ePO\.The eProcurement Ontology \(ePO\)\[[19](https://arxiv.org/html/2608.07530#bib.bib9)\]models public procurement processes in the European Union, covering multiple stages such as ordering, invoicing, and payment\. The shapes specify constraints on entities and relationships across these stages, resulting in a wide variation in complexity\. Additional filtering and sampling are applied to ensure balanced and tractable descriptions\.

Invoice\.The Invoice data source\[[10](https://arxiv.org/html/2608.07530#bib.bib12)\]is designed for validating invoicing data conforming to the EDIFACT standard666Electronic Data Interchange for Administration, Commerce and Transport\. The shapes specify constraints such as required fields, data formats, and identifier structures\. Notably, several shapes employsh:orandsh:notto express alternative and negation constraints\.

SNIK\.The SNIK data source\[[20](https://arxiv.org/html/2608.07530#bib.bib10)\]models information management in healthcare systems using a meta\-level ontology\. The shapes primarily define semantic constraints through class assignments and target declarations \(e\.g\.,sh:class,sh:targetClass\), with relatively few property\-level restrictions\. This results in structurally simple but semantically expressive constraints\.

DBpedia\.The DBpedia\[[2](https://arxiv.org/html/2608.07530#bib.bib14)\]subset is curated using the DBpedia ontology and focuses on a wide range of constraint types\. The shapes include not only existence, cardinality, and datatype constraints, but also logical and conditional rules\. Unlike other subsets, the natural language descriptions are derived from and reviewed based on existingsh:descriptionannotations\.

These subsets cover a diverse range of constraint patterns, from structural and datatype validation to semantic and logical constraints, providing a comprehensive testbed for NL2SHACL\.

### 4\.3NL2SHACL\-Metrics

In NL2SHACL, Evaluating the correctness of a translated shapes graph𝒮i=f​\(Ti∗,Oi\)\\mathcal\{S\}\_\{i\}=f\(T\_\{i\}^\{\*\},O\_\{i\}\)w\.r\.t\. the reference𝒮∗\\mathcal\{S\}^\{\*\}requires going beyond surface\-level comparison, as syntactically different shapes may still be semantically equivalent\. We therefore design a set of metrics that assess generated shapes from three complementary perspectives: validity, structure, and semantics\. Validity metrics ensure that outputs are well\-formed and conform to SHACL standards\. Structural metrics measure their similarity to reference shapes\. Semantic metrics evaluate their equivalence in terms of validation conformance\. Together, these metrics provide a comprehensive and fine\-grained evaluation of NL2SHACL systems\.

#### 4\.3\.1Validity Metrics

We define three validity metrics capturing different levels of correctness, organized as a layered pipeline where each stage is only reached if the previous one passes:

1. 1\.RDF Parsing Validity:Checks whether𝒮i\\mathcal\{S\}\_\{i\}is a valid RDF graph via standard parsing\. Outputs that fail this check are recorded as aParsing Error\.
2. 2\.SHACL Specification Validity:Validates𝒮i\\mathcal\{S\}\_\{i\}against the W3C SHACL\-SHACL meta\-shapes, applied only to outputs that pass RDF parsing\. Outputs that fail this check are recorded as aSpecification Error\.
3. 3\.SHACL Vocabulary Validity:Checks allsh:\-prefixed predicates in𝒮i\\mathcal\{S\}\_\{i\}against the set of standard SHACL constraint components, applied to outputs that pass both previous checks\. This layer targets a failure mode specific to LLM translators: hallucinated non\-standard vocabulary \(e\.g\.,sh:if,sh:then,sh:condition\)\. Shapes that fail this check are recorded as aVocabulary Error\.

For each data point, a binary result is assigned at each stage\. Each validity rate \(VR\) is computed as the proportion of outputs that pass the corresponding check among those that reach that stage\. LetNNbe the total number of data points,NrdfN\_\{\\text\{rdf\}\}the number that pass RDF parsing, andNspecN\_\{\\text\{spec\}\}the number that additionally pass the specification check\. LetPi,Si,Vi∈\{0,1\}P\_\{i\},S\_\{i\},V\_\{i\}\\in\\\{0,1\\\}denote whether𝒮i\\mathcal\{S\}\_\{i\}satisfies RDF parsing, SHACL specification, and vocabulary validity, respectively\. The validity rates are defined as follows:

RDF\-VR=1N​∑i=1NPi,Spec\-VR=1Nrdf​∑i=1NPi​Si,Vocab\-VR=1Nspec​∑i=1NPi​Si​Vi\\text\{RDF\-VR\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}P\_\{i\},\\ \\text\{Spec\-VR\}=\\frac\{1\}\{N\_\{\\text\{rdf\}\}\}\\sum\_\{i=1\}^\{N\}P\_\{i\}S\_\{i\},\\ \\text\{Vocab\-VR\}=\\frac\{1\}\{N\_\{\\text\{spec\}\}\}\\sum\_\{i=1\}^\{N\}P\_\{i\}S\_\{i\}V\_\{i\}
For the casesNrdf=0N\_\{\\text\{rdf\}\}=0orNspecN\_\{\\text\{spec\}\}, the metricsSpec\-VRandVocab\-VRare considered undefined, respectively\.

#### 4\.3\.2Structural Metrics

We define two structural metrics to measure graph\-level structural similarity between the generated shape𝒮i\\mathcal\{S\}\_\{i\}and the reference shape𝒮i∗\\mathcal\{S\}^\{\*\}\_\{i\}\. To enable comparison, both graphs are canonicalized to handle blank nodes, and their triple sets are compared\. The defined metrics are:

- •Exact Matching:A binary metric indicating whether the generated and reference shapes are structurally identical after canonicalization, i\.e,𝟏\[𝒮i=𝒮i∗\]\\mathbf\{1\}\_\{\[\\mathcal\{S\}\_\{i\}=\\mathcal\{S\}^\{\*\}\_\{i\}\]\}\.
- •Partial Matching:A continuous metric measuring the degree of overlap between the two shapes graphs based on triple\-level similarity, computed using precisionP=\|𝒮i∩𝒮i∗\|\|𝒮i\|P=\\frac\{\|\\mathcal\{S\}\_\{i\}\\cap\\mathcal\{S\}^\{\*\}\_\{i\}\|\}\{\|\\mathcal\{S\}\_\{i\}\|\}, recallR=\|𝒮i∩𝒮i∗\|\|𝒮i∗\|R=\\frac\{\|\\mathcal\{S\}\_\{i\}\\cap\\mathcal\{S\}^\{\*\}\_\{i\}\|\}\{\|\\mathcal\{S\}^\{\*\}\_\{i\}\|\}, andF​1=2​P​RP\+RF1=\\frac\{2PR\}\{P\+R\}\.

Exact matching provides a strict criterion, while partial matching offers a finer\-grained similarity signal\. However, both metrics are inherently structural and do not capture whether two shapes enforce the same constraints, which motivates the semantic metrics introduced next\.

At the dataset level, we report theExact Matching Rate\(EMR\), defined as the proportion of records with an exact match\. We also report thePartial Matching Score\(PMS\) as the macro F1\-score across all records\.

#### 4\.3\.3Semantic Metrics

We define semantic metrics to assess whether the translated shapes𝒮i\\mathcal\{S\}\_\{i\}and the reference shapes𝒮i∗\\mathcal\{S\}^\{\*\}\_\{i\}enforce the same constraints\. For this, we compare the validation conformance of𝒮i\\mathcal\{S\}\_\{i\}and𝒮i∗\\mathcal\{S\}^\{\*\}\_\{i\}on RDF data graphs\.

For each record, we generate a synthetic RDF data graph𝒢i\\mathcal\{G\}\_\{i\}usingRDFGraphGen\[[7](https://arxiv.org/html/2608.07530#bib.bib13)\], with the reference shape serving as the schema\. This ensures that𝒢i\\mathcal\{G\}\_\{i\}contains nodes that are relevant for the validation\. Then, we assess the validation conformance, i\.e\.,𝒢⊧𝒮i∗\\mathcal\{G\}\\models\\mathcal\{S\}^\{\*\}\_\{i\}and𝒢⊧𝒮i\\mathcal\{G\}\\models\\mathcal\{S\}\_\{i\}\. For non\-conformance cases, we extract the non\-conforming nodes from𝒢\\mathcal\{G\}and compare them as sets\. A binarySemantic Equivalencescore is assigned: two shapes are considered equivalent if and only if the sets of violating nodes are identical\.

At the dataset level, we report theSemantic Equivalence Rate\(SER\), defined as the fraction of records for which the generated and reference shapes are semantically equivalent, i\.e\., the number of data points with equivalent validation outcomes divided by the total number of evaluated records\.

Importantly, structural and semantic metrics capture distinct properties\. Translated shapes may enforce equivalent constraints \(measured with the semantic metrics\) despite different graph structures\. Yet, EMR and PMS should not be read as proxies for semantic correctness\. The structural metrics are better understood as measures of stylistic conformance, i\.e\., whether the translator reproduces the organizational conventions of the reference shape\. This matters when shapes are maintained, audited, or extended by humans, or composed with other shapes that assume particular structural patterns\.

## 5Evaluation

We evaluate the benchmarking capabilities of NL2SHACL\-Bench, using the constructed datasets \(Section[4\.2](https://arxiv.org/html/2608.07530#S4.SS2)\) and defined metrics \(Section[4\.3](https://arxiv.org/html/2608.07530#S4.SS3)\)\. In our experiments, we instantiate the NL2SHACL translator module using LLMs and assess their performance in the NL2SHACL task\. We introduce our evaluation pipeline and experimental results in this section\.

![Refer to caption](https://arxiv.org/html/2608.07530v1/x3.png)Figure 4:Overview of the evaluation pipeline for the NL2SHACL\-Bench\.![Refer to caption](https://arxiv.org/html/2608.07530v1/x4.png)

Figure 5:Illustration of the NL2SHACL prompt template used in our evaluation pipeline\. The left part shows thesystem prompt, and the right part shows theuser prompt\.

### 5\.1Evaluation Set Up

Figure[4](https://arxiv.org/html/2608.07530#S5.F4)shows our evaluation pipeline\. We use NL2SHACL\-Dataset as test data and prompt selected LLMs with pre\-defined prompt templates for the task\. We use the NL2SHACL\-Metrics, to assess the shapes generated by LLMs from the given natural language descriptions\. The details are described as follows\.

PromptWe define a prompt template consisting of a staticsystem promptand dynamically generateduser prompts\. Figure[5](https://arxiv.org/html/2608.07530#S5.F5)illustrates the overall prompt structure\. The system prompt defines the task, role, and generation rules\. The user prompt has aone\-shot exampletogether with the test input and output indicator\. Following our task definition, the models are provided with thenatural language descriptionTi∗T^\{\*\}\_\{i\}, and relevantontology snippetOiO\_\{i\}of the target shape fragment𝒮i∗\\mathcal\{S\}^\{\*\}\_\{i\}\. The one\-shot example additionally includes the expected SHACL output to demonstrate the desired generation format and support in\-context learning, following recent work on LLM\-based SHACL generation\[[22](https://arxiv.org/html/2608.07530#bib.bib35)\]\. All test instances within the same subset share the same example, constructed from the last data item of that subset, which is excluded from evaluation\.

ModelsWe select four state\-of\-the\-art LLMs to stress\-test our benchmark, covering both proprietary \(Gemini 3\.1 Pro Preview, Claude Opus 4\.7\) and open\-source \(GLM\-5\.1, Qwen3\.5\-397B\-A17B\) providers, chosen based on their strong performance on the Coding Index Score in OpenRouter\[[11](https://arxiv.org/html/2608.07530#bib.bib5)\]\. All models are accessed via the LLM Interactor using the OpenRouter API\.

### 5\.2Results

We evaluated a total of 234 records on four LLMs, resulting in 926 translated shape fragments overall\. Reported results are based on post\-processed outputs\.

![Refer to caption](https://arxiv.org/html/2608.07530v1/x5.png)Figure 6:Validity results of the evaluated LLMs on NL2SHACL\-Bench\.#### 5\.2\.1Validity Results

Figure[6](https://arxiv.org/html/2608.07530#S5.F6)presents the validity results of the evaluated models across all datasets\. Overall, all four models achieved very strong validity performance, with Gemini 3\.1 Pro obtaining perfect scores on all three metrics and the remaining models producing only a few invalid shapes\.

Most RDF parsing failures are low\-level formatting issues rather than fundamental misunderstandings of SHACL syntax\. Claude Opus 4\.7 consistently failed due to missing declarations of therdfs:prefix, whereas Qwen 3\.5 and GLM 5\.1 produced more isolated errors such as malformed string literals, incorrect escaping, and mismatched bracket structures\. The latter three models also occasionally hallucinated non\-standard components such assh:if,sh:then, andsh:conditionwhen translating logical constraints expressed throughsh:orandsh:not, replacing valid SHACL patterns with invented rule\-like constructs\.

#### 5\.2\.2Structural Results

![Refer to caption](https://arxiv.org/html/2608.07530v1/x6.png)Figure 7:Structural and semantic evaluation results across datasets\.Overall, the structural scores vary substantially across datasets as shown in Figure[7](https://arxiv.org/html/2608.07530#S5.F7)\. DBpedia and Invoice achieve comparatively higher EMR and PMS scores, whereas SNIK, DCAT, and EPO show near\-zero EMR and consistently low PMS values\. These differences mainly reflect variations in the structural organization of the reference shapes\. For SNIK, DCAT, and EPO, the reference shapes often contain multiple interconnected node and property shapes, while the evaluated LLMs typically generate a single node shape with nested constraints\. As a result, the generated graphs differ substantially from the references at the structural level even when the intended constraints are similar\. In contrast, DBpedia and Invoice follow more regular and comparatively simple structural patterns, making them easier to reproduce at the graph level\.

A clear gap can be observed between EMR, PMS, and SER across most datasets\. EMR consistently yields the lowest scores due to its strict exact\-matching requirement, while PMS is slightly higher by allowing partial structural overlap\. At the same time, the comparatively higher SER scores indicate that structural mismatch does not necessarily imply semantic mismatch\. Therefore, EMR and PMS should mainly be interpreted as lower\-bound indicators of graph\-level similarity rather than direct measures of semantic correctness\.

#### 5\.2\.3Semantic Results

![Refer to caption](https://arxiv.org/html/2608.07530v1/x7.png)Figure 8:Semantic equivalence results across datasets and models\.Table 2:Representative semantic error patterns observed in generated SHACL shapes\.Error TypeReference PatternTypical Generated PatternInverse Path Omissionsh:path \[ sh:inversePath rdf:type \]:DatasetShape a sh:NodeShape ; sh:targetClass dcat:Dataset \.Sequential Path Truncationsh:path \( rdfs:subClassOf rdf:type \)sh:path \( rdfs:subClassOf \)Constraint Semantics Substitutionsh:hasValue "http://example\.com/BuyerRole" ; sh:path rdf:typesh:class ex:BuyerRoleConditional Constraint Omissionsh:or \( \[ sh:not \(\.\.\.\) \] \[ \.\.\. \] \)plain property shapes without conditional logicNon\-standard Vocabulary Hallucinationsh:or / sh:notsh:conditionDatatype Omissionsh:datatype xsd:floatdatatype constraint missing

Figure[7](https://arxiv.org/html/2608.07530#S5.F7)presents the overall Semantic Equivalence Rate \(SER\) alongside the structural metrics\. Chemrof, DBpedia, DCAT, and EPO achieve high SER scores across most models, often approaching 100%, whereas Invoice and SNIK show substantially lower scores, with only around half of the records being semantically equivalent\. Figure[8](https://arxiv.org/html/2608.07530#S5.F8)further breaks down the distribution of semantic evaluation cases\. The gray bars indicate records that could not be automatically evaluated due to failed validity checks or limitations ofRDFGraphGenfor certain logical constraints involving constructs such assh:notandsh:xone, mainly affecting Invoice and SNIK\. We additionally performed manual inspection for these cases, and all manually inspected cases were identified as non\-equivalent, as reflected by the red bars in Figure[8](https://arxiv.org/html/2608.07530#S5.F8)\.

Among all evaluated models, Gemini 3\.1 Pro performs best, achieving semantic equivalence on all automatically evaluated records without producing non\-equivalent cases\. In particular, for inverse path constraints in DCAT, Gemini 3\.1 Pro is the only model that consistently preserves the inverse path semantics, whereas the other models reduce the constraint to simpler target class declarations\. Claude Opus 4\.7 performs second best, while Qwen 3\.5 and GLM 5\.1 produce more semantic mismatches across multiple datasets\. All four models achieve full semantic equivalence on EPO and DBpedia, whose constraints mainly consist of comparatively simple cardinality and datatype restrictions\. Qwen 3\.5 additionally omits a datatype constraint in one Invoice case, while the other three models preserve the datatype semantics correctly\.

Table[2](https://arxiv.org/html/2608.07530#S5.T2)summarizes representative semantic failure patterns\. The models mainly struggle with complex path constraints, such as inverse and sequential paths, as well as logical constraints expressed through combinations ofsh:orandsh:not\. Additional errors include semantic substitutions, such as replacingsh:hasValuewithsh:class, and hallucinated vocabulary such assh:condition\. Most of these complex semantic failures were identified through manual inspection and are reflected in the comparatively low SER scores on Invoice and SNIK\. Overall, the results indicate that LLMs perform well on simple SHACL shapes, but complex logical and structural patterns remain challenging\.

## 6Conclusion and Future Work

In this work, we presented NL2SHACL\-Bench, the first benchmark suite specifically designed for natural language to SHACL translation\. We formally define the NL2SHACL task and introduce a unified framework for dataset construction and evaluation, including semantic metrics based on validation behavior beyond purely structural comparison\. Our experiments provide a systematic evaluation of current LLMs on this task\. The results show that while modern LLMs are already highly capable of generating syntactically valid SHACL shapes, substantial challenges remain in producing structurally accurate and semantically equivalent constraints, especially for complex logical patterns and path constraints\.

As a sampling\-based approximation, our SER metric has a narrow blind spot: if a generated shape has an incorrect target declaration and produces no violations alongside the reference shape on the sampled data, SER may incorrectly judge them as equivalent\. A promising future direction is converting shapes to a common logical representation, e\.g\., via SHACL2FOL\[[12](https://arxiv.org/html/2608.07530#bib.bib4)\], to complement current evaluation with correctness\-guaranteed judgments on decidable fragments\.

Several other directions remain for future work: extending the benchmark with more expressive SHACL features such as SHACL\-SPARQL; evaluating NL2SHACL under more practical settings, since our current evaluation assumes exact ontology terms and in\-context examples are explicitly provided; and extending the largely constraint\-language\-independent framework to Shape Expressions \(ShEx\)\[[13](https://arxiv.org/html/2608.07530#bib.bib6)\]\.

\{credits\}

##### Resource Availability Statement:

The source code for the NL2SHACL\-Bench, including the software component and implemented metrics are available on GitHub777SHACL2NL\-Bench Code:[https://github\.com/DE\-TUM/NL2SHACL\-Framework](https://github.com/DE-TUM/NL2SHACL-Framework)and is licensed under the MIT license\. The constructed datasets are publicly available on Zenodo \(DOI: 10\.5281/zenodo\.20082565\) under the CC\-BY\-SA 4\.0 license\.

##### Maintainability Statement:

We plan to actively maintain NL2SHACL\-Bench alongside its ongoing development\. Future updates will include additional datasets, more expressive SHACL features, and improvements to the semantic evaluation pipeline\. We also plan to publish NL2SHACL\-Framework as a PyPI package to lower the barrier for adoption\. We further aim to extend the framework toward more realistic NL2SHACL settings and additional RDF validation formalisms such as ShEx, ensuring its extensibility and long\-term usability for both research and practical applications\.

#### Acknowledgements

This work has been supported by the Deutsche Forschungsgemeinschaft \(DFG, German Research Foundation\) \- SFB 1625 \- 506711657, subproject A06\. Maribel Acosta is supported by a Google Gemini Academic Program Award\.

## Declaration of Use of Generative AI

Generative AI tools were used to assist with language refinement and parts of the dataset construction process\. In particular, some initial NL–SHACL pairs were generated with AI assistance and subsequently manually verified and corrected by the authors\. All scientific content, including the research design, methodology, evaluation, analysis, and final dataset curation, was developed and validated by the authors\. The authors take full responsibility for the content of this paper and the released resources\.

## References

- \[1\]S\. Ahmetaj, I\. Boneva, J\. Hidders, K\. Hose, M\. Jakubowski, J\. E\. Labra Gayo, W\. Martens, F\. Mogavero, F\. Murlak, C\. Okulmus,et al\.\(2025\)Common foundations for shacl, shex, and pg\-schema\.InProceedings of the ACM on Web Conference 2025,pp\. 8–21\.Cited by:[§3](https://arxiv.org/html/2608.07530#S3.p3.7),[§3](https://arxiv.org/html/2608.07530#S3.p4.1)\.
- \[2\]S\. Auer, C\. Bizer, G\. Kobilarov, J\. Lehmann, R\. Cyganiak, and Z\. Ives\(2007\)Dbpedia: a nucleus for a web of open data\.Ininternational semantic web conference,pp\. 722–735\.Cited by:[§4\.2\.3](https://arxiv.org/html/2608.07530#S4.SS2.SSS3.p7.1),[Table 1](https://arxiv.org/html/2608.07530#S4.T1.1.8.7.1)\.
- \[3\]A\. Cimmino, A\. Fernández\-Izquierdo, and R\. García\-Castro\(2020\)Astrea: Automatic Generation of SHACL Shapes from Ontologies\.InThe Semantic Web,A\. Harth, S\. Kirrane, A\. Ngonga Ngomo, H\. Paulheim, A\. Rula, A\. L\. Gentile, P\. Haase, and M\. Cochez \(Eds\.\),Vol\.12123,pp\. 497–513\.External Links:ISBN 978\-3\-030\-49460\-5 978\-3\-030\-49461\-2Cited by:[§1](https://arxiv.org/html/2608.07530#S1.p3.1),[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]X\. Duan, D\. Chaves\-Fraga, O\. Derom, and A\. Dimou\(2024\)SCOOP all the constraints’ flavours for your knowledge graph\.InEuropean Semantic Web Conference,pp\. 217–234\.Cited by:[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]X\. Duan, D\. Chaves\-Fraga, and A\. Dimou\(2023\)XSD2SHACL: capturing rdf constraints from xml schema\.InProceedings Of The 12th Knowledge Capture Conference 2023,pp\. 214–222\.Cited by:[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]D\. Fernandez\-Álvarez, J\. E\. Labra\-Gayo, and D\. Gayo\-Avello\(2022\-02\)Automatic extraction of shapes using sheXer\.Knowledge\-Based Systems238,pp\. 107975\.External Links:ISSN 0950\-7051Cited by:[§1](https://arxiv.org/html/2608.07530#S1.p3.1),[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]M\. Jovanovik, M\. Vecovska, M\. Jakubowski, and K\. Hose\(2025\)RDFGraphGen: an rdf graph generator based on shacl shapes\.InInternational Joint Conference on Knowledge Graphs,pp\. 111–125\.Cited by:[§4\.3\.3](https://arxiv.org/html/2608.07530#S4.SS3.SSS3.p2.5)\.
- \[8\]M\. J\. Luthfi, F\. Darari, and A\. C\. Ashardian\(2022\)SoCK: SHACL on Completeness Knowledge\.InInternational Semantic Web Conference,Cited by:[§1](https://arxiv.org/html/2608.07530#S1.p2.1),[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]J\. Mäkelburg, C\. John, and M\. Acosta\(2024\)Automation of Electronic Invoice Validation Using Knowledge Graph Technologies\.InThe Semantic Web,A\. Meroño Peñuela, A\. Dimou, R\. Troncy, O\. Hartig, M\. Acosta, M\. Alam, H\. Paulheim, and P\. Lisena \(Eds\.\),Vol\.14664,pp\. 253–269\.External Links:ISBN 978\-3\-031\-60625\-0 978\-3\-031\-60626\-7Cited by:[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]Mäkelburg, Johannes and John, Christian and Acosta, MaribelEDIFACT\-val\.Note:[https://github\.com/DE\-TUM/EDIFACT\-VAL](https://github.com/DE-TUM/EDIFACT-VAL)Accessed: 2026/05/07Cited by:[§4\.2\.3](https://arxiv.org/html/2608.07530#S4.SS2.SSS3.p5.1),[Table 1](https://arxiv.org/html/2608.07530#S4.T1.1.6.5.1)\.
- \[11\]OpenRouter\.Note:[https://openrouter\.ai/rankings?benchmark=coding](https://openrouter.ai/rankings?benchmark=coding)Accessed: 2026/05/07Cited by:[§5\.1](https://arxiv.org/html/2608.07530#S5.SS1.p3.1)\.
- \[12\]P\. Pareti\(2024\)SHACL2FOL: an fol toolkit for shacl decision problems\.arXiv preprint arXiv:2406\.08018\.Cited by:[§6](https://arxiv.org/html/2608.07530#S6.p2.1)\.
- \[13\]E\. Prud’hommeaux, I\. Boneva, J\. E\. L\. Gayo, and G\. Kellogg\(2019\-10\)Shape expressions language 2\.1\.Note:W3C Community Group ReportAvailable at[https://www\.w3\.org/ns/shex](https://www.w3.org/ns/shex)External Links:[Link](https://shex.io/shex-semantics/)Cited by:[§6](https://arxiv.org/html/2608.07530#S6.p3.1)\.
- \[14\]K\. Rabbani, M\. Lissandrini, and K\. Hose\(2022\-04\)SHACL and ShEx in the Wild: A Community Survey on Validating Shapes Generation and Adoption\.InCompanion Proceedings of the Web Conference 2022,pp\. 260–263\.External Links:ISBN 978\-1\-4503\-9130\-6Cited by:[§1](https://arxiv.org/html/2608.07530#S1.p1.1),[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]K\. Rabbani, M\. Lissandrini, and K\. Hose\(2023\-01\)Extraction of Validating Shapes from Very Large Knowledge Graphs\.Proceedings of the VLDB Endowment16\(5\),pp\. 1023–1032\.External Links:ISSN 2150\-8097Cited by:[§1](https://arxiv.org/html/2608.07530#S1.p3.1),[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px2.p1.1)\.
- \[16\]R\. Schaffenrath, D\. Proksch, M\. Kopp, I\. Albasini, O\. Panasiuk, and A\. Fensel\(2020\)Benchmark for Performance Evaluation of SHACL Implementations in Graph Databases\.InRules and Reasoning,V\. Gutiérrez\-Basulto, T\. Kliegr, A\. Soylu, M\. Giese, and D\. Roman \(Eds\.\),pp\. 82–96\.External Links:ISBN 978\-3\-030\-57977\-7Cited by:[§1](https://arxiv.org/html/2608.07530#S1.p3.1),[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px2.p1.1)\.
- \[17\]UnknownChemkg/chemrof\.Note:[https://github\.com/chemkg/chemrof](https://github.com/chemkg/chemrof)Accessed: 2026/05/07Cited by:[§4\.2\.3](https://arxiv.org/html/2608.07530#S4.SS2.SSS3.p2.1),[Table 1](https://arxiv.org/html/2608.07530#S4.T1.1.4.3.1)\.
- \[18\]UnknownDcat\-ap\_shacl\.Note:[https://github\.com/SEMICeu/dcat\-ap\_shacl](https://github.com/SEMICeu/dcat-ap_shacl)Accessed: 2026/05/07Cited by:[§4\.2\.3](https://arxiv.org/html/2608.07530#S4.SS2.SSS3.p3.1),[Table 1](https://arxiv.org/html/2608.07530#S4.T1.1.5.4.1)\.
- \[19\]UnknownEProcurement ontology\.Note:[https://github\.com/OP\-TED/ePO/tree/master](https://github.com/OP-TED/ePO/tree/master)Accessed: 2026/05/07Cited by:[§4\.2\.3](https://arxiv.org/html/2608.07530#S4.SS2.SSS3.p4.1),[Table 1](https://arxiv.org/html/2608.07530#S4.T1.1.1.2)\.
- \[20\]UnknownSnikproject\.Note:[https://github\.com/snikproject](https://github.com/snikproject)Accessed: 2026/05/07Cited by:[§4\.2\.3](https://arxiv.org/html/2608.07530#S4.SS2.SSS3.p6.1),[Table 1](https://arxiv.org/html/2608.07530#S4.T1.1.7.6.1)\.
- \[21\]W3CShapes constraint language \(shacl\)\.Note:[https://www\.w3\.org/TR/shacl/](https://www.w3.org/TR/shacl/)Accessed: 2026/05/07Cited by:[§1](https://arxiv.org/html/2608.07530#S1.p1.1),[§3](https://arxiv.org/html/2608.07530#S3.p1.5)\.
- \[22\]T\. Westermann, A\. Köcher, and F\. Gehlhoff\(2025\-09\)Automated Validation of Textual Constraints Against AutomationML via LLMs and SHACL\.In2025 IEEE 30th International Conference on Emerging Technologies and Factory Automation \(ETFA\),pp\. 1–4\.External Links:ISSN 1946\-0759Cited by:[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2608.07530#S5.SS1.p2.3)\.
- \[23\]B\. Zhang, Y\. He, L\. Pintscher, A\. M\. Peñuela, and E\. Simperl\(2025\)Schema Generation for Large Knowledge Graphs Using Large Language Models\.InEMNLP,Cited by:[§2](https://arxiv.org/html/2608.07530#S2.SS0.SSS0.Px3.p1.1)\.