PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Hugging Face Daily Papers Papers
benchmark-generation text2cypher knowledge-graphs property-graphs nl-to-cypher schema-profiling enterprise

Summary

PIPE-Cypher is a pipeline that automatically generates balanced NL-to-Cypher benchmarks from live property graphs and seed queries, using techniques like schema profiling, reverse-query grounding, and local LLM judging to create discriminative, deployment-relevant benchmarks.

Enterprise property graphs vary widely in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. A deployment-relevant Text2Cypher benchmark therefore reflects the questions users and agents actually ask of that graph. Creating such a benchmark is difficult because schemas and values are unique, and graph structure changes over time. Each NL-query pair must also be executable, use real graph entities, preserve diversity, and remain balanced across query types and difficulty levels. We present PIPE-Cypher, a local benchmark-generation pipeline that turns a live property graph and optional seed queries from customer questions, analyst logs, or agent tool calls into balanced NL-to-Cypher benchmarks. PIPE-Cypher combines schema profiling, reverse-query grounding, constrained generation, deterministic Cypher governance, execution validation, redaction, diversity controls, and a calibrated local LLM judge. Using local Qwen3.5-9B generation and judging, PIPE-Cypher exports 3,000 accepted FinBench/SNB examples, completes three audited ablation suites, calibrates judge behavior with human labels, and evaluates 11 local downstream models. The resulting benchmark is deliberately discriminative: zero-shot transfer is weak, while a few-shot control shows that schema-specific example banks can help compatible model families. Together, PIPE-Cypher makes Text2Cypher benchmarking a repeatable process that evolves with the graph, its users, and its target workloads.
Original Article
View Cached Full Text

Cached at: 06/09/26, 08:41 AM

Paper page - PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Source: https://huggingface.co/papers/2606.08481

Abstract

A local benchmark-generation pipeline transforms live property graphs and seed queries into balanced NL-to-Cypher datasets for enterprise knowledge graphs, incorporating schema profiling, reverse-query grounding, and execution validation.

Enterpriseproperty graphsvary widely in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. A deployment-relevantText2Cypherbenchmark therefore reflects the questions users and agents actually ask of that graph. Creating such a benchmark is difficult because schemas and values are unique, and graph structure changes over time. Each NL-query pair must also be executable, use real graph entities, preserve diversity, and remain balanced across query types and difficulty levels. We present PIPE-Cypher, a localbenchmark-generation pipelinethat turns a live property graph and optional seed queries from customer questions, analyst logs, or agent tool calls into balanced NL-to-Cypher benchmarks. PIPE-Cypher combinesschema profiling,reverse-query grounding,constrained generation, deterministicCypher governance,execution validation,redaction,diversity controls, and a calibratedlocal LLM judge. Using local Qwen3.5-9B generation and judging, PIPE-Cypher exports 3,000 acceptedFinBench/SNBexamples, completes three audited ablation suites, calibrates judge behavior with human labels, and evaluates 11 local downstream models. The resulting benchmark is deliberately discriminative:zero-shot transferis weak, while a few-shot control shows that schema-specific example banks can help compatible model families. Together, PIPE-Cypher makesText2Cypherbenchmarking a repeatable process that evolves with the graph, its users, and its target workloads.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2606\.08481

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.08481 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.08481 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.08481 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

CYGNET: Cypher Gate for Neural Execution Triage and Cost Containment

arXiv cs.CL

CYGNET is a pre-execution gate system that validates and corrects Cypher queries generated by LLM agents over knowledge graphs, catching structural failures before they hit production databases with near-zero false positives and achieving 81–95% success in repairing broken queries across five language models.

buildpipe

Product Hunt

Buildpipe is a tool for composing, running, and automating multi-step AI developer workflows.

ProgramBench (5 minute read)

TLDR AI

ProgramBench is a new benchmark that evaluates AI agents' ability to reconstruct complete software projects from compiled binaries and documentation without access to source code or decompilation tools.