Benchy:迈向面向任务的AI基准测试的通用语言

arXiv cs.AI 论文

摘要

Benchy 引入了一种语义语言和执行引擎,用于基准测试AI程序,通过规范表示来标准化基准测试的定义和执行。

arXiv:2609.30550v1 Announce Type: new Abstract: Benchy is a semantic language and execution engine for benchmarking AI programs. A benchmark is completely specified by a program, a scoring function, and a dataset, B=(P,S,D), and is separate from the AI-system taking it; a run binds the two, R=(B,AI). Benchmarks are authored as canonical YAML in which each semantic concept has one valid syntax, classified by a shared task/domain/language ontology, and deterministically compiled into a canonical JSON intermediate representation that the engine executes. Compilation changes representation, not meaning: it does not repair invalid definitions or inject hidden defaults. Programs use fixed schemas of named input and output fields, the leaf output fields are the scoring dimensions, and the engine exposes one universal runtime contract --- a named-field input object in, a named-field output object out --- to which external AI-systems adapt at the boundary, so integration mechanics never propagate into benchmark semantics. This paper gives the semantic object model, the ontology and task-to-program validation rule, the scoring and failure semantics, the compilation and execution architecture, and the scope of the current language. An appendix fixes the normative engineering contract for the first engine implementation.
查看原文
查看缓存全文

缓存时间: 2026/09/28 09:38

# Benchy: towards a universal language for task-oriented AI benchmarks
Source: [https://arxiv.org/html/2609.30550](https://arxiv.org/html/2609.30550)
September 23, 2026

###### Abstract

Benchy is a semantic language and execution engine for benchmarking AI programs\. A benchmark is completely specified by a program, a scoring function, and a dataset,B=\(P,S,D\)B=\(P,S,D\), and is separate from the AI\-system taking it; a run binds the two,R=\(B,AI\)R=\(B,\\mathrm\{AI\}\)\. Benchmarks are authored as canonical YAML in which each semantic concept has one valid syntax, classified by a shared task/domain/language ontology, and deterministically compiled into a canonical JSON intermediate representation that the engine executes\. Compilation changes representation, not meaning: it does not repair invalid definitions or inject hidden defaults\. Programs use fixed schemas of named input and output fields, the leaf output fields are the scoring dimensions, and the engine exposes one universal runtime contract — a named\-field input object in, a named\-field output object out — to which external AI\-systems adapt at the boundary, so integration mechanics never propagate into benchmark semantics\. This paper gives the semantic object model, the ontology and task\-to\-program validation rule, the scoring and failure semantics, the compilation and execution architecture, and the scope of the current language\. An appendix fixes the normative engineering contract for the first engine implementation\.

1SURUS2Independent

Contact:francis@surus\.lat

Keywords:AI evaluation, benchmarks, semantic language, intermediate representation, task\-oriented AI

###### Contents

1. [1Overview](https://arxiv.org/html/2609.30550#S1)
2. [2Semantic object model](https://arxiv.org/html/2609.30550#S2)
3. [3Benchmark classification and the shared SURUS ontology](https://arxiv.org/html/2609.30550#S3)
4. [4Scoring](https://arxiv.org/html/2609.30550#S4)
5. [5Data](https://arxiv.org/html/2609.30550#S5)
6. [6AI\-system](https://arxiv.org/html/2609.30550#S6)
7. [7Compilation and execution architecture](https://arxiv.org/html/2609.30550#S7)
8. [8Execution and results](https://arxiv.org/html/2609.30550#S8)
9. [9Canonical YAML](https://arxiv.org/html/2609.30550#S9)
10. [10Scope and future extensions](https://arxiv.org/html/2609.30550#S10)
11. [AEngine 1\.0 engineering contract](https://arxiv.org/html/2609.30550#A1)1. [A\.1YAML grammar and strictness](https://arxiv.org/html/2609.30550#A1.SS1) 2. [A\.2Strict object validation](https://arxiv.org/html/2609.30550#A1.SS2) 3. [A\.3Weight coverage](https://arxiv.org/html/2609.30550#A1.SS3) 4. [A\.4Task\-to\-program validation](https://arxiv.org/html/2609.30550#A1.SS4) 5. [A\.5Dataset encoding](https://arxiv.org/html/2609.30550#A1.SS5) 6. [A\.6JSON representation of semantic values](https://arxiv.org/html/2609.30550#A1.SS6) 7. [A\.7Exact\-match equality](https://arxiv.org/html/2609.30550#A1.SS7) 8. [A\.8Compile\-time versus execution\-time validation](https://arxiv.org/html/2609.30550#A1.SS8) 9. [A\.9Canonical JSON IR](https://arxiv.org/html/2609.30550#A1.SS9) 10. [A\.10Adapter protocol](https://arxiv.org/html/2609.30550#A1.SS10) 11. [A\.11AI\-system execution setup](https://arxiv.org/html/2609.30550#A1.SS11) 12. [A\.12Instance statuses](https://arxiv.org/html/2609.30550#A1.SS12) 13. [A\.13Scoring failed examples](https://arxiv.org/html/2609.30550#A1.SS13) 14. [A\.14Benchmark result schema](https://arxiv.org/html/2609.30550#A1.SS14) 15. [A\.15Dataset failures](https://arxiv.org/html/2609.30550#A1.SS15) 16. [A\.16Error model](https://arxiv.org/html/2609.30550#A1.SS16)
12. [BShared SURUS ontology registry](https://arxiv.org/html/2609.30550#A2)
13. [CArtifact runtime representation](https://arxiv.org/html/2609.30550#A3)
14. [DCompact architecture](https://arxiv.org/html/2609.30550#A4)

SURUS Team · Buenos Aires · 2026

## 1Overview

Benchy is a semantic language and execution engine for benchmarking AI programs\.

Aprogramdefines a typed input/output contract\. AnAI\-systemis any AI\-based implementation of that program: a single AI model; an AI\-node such as a model with an optimized prompt and fixed task behavior; a composition of AI models or AI programs; AI components intermingled with explicit or deterministic code; or an agent/workflow composed from those elements\.

A benchmark is separate from the AI\-system taking it\.

B=\(P,S,D\)\\boxed\{B=\(P,S,D\)\}\(1\)> BenchmarkBBis completely specified by programPP, scoring functionSS, and datasetDD\.

A benchmark run binds that benchmark to an AI\-system:

R=\(B,AI\)\\boxed\{R=\(B,\\mathrm\{AI\}\)\}\(2\)> RunRRevaluates AI\-systemAI\\mathrm\{AI\}on benchmarkBB\.

The design follows a small number of decisions:

- •YAML is the canonical semantic definition authored by humans, agents, and the UI\.
- •Each semantic concept has one valid YAML syntax\.
- •Valid YAML is deterministically compiled into a canonical JSON intermediate representation \(IR\)\.
- •Compilation changes representation, not meaning\. It does not repair invalid definitions or inject hidden defaults\.
- •Types express semantic constraints, not storage representation\.
- •Programs use fixed schemas composed of named input and output fields\.
- •Variable\-length output collections are outside the current program model\.
- •The output schema determines the fixed scoring dimensions\.
- •Task, domain, and language come from one shared SURUS ontology registry\.
- •Benchy defines one universal runtime contract from the declared program schema\.
- •External AI\-systems adapt to that contract at the boundary\.
- •Integration\-specific mechanics do not propagate into benchmark semantics or the engine\.

The authoring and execution path is:

Human / Agent / UIYAMLparsevalidatecompileJSON IREngineAdapterAI\-systemruntime contract

Theengineis Benchy’s execution machinery\. It consumes the compiled JSON IR, reads exam examples, invokes the AI\-system through an adapter, validates outputs, computes scores, and produces results\.

Theruntime contractis the interface rule the engine expects at its boundary:

named\-field input object→named\-field output object\\boxed\{\\text\{named\-field input object\}\\rightarrow\\text\{named\-field output object\}\}\(3\)> Every Benchy\-compliant AI\-system is presented to the engine as something that consumes an object matching the program’s named input fields and produces an object matching its named output fields\.

An adapter translates between that contract and an AI\-system’s native interface\.

## 2Semantic object model

The program defines what must be done\.

Program=Input Schema\+Output Schema\\boxed\{\\text\{Program\}=\\text\{Input Schema\}\+\\text\{Output Schema\}\}\(4\)> A program consists of one schema describing its inputs and another describing its outputs\.

Conceptually:

P:X→YP:X\\rightarrow Y\(5\)> ProgramPPmaps values conforming to input schemaXXinto values conforming to output schemaYY\.

A schema combines field structure with semantic types:

Schema=Field Structure\+Semantic Types\\boxed\{\\text\{Schema\}=\\text\{Field Structure\}\+\\text\{Semantic Types\}\}\(6\)> A schema says which named fields exist, how they are nested, and what each field semantically means\.

The semantic type vocabulary is:

For an enum:

Y=\{v1,v2,…,vk\}Y=\\\{v\_\{1\},v\_\{2\},\\ldots,v\_\{k\}\\\}\(7\)> An enum field may take exactly one value from the finite declared set\.

Every input and output value is represented through one or more named fields\.

```
program:
  input:
    audio: audio
  output:
    transcription: string
```

A classification program can be:

```
program:
  input:
    text: string
  output:
    sentiment:
      enum: [positive, neutral, negative]
```

A structured extraction program can be:

```
program:
  input:
    image: image
  output:
    invoice_number: string
    date: date
    supplier: string
    subtotal: float
    total: float
```

Fixed nested structures are allowed:

```
program:
  input:
    document: document
  output:
    supplier:
      name: string
      tax_id: string
    date: date
    total: float
```

Aleaf output fieldis an output field whose value is a semantic type rather than another nested field structure\.

The leaf output fields are the scoring dimensions\.

fixed named output schema⇒fixed scoring dimensions⇒fixed field weights\\boxed\{\\text\{fixed named output schema\}\\Rightarrow\\text\{fixed scoring dimensions\}\\Rightarrow\\text\{fixed field weights\}\}\(8\)> Once the output fields are fixed, Benchy knows exactly which dimensions can be scored and weighted\.

## 3Benchmark classification and the shared SURUS ontology

Benchmarks are classified with the shared SURUS task\-oriented AI ontology:

/task/domain/language/\\boxed\{/\\text\{task\}/\\text\{domain\}/\\text\{language\}/\}\(9\)> The ontology identifies what operation is performed, the domain/distribution in which it is evaluated, and the linguistic context of the data\.

```
benchmark:
  task: extract
  domain: finance
  language: es
```

The three coordinates have different roles:

- •taskdescribes the operation performed;
- •domaindescribes the world or distribution represented by the exam data;
- •languagedescribes the linguistic distribution represented by the exam data\.

Tasks, domains, and languages live in one shared, versioned registry used by Benchy, DataHub, EvalsHub, and other SURUS systems\.

A task defines a family of admissible programs\.

Let:

- •TT= a task;
- •𝒫T\\mathcal\{P\}\_\{T\}= the family of programs admitted by taskTT;
- •PP= a concrete program\.

Then:

T→𝒫TT\\rightarrow\\mathcal\{P\}\_\{T\}\(10\)> TaskTTdetermines the family of programs considered valid instances of that task\.

The compiler checks:

P∈𝒫T\\boxed\{P\\in\\mathcal\{P\}\_\{T\}\}\(11\)> ProgramPPmust belong to the family admitted by taskTT\.

The program schema alone cannot always determine the task\.

string→string\\text\{string\}\\rightarrow\\text\{string\}\(12\)> The same input/output shape could represent translation, summarization, rewriting, question answering, or another operation\.

Task is therefore explicit rather than inferred\.

The ontology registry carries shared identifiers and descriptions\. Benchy implements the finite task\-to\-program structural validation rules for the ontology versions it supports\. The registry does not contain a separate constraint language\.

Translation uses an ordered source/target language relation:

```
benchmark:
  task: translate
  domain: general
  language:
    source: es
    target: en
```

which may be represented externally as:

## 4Scoring

The output schema determines the dimensions over which correctness can be evaluated\.

A scoring function specifies the relative importance of output fields and the rule by which field\-level correctness is aggregated into program\-level performance\.

Scoring Function=Field Weights\+Aggregator\\boxed\{\\text\{Scoring Function\}=\\text\{Field Weights\}\+\\text\{Aggregator\}\}\(13\)> Field weights express how important each output dimension is; the aggregator defines how those field scores become one instance score\.

```
scoring:
  weights:
    invoice_number: 1
    date: 1
    supplier: 1
    subtotal: 1
    total: 5
  aggregator: weighted_mean
```

Nested weights mirror the output schema:

```
program:
  output:
    supplier:
      name: string
      tax_id: string
    total: float

scoring:
  weights:
    supplier:
      name: 1
      tax_id: 0
    total: 5
  aggregator: weighted_mean
```

Every leaf output field has exactly one explicit weight\. There are no missing weights, extra weights, or weights on intermediate objects\.

Weights are non\-negative:

> The weightwjw\_\{j\}of scoring dimensionjjmay be zero or positive, but never negative\.

A zero weight means the field is present and validated but does not affect the instance score\.

For exam exampleiiand scoring dimensionjj, field correctness is:

ci​j=𝟏\[y^i​j=yi​j∗\]c\_\{ij\}=\\mathbf\{1\}\[\\hat\{y\}\_\{ij\}=y\_\{ij\}^\{\*\}\]\(15\)> Field correctnessci​jc\_\{ij\}is 1 when the predicted semantic value exactly equals the expected semantic value and 0 otherwise\.

Field correctness is currently produced by exact match\. The evaluator is not exposed as a benchmark configuration option\.

All field scores for exampleiiform:

𝐜i=\(ci​1,ci​2,…,ci​n\)\\mathbf\{c\}\_\{i\}=\(c\_\{i1\},c\_\{i2\},\\ldots,c\_\{in\}\)\(16\)> 𝐜i\\mathbf\{c\}\_\{i\}contains one correctness value for each scored output dimension of exam exampleii\.

The current aggregator is normalized weighted mean:

si=∑jwj​ci​j∑jwjs\_\{i\}=\\frac\{\\sum\_\{j\}w\_\{j\}c\_\{ij\}\}\{\\sum\_\{j\}w\_\{j\}\}\(17\)> Instance scoresis\_\{i\}is the weighted average of field correctness values for exam exampleii\.

Therefore:

∑jwj\>0\\sum\_\{j\}w\_\{j\}\>0\(18\)> At least one scoring dimension must have positive weight\.

The weights encoderelative importance\. Multiplying every weight by the same positive constant does not change the instance score\.

Normalization is a property of this aggregator, not a universal property of Benchy\.

The aggregator defines the semantics and scale of the score\.\\boxed\{\\text\{The aggregator defines the semantics and scale of the score\.\}\}\(19\)> A normalized aggregator can produce a score in\[0,1\]\[0,1\]; future aggregators may naturally use another scale\.

## 5Data

The dataset is the exam\.

Each exam exampleiicontains:

\(xi,yi∗\)\(x\_\{i\},y\_\{i\}^\{\*\}\)\(20\)> xix\_\{i\}is the input for exampleii, andyi∗y\_\{i\}^\{\*\}is its expected or ground\-truth output\.

The program schema constrains both sides:

> Exam inputxix\_\{i\}must conform to the program’s input schemaXX\.

> Expected outputyi∗y\_\{i\}^\{\*\}must conform to the program’s output schemaYY\.

Thus:

```
Input Schema  ---> exam inputs
Output Schema ---> expected outputs
Output Schema ---> scoring dimensions
```

The benchmark definition references the dataset:

```
data:
  path: ./data/invoices.jsonl
```

The program schema defines the semantic structure of every example\. The engine specification in Appendix[A](https://arxiv.org/html/2609.30550#A1)defines the concrete dataset encoding used by the current implementation\.

## 6AI\-system

The AI\-system is the benchmark taker\.

AI:X→Y\\mathrm\{AI\}:X\\rightarrow Y\(23\)> AI\-systemAI\\mathrm\{AI\}implements the declared program contract: it consumes values fromXXand produces values intended to conform toYY\.

An AI\-system may be:

```
single AI model
AI-node: model + optimized prompt + fixed task behavior
composition of AI models or AI programs
AI components + explicit deterministic code
agent or workflow
```

For a directly specified model:

```
ai-system:
  type: model
  provider: openai
  model: <model>
  prompt: ./prompts/invoice.md
  parameters:
    temperature: 0
```

These fields identify the AI\-system under evaluation\.

Credentials, SDK construction, HTTP mechanics, response extraction, and similar integration details are outside benchmark semantics\.

An arbitrary external AI\-system can instead be identified as:

```
ai-system:
  type: external
  id: invoice-extractor-v7
```

The runtime environment binds that identifier to an adapter\.

The distinction is:

```
AI-system definition
= what is being evaluated

adapter/runtime binding
= how this environment invokes it
```

## 7Compilation and execution architecture

The canonical compilation path is:

YAML→parse→validate→compile→JSON IR→engine\\boxed\{\\text\{YAML\}\\rightarrow\\text\{parse\}\\rightarrow\\text\{validate\}\\rightarrow\\text\{compile\}\\rightarrow\\text\{JSON IR\}\\rightarrow\\text\{engine\}\}\(24\)> Benchy parses the canonical source, validates its semantics, compiles it once into a machine\-facing representation, and executes from that representation\.

There is no normalization stage\.

Let:

- •YyamlY\_\{\\text\{yaml\}\}= valid Benchy YAML;
- •CC= compiler;
- •JJ= canonical JSON IR\.

Then:

J=C⁡\(Yyaml\)\\boxed\{J=C\(Y\_\{\\text\{yaml\}\}\)\}\(25\)> CompilerCCdeterministically produces JSON IRJJfrom valid YAMLYyamlY\_\{\\text\{yaml\}\}\.

Compilation preserves meaning:

Semantics⁡\(J\)=Semantics⁡\(Yyaml\)\\boxed\{\\operatorname\{Semantics\}\(J\)=\\operatorname\{Semantics\}\(Y\_\{\\text\{yaml\}\}\)\}\(26\)> JSON IR and YAML express the same benchmark/run semantics; the IR is the executable representation, not another semantic source\.

The engine does not reinterpret YAML\.

At runtime, the engine knows one AI\-system interface:

named\-field input object→named\-field output object\\boxed\{\\text\{named\-field input object\}\\rightarrow\\text\{named\-field output object\}\}\(27\)> External AI\-system differences are translated at the adapter boundary instead of being spread through the engine\.

## 8Execution and results

For exam exampleii:

y^i=AI⁡\(xi\)\\hat\{y\}\_\{i\}=\\mathrm\{AI\}\(x\_\{i\}\)\(28\)> AI\-systemAI\\mathrm\{AI\}receives inputxix\_\{i\}and produces predictiony^i\\hat\{y\}\_\{i\}\.

The output is validated against schemaYYbefore scoring\.

A valid but completely incorrect output can have:

An output that does not satisfy the declared schema is structurally different:

```
status: invalid_output
score: null
```

An invocation failure is different again:

```
status: execution_error
score: null
```

This distinction preserves the difference between a valid zero score and a failure to produce a valid program output\.

For final aggregation, define each example’s contributionqiq\_\{i\}:

qi=\{si,if the AI\-system produced a valid output0,if the output was invalid or execution failedq\_\{i\}=\\begin\{cases\}s\_\{i\},&\\text\{if the AI\-system produced a valid output\}\\\\ 0,&\\text\{if the output was invalid or execution failed\}\\end\{cases\}\(29\)> Invalid outputs and execution failures retainnullas their stored instance score, but contribute zero to the benchmark aggregate\.

ForNNexam examples:

B⁡\(AI\)=1N​∑i=1Nqi\\boxed\{B\(\\mathrm\{AI\}\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}q\_\{i\}\}\(30\)> The benchmark score is the arithmetic mean across all example contributions, so failed examples remain in the denominator\.

## 9Canonical YAML

```
version: "1.0"
ontology_version: "1.0"

benchmark:
  task: extract
  domain: finance
  language: es

program:
  input:
    image: image
  output:
    invoice_number: string
    date: date
    supplier: string
    subtotal: float
    total: float

scoring:
  weights:
    invoice_number: 1
    date: 1
    supplier: 1
    subtotal: 1
    total: 5
  aggregator: weighted_mean

data:
  path: ./data/invoices.jsonl

ai-system:
  type: external
  id: invoice-extractor-v7
```

versionpins the Benchy semantic specification\.

ontology\_versionpins the shared SURUS task/domain/language registry used to interpret the benchmark classification\.

## 10Scope and future extensions

The current language deliberately does not expose:

```
variable-length output collections
custom field evaluators
aggregators other than weighted_mean
adapter configuration inside benchmark YAML
implicit field weights
anonymous root scalar inputs or outputs
```

These are extension points rather than undefined behavior\.

The present architecture is intended to let those capabilities be introduced without changing the meaning of existing benchmark definitions\.

## Appendix AEngine 1\.0 engineering contract

This appendix is normative for the first implementation\. The main paper defines semantics; this appendix fixes the engineering representation required to build the engine\.

### A\.1YAML grammar and strictness

The top\-level YAML object contains exactly:

```
version
ontology_version
benchmark
program
scoring
data
ai-system
```

Unknown top\-level keys are errors\.

Mappings must not contain duplicate keys\.

YAML anchors, aliases, merge keys, and custom tags are rejected\.

The program grammar is:

```
program:
  input:
    <field-name>: <type-or-nested-object>
  output:
    <field-name>: <type-or-nested-object>
```

Rules:

- •inputandoutputare mappings;
- •each contains at least one named field;
- •field names are non\-empty strings;
- •all fields are required;
- •nullis not a valid field value;
- •lists/arrays are not supported as field values;
- •nested objects are allowed;
- •scalar type declarations must be one of the supported semantic type names;
- •enum is declared only as:

```
sentiment:
  enum: [positive, neutral, negative]
```

Enum values are non\-empty, distinct strings\.

Unknown type declarations are errors\.

### A\.2Strict object validation

Program schemas are closed\.

For dataset records and AI\-system outputs:

```
missing required field -> invalid
extra field            -> invalid
wrong semantic type    -> invalid
```

Nested objects are validated recursively\.

### A\.3Weight coverage

The weight tree must mirror the output schema down to every leaf output field\.

There must be:

```
exactly one explicit weight per leaf output field
no missing weights
no extra weights
no weights on intermediate objects
```

Weights are finite numbers satisfyingwj≥0w\_\{j\}\\geq 0, and their total must be positive\.

### A\.4Task\-to\-program validation

The shared ontology registry contains identifiers and descriptions only\.

Benchy implements the structural validators for the task vocabulary of the ontology version it supports\.

For ontology version1\.0:

```
extract
  no task-specific structural constraint beyond normal program rules

classify
  exactly one leaf output field
  that leaf field is enum

transcribe
  input contains at least one audio leaf field
  exactly one leaf output field
  that leaf field is string

translate
  input contains at least one string leaf field
  exactly one leaf output field
  that leaf field is string
  benchmark.language is {source, target}
```

If an ontology task is not supported by the loaded Benchy specification, compilation fails rather than silently skipping task validation\.

### A\.5Dataset encoding

The engine uses JSONL\.

Each non\-empty line is one exam example:

```
{
  "input": {
    "image": "assets/invoice-001.png"
  },
  "expected": {
    "total": 121.0
  }
}
```

The row contains exactlyinputandexpected\.

Blank lines may be ignored\.

The dataset must contain at least one example\.

Dataset examples are streamed\. They are not embedded into JSON IR\.

Each example is validated at execution time against the already\-compiled schemas in the IR\.

An invalid dataset record is a benchmark/data error and aborts the run\. It is not scored as an AI\-system failure\.

### A\.6JSON representation of semantic values

boolis not accepted asintorfloat\.

#### Artifact path rule

In a dataset,image,audio, anddocumentare encoded as relative path strings\.

The engine:

1. 1\.resolves the path relative to the JSONL file’s directory;
2. 2\.normalizes it;
3. 3\.verifies that it exists and is a regular readable file;
4. 4\.passes the resolved absolute path string through the runtime contract\.

The adapter decides whether to open the file, upload it, decode it, or convert it to another native representation\.

For artifact outputs, an adapter returns a local path string to the produced file\. The engine validates that the file exists before treating the output as schema\-valid\.

This is an implementation representation of the semantic artifact types, not a redefinition of those types\.

### A\.7Exact\-match equality

Exact match compares schema\-valid semantic values\.

There is no trimming, case folding, Unicode normalization, numeric tolerance, or semantic similarity\.

### A\.8Compile\-time versus execution\-time validation

Compile\-time validation covers:

```
YAML syntax
specification version
ontology version
task/domain/language registry membership
task <-> program compatibility
program schema grammar
weight coverage and validity
data configuration
AI-system semantic definition
```

Execution\-time validation covers:

```
each dataset input against compiled input schema
each expected value against compiled output schema
each AI-system output against compiled output schema
```

The engine uses schemas already present in the JSON IR\. It never reinterprets source YAML\.

### A\.9Canonical JSON IR

The IR contains the validated executable semantics needed by the engine, including:

```
versions
benchmark classification
typed input/output schemas
resolved scoring dimensions
field evaluator
instance aggregator
benchmark aggregator
data configuration
AI-system semantic definition
```

Nested output fields become scoring paths represented as string arrays, for example:

```
{
  "path": ["supplier", "name"],
  "weight": 1
}
```

Paths are an IR representation detail\. Authors do not write them\.

The IR includes:

```
{
  "scoring": {
    "evaluator": "exact_match",
    "instance_aggregator": "weighted_mean",
    "benchmark_aggregator": "mean"
  },
  "data": {
    "format": "jsonl"
  }
}
```

These fields make fixed specification semantics explicit to the engine; they do not introduce new benchmark meaning\.

### A\.10Adapter protocol

The engine binds one adapter before execution\.

Conceptually:

```
class Adapter:
    async def invoke(
        self,
        input_object: dict[str, object],
    ) -> dict[str, object]:
        ...
```

The adapter instance may be created with the compiled program contract and AI\-system definition\.

The engine owns:

```
dataset loading
input validation
output validation
scoring
aggregation
result/error recording
```

The adapter owns:

```
mapping Benchy inputs to the native AI-system interface
invoking the AI-system
mapping native outputs back to the Benchy named-field object
```

Adapter exceptions become execution errors\.

If the adapter returns an object that violates the output schema, the result isinvalid\_output\.

Credentials, endpoint mappings, response paths, Python imports, SDK construction, and other integration mechanics are runtime configuration outside benchmark YAML\.

The default engine may execute sequentially\. Concurrency is an execution optimization and must not change benchmark semantics\.

### A\.11AI\-system execution setup

For:

```
ai-system:
  type: external
  id: invoice-extractor-v7
```

the runtime must provide an adapter binding for that identifier before the run begins\.

Missing binding is a run setup error and aborts before evaluating examples\.

For:

```
ai-system:
  type: model
  provider: ...
  model: ...
```

a provider adapter may be selected by the runtime\. Provider integrations are built on the same adapter protocol and are outside the core engine\.

### A\.12Instance statuses

Every evaluated example has one of:

validinvalid\_outputexecution\_error

validmeans the AI\-system produced a schema\-valid output, regardless of score\.

invalid\_outputmeans the adapter returned an output value but it failed the compiled output schema\.

execution\_errormeans invocation failed before a Benchy output object was successfully produced\.

Examples include adapter exceptions, HTTP/provider failures, and runtime timeouts\.

### A\.13Scoring failed examples

Stored instance score:

```
valid            -> numeric s_i
invalid_output   -> null
execution_error  -> null
```

Aggregation contribution:

qi=\{si,valid0,invalid\_output or execution\_errorq\_\{i\}=\\begin\{cases\}s\_\{i\},&\\text\{valid\}\\\\ 0,&\\text\{invalid\\\_output or execution\\\_error\}\\end\{cases\}\(31\)> nullpreserves the diagnostic distinction, while zero contribution prevents failed examples from disappearing from the aggregate\.

### A\.14Benchmark result schema

A run result contains:

```
{
  "version": "1.0",
  "benchmark_score": 0.86,
  "summary": {
    "examples": 100,
    "valid": 96,
    "invalid_outputs": 2,
    "execution_errors": 2
  },
  "results": []
}
```

A valid example result:

```
{
  "index": 0,
  "status": "valid",
  "prediction": {
    "total": 100.0
  },
  "field_scores": [
    {
      "path": ["total"],
      "score": 0,
      "weight": 5
    }
  ],
  "score": 0.0,
  "contribution": 0.0,
  "error": null
}
```

An invalid output:

```
{
  "index": 0,
  "status": "invalid_output",
  "prediction": {
    "debug": "foo"
  },
  "field_scores": null,
  "score": null,
  "contribution": 0.0,
  "error": {
    "code": "missing_field",
    "path": ["total"],
    "message": "required output field is missing"
  }
}
```

An execution error:

```
{
  "index": 0,
  "status": "execution_error",
  "prediction": null,
  "field_scores": null,
  "score": null,
  "contribution": 0.0,
  "error": {
    "code": "adapter_error",
    "path": null,
    "message": "..."
  }
}
```

### A\.15Dataset failures

Dataset failures are run\-level benchmark errors rather than example scores\.

Examples:

```
data file missing
malformed JSONL
row lacks input/expected
input does not conform to input schema
expected output does not conform to output schema
artifact path does not exist
empty dataset
```

The run aborts and does not produce a benchmark score\.

### A\.16Error model

Compiler and runtime diagnostics should use:

phasecodepathmessage

pathis an array of semantic/object keys when applicable\.

Representative codes:

invalid\_yaml duplicate\_key unknown\_key unsupported\_version unsupported\_ontology\_version unknown\_task unknown\_domain unknown\_language task\_program\_mismatch invalid\_schema missing\_weight extra\_weight invalid\_weight data\_not\_found invalid\_dataset\_record artifact\_not\_found missing\_field extra\_field wrong\_type invalid\_enum invalid\_value adapter\_error timeout provider\_error

The exact message text is not part of benchmark semantics\.

## Appendix BShared SURUS ontology registry

The ontology is one shared YAML file containing:

tasksdomainslanguages

Example:

```
version: "1.0"

tasks:
  extract:
    description: Extract named information from an input.
  classify:
    description: Assign one declared categorical label.
  transcribe:
    description: Convert spoken audio into text.
  translate:
    description: Transform text from a source language to a target language.

domains:
  general: {}
  finance: {}
  healthcare: {}
  legal: {}
  retail: {}

languages:
  es: {}
  pt: {}
  en: {}
```

Benchy code, not the registry, implements the structural task validators for each supported ontology version\.

This avoids prematurely introducing a task\-constraint DSL while preserving:

P∈𝒫TP\\in\\mathcal\{P\}\_\{T\}\(32\)> The declared program must satisfy the structural semantics associated with its task\.

## Appendix CArtifact runtime representation

Two designs were considered forimage,audio, anddocument\.

### Typed artifact object

The engine could convert a dataset path into an internal object such as:

```
Artifact(type=image, path=...)
```

This gives the runtime value an explicit host\-language wrapper, but introduces a new abstraction that every adapter and language binding must understand\.

### Path string

The dataset may keep the artifact as a path string and let the adapter decide how to consume it\.

This is simpler and fits the adapter boundary: one adapter may open the file, another may upload it over HTTP, and another may decode it into a provider\-specific object\.

The first implementation therefore usesresolved path stringsat the runtime boundary\.

The semantic meaning still comes from the program schema:

not from the host\-language type of the value\.

A future runtime representation may replace path strings with a richer artifact handle without changing the benchmark’s semantic type system\.

## Appendix DCompact architecture

shared SURUS ontologycanonical YAMLHuman / Agent / UIparsevalidatecompilecanonical JSON IREngineAdapterAI\-systemruntime contract

The central rule is:

> Benchmark meaning is defined once in the canonical YAML, compiled once into JSON IR, and executed through one runtime contract\.

相似文章

好的基准

arXiv cs.AI

这篇2026年7月的论文基于Terminal Bench项目的经验,定义了为AI智能体设计有效基准任务的原则。好的任务应当正确、可解、可验证、明确规范,并且因有趣的原因而具有难度,强调现实世界相关性和基于结果的验证。

基准测试变得简单

Reddit r/AI_Agents

作者创建了Any-Bench,这是一个用于在个人代码库上对AI模型进行基准测试的工具,解决了现有基准测试如SWE-Bench的不足。

ProgramBench(5分钟阅读)

TLDR AI

ProgramBench 是一项全新的基准测试,用于评估 AI 智能体在无法获取源代码或反编译工具的情况下,仅凭编译后的二进制文件和文档重建完整软件项目的能力。