A model score is not an agent score.

Model benchmarks usually isolate the foundation model.

However, production agents do not run in isolation. Instead, they run inside a system that parses tool calls, stores history, manages context, retries failed actions, enforces budgets, and decides when the task is complete.

A failed benchmark run does not reveal whether the model or the surrounding system caused the failure.

Say the model selects the correct tool and arguments, but the adapter serializes one field incorrectly. The environment rejects the call.

The retry policy then submits the same malformed request three times. History grows, useful observations fall out of context, and the run stops after reaching its action budget.

The final score is zero, even though the model selected the right action.

Replacing the model might change the next run, but it would not isolate the original failure. The tool adapter, retry policy, context policy, or stopping rule could produce the same result.

A controlled agent evaluation should hold the task, data, tools, budget, and verifier fixed. The same model can then run under two configurations.

The reference configuration measures the model inside a pinned control loop. The submitted configuration measures the complete system an engineering team plans to deploy.

The difference between those results estimates what the harness contributed for that model and task.

Apodex built this comparison into TRACES.

TRACES stands for Tools, Repair, Alternatives, Coherence, Evidence, and Scope, the six process dimensions used in its HDS6 evaluation.

Devs can submit a hosted model, a checkpoint, or a complete agent system.

Model submissions run through a pinned reference harness. Full-system submissions can also run with the submitted harness, producing a comparable reference result and an end-to-end system result.

The accompanying paper tested four model-agnostic harnesses across fixed model configurations on seven selected LLM environments, and I worked with Apodex on this post to show how this works.

ApodexHarness led with Opus at 0.611, while A-Evolve led with GPT at 0.648. The harness ordering changed with the model, so those experiments did not produce one universal winner.

You can read the paper here: https://huggingface.co/papers/2608.11341…

Model choice and harness choice interact. A leaderboard that considers them as a single variable cannot show which component needs work.

Interpreting that difference also requires understanding what an agent harness contains.

I broke down its 11 production components, including the orchestration loop, tools, memory, state persistence, and guardrails.

Read it below.

Paper page - Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

Source: https://huggingface.co/papers/2608.11341 Published on Aug 11

#3 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Apodex Discovery introduces a framework for verifiable, extended AI investigations using a heavy-duty solver and structured evaluation across real-world scientific tasks.

Apollo did not reach the Moon merely because its engineers could solve difficult equations. It succeeded by turning a distant ambition into a mission architecture of explicit objectives, simulation, verification, and repeated correction. AI now faces a similar transition: frontier models can solve difficult tasks once the problem, tools, and success criteria are specified, yet consequential real-world challenges rarely arrive in an executable or verifiable form. We introduceApodex Discovery, a framework for building and evaluating discoverative AI through theheavy-duty solver, a system comprising afoundation model,harness, tools, andcontrol policiesthat pursues extended, stateful, verifiable investigations. It has three core components. First, aproblem-scoutingprocess surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release. Second, a commonenvironment-task-episode abstractionprovides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions. Third,HDS6evaluates Tools, Repair, Alternatives, Coherence, Evidence, and Scope independently of final-task success. InAAV capsid design, Apodex surpassed the published state of the art by 7% across viability, tropism, structure prediction, and generative design. Indrug repurposingand reformulation, a task-specific biomedical environment improved the mean normalized prediction score of GPT-5.5 and GPT-5.6-sol by 2.5 and 7.6 points over the same closed-book backbone. Controlled ablations show that the fixedTRACESepisode interface enables attribution of performance differences to specific solver components.Apodex Discoverymoves AI evaluation beyond predefined benchmarks toward verifiable investigations aimed at genuine discovery.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2608\.11341

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.11341 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.11341 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.11341 in a Space README.md to link it from this page.

Collections including this paper2