COMPOSITE-Stem

# Introduction Source: https://arxiv.org/html/2604.09836 ![[Uncaptioned image]](https://arxiv.org/html/2604.09836v2/orb_avatar.png) COMPOSITE-STEM 70 expert-curated agentic tasks across Physics, Biology, Chemistry, and Math. Portex Organizing Team Kyle Waters, Lucas Nuzzi, Tadhg Looram PortexAI, PortexAI, PortexAI [email protected], [email protected], [email protected] April 2026 Dataset Contributors Alessandro Tomasiello (University of Milano-Bicocca), Ariel Ghislain Kemogne Kamdoum (University of Calgary), Bikun Li (University of Chicago), Damien Sileo (Inria), Egor Kretov (Fraunhofer Institute for Individualized Medical Technology IMTE), Francesco Fournier-Facio (University of Cambridge), Georgios Soloupis (Independent), Haile Kassahun (McGill University), Hew Wolff (Independent), Jiaqi Cai (Massachusetts Institute of Technology), Lianghui Li (École Polytechnique Fédérale de Lausanne), Marc Roth (Queen Mary University of London), Mohinder Naiya (Dot Ingredients), Naixu Guo (National University of Singapore), Qicheng Tang (Georgia Institute of Technology), Richard Wheeler (University of Edinburgh), Samuele Sala (Murdoch University), Serguei Popov (University of Porto), Steven Dillmann (Stanford University), Yuqi Li (Stony Brook University)

## Abstract

AI agents hold growing promise for accelerating scientific discovery; yet, a lack of frontier evaluations hinders adoption into real workflows. Expert-written benchmarks have proven effective at measuring AI reasoning, but most at this stage have become saturated and only measure performance on constrained outputs. To help address this gap, we introduce COMPOSITE-STEM, a benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics, curated by doctoral-level researchers. Our benchmark combines exact-match grading and criterion-based rubrics with an LLM-as-a-jury grading protocol, allowing more flexible assessment of scientifically meaningful outputs. Using an adapted multimodal Terminus-2 agent harness within the Harbor agentic evaluation framework, we evaluate four frontier models. The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current agent reach. All tasks are open-sourced with contributor permission to support reproducibility and to promote additional research towards AI's acceleration of scientific progress in these domains.

Scientific evaluations are central to advancing frontier AI for real scientific workflows. In this work, we introduce COMPOSITE-STEM, a cross-domain STEM task bundle compatible with Harbor (https://harborframework.com/) (TerminalBench-style agentic evaluation). This paper documents benchmark construction, task curation, and model performance. We evaluate 4 models using a modified Terminus-2 agent harness adapted for multimodal support in Harbor. claude-opus-4.6 leads at 21.4% (Pass@1). The benchmark is designed to test more than isolated scientific reasoning by pairing expert-authored tasks with executable environments and flexible grading.

## Leaderboard Snapshot

![[Uncaptioned image]](https://arxiv.org/html/2604.09836v2/frontpage_leaderboard.png)

Figure 1: COMPOSITE-STEM leaderboard on 70 tasks (Pass@1).

## Related Work

Benchmark design has been shifting from short, static reasoning tests toward harder, longer-horizon, and more realistic tasks that better reflect real-world work. This has been necessitated by the quick advancement in capabilities, coupled with the saturation and contamination of many expert-grade benchmarks. When GPQA (Rein et al. 2023) was introduced in 2023 as a benchmark of "Google-proof" multiple-choice science questions written by PhD-level experts, GPT-4 scored 39%, but only two years later, GPT-5.2 scored 92%. A milestone in benchmark design was the release of Humanity's Last Exam or HLE, which established a multi-subject, academic-focused benchmark at scale. HLE covered over 2,500 questions from 500+ contributors sourced from top global research institutions, and revealed clear limits to SOTA model capabilities at the time of its release (Center for AI Safety et al. 2026). However, HLE remains primarily a static QA-style benchmark with exact-match and multiple-choice style evaluation. Moreover, despite efforts to prevent training on benchmark data, data contamination has become a real concern as HLE is a high-profile benchmark that has garnered attention and is cited in almost all model releases.

As agentic systems like Claude Code and Codex gained popularity, benchmark efforts increasingly moved into executable environments. Terminal-Bench Core and Terminal-Bench 2.0 were important steps in evaluating agents in terminals with reproducible containerized settings. The current 2.0 benchmark includes 89 curated tasks spanning software engineering, ML, security, and data workflows (Merrill et al. 2026). In parallel, GDPval expanded realism from an economic-work perspective with 1,320 professionally grounded tasks curated by domain experts, emphasizing high-value deliverables (e.g. excel spreadsheets, presentations, PDF documents) rather than only abstract reasoning (Patwardhan et al. 2025).

Mercor contributes two distinct and relevant papers in this space. First, APEX-v1-extended evaluates economically valuable professional tasks (law, finance, consulting, and general medicine) with prompt-specific grading rubrics and LM-judge scoring, illustrating how rubric-based evaluation can support richer, less brittle assessment than exact-match-only benchmarks (Vidgen et al. 2025). Second, APEX-Agents pushes further into fully agentic workflows with 480 tasks across realistic multi-application professional environments (called "worlds"), bringing rubric-based scoring into longer-horizon agent execution settings (Vidgen et al. 2026). Finally, OpenAI's introduction of FrontierScience (Wang et al. 2026) further advanced expert-sourced scientific benchmarking by pairing difficult, original science problems with a more structured evaluation framework for open-ended answers. The benchmark spans physics, chemistry, and biology, and is split into two tracks: an Olympiad track built from expert-written, verifiable short-answer problems, and a Research track composed of PhD-level research subproblems designed to reflect authentic scientific reasoning tasks. Most relevant here, FrontierScience's Research track moves beyond exact-match grading by assigning each task a 10-point rubric made up of multiple independent, objectively assessable criteria, allowing evaluation of intermediate reasoning steps rather than only final-answer correctness. A response is treated as successful if it earns at least 7 out of 10 rubric points, enabling finer-grained analysis of partial progress and failure modes. To scale grading, FrontierScience uses a single LLM judge, GPT-5 with high reasoning effort, to score submissions against these expert-authored rubrics.

COMPOSITE-STEM sits at the intersection of these directions: expert-authored STEM tasks paired with reproducible Harbor-compatible agent execution. The goal is to preserve the academic rigor of difficult expert benchmarks while evaluating performance in realistic settings across physics, chemistry, biology, and math.

## Task Composition

COMPOSITE-STEM contains 70 tasks: 20 in physics, 23 in chemistry, 20 in biology, and 7 in math. The tasks mix structured problem solving, multi-step reasoning, and domain-specific constraints, so strong performance requires both conceptual understanding and knowledge of execution (e.g. most appropriate Python packages). Reference assets are also a meaningful part of the task mix: 18/70 tasks include files mounted into the agent sandbox (under /app/refs), and these are mostly images (17 image files, primarily PNG, plus 1 PDF).

## Expert Background

Contributors for COMPOSITE-STEM were sourced from top research institutions and global universities and primarily included contributors to Humanity's Last Exam (HLE) (https://agi.safe.ai/) and adjacent frontier benchmarks. Across domains, the contributor group includes doctoral-level researchers, distinguished faculty members, postdoctoral scientists, and industry practitioners with publication records and prior benchmark design experience. Beyond sourcing, the Portex team worked closely with all contributors through detailed calls and review cycles to shape task design, clarify grading intent, and identify specification issues before release.

## Environment

We use an adapted multimodal Terminus-2 (https://harborframework.com/docs/agents/terminus-2) agent harness in the Terminal-Bench (https://www.tbench.ai/) and Harbor ecosystem, to minimize harness-specific variance while better supporting visual-based tasks such as analyzing x-rays and microscopy imagery (more details about the Harbor framework are provided later in the paper). The harness preserves the standard Terminus-2 execution loop, but when a task specifies a reference file, it downloads that file from the environment and supplies it to the model in the first turn as native multimodal input. Images are attached directly, and common text-based files are inlined as text. This modification improves evaluation fidelity on tasks requiring visual or document understanding by avoiding extra turns spent on indirect file inspection, and better aligns agent evaluation with standard multimodal LLM evaluation.

Evaluations run in a Modal (https://modal.com/)-provided sandbox with a controlled runtime envelope:
- timeout_sec = 3600.0 for the agent loop
- build_timeout_sec = 1200.0
- cpus = 1
- memory_mb = 2048
- storage_mb = 10240

The Harbor task image is bootstrapped from the following base Dockerfile:

```
FROM python:3.12-slim
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    bash \
    tmux \
    asciinema \
    curl \
    ripgrep \
    git \
    && rm -rf /var/lib/apt/lists/*
RUN python -m pip install --no-cache-dir "litellm>=1.67.0"
WORKDIR /app
# Ensure reference assets are available inside the container.
COPY refs/ /app/refs
# Default shell
CMD ["/bin/bash"]
```

Verification combines exact-match checks with semantic rubric grading using an LLM jury (more details explained below in AsymmetryZero Grading Protocol). In COMPOSITE-STEM (n=70), 35 tasks are graded with exact match, 34 use semantic LLM-jury grading, and 1 uses a hybrid setup. Experts design rubrics as sets of criteria that are graded either by exact-match parsing or via an LLM-as-a-jury for semantic correctness; rubric size ranges from 1 to 40 criteria, with an average of 2.6 criteria per rubric.

The LLM jury is composed of:
- deepseek/deepseek-v3.2
- z-ai/glm-5
- openai/gpt-oss-120b
- meta-llama/llama-3.3-70b-instruct
- moonshotai/kimi-k2.5

At a high level, `portex_grade.py` loads task config and criteria, reads the submission, and grades each criterion either via exact-match extraction or via multi-judge LLM scoring. It applies strict majority voting for semantic criteria, awards weighted points, aggregates to a raw score, normalizes to Harbor's reward scalar, and writes both `reward.json` and a detailed `portex_detail.json`. The grader includes robust request handling: retries transient judge failures a limited number of times with short delays, fails fast on clear client errors, and maps persistent judge failures to explicit error outcomes with zero score so majority voting and verifier rewards stay well-defined.

The environment is a bare Python-based instance and we observe all agents using relevant packages for scientific computation and exploration such as rdkit-pypi, scipy, rd-kit, sympy, googlesearch-python, numpy, and others. Agents are allowed to effectively use web search via installed packages like DuckDuckGo but there is no native websearch tool in the Terminus-2 harness we use. We do not supply any additional custom tooling beyond the base Dockerfile.

## Task Curation and Portex Datalab

Task development was conducted in the Portex Datalab (https://datalab.portexai.com/), where experts iterated on prompts, rubrics, and scoring assumptions while inspecting frontier-model behavior in near real time. The Portex team spoke extensively with contributors, collaborated with them to design and refine their evals, tested tasks against state-of-the-art models to gauge difficulty, and returned detailed reports and trial data so experts could iterate on task wording, rubric structure, and latent errors. This yielded a structured, multi-round curation loop instead of one-shot task drafting.

The curation workflow on the Datalab combined:
- Eval builder workflow: experts drafted tasks, attached references, and specified weighted criteria with explicit grading intent.
- Live model feedback: experts observed frontier-model outcomes and failure patterns while iterating on task clarity and discriminative power.
- Portex review loop: contributors received detailed run reports, artifacts, and error signals that helped them debug task specs, tighten rubrics, and correct edge-case mistakes.
- Public Leaderboards: shared leaderboards introduced light gamification that encouraged repeated quality improvements and sharper task specifications.
- Cross-domain consistency: a common Harbor execution substrate reduced harness variance while preserving domain-specific task content.

Figure 1: Portex Datalab eval-builder interface used by contributors to draft task instructions, attach reference files, and define grading rubrics.

Figure 2: Portex Datalab audit view showing model traces, jury decisions, and criterion-level outcomes used to inspect failures and refine evals.

### Domain Expert Backgrounds and Task Design

Here, we summarize expert credentials and task-design coverage for each domain.

### Physics

Physics contributors include advanced profiles across theoretical and experimental research, including doctoral and postdoctoral work in mathematical physics, string theory, condensed matter, and quantum systems, with affiliations spanning leading universities and research institutes. Task types include symbolic derivations, quantum/open-system reasoning, many-body and topological analyses, and device-level or measurement-grounded calculations.

### Chemistry

Chemistry contributors include researchers and practitioners in organometallic, medicinal, analytical, and computational chemistry from both academic and applied R&D settings. Task types include synthetic-mechanism reasoning, representation conversion (e.g., SMILES/InChI), spectroscopy interpretation (NMR/MS), and concept-heavy chemical judgment.

### Biology

Biology contributors include experts in infection biology, biomedical engineering, medical imaging, and microscopy-driven interpretation with faculty/postdoctoral and industry-adjacent backgrounds, including wet-lab settings. Task types include imaging diagnosis, MRI engineering reasoning, electron microscopy interpretation from labs, spatial-structure inference, and mechanism-aware biological analysis.

### Mathematics

Math contributors include doctoral-level researchers and academics across pure and applied mathematics, theoretical computer science, and statistics with records in peer-reviewed venues. Task types include proof-oriented derivations, combinatorial and algebraic reasoning, stochastic-process analysis, and invariant/structure computation.

Figure 3: COMPOSITE-STEM domain-level task counts.

### AsymmetryZero Grading Protocol