Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

Hugging Face Daily Papers Papers
agentic-evaluation base-models pretraining agent-scaffolds benchmarks llm

Summary

The paper introduces ASAP, a lightweight framework for evaluating agentic behavior in pretrained base models through interactive end-to-end tasks without converting them into chat agents, demonstrating discriminative performance across software-engineering and deep-research benchmarks.

Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior.
Original Article
View Cached Full Text

Cached at: 10/21/26, 10:21 AM

Paper page - Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

Source: https://huggingface.co/papers/2610.19425 Published on Oct 14

·

Submitted byhttps://huggingface.co/qinjr

JR QINon Oct 21

Abstract

ASAP is a lightweight framework that evaluates agentic behavior in pretrained base models via interactive end-to-end tasks without converting them to chat agents.

Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into howbase modelsbehave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluatingagentic behaviorfrom pretrainedbase modelson end-to-end tasks. Rather than convertingbase modelsinto chat-style agents, ASAP preserves the completion interface and adapts existingagent scaffoldsthrough structuredtrajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering anddeep-research benchmarks, ASAP yields discriminative end-to-end performance acrossbase modelsandpretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generatedexecution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.19425

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.19425 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.19425 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.19425 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Scaling Agents via Continual Pre-training

Papers with Code Trending

Proposes Agentic Continual Pre-training to build agentic foundation models, achieving state-of-the-art results on 10 benchmarks with AgentFounder-30B, including 39.9% on BrowseComp-en and 43.3% on BrowseComp-zh.

Is it agentic enough? Benchmarking open models on your own tooling

Hugging Face Blog

This blog post introduces a benchmark methodology for evaluating how well open models perform on agentic coding tasks, focusing not just on accuracy but on the efficiency of the agent's process. It provides a customizable tooling harness using the pi coding agent and tests across models and library revisions.