onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

Hugging Face Daily Papers Papers
llm-alignment token-level-correction annotation-tool on-policy-data agents interactive-tool

Summary

onPanda is an interactive tool that uses token-level correction to efficiently annotate LLM alignment data and agent trajectories, reducing median annotation time by 52%.

We present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model's candidate tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained. This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing. Since the vast majority of tokens in the final response are generated by the model itself, the resulting data largely preserves the model's sampling distribution and is well suited for constructing on-policy SFT and preference data. Furthermore, the token-level corrections recorded during annotation provide fine-grained supervision with precise positions and naturally paired positive--negative samples. onPanda also connects to external tools and harnesses, enabling interactive trajectory annotation in realistic environments. In addition, we release Panda-CVL, a dataset annotated with onPanda, together with a benchmark for token-level correction.
Original Article
View Cached Full Text

Cached at: 09/22/26, 07:27 AM

Paper page - onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

Source: https://huggingface.co/papers/2609.24983 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

WepresentonPanda,aninteractivetoolforefficientlyannotatingLLMalignmentdataandagenttrajectories.onPandaadoptstoken-levelcorrectionasitscoreinteraction:whilereadingamodelresponse,theannotatorlocatesthefirstinappropriatetokenandeitherpicksasubstitutefromthemodel’scandidatetokensortypesthecorrecttextviafree-formediting.Thesystemthentruncateseverythingafterthatpositionandcontinuesgenerationfromthecorrectedprefix,repeatingthislocate-correct-continueloopuntilasatisfactoryresponseisobtained.Thismechanismletsannotatorspreciselysteermodeloutputsatlowcost:asmallcontrolledstudysuggeststhatonPandareducesmedianannotationtimeby52%overmanualpost-editing.Sincethevastmajorityoftokensinthefinalresponsearegeneratedbythemodelitself,theresultingdatalargelypreservesthemodel’ssamplingdistributionandiswellsuitedforconstructingon-policySFTandpreferencedata.Furthermore,thetoken-levelcorrectionsrecordedduringannotationprovidefine-grainedsupervisionwithprecisepositionsandnaturallypairedpositive--negativesamples.onPandaalsoconnectstoexternaltoolsandharnesses,enablinginteractivetrajectoryannotationinrealisticenvironments.Inaddition,wereleasePanda-CVL,adatasetannotatedwithonPanda,togetherwithabenchmarkfortoken-levelcorrection.

View arXiv pageView PDFProject pageGitHub14Add to collection

Get this paper in your agent:

hf papers read 2609\.24983

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.24983 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.24983 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.24983 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

PrivacyAlign: Contextual Privacy Alignment for LLM Agents

Hugging Face Daily Papers

PrivacyAlign introduces a human-annotated dataset and training framework for aligning LLM agents to respect contextual privacy norms, showing that frontier models still leak sensitive information and that human-grounded evaluation improves alignment.

Measuring, Localizing, and Ablating Alignment Signatures in LLMs

arXiv cs.LG

This paper investigates how post-training of LLMs introduces AI-like stylistic regularities and proposes PASTA, a training-free method to localize and ablate these alignment signatures, reducing AI detection rates while maintaining coherence across 11 models and 6 detectors.