Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness
Summary
Goldsmith, an agentic annotation pipeline presented in an arXiv paper, converts small expert-annotated gold sets into structured annotation definitions optimized via an executable loss-guided loop, outperforming direct rewriting, OPRO, APE, and PromptBreeder.
View Cached Full Text
Cached at: 10/08/26, 12:42 PM
# Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness Source: [https://arxiv.org/abs/2610.09489](https://arxiv.org/abs/2610.09489) [View PDF](https://arxiv.org/pdf/2610.09489)[HTML \(experimental\)](https://arxiv.org/html/2610.09489v1) > Abstract:Many annotation projects begin before experts have a stable guideline or enough labels to train a task\-specific model\. We present Goldsmith, an agentic pipeline that turns a small gold set\-\-\-expert\-annotated calibration examples representing the intended task boundaries\-\-\-into a reusable structured annotation definition\. Goldsmith treats this definition as a trainable textual object\. Candidate definitions are run on the same gold examples and scored with an executable structured loss, while the output schema, formatting, retrieval, repair, judging, and human review remain in an external harness\. A large language model \(LLM\) editor converts the highest\-loss failures into textual\-gradient revisions, which are accepted only when the measured loss decreases\. In prompt\-optimization comparisons, Goldsmith improves over direct rewriting, OPRO, APE, and PromptBreeder under matched evaluation protocols\. The resulting definition also improves downstream annotation when combined with retrieval, score\-based routing, and human review across typed span, pair\-level relation, and fixed\-trigger event\-argument tasks\. These results show that scarce expert supervision can support both task\-definition learning and scalable annotation\. ## Submission history From: Yihan Li \[[view email](https://arxiv.org/show-email/ac6effbb/2610.09489)\] **\[v1\]**Wed, 7 Oct 2026 05:45:25 UTC \(548 KB\)
Similar Articles
Making Optimization Work When Labels Are Scarce [R]
Gnosys Labs introduces an autonomous model engineering method that improves classifiers under label scarcity, outperforming standard optimizers like GEPA on the ToxicChat benchmark.
A^2TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
This paper introduces A^2TGPO, a reinforcement learning method for agentic LLMs that uses adaptive turn-level clipping and information gain normalization to improve process credit assignment in multi-turn interactions.
PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks
PGPO proposes potential-guided policy optimization for multi-turn agentic tasks, enabling finer-grained credit assignment in LLM post-training and showing strong results on ALFWorld and WebShop benchmarks.
GAGPO: Generalized Advantage Grouped Policy Optimization
GAGPO proposes a critic-free RL method that uses a non-parametric grouped value proxy for step-level credit assignment in multi-turn agentic tasks, outperforming strong baselines on ALFWorld and WebShop.
Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
ProGPO is a learned-critic-free method for step-level advantage estimation in group-based RL for LLM agents, using exact-prefix action comparisons and rollout-based state potentials to improve credit assignment on long-horizon tasks. Experiments on ALFWorld and WebShop with Qwen2.5 models show it outperforms existing agentic RL baselines.