The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
Summary
This paper introduces Taste-Bench, a benchmark for measuring taste in LLM agents' long-horizon decisions, finding that frontier models have low accuracy and that taste can be improved through distillation training.
View Cached Full Text
Cached at: 09/23/26, 07:31 AM
Paper page - The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
Source: https://huggingface.co/papers/2609.25804
Abstract
LLMagentsincreasinglyworkonlong-horizontasks,andthedecisionstheymakealongtheway,suchaswhichhypothesistotestorwhichimplementationtobuildon,determinetheoutcomeofthewholerun.Makingthesedecisionswellisbecomingakeycapabilityforbothengineeringandresearchagents.Werefertotheabilitytomakegoodlong-horizondecisionsasthetasteofanagent.Whileexistingbenchmarksmeasuretheend-to-endsuccessofagentsonlong-horizontasks,noneofthemmeasuresthetasteofanagent.Toaddressthisproblem,webuildTaste-Bench,abenchmarkoftastequestionsconstructedautomaticallyfromtrajectoriesthatagentsproducedinengineeringandresearchtasks.Eachquestionpresentsadecisionfork,apointinatrajectorywheremultipledirectionsareavailableandoneofthemleadstoabetteroutcome,andtheevaluatedmodelchoosesamongthesedirectionswithoutseeingwhathappensafterthefork.Weminetheseforksautomaticallyfromparallelattemptsatthesametaskandfromdetoursinsideasingletrajectory,withoutneedinghumanannotation.WeevaluatefrontiermodelsonTaste-Benchandfindthatthebestmodelanswersonly59.7%ofthequestionscorrectly.Wefurtherfindthatforkswhosedecidingevidenceappearslaterinthetrajectoryaremuchharderforeverymodel,andthatalargerreasoningbudgetdoesnotimprovetheaccuracy.Finally,weshowthattastecanbetrained.Wedistillthejudgmentofateacherthathasseentheoutcomeintoastudentmodel,andthestudentmakesbetterdecisionsonunseentasksandimprovesend-to-endsuccessonheld-outSWE-benchProtasks.
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2609\.25804
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.25804 in a model README.md to link it from this page.
Datasets citing this paper1
#### wenbopan/taste-bench Viewer• Updatedabout 4 hours ago • 502 • 7
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.25804 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
How to Build a Benchmark That Actually Measures Taste
The paper introduces TasteBench, an expert-gold pairwise benchmark for measuring 'taste' in VLMs across visual, brand, book, and tech domains, along with a control protocol that quantifies and corrects evaluation artifacts like position bias, lexical leakage, and difficulty ceilings.
A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
TASTE is an automated method for generating challenging agent benchmarks with broader tool-use coverage by evolving tool sequences through adaptive contrastive n-gram modeling and iterative difficulty refinement. The resulting τ^c-Bench reveals that models nearly saturating existing benchmarks suffer severe performance drops, indicating saturation rather than robust skill.
rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment
rTaste evaluates whether frontier LLM research agents can exercise good early-stage research judgment by having six models propose and rank 415 experiments derived from recent NLP papers, finding only 73 experiments were both sound and beyond the seed paper, and that agent forecasts and rankings did not clearly beat simpler baselines.
TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods
TasteBench introduces a multimodal benchmark and privacy-preserving competition for sensory prediction, spanning food-level ranking (21K+ human evaluations across 215 plant-based foods) and molecular-level taste classification (15K flavor molecules), with baselines reaching pairwise accuracy competitive with median human panelists. It aims to provide computational proxies for sustainable protein discovery, analogous to molecular docking in drug design.
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
CoffeeBench is a benchmark for evaluating LLM agents in a long-horizon multi-agent economic simulation where firms interact over 90 days to maximize profits, revealing differences in communication patterns and performance among various models.