wbopan/tastebench

Source: https://github.com/wbopan/tastebench

📄 Paper  |  🤗 Dataset  |  🏆 Leaderboard  |  🚀 Quick start

Taste-Bench measures the taste of an LLM agent: its ability to choose the better direction at a real decision fork in a long-horizon task. Given the task, the trajectory up to the fork, and two candidate next steps, the model must pick the step that the hidden rest of the trajectory proves right. A wrong choice often looks reasonable at the moment and costs the agent most of its budget later. The 502 questions are mined from software engineering and machine-learning research trajectories, with no expert annotation, and the best frontier model answers 59.7% of them correctly.

_{A decision fork from a machine-learning trajectory. The model chooses before the later losses reveal that A is better.}

Leaderboard

A question counts as correct only when the model answers it correctly in both option orders, so random guessing scores 25 and a model that always picks the same position scores 0. Average is the 1:1 mean of the research and engineering subsets.

_{D = detour, P = parallel; Eng = engineering (390 questions), Res = research (112). Unparsed presentations, out of 1,004, count as wrong. All rows use protocol paired_order_v1, August 2026. To add a model, run the full protocol and open a pull request with its summary.json under results/<model>/.}

How the questions are built

The later part of a trajectory is hindsight evidence for the decision made at a fork, so the trajectories label themselves. Questions come in two constructions and two domains.

• Parallel forks. Independent attempts at the same task diverge at the same point and end with different recorded outcomes. The outcome of each attempt labels the better direction.
• Detour forks. An agent takes a direction, abandons it after an observed failure, and recovers inside the same run. The abandoned direction and the later recovery are the candidates.
• Filtering. A question is dropped as trivial when every judge model answers it from the candidate wording alone, and as undecidable when any judge disagrees with its label after reading the full record. Of 4,657 mined forks, 502 survive: 266 detour and 124 parallel questions in engineering (SWE-bench and SWE-bench Pro rollouts), 64 detour and 48 parallel in research (METR’s MALT release of RE-Bench and HCAST runs).

Evaluation protocol

Specified in protocol/paired_order_v1.yaml. The model sees the task, the full pre-decision trajectory as published in prefix_text (credentials and usernames redacted, 64K-token budget with an explicit omission marker on overflow), and the two candidates. The prompt asks for one line, ANSWER: X, without visible chain of thought. Every question is asked in the published option order and in its exact reverse, with letters recomputed each time. The denominator is every released question, and request errors and unparseable outputs count as wrong. tb validate checks the release hashes and every question’s invariants.

Quick start

The dataset is gated to limit training contamination: request access on its Hugging Face page, then run hf auth login once.

git clone https://github.com/wbopan/tastebench && cd tastebench && uv sync
uv run tb download                                   # wenbopan/[email protected] into ./data
export TASTEBENCH_API_KEY=...                        # any OpenAI-compatible endpoint
uv run tb run --model gpt-5.5 --api https://api.openai.com/v1/chat/completions
uv run tb score runs/gpt-5.5/<date>                  # prints per-cell accuracy and Average

tb run writes one record per question and order, is resumable, and takes --limit N for smoke tests. A full run is 1,004 requests and about 8M input tokens. Per-model request settings live in configs/models.yaml.

The data itself is two parquet configs, engineering (390 rows) and research (112 rows), with one test split each. A row holds query, the full prefix_text, the two choices in the published order, and the answer letter, plus cell, task_id, and a contamination canary. It loads without this repository:

from datasets import load_dataset
rows = load_dataset("wenbopan/taste-bench", "engineering", split="test", revision="v1.0")

Citation

@article{pan2026tasteful,
  title   = {The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks},
  author  = {Pan, Wenbo and Liu, Zhichao and Liu, Shujie and Zeng, Jingying and Lin, Chin-Yew and Tang, Xianfeng and Lu, Yan and He, Qi and Jia, Xiaohua},
  journal = {arXiv preprint arXiv:2609.25804},
  year    = {2026}
}

Trajectories come from SWE-bench, SWE-bench Pro, and METR’s MALT release. The code is released under the MIT License and the dataset text under CC BY 4.0.