Tag
Eli Gottlieb shares insights on designing effective RL training tasks, framing task creation as a constraint-satisfaction problem across four components—prompt, world, verifier, and intent—and outlining alignment axes like prompt-verifier and verifier-intent to avoid reward hacking.
This paper introduces 'instrument uncertainty' in LLM-based annotation, showing that while repeated LLM runs under one task design are highly reliable, reasonable changes in task design or model choice can substantially shift labels and prevalence estimates for offensive language and hate speech.
The article explores whether AI's biggest challenge is interface design rather than model improvements, noting that current chat-based interactions fail to capture the rich context humans use for tasks.
This article details recruitment requirements for expert question designers to develop non-code, long-cycle tasks for AI model validation, stressing authenticity, quality control, and relevant professional experience.
This paper from July 2026 defines principles for designing effective benchmark tasks for AI agents, drawing on experience from the Terminal Bench project. Good tasks are correct, solvable, verifiable, well-specified, and hard for interesting reasons, with an emphasis on real-world relevance and outcome-based verification.