@yibie: Recommend this: Hamel Husain—one of the world's most experienced AI eval practitioners—said: "It's hard to evaluate" is itself a product smell. If you don't provide users with inspectable intermediate artifacts, they'll redo your work to verify the output. Then what value does your product bring them?...

Recommend this: Hamel Husain—one of the world's most experienced AI eval practitioners—said: "It's hard to evaluate" is itself a product smell. If you don't provide users with inspectable intermediate artifacts, they'll redo your work to verify the output. Then what value does your product bring them? "It's Hard to Eval" Is a Product Smell I've been doing AI eval for three years. The most common objection I hear is "our product is hard to evaluate." This objection itself is a product smell—what's hard for you to verify is also hard for your users. More importantly: design a product that is easy to verify, and do it before writing any eval. Example 1: AI Data Analysis Agent Almost every company is building an internal analysis agent. Ask it a business question, it finds data sources, runs queries, gives an answer. The goal is to reduce dependency on data analysts. The most common mistake: making the "answer" the only output. The interface shows only a single number—"Product A's Net Revenue Last Quarter: $4.21M." Users have no way to verify other than redoing everything themselves. A better design doesn't hide the process; it gives users inspectable artifacts. When a data analyst verifies a metric, they do six things: cross-check intermediate calculations and totals against trusted sources, confirm the precise definition of the metric, verify plausibility of related quantities, disaggregate the aggregate numbers by dimensions, read the SQL, and document what cannot be verified. A good AI data analysis agent should do the same—show sources, assumptions, and unverifiable questions in its answer. Users can click to open a notebook and see the full analysis: the assumptions the agent made, the queries it ran, the breakdown by region, and explicitly listed "parts that cannot be verified." Each item is preserved as a runnable cell so users can drill down further. What This Means for Eval If your product is designed for easy verification, annotation costs drop and eval can extract better signals. More importantly, you'll ship a better product. Example 2: PE Lesson Plan Generator A founder built a product: teachers input constraints (grade level, duration, indoor/outdoor, equipment), and the AI writes a lesson plan. "How do you evaluate a good lesson plan?"—Hamel flipped the question: what do teachers care about? The fastest way is to see "a lesson plan that a teacher like you has already used." A better design: instead of generating a plan from scratch, start from a trusted plan that has been reviewed, is in use at 14 schools, and has run 30+ times this year. Then only show what has changed—"Shortened to 45 min, changed warmup from 2 laps to 1" "Matched your equipment, 4 stations to 3." The teacher doesn't review a full plan but a diff—cognitive load drops dramatically. Eval also becomes more traceable: you only need to verify that the retrieved plan is reasonable and that every modification respects the constraints. Example 3: Workplace Injury Medical Report A product generates a 50-page expert opinion from medical records (MRI, physiotherapy notes, exam reports). The problem: doctors are responsible for the report, so they re-read all the records and check each point—taking as long as writing from scratch. The product provides no value. Hamel suggested turning the product into a research assistant rather than a report generator. First, read each record, extract relevant facts, and attach links for the doctor to verify. Two contradictory findings? Flag them. Missing imaging data leading to uncertainty? Flag it. Once fact-checking is done, a "generate report" button appears—the report is assembled from already-verified facts. This design allows each unit to be scored independently: Is this contradiction real? Does this citation support this claim? This Pattern Generalizes Ask these four questions: What do users actually need to check? What trusted references can they compare against? What signals or heuristics do experts use for verification? What smaller units can be accepted, edited, or rejected? A thread running through all examples is provenance. The fastest way to make output inspectable is to show where each piece came from, with links to details. Use progressive disclosure so these sources don't overwhelm the user. Even for products that seem "easy to evaluate" (e.g., you produce working code), the best products still make work more inspectable—Cursor and Devin both record short UI change videos so you can confirm correctness without replaying everything. None of This Is New These are old design principles—observing experts in their domain is called needfinding, and structured sensemaking in medical diagnosis is sensemaking. But before the AI era, verification typically happened incidentally during product creation. In the AI era, verification is the bottleneck. It's time to think about it more explicitly. Original: Hamel Husain, "'It's Hard to Eval' Is a Product Smell", 2026-06-29 https://hamel.dev/blog/posts/eval-smell/… #AIProduct #Eval #ProductDesign