@undefinedKi: DoorDash just published the full structure behind Ask DoorDash, their new AI assistant. And it's the clearest picture o…

DoorDash just published the full structure behind Ask DoorDash, their new AI assistant. And it's the clearest picture of the AI job nobody advertises Ask DoorDash is the assistant inside their app. You type a question, it answers, finds things and places the order. Millions of conversations so far. Their problem at the start: no way to tell whether a change made it better. Checking quality meant employees writing feedback by hand, around one note a day. A full test round took over six hours, so it almost never ran. So they built something that grades the assistant for them: 1. Write down what a good answer looks like, as explicit checks. 2. Rebuild real sessions from logs, so the grader sees what actually happened. 3. Replay them with simulated users and frozen tool responses, so a rerun measures your change and nothing else. 4. Let a model do the grading, after calibrating it against human labels. Now it grades 2,000 sessions a day, a full test round takes 20 minutes, and error rates dropped nearly by half before the national launch. Nobody at DoorDash has eval engineer on a business card. Someone still writes the rubric and calibrates the judge. That job is arriving at every company that ships a model, food delivery included. Bookmark this