@_philschmid: At the @aiDotEngineer World's Fair, I gave a talk on why vibe-checking agent skills breaks in production and how to bui…

X AI KOLs Following Events
agent-skills evals production testing ai-engineer talk conference

Summary

At the AI Engineer World's Fair, Phil Schmid gave a talk on why vibe-checking agent skills break in production and how to build reliable automated evals using negative test cases, skill limits, and ablation tests.

At the @aiDotEngineer World's Fair, I gave a talk on why vibe-checking agent skills breaks in production and how to build reliable automated evals before your users find the bugs. I talked about:  Why vague descriptions cause skill failures and how adding negative test cases prevents keyword hijacking on unrelated prompts  Why skill files over 500 lines degrade model reasoning  How to validate outcomes using simple, millisecond regex assertions across 10 to 20 production prompts.  How to run ablation tests (with vs. without skills) to know when models have caught up and you can retire skills. Check out the full talk and resources below:
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:37 PM

At the @aiDotEngineer World’s Fair, I gave a talk on why vibe-checking agent skills breaks in production and how to build reliable automated evals before your users find the bugs. I talked about:

 Why vague descriptions cause skill failures and how adding negative test cases prevents keyword hijacking on unrelated prompts  Why skill files over 500 lines degrade model reasoning  How to validate outcomes using simple, millisecond regex assertions across 10 to 20 production prompts.  How to run ablation tests (with vs. without skills) to know when models have caught up and you can retire skills.

Check out the full talk and resources below:

Similar Articles

@aiDotEngineer: !!!!

X AI KOLs Following

Paul Bakaus will give two talks on advanced agent skill engineering at the @aiDotEngineer World's Fair next week.