Tag
An experienced developer reflects on the gap between AI agent demos and real-world performance, highlighting issues like poor documentation, naive permission expectations, and the misconception that probabilistic software becomes deterministic in production.
The author, running a small dev team, shares mixed real-world results from using AI coding tools: they speed up boilerplate and onboarding, but produce confident wrong answers on complex problems and increase code review workload, yielding modest net gains far below the often-cited 10x improvement.
When several AI models were priced equally for a week, actual token usage revealed preference differences from leaderboard rankings, showing that coding and general chat have different top models and long context usage concentrated on two trusted models.