Combining multiple team priorities in agent evaluation

I think we’re past the point where CTOs are bragging about their token spend and actually want cost efficient agents, not just flashy chat and automations. I wrote an article in towards data science that addresses how to turn cost, quality and latency requirements based on business concerns into math that drives agent evaluation. Over and over I’ve been thinking about the prompt grind. We build, then tweak, then regress, then optimize, then realize its too expensive, then look for workarounds. AI harnesses make that process doable, but I think there are better ways to speed up the process, especially when its more than you and Claude at the table. Its about building a box that determines an acceptable agent, staying in the box and then optimizing. In a nutshell, for cost, determine: 1) the threshold cost per trace you can accept 2) penalty for crossing threshold 3) weight vs other objectives, which matters more in the box Repeat for quality, latency, turns, etc. combine dimensions using an exponential to create a smooth, convex score. Take the root squared loss to handle both average loss and variance. Evaluate configs (models, prompts, workflows, skills, tools) on a single number that captures real business needs. Enjoy, BUILD and let me know if it resonates.