What breaks the most when you call LLM APIs in production?
Summary
A discussion of common errors when calling LLM APIs in production, including rate limits, format mismatches, malformed responses, context overflow, model deprecation, and silent failures, with statistics from Datadog and a cited paper.
Similar Articles
After talking to 20+ teams running LLMs in production, 3 pain points kept coming up independently
Based on conversations with over 20 teams, the author identifies three recurring pain points when using LLMs in production: enterprise-only basics, lack of agent observability, and slow support for new models.
Effective use-cases for LLMs
This article shares practical, real-world use cases for LLMs in software engineering, including searching through customer conversations via RAG, triaging API failures from logs, and shortening content. It emphasizes efficiency gains and reducing manual sifting.
10 Ways To Reduce Your LLM API Costs
A practical guide listing 10 strategies to reduce costs when using LLM APIs, including model selection, prompt caching, batch processing, and monitoring expenses.
End of an Agony. Real production service that uses LLM to earn money my team had made and now we are so happy that it will die. Here are some of my final "experiences".
A developer recounts the painful experience of building and eventually shutting down a production LLM-based service for medical appointment scheduling, highlighting issues with model reliability, structured output validation, and provider uptime.
One line system prompt change dropped model quality from 84% to 52%. How are people monitoring semantic quality in production?
A developer shares their experience of a single system prompt change degrading LLM response quality without triggering traditional monitoring alerts, and describes internal tooling they built to monitor semantic quality in production LLM applications.