braintrust

The article discusses methods for detecting new prompt injection patterns in AI systems after launch, including semantic search, trace-level safety scores, and tools like Braintrust, while highlighting challenges with false positives and attack taxonomy.

A team describes how their prompt-injection eval suite caught a regression in a document assistant before shipping, emphasizing the importance of maintaining a strict hierarchy between system instructions and retrieved data.

A developer shares the challenge of creating a gold standard evaluation dataset for an AI product with no users, considering synthetic data generation and adversarial testing to avoid post-launch restructuring.

The team discovered that 80% of RAG hallucinations were caused by poor retrieval, not the generation model, emphasizing the need to evaluate retrieval and generation separately to effectively debug bad answers.

Braintrust's Topics feature uses LLM summarization to make production agent traces tractable for clustering and classification at scale, inspired by Anthropic's Clio approach.

Braintrust uses OpenAI's Codex with GPT-5.5 to turn customer feature requests into preview branches in minutes, enabling real-time iteration and faster feedback loops.

Braintrust CEO Ankur Goyal demonstrates how OpenAI Codex enables his team to generate 100 feature prototypes in the time it previously took to build one, turning Slack requests into customer-testable previews within 10 minutes.