What does it take to get to an agent that catches its own mistake and writes the eval and the fix?

Catch @hwchase17’s Interrupt London session on building agents that improve over time: https://t.co/0l2gxKtdxd https://t.co/J6xe9At0fX

@LangChain: What does it take to get to an agent that catches its own mistake and writes the eval and the fix? Catch @hwchase17’s I…

Channel: @LangChain Source: https://www.youtube.com/watch?v=xy_7SzwsIjM&feature=youtu.be

Description

Harrison Chase, Co-Founder and CEO of LangChain, splits an agent into three parts, the model, the harness, and the context, and explains how each one can improve over time: the model’s weights through supervised fine-tuning with LangSmith Fine-Tuning and its smithtune CLI, or through RL for more complex tasks; the harness through LangSmith Engine, a deep agent that runs over traces, clusters issues, and proposes fixes with a human in the loop; and the context through memory, where he points to Hex and Profound as companies he thinks do this well and to LangChain’s open source OpenWiki project. He argues that traces and evals are the common foundation of all three loops, which is why LangChain built SmithDB and the new LangSmith AI Search for trace data, then covers what makes evals hard (realistic inputs, defining what good looks like, simulation environments) and how red teaming in LangSmith Engine looks for agent mistakes proactively. He closes with what he calls a North Star vision: an agent that notices its own mistakes, creates its own eval data points, and suggests its own fixes.

This session was captured at Interrupt London, the agent conference by LangChain.

Chapters: 0:00 What is an agent, and how do you improve it? 3:22 When should you fine-tune the model behind your agent? 5:52 How LangSmith Engine proposes fixes to your harness 7:13 Why context is the easiest part of an agent to improve 8:42 How OpenWiki gives coding agents a brain that updates 9:11 Why traces and evals are the foundation of improvement 10:44 Why LangChain built SmithDB and AI Search for traces 12:02 Why evals are the new PRD 12:50 Why is building evals so hard? 14:57 How to run evals without hitting production systems 16:01 How red teaming finds agent mistakes ahead of time 16:54 How do you judge correctness without a universal judge? 18:54 The flywheel that improves your agent over time 19:28 What if the agent ran its own improvement loop?

Resources: → LangSmith Engine: https://www.langchain.com/langsmith/engine → LangSmith Fine-Tuning (smithtune): https://docs.langchain.com/langsmith/smithtune → OpenWiki on GitHub: https://github.com/langchain-ai/openwiki → LangSmith: https://www.langchain.com/langsmith → Interrupt, the agent conference by LangChain: https://interrupt.langchain.com