@LangChain: Point a coding agent at the LangSmith CLI and it can build you an online LLM-as-a-judge: define the rubric, wire it to …
Summary
A tutorial on using a coding agent with LangSmith CLI to set up an online LLM-as-a-judge for evaluating LLM traces, all configurable from the terminal.
View Cached Full Text
Cached at: 09/11/26, 12:25 AM
Point a coding agent at the LangSmith CLI and it can build you an online LLM-as-a-judge: define the rubric, wire it to every incoming trace, adjust the sampling rate, all from your terminal.
Here’s a 5 minute tutorial. https://t.co/m8IBLK4pH2
Similar Articles
@LangChain: Jev-as-a-judge is now available in LangSmith. Score every production trace instead of a sample. Check more criteria per…
LangSmith now integrates Jev, a System One model, for fast and cost-effective online evaluations of agent traces, enabling broader scoring and safety checks without high costs.
@LangChain: Can you trust an LLM judge? Join @divyasan0 Oct 28th for a virtual workshop on: How to score a trajectory with the traj…
LangChain is hosting a virtual workshop on Oct 28th about LLM-as-a-judge reliability, trajectory scoring, and wiring evaluations into development loops to catch regressions before shipping.
@LangChain: You can use the LangSmith CLI to pull your flagged traces, classify each one by issue type, and build an eval dataset w…
This article explains how to use the LangSmith CLI to manage traces and build evaluation datasets through a quick tutorial.
@akshay_pachaar: If you use LLM-as-judge, this one is for you. (bookmark it) Most teams validate their agent's outputs by calling a fron…
Details an approach to train a small LLM judge for evaluating agent outputs, replacing costly frontier models, with a Claude Code plugin for deployment.
@LangChain: Great thread from @AdamRLucek on LangSmith Engine!
Adam Łucek discusses LangSmith Engine, an agent for automating agent development using trace data.