@LangChain: Watch the full conversation:

@clay @jeffbarg Watch the full conversation: https://t.co/JStaVgW9pH

Clay uses AI agents to help users discover, convert, and grow high-quality customers, and leverages LangSmith extensively for agent development and cost monitoring.

Clay’s Positioning as a Growth Hacking Tool

Clay is a growth hacking tool whose core value lies in helping users build various growth solutions within the platform. Specific capabilities include creating lists of companies and contacts, enriching information through traditional data providers or AI-powered web research, and conducting outbound marketing, lead qualification, and lead routing based on those lists.

Evolution of AI in Clay

Clay initially started as a wrapper around conversational completion for summarizing long documents or writing marketing copy. By mid-2023, in the early days of GPT-4, the company observed users spontaneously performing web research and using AI to extract information, which led to the launch of Claygent (an AI web research agent). This development was driven by user behavior.

Current State of the AI Agent System

At its core, Clay focuses on discovering, converting, and growing high-quality customers, with specialized agents for each stage:

• Smart Search Agent: Used for target market filtering, locating ideal customers via the web or databases.
• Account Development Agent: Conducts in-depth research or evaluates company fit.
• Account Follow-Up Agent: Executes outbound communication to drive conversion.

In terms of scale, Clay runs approximately 300 million agent runs per month, with each run averaging 10 to 30 steps, involving operations like web document integration, web search, and page scraping. The proportion of AI-driven operations continues to grow.

Development Process and Quality Control

As the agent system expanded, Clay evolved from a pure software application to a “software + AI agents” model. Development primarily focuses on three dimensions:

1. Agent Run Quality: Users have extremely high expectations for data accuracy, with diverse evaluation criteria.
2. Processing Efficiency: How to execute a large volume of runs efficiently.
3. Cost Control.

Quality Evaluation Methods

Clay employs a hybrid evaluation approach:

• Some assessments are based on deterministic fact verification, such as comparing outputs against a standard answer bank.
• Others use large language models for judgment, suitable for paragraph-style answers or generating recommendations.

Model Selection and Management

Clay follows a model-agnostic strategy, allowing users to choose their preferred model providers. However, different models vary in writing style and task suitability. The company developed a meta-prompt tool that automatically matches the most suitable model based on question characteristics. When new models are released, they are evaluated and added to the mapping relationships within the meta-prompt tool.

Application and Value of LangSmith

LangSmith is primarily used in two major areas at Clay:

Production Environment Monitoring

• Analyzing traffic patterns: Monitoring discrepancies between actual usage and expectations, as well as variations among customers.
• Deep customer-level insights: Resolving mismatches between customer feedback and actual situations, determining whether issues stem from tool malfunctions, reasoning errors, or other parts of the system.
• Since user inputs are in pure natural language and applied across a wide range of scenarios, the production environment involves a high degree of unpredictability.

Agent Development

• Building new agents from scratch (e.g., smart search agents, CRM account agents) with real-time tracking of run trajectories and outputs.
• Integration with Vercel AI SDK requires just a one-line code change to export trace data to LangSmith.
• Tracing is enabled from the first day of development, and receiving streaming trace trajectories is part of the iterative process.

Key Features and Impact

• Cost Observability: LangSmith data matches actual billing records with 99%–99.5% accuracy, providing full coverage of cost expenditures and high satisfaction from the finance team.
• Visualization and Usability: Clearly displays sub-agent nesting relationships or tool call paths, with a platform that is easy for engineers to adopt.
• Data Export: Supports exporting trace data to Snowflake for long-term analysis.
• Performance and Speed: For large-scale users, quickly accessing trace data and ensuring reliable data ingestion are critical.
• Team Usage: Approximately 25 to 50 people use LangSmith, primarily engineers, but it is gradually being opened up to other teams like customer support for debugging purposes.

Intangible benefits include reduced development time, access to tracing capabilities, and traditional observability features.

Future Trends and Recommendations

Clay has observed two significant trends:

1. Long-running agents: Agents that execute more steps and run for longer periods—LangSmith’s thread tracking capability is particularly helpful here.
2. Increased sub-agent usage: The centralized display of all agent trace data in LangSmith is highly practical.

The company is exploring providing production trace data to coding agents so they can identify issues and self-repair, though a fully closed-loop integration has not yet been achieved. Inspired by the Ramp team, Clay is incorporating more coding agents into the software development lifecycle.

For other companies aiming for similar scale, Clay’s advice touches on considerations for large-scale operations (though specific details were not elaborated in the transcription).

Source: YouTube Video (https://www.youtube.com/watch?v=cx6_tb6HCeY)