@AdamRLucek: I'm bullish on agent swarms (aka workflows). Agents are increasingly being used to analyze and collate massive amounts …

I’m bullish on agent swarms (aka workflows). Agents are increasingly being used to analyze and collate massive amounts of unstructured data in repetitive ways (e.g. document extraction, reading emails, parsing logs), but as these tasks and data inputs scale we’ve seen reliable execution decrease, even from the most capable models. Specifically, the consistency of sub agent dispatches from filesystem-based agents drops dramatically when attempting to deploy more than 30+ sub agents in parallel.

So… how can you harness the best of an agent’s intelligent decision making with reliable sub agent task execution at scale? Here’s how 1/5

First, it all starts with breaking down the task into individual units of work that can be run programmatically. The core idea behind this setup is to orchestrate agent execution and context as code, which requires structuring and saving that context into a format like a task table. For example, this could look like an agent using a REPL tool to fetch all unread emails from an inbox and save each as an entry in a JSON file. This organizes the individual tasks in a persistent way that can be iterated through in the next step: 2/5

Once we have the context and tasks organized, creating a swarm is as simple as running a dispatch script against the table to package each row’s instructions + system prompting + additional saved context together and send it all to a single agent. Often these scripts are bundled within skills so the orchestrating agent can do both the task table building and the dispatching itself, automatically! 3/5

Once each individual sub agent completes its run, the results can either be joined back to the main table or acknowledged as completed. We can use structured outputs to do this join automatically via the same dispatch script as results stream in. 4/5

Finally, the output can be navigated and ingested by the orchestrating agent for any downstream follow up! From testing and implementation, we’ve seen these techniques begin to deliver incredibly reliable sub agent execution at scale, where relying on a single LLM’s function calling ability and logic would fail. It will be critical to watch how this evolves as more agent harnesses begin to rely on both code execution and recursive agent runs behind the scenes! 5/5

Simplicity is always key! Very easy to overdo it once code execution and scripts come into play

I use https://github.com/langchain-ai/deepagents… with the quickjs add on and https://github.com/langchain-ai/langchain-skills/tree/main/config/skills/swarm…