The article investigates whether recurring LLM workloads can be replaced by automatically synthesized deterministic pipelines of typed ML/NLP operators, and asks for feedback on feasibility and approaches.
We are investigating whether recurring LLM workloads can be replaced, where appropriate, by automatically constructed pipelines of regexes, deterministic parsers, traditional ML and NLP models. As an example, suppose an application repeatedly asks a frontier model to read an annual report and return all customer–supplier relationships as structured records containing a customer, supplier, and supporting evidence. A possible replacement pipeline might run named-entity recognition, entity normalization, candidate generation, entity linking, relation extraction, and schema validation. A calibrated uncertainty or out-of-distribution gate would use the pipeline for inputs inside its validated domain and escalate other cases to the original frontier model. NER → entity normalization → candidate generation → entity linking → relation extraction → schema validation Our current action space is a taxonomy of 41 atomic task types spanning classification, token and span labeling, structured extraction, retrieval and entity resolution, similarity, normalization, reshaping, and deterministic computation. The idea is that we would first cluster repeated traces into workload families and induce an end-to-end typed contract for each family. It would then generate candidate DAGs using the 41 task types as building blocks, instantiate each node with an appropriate implementation, and optimize the composition for quality, cost, and latency. Candidate pipelines would be tested on time-separated and group-separated holdouts before being deployed behind abstention and fallback. The problem is quite likely undetermined based on just the input and output contracts alone even if inferred correctly. The intermediate graph is therefore not a recovered latent reasoning trace. It is a synthesized program hypothesized to be behaviorally equivalent over a bounded input distribution. A fixed task taxonomy may help by constraining the search space and supplying type signatures, candidate implementations, and task-specific evaluators. And we are thinking about this problem as a form of program synthesis and formal verification for now, but wondering if this is the right approach and if there is a better way. Looking to speak with people who have worked in this problem space and/or the program synthesis domain for insights. TL;DR: We want to synthesize executable DAGs composed of regexes, deterministic parsers and ML/NLP models from LLM traces for appropriate tasks. Does this seem feasible and what might be some good approaches?
This paper systematically measures behavioral reproducibility of LLM agents in multi-step tool-calling pipelines across 1,140 traces, finding a 'structural consistency, parametric variance' pattern where agents reliably select tools in the same order but vary in arguments, and that structural consistency predicts task success.
This article explores the theoretical and practical challenges of training LLMs to produce typed outputs natively, rather than relying on post-hoc typechecking, with a focus on formally typed languages like Idris, Lean, and Agda. It analyzes current ad-hoc approaches to enforcing types during inference and proposes rebuilding LLMs from the ground up to generate inherently typed outputs.
Argues that LLMs should be used for reasoning within coding-agent workflows, while deterministic infrastructure handles queues, state, retries, and recovery, so the process doesn't break when usage limits hit.
The article discusses the philosophy of combining non-deterministic LLMs with deterministic tools and formal workflows to automate AI development, using Beagle SCM as an example. It suggests letting LLMs automate themselves away in favor of reliable deterministic processes.
This paper introduces AutoTTS, an environment-driven framework that automates the discovery of test-time scaling strategies for LLMs by formulating it as controller synthesis. It demonstrates improved accuracy-cost tradeoffs on mathematical reasoning benchmarks with minimal computational overhead.