RecSys Factory: Bounding LLM Agent Autonomy to Decision Points in the Industrial Recommender Lifecycle

# RecSys Factory: Bounding LLM Agent Autonomy to Decision Points in the Industrial Recommender Lifecycle
Source: [https://arxiv.org/html/2608.11241](https://arxiv.org/html/2608.11241)
###### Abstract\.

Deploying LLM agents into industrial recommender operations exposes a three\-way tension we frame as the autonomy–determinism–efficiency trilemma: general autonomy \(interpreting operator intent, generating glue code zero\-shot\), industrial determinism \(schema\-conforming feature extraction, non\-crashing A/B, zero compliance\-path hallucination\), and end\-to\-end efficiency \(an author\-recalled new\-business\-line onboarding compression on two lines that we report as a case study rather than a generalization claim — see §6\.3; controlled measurement pending v1\.1\)\. Any two can be maximized against the third; the failure modes at each edge are documented in concurrent industrial\-agent work\. We presentRecSys Factory, an LLM\-agent platform deployed for 78 days across three heterogeneous Tencent recommender business lines\. The design principle is*autonomy at decision points, not over pipelines*, made concrete through three deconstructions that each discharge one vertex of the trilemma\.Runtimeis deconstructed into three host\-emitted event sources \(Claude CodeStophooks, the corporate\-IM webhooks, the workflow scheduler APIs\), giving determinism: the platform carries no long\-running daemon*during the wait phase*\(a transient LangGraph process exists during the≈\\approx6% of wall\-clock spent on agent\-side reasoning\) and consumes zero CPU during the 94% of wall\-clock spent waiting on Spark or GPU jobs\.Capabilityis deconstructed into a 29\-file skill ecosystem \(8 971 lines ofSKILL\.md\) whose per\-skill pitfall tables mechanically compile into a 400\-entryPitfallStore, confining autonomy to bounded typed decision surfaces inside pre\-committed pipelines\.Deploymentis deconstructed across three business lines with disjoint label semantics, A/B layer topologies, and operator personas; the onboarding compression is observed on two of the three \(Business A recommendation and Business C growth\-marketing\) as a case\-study observation — not a generalization claim, and not measured against a controlled pre\-platform baseline \(§6\.3 flags this explicitly\)\. The human is retained at the diagnostic\-vs\-execution boundary via a corporate\-IM human\-in\-the\-loop card protocol — deployed as an audit\-trail primitive \(schema\-validated, idempotent, replayable\) rather than a full HCI contribution, and reported here from an 8\-day 16\-run pilot with production rollout scheduled for v1\.1\. Across the 78\-day window the platform recorded 1 624 CLI\-tool dispatches \(78\.6% aggregate success rate countingWAITINGoutcomes as non\-success;≈\\approx83\.7% ifWAITINGis treated as a correct signal, i\.e\., end\-to\-end platform\-error rate≈\\approx16\.3%\); a companion paper \(AutoResearch, P3b\) instantiates the same substrate for autonomous research and produced the IOSkip CIKM 2026 submission through a 196\-round campaign on this platform\.

industrial recommender systems, LLM agents, ChatOps, autonomous research, human\-in\-the\-loop, skill ecosystem, memory sedimentation

††copyright:none## 1\.Introduction

A modern industrial recommender platform is not a single model but a portfolio of pipelines that span sample generation, feature engineering, model training, evaluation, A/B testing, and online serving\. In a typical Tencent business unit, three or more recommender business lines \(telecom payment recommendation, wealth\-management product CVR, credit\-card scenario decisioning\) run side by side, each with its own labels, feature schema, A/B layer topology, and operator persona\. Algorithm engineers, business operators, and data analysts all need to drive the same underlying machinery, but at different abstraction levels — engineers debug feature\-store joins, operators tune item weights, analysts attribute A/B effects across layers\. The cognitive load is not where most papers focus \(model architecture\); it is in the*operational substrate*— knowing which weight to nudge, which sample table to refresh, which causal\-attribution layer to inspect when a metric moves\.

LLM agents promise to absorb this operational load\. Deploying them into industrial recommender operations, however, exposes a three\-way tension we frame as theautonomy–determinism–efficiency trilemma\.*General autonomy*— the model’s ability to interpret unstructured operator intent, decompose a novel request into steps, and generate glue code zero\-shot — is the reason to invoke an LLM at all; without it a bash script suffices\.*Industrial determinism*— schema\-conforming feature extraction, non\-crashing A/B rollouts, revenue\-floor protection, and zero hallucination on the compliance path — is what production actually requires; a 42%\-failure agentic pipeline\(Vintschger et al\.,[2025](https://arxiv.org/html/2608.11241#bib.bib18)\)is not deployable regardless of how novel its reasoning is\.*End\-to\-end efficiency*— compressing the new\-business\-line baseline onboarding from a≈\\approx14\-day glue\-code sprint to≈\\approx3 days of skill\-authored setup on Business A and Business C — is what justifies the LLM overhead against a hand\-crafted alternative\. We report this as an author\-recalled case\-study observation, not a generalization claim: no controlled pre\-platform baseline was tracked, and §6\.3 flags the same limitation for Business C\.

Any two vertices can be maximized against the third under a single forcing resource: theper\-business\-line engineering plus human\-oversight budget, which at Tencent scale runs≈\\approx3–6 person\-months to onboard a new recommender business line and 40–80 review\-hours per month to keep the resulting pipelines running\. A dollar spent hardening determinism \(schema validation, sandboxes, HITL gates, audit trails\) is a dollar not spent authoring skills \(autonomy\) or writing one\-off glue code \(efficiency\); every skill added to broaden autonomy is a skill that must be maintained, regression\-tested, and eventually reviewed by an engineer\. The three vertices therefore compete for the same budget rather than acting as independent knobs, and each edge of the triangle names the failure mode a system falls into when the budget forced it to abandon the opposite vertex\. Trading determinism for autonomy plus efficiency is the failure mode of maximally agentic systems: AI Scientist v2’s free\-form ReAct planner posts a 42% divergence rate on the MLE\-Bench held\-out set\(Vintschger et al\.,[2025](https://arxiv.org/html/2608.11241#bib.bib18)\)\. Trading autonomy for determinism plus efficiency is the failure mode of classic workflow\-fixed platforms like EasyRec\(Cheng et al\.,[2023](https://arxiv.org/html/2608.11241#bib.bib5)\)or Monolith\(Liu et al\.,[2022](https://arxiv.org/html/2608.11241#bib.bib14)\)— extremely stable and fast, but schema drift or long\-tail intent immediately re\-engages a human engineer\. Trading efficiency for autonomy plus determinism is the LLM\-Copilot failure mode: Devin\(Cognition AI,[2024](https://arxiv.org/html/2608.11241#bib.bib6)\), OpenHands\(Wang et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib19)\), or a review\-every\-line assistant will produce correct code, but the multi\-week paper\-to\-model tax is untouched because every generated line still requires manual review before it enters production\.

Concurrent industrial\-agent work responds to the same tension along different axes\. Kuaishou’s AgentX\(Kuaishou Team,[2026](https://arxiv.org/html/2608.11241#bib.bib10)\)keeps the agent as a self\-evolving loop centred on one product surface; Tencent’s NOVA\(Liu et al\.,[2026](https://arxiv.org/html/2608.11241#bib.bib13)\)attacks the determinism vertex with a verification\-aware harness but does not treat efficiency as an independent variable\.RecSys Factory takes a different design stance: instead of maximizing a single vertex, we commit to a specific interior compromise point and instrument the platform to make the compromise legible\.Figure[1\.1](https://arxiv.org/html/2608.11241#S1.F1)sketches the three vertices, the failure mode of pinning any single edge, and our compromise point\.

![Refer to caption](https://arxiv.org/html/2608.11241v1/figures/fig-p3a-1-1-trilemma.png)Figure 1\.1\.The*autonomy–determinism–efficiency trilemma*for industrial LLM agents\. Each vertex names a design goal; each edge names the failure mode of pinning that edge \(equivalently, of trading away the opposite vertex\)\. The three vertices compete under a single forcing resource — the per\-business\-line engineering plus human\-oversight budget \(≈\\approx3–6 person\-months per new line at Tencent scale, plus 40–80 review\-hours per month\) — which is what makes them pairwise conflicting rather than three independent knobs\. RecSys Factory commits to an interior compromise point rather than maximizing any single vertex, and instruments the platform to make the compromise legible; the interior\-compromise principle collapses the trilemma into three concrete engineering commitments described in §3–§6\.Design principle: autonomy at decision points, not over pipelines\.The compromise becomes concrete through three*deconstructions*of the monolithic\-agent stance, and each deconstruction discharges one vertex of the trilemma\.*Runtime is deconstructed*into host\-emitted event sources \(enumerated below\), which fixes the pipeline shape and gives usdeterminism: the agent has no long\-running process to supervise*during the wait phase*\(only a transient LangGraph process during≈\\approx6% agent\-reasoning wall\-clock\) and inherits the host’s own uptime\.*Capability is deconstructed*into a 29\-fileSKILL\.mdecosystem, each skill presenting the LLM with a small typed decision surface \(which sample table to refresh, which feature dimension to compress, which override to propose\) rather than a general reasoning problem — this is where we spend ourautonomybudget, at pre\-committed decision points inside pre\-committed pipelines\.*Deployment is deconstructed*across three heterogeneous business lines, stress\-testing whether the compromise preservesefficiencyacross recommendation, decision\-support, and growth\-marketing operational regimes rather than just one flagship product\.

Around this compromise we adopt four quality\-defence practices — schema validation at dispatch, sandboxed unit tests in the training subgraph, a working A/B≥\\geqbaseline⋅\\cdot80% threshold, and engineer\-in\-the\-loop code review at the sensitive boundary — as team conventions the platform makes cheap enough to actually follow, not as production hard gates\. The human is retained at the diagnostic\-vs\-execution boundary \(§4’s IM HITL card protocol\): the Analyzer proposes an override, the operator decides\. Removing that boundary erases the compromise; over\-weighting it collapses efficiency\.

Contributions\.Each contribution defends one vertex of the trilemma with a concrete engineering artifact and observable evidence\.

1. 1\.Lifecycle\-Aware Agent Framework \(§3\) — defending determinism\.Three design principles turn the recommender lifecycle into first\-class agent abstractions: \(a\)*host\-event coupling*— the agent attaches toStophooks, the corporate\-IM webhooks, and the workflow scheduler APIs, firing on host lifecycle events and sleeping otherwise; \(b\)*single\-source state*— onePipelineStateper run, persisted as inspectable JSON in SQLite, is the single truth across sessions; \(c\)*skill subgraphs*— domain knowledge is packaged as LangGraph\-dispatchable subgraphs, not retrieval passages\. Parasitic execution means*no long\-running daemon during the wait phase*\(a transient LangGraph process runs during the≈\\approx6% of wall\-clock devoted to agent\-side reasoning; the remaining 94% is actual zero\-CPU wait\)\. The platform ran 1 624 CLI\-tool dispatches over 78 days without a scheduler process\.
2. 2\.29\-Skill Ecosystem as Working Memory \(§5\) — defending bounded autonomy\.29 explicit skills totaling 8 971 lines of curatedSKILL\.mddocumentation, each a directory with a procedural body, bound script wrappers, and a machine\-readable pitfall/problem/trap table\. A rule\-based extractor compiles the tables into a 400\-entryPitfallStore\(SQLite\), which the platform’s Plan node consumes at every planning step\. the skill library becomes*executable working memory*rather than a prompt library — every operator gesture \(“re\-run business\-A yesterday’s training”\) deterministically dispatches a single skill subgraph, and every documentation contribution \(skill or changelog\) mechanically enters the agent’s runtime context at zero marginal effort\. A business\-specific skill onboarded post\-launch \(domain\-decision\-c, business C\) added 12 pitfall entries that the extractor absorbed without any code change\.
3. 3\.Three\-Business\-Line Deployment Case Studies \(§6\) — reporting efficiency evidence in situ\.We report onboarding evidence from three heterogeneous business lines as case studies rather than as a generalization claim\. Author\-recalled onboarding time compressed from≈\\approx14 days to≈\\approx3 days on two of the three lines \(recommendation product Business A and growth\-marketing product Business C\); no controlled pre\-platform baseline was tracked, so this is a directional observation not a measurement\. Business A \(telecom personalization\) reports \+10–31 % CPM lift across regional cohorts of one carrier plus \+14–45 % expected lift from rule\-fallback identification on 9 sub\-cohorts of a second carrier\. Business B \(telecom reranking decision support\) reports a bootstrap 95 % CI on the top\-3 subset weight\-tuning recommendation adopted by operators \(P\(Δ\\Delta\> 0\) = 100 %\)\. Business C \(wealth\-management new\-customer CVR audience\-set orchestration\) detected three simultaneous upstream data\-integrity issues in a single pre\-flight session that would have silently corrupted training\. three heterogeneous business lines under one platform demonstrate that the trilemma compromise survives the operational realities of*different*label semantics, A/B layer topologies, and operator personas\.
4. 4\.Silent\-drift detection at production seams \(§6\.2, §6\.3\)\.A methodological byproduct of embedding a ChatOps agent inside real business\-line workflows is that the agent*observes*points of silent divergence between what a configuration table says and what serving\-time infrastructure does\. In Business B we surface a “configured weight≠\\neqeffective weight” diff \(§6\.2 C3\) by reverse\-engineering the effective weight from\(line\_score / raw\_pctr\)ratios in the daily snapshot — exposing one item whose effective weight was 67×\\timeslower than the operator\-visible configured value\. In Business C \(§6\.3\) the pre\-flight upstream\-integrity check surfaces three simultaneous data\-integrity failures in a single onboarding session\. These are not features we designed for; they are what a platform earns when its agent is close enough to production seams to see them\. Documenting the pattern is itself a contribution\.

Human\-in\-the\-loop as audit\-trail primitive, not fallback\.The corporate\-IM card protocol described in §4 is not a graceful\-degradation channel: it is the explicit locus at which the Analyzer’s diagnosis and drafted override are recorded before an operator judgment is applied\. §4’s protocol contributes an*engineering*commitment \(schema\-validated, idempotent, replayable\) rather than an HCI evaluation; the v1\.0 statistics reported here are from an 8\-day 16\-run test\-user pilot \(§4\.4\), and full production\-operator HITL rollout is scheduled for v1\.1\. §7 documents the two failure modes that motivated retaining this boundary: \(i\) LLM diagnoses that were locally coherent but violated business\-line guardrails invisible to the agent, and \(ii\) operator decisions whose rationale \(upcoming campaign, regulatory freeze\) was never encoded in any skill or pitfall\.

Companion paper\.AutoResearch \(P3b\) instantiates the same lifecycle\-aware framework for autonomous research, contributing surprise\-weighted memory retrieval, two\-tier knowledge sedimentation, and a 5\-mode memory ablation\. The two papers share §3’s framework description and §2’s related\-work survey\.

## 2\.Related Work

We position RecSys Factory against four lines of work: industrial RecSys frameworks, general\-purpose LLM agent platforms, enterprise ChatOps and multi\-agent systems, and concurrent industrial RecSys agents\. P3b \(AutoResearch\) discusses the parallel literature on autonomous research agents and agent memory architectures — we cross\-reference rather than duplicate that section\.

### 2\.1\.Industrial Recommender Frameworks \(no agent layer\)

A first cluster establishes the*modeling and infrastructure*substrate that recommender platforms run on, but does not address agent\-level orchestration: EasyRec\(Cheng et al\.,[2023](https://arxiv.org/html/2608.11241#bib.bib5)\)\(modular TF \+ model zoo on Alibaba PAI\), Monolith\(Liu et al\.,[2022](https://arxiv.org/html/2608.11241#bib.bib14)\)\(collisionless cuckoo\-hash embeddings\), Persia\(Lian et al\.,[2022](https://arxiv.org/html/2608.11241#bib.bib11)\)\(hybrid sparse\-async/sync\-dense at 100T parameters\), Angel\(Jiang et al\.,[2018](https://arxiv.org/html/2608.11241#bib.bib9)\)\(parameter server\), LiRank\(LinkedIn Ranking Team,[2024](https://arxiv.org/html/2608.11241#bib.bib12)\)\(unified ranking stack\), and TransAct V2/PinFM/PinRec\(Pinterest Ranking Team,[2025](https://arxiv.org/html/2608.11241#bib.bib16)\)\(transformer ranking\)\. All report A/B\-tested deployment but each sits*under*the agent layer — they are the engines our agent dispatches against, not competitors at the orchestration tier\.

### 2\.2\.LLM Agent Platforms \(general\-purpose\)

A second cluster builds*general\-purpose*LLM agent infrastructure without recommender\-specific specialization: LangGraph and AutoGen\(Wu et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib20)\)provide state\-machine and multi\-agent orchestration but ship without domain primitives; Devin\(Cognition AI,[2024](https://arxiv.org/html/2608.11241#bib.bib6)\), OpenHands\(Wang et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib19)\), and SWE\-Agent\(Yang et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib22)\)target software engineering in sandboxed environments — precisely the architectural choice we depart from when we couple to host\-system events; MetaGPT\(Hong et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib7)\)and ChatDev\(Qian et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib17)\)coordinate role\-specialized agent teams for software\-development tasks\. None target the recommender lifecycle or expose a chat interface for non\-engineer operators\.

### 2\.3\.Enterprise ChatOps and Multi\-Agent Platforms

A third cluster builds general enterprise multi\-agent platforms with chat or DAG interfaces\. JoyAgent\-JDGenie\(JD JoyAgent Team,[2025](https://arxiv.org/html/2608.11241#bib.bib8)\)is the closest enterprise comparator \(open\-sourced multi\-agent ChatOps reportedly serving 20K\+ internal agents at JD\), but treats recommendation as a generic “task” with no domain primitives \(no CTR/CVR labels, no feature\-store schemas, no A/B layer topology\)\. Academic ChatOps work — ChatOps4Msa\(ChatOps4Msa Authors,[2024](https://arxiv.org/html/2608.11241#bib.bib4)\), IMPROVE\(Xue et al\.,[2025](https://arxiv.org/html/2608.11241#bib.bib21)\)— proposes NL\-fronted DAG execution but evaluates on synthetic microservices, not recommender pipelines\. RecSys Factory is, to our knowledge, the first published agent platform that combines ChatOps NL access, recommender\-specific skill subgraphs, and multi\-business\-line industrial deployment\.

### 2\.4\.LLM\-as\-Recommender vs LLM\-as\-MLOps\-Agent \(disambiguation\)

A line sometimes confused with our setting uses LLMs as the reasoning brain*inside the recommender product itself*: InteRecAgent\(Microsoft AI4Science Team,[2023](https://arxiv.org/html/2608.11241#bib.bib15)\)and RecMind\(Amazon,[2024](https://arxiv.org/html/2608.11241#bib.bib2)\)expose LLM\-based conversational recommenders to end users with traditional recommenders as tools\. Those systems live inside the product surface \(LLM = user\-facing recommender\)\. RecSys Factory operates one layer above: LLM = engineering/operations agent that drives the modeling pipeline\. Complementary rather than comparable\.

### 2\.5\.Concurrent Industrial Work

Two closely\-related industrial\-agent papers arrived on arXiv shortly before this submission\.AgentX\(Kuaishou Team,[2026](https://arxiv.org/html/2608.11241#bib.bib10)\)\(Kuaishou, 2026\-06\-26\) is a 4\-stage closed\-loop production agent for Kuaishou App feature engineering, with a Monitoring Platform for engineers and a runtime playbook accumulated online\.NOVA\(Liu et al\.,[2026](https://arxiv.org/html/2608.11241#bib.bib13)\)\(Tencent, 2026\-06\-29\) is a verification\-aware harness for architecture evolution on a billion\-user ad ranking backbone, with L1–L4 task\-complexity levels and an AutoRun/Copilot mode\. Both target production ranking systems at team\-scale infrastructure with vertical depth on one product surface\. Table[2\.1](https://arxiv.org/html/2608.11241#S2.T1)compares the three systems along five design axes to operationalize the residual differentiation\.

Table 2\.1\.Five\-axis comparison of concurrent industrial agent systems\.When each design wins\.A team onboarding an additional recommender business line whose failure mechanisms overlap existing ones benefits most from RecSys Factory: the day\-0 PitfallStore short\-circuits the “blunder through first 50 experiments” phase that both concurrent systems rely on team curation to avoid\. A team maximizing offline AUC on a mature ad\-ranking backbone benefits most from NOVA’s architecture\-gradient formalism \(modular search over discrete architectural moves is more efficient than skill\-dispatch when depth on one system is the goal\)\. A team pushing candidate\-scoring quality under a shared 24/7 monitoring surface benefits most from AgentX’s SGPO harness \(closed\-loop online playbook accumulation is more expressive than our static PitfallStore when harness self\-evolution is primary\)\. The three systems compose rather than compete; a systematic head\-to\-head is v2 work\.

## 3\.Lifecycle\-Aware Agent Framework

This section describes the framework that grounds theSkill Ecosystem\(§5\) and theThree Business Lines deployment\(§6\)\. It is*the*paper’s main architectural contribution and is shared in spirit — though not in detail — with our companion paper AutoResearch \(P3b §3, which describes the research\-side instantiation of the same framework\)\.

### 3\.1\.Three\-Tier Architecture

RecSys Factory is decomposed into three loosely\-coupled tiers that share a single state object and communicate only through it \(Figure[3\.1](https://arxiv.org/html/2608.11241#S3.F1)\):

- •User Tier— Corporate\-IM \(the corporate IM\) ChatOps surface\. Operators \(campaign managers, A/B reviewers\) and engineers \(algorithm, platform\) submit natural\-language requests through chat; the agent replies with structured human\-in\-the\-loop \(HITL\) cards that carry an audit trail of which user approved which action at which time\.
- •Agent Tier— A LangGraph\-based stateful DAG that decomposes a request into a sequence of skill invocations \(sample construction, feature engineering, model training, evaluation, A/B attribution, decision support\)\. Each node is a pure Python function over an immutablePipelineState\.
- •Infrastructure Tier—*Borrowed*, not built: an internal GPU scheduler workflow engine for SQL / Spark / GPU scheduling, distributed SQL / Spark for sample tables, distributed FS / the shared storage tier for sample data, the online\-serving platform for online serving\. The agent’s only writes are JSON\-RPCcreate\_task/start\_task/ the workflow API /resource uploadcalls; subsequent reads are pure polling\.

The architecture’s central design commitment is thatno tier owns a long\-running process\. The agent tier is invoked by lifecycle events \(IM webhook, HITL card callback, training\-completion sentinel\); the infrastructure tier is the host platform’s own scheduler; the user tier is the end\-user’s own IM client\. There is no agent daemon, no cron, no message queue we own\. This commitment — “lifecycle\-coupled execution” — is what lets the platform survive month\-long deployment windows without operator supervision\.

![Refer to caption](https://arxiv.org/html/2608.11241v1/figures/fig-p3a-3-1-three-tier-architecture.png)Figure 3\.1\.*Three\-tier architecture with lifecycle\-coupled execution\.*The User Tier surfaces Corporate\-IM chat \+ HITL card audit trail; the Agent Tier is a LangGraph DAG over an immutablePipelineState, with a Knowledge Tier \(SQLite\) that the Analyzer LLM queries by tag at diagnosis time; the Infrastructure Tier is borrowed \(distributed SQL / Spark / the shared storage tier / the GPU scheduler / the online\-serving platform\)\. Dashed red arrows mark the failure path \(Train→\\toAnalyzer→\\toHITL card→\\toRouter\)\. Note\. No tier owns a long\-running process; agent invocation is event\-driven\.
### 3\.2\.Design Principle I — Host\-Event Coupling

Conventional agent platforms \(LangChain, AutoGen, MetaGPT\) run the agent loop inside a continuously\-executing Python process\. Sandboxed execution platforms \(Devin, OpenHands\) run inside a long\-lived Docker container\. Both choices require dedicated infrastructure \(a process or a container\) for the agent itself\. We insteadcouple the agent to host\-system lifecycle events:

- •An IM webhook arriving atPOST /webhook/imtriggers a Router LLM call followed by a LangGraph DAG dispatch\. When the DAG completes \(success or HITL pause\), the process exits\.
- •A HITL card callback arriving atPOST /callback/im/cardresumes a paused DAG by readingPipelineStatefrom SQLite, applying the operator’s approval, and continuing from the paused node\.
- •A training\-completion sentinel file \(written by the GPU scheduler at job completion\) triggers the next post\-training step via a Stop hook in the operator’s CLI\. \(This sentinel mechanism is also AutoResearch’s only out\-of\-band signal; see P3b §6\.\)

This pattern —parasitic coupling, in the sense of consuming the host’s existing event grid rather than running our own — is one of the key reasons RecSys Factory deploys with operational footprint comparable to a cron job rather than to a microservice\.

### 3\.3\.Design Principle II — Single\-Source State Across Sessions

A request that crosses a HITL gate or a long\-running training job*must*survive process exit\. We use a single Pydantic state object \(PipelineState\) persisted as JSON in SQLite viaaiosqlite\. The state carries: the request’s lifecycle stage, all task handles \(the GPU schedulertask\_flag,instance\_id,job\_id,exec\_id,yarn\_app\_id\), allyaml\_overridesaccumulated through HITL approvals, thehuman\_approvalfield that gates retry\-vs\-end at the Analyzer node, anddry\_run/dry\_run\_scenarioflags that switch the executor to a mock registry for offline testing\.

Updates are immutable — every node returnsstate\.model\_copy\(update=\{\.\.\.\}\)— which means a re\-execution of any node from the persisted state is well\-defined\. This in turn means the agent isidempotent at the node granularity: a Webhook redelivery, a CLI restart, or a HITL callback retry never corrupts in\-flight work\. The same property is what makesdry\_runmode \(used bycli\.py run\) trustworthy as a regression test of the DAG itself\.

### 3\.4\.Design Principle III — Skill Subgraphs as Composability Unit

Where AgentX organizes domain knowledge as four queryable evidence sources blended via attention\-styleα\\alpha\-weighting into the prompt context, RecSys Factory organizes domain knowledge as 29 explicitlydispatchable skill subgraphs\. A skill is a triple of \(a\) aSKILL\.mddeclaring inputs, outputs, side\-effects, and validation predicates; \(b\) a set of bound tool calls that implement the skill \(SQL templates, fid\-xgb config templates, platform\_cli wrappers\); \(c\) optionally, sub\-skills referenced by name for chained execution\.

This distinction matters because operational recommender knowledge often has a*procedural*shape — “to attribute an A/B effect across module\-codes, run this 4\-step procedure” — that is more naturally expressed as a subgraph than as retrievable text\. AgentX’sα\\alpha\-weighted retrieval mixture is appropriate when knowledge is fluid context; our dispatchable subgraph is appropriate when knowledge is structured procedure\. The two paradigms answer different questions: AgentX asks*what context to retrieve*, we ask*what subgraph to invoke*\. We discuss the design’s empirical trace in §5 \(skill\-call statistics over the 10\-week window, including hit rate, average chain length, and cross\-skill composition\)\.

The subgraph formulation also gives us a research\-autopilotas a special case: the AutoResearch P2M subgraph \(P3b §3\) is itself a skill registered in the same ecosystem, withpaper\_to\_modelas its dispatch key\. From the platform’s perspective, autonomous research is one more business line\.

### 3\.5\.Recovery Semantics and Adversarial Failure Modes

The framework’s “zero long\-running daemon” promise \(§3\.2\) and “single\-source state” promise \(§3\.3\) invite a natural systems\-review question: what actually happens when the assumed invariants meet the adversarial reality of distributed infrastructure? We enumerate the concrete recovery\-semantics contract v1\.0 provides, and explicitly name the failure modes for which no formal guarantee currently holds\.

Task submission \(idempotency\)\.create\_taskreceives a client\-side deterministictask\_flagslug derived from\(user, stage, run\_id, dispatch\_seq\); on a redelivery storm the workflow scheduler API rejects duplicate flags rather than minting a second task, giving effective idempotency at the create step\. However thestart\_taskcall that returns theinstance\_idis itself*at\-least\-once*: a network retry after a successful launch would in principle produce two identical training runs\. In v1\.0 we detect this via alast\_instance\_idfield inPipelineStateand abort if a start\-response arrives with a mismatching instance ID; a proper exactly\-once contract via server\-side dedup is not in v1\.0\.

State journaling\.PipelineStateis persisted to SQLite withPRAGMA journal\_mode = WALandPRAGMA synchronous = NORMAL; every state transition is committed inside a single transaction\.state\.jsonsnapshots \(used by AutoResearch’s Stop\-hook loop\) are written to a temp file and renamed viaos\.replace— POSIX\-atomic on the same filesystem, but we do not enforce that both files live on the same volume in v1\.0\. A torn write across mounts is possible; the loop detects it by verifying a SHA\-256 checksum on read and refusing to resume with an unclear message rather than continuing with a partial state\.

Callback and dual\-account concurrency\.The corporate\-IM card callback endpoint carries acard\_id\(payload\-hash\-deterministic\) and aballot\_state\(approve / reject / custom\); the store rejects a second callback for the same\(card\_id, ballot\_state\)tuple, so retries are safe\. Split approvals across two channels are deduplicated at the endpoint but not ordering\-guaranteed — the audit trail records both events; we do not guarantee monotonic single\-operator intent\. Two concurrent Stop hooks under the P3b §6\.4 dual\-account pattern serialize optimistically via a compare\-and\-swap against\(schema\_version, round\_counter\); the loser retries after re\-reading, and a hook crashing between plan and write recovers on the next fire\. No cross\-process lock; livelock\-free empirically over 196 rounds but not formally proven\.

Where guarantees do not hold\.We explicitly do*not*claim: \(i\) exactly\-once delivery of training\-completion sentinels across scheduler outages spanning\>\>5 minutes \(we depend on the scheduler’s own at\-least\-once semantics and treat duplicates as idempotent no\-ops whenstate\.jsonshows the round already committed\); \(ii\) partition\-tolerant serializability of the two\-tier memory reads \(a rare cross\-project write during a Plan\-time retrieval is served stale, then reconciled on the next round\); \(iii\) formal livelock\-freedom of the dual\-account CAS\. Where these matter for downstream users, we recommend running under a single\-account configuration until v1\.1 tightens the invariants\.

Substrate mapping across the companion pair\.RecSys Factory’s ChatOps DAG \(§3\.6 below\) and AutoResearch’s P2M subgraph \(P3b §3\) share this framework but consume different combinations of the host\-emitted event sources\. Table[3\.1](https://arxiv.org/html/2608.11241#S3.T1)enumerates the assignment; the operator\-triggered ChatOps DAG is a “poll\-then\-diagnose” pattern because it is invoked by an external IM message and then polls the scheduler at each stage, while the autonomous P2M loop is a “Stop\-hook parasitism” pattern because it self\-triggers via the Claude Code Stop hook and polls the scheduler only inside a single round\. The two patterns share the same event grid; they differ in which vertex of the grid supplies the round\-boundary trigger\.

Table 3\.1\.Substrate mapping for the two DAGs served by the same framework\. The ChatOps DAG \(P3a §3\.6\) is externally triggered by operator IM messages; the P2M subgraph \(P3b §3\) is self\-triggered byStophooks\. Both share the workflow\-scheduler status poll but differ in what supplies the round\-boundary trigger\.
### 3\.6\.The ChatOps DAG

The agent tier’s main DAG realizes the standard recommender pipeline with HITL fallback \(see Figure[3\.1](https://arxiv.org/html/2608.11241#S3.F1)for the tier topology\)\. Sensor polls the GPU scheduler for upstream job completion\. Sample\-Labeling and Feature\-Join dispatch SQL templates to a general\-purpose Spark workflow via the workflow API\. Training and Evaluation submit JSON\-RPC tasks via the workflow scheduler API\. On any node failure, control transfers toAnalyzer— the LLM\-driven diagnosis node — which fetches a two\-layer log \(the scheduler\-log service for scheduler diagnostics, YARN for container diagnostics\) and emits a structureddiagnosisplus a candidatesuggested\_overridespayload\. The diagnosis is rendered as a HITL card; the operator’sapprovedecision merges the overrides intoyaml\_overridesand re\-routes the DAG back to Training, whilerejectends the run with a recorded failure record\.

We detail the HITL contract — request format, card schema, callback semantics, and the 10\-week \+ 8\-day\-pilot statistics on operator approval rate and Analyzer diagnostic accuracy — in §4\.

### 3\.7\.Relation to Existing Agent Platforms

Compared withLangGraphalone \(which is the orchestrator we use\), RecSys Factory adds a domain\-specific node library, a HITL card protocol, persistent state across sessions, and the host\-event coupling pattern of §3\.2\. Compared withAutoGen\(Wu et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib20)\)andMetaGPT\(Hong et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib7)\), we share the multi\-step orchestration pattern but discard the role\-specialized agent metaphor — our nodes are pipeline stages, not personas\. Compared withDevin\(Cognition AI,[2024](https://arxiv.org/html/2608.11241#bib.bib6)\)/OpenHands\(Wang et al\.,[2024](https://arxiv.org/html/2608.11241#bib.bib19)\), we discard the sandboxed\-Docker abstraction in favor of explicit infrastructure\-tier coupling — the recommender lifecycle*is*the substrate we couple to, not something to be reproduced inside a sandbox\.

## 4\.ChatOps DAG with Human\-in\-the\-Loop Cards

§3\.5 sketched the LangGraph DAG that orchestrates a training request; this section specifies therequest contract, theHITL card protocol, theAnalyzer diagnosis node, and the 10\-week operational statistics plus the 8\-day HITL DAG pilot statistics that quantify each of these\. We consider §4 the*operator\-facing*face of the framework: everything a non\-engineer campaign manager or A/B reviewer perceives lives in this section, which is why we treat it as first\-class rather than as an appendix to §3\.

![Refer to caption](https://arxiv.org/html/2608.11241v1/figures/fig-p3a-4-1-im-hitl-sequence.png)Figure 4\.1\.*End\-to\-end the corporate IM\-to\-HITL\-card lifecycle\.*An operator’s natural\-language request enters through the IM webhook, the Router LLM dispatches a LangGraph DAG that submits the GPU scheduler tasks, and on failure the Analyzer LLM produces a structured diagnosis fetched againstPitfallStorebefore emitting a HITL card\. The operator’sapprovemerges suggested overrides intoPipelineStateand resumes the DAG at Training\. The highlighted band \(steps 10–17\) is the Analyzer \+ HITL loop that separates diagnosis from execution\. Note\. Numbering is autonumbered by sequence order, not by DAG topology\.### 4\.1\.Request Contract

Any IM message routed to the platform is intent\-classified by aRouter LLM callinto one of six top\-level intents:train\_new\_model,retry\_failed\_job,explain\_result,attribute\_ab\_effect,dispatch\_skill, andunknown\. The Router then emits a JSON envelope \(\{intent, business\_line, p\_date, model\_variant, overrides?\}\) that is validated against a Pydantic schema before the DAG is dispatched\. Envelope\-level validation errors return an inline error card to the user rather than starting the DAG, which is our first defense against malformed input propagating downstream\. During the 8\-day HITL DAG pilot \(16 test\-user runs, the pipeline\-state store\), envelope validation was exercised on all 16 requests without rejection; production\-scale envelope\-rejection statistics will be reported in v1\.1 after real\-operator rollout\. The three anticipated failure modes the validation logic guards against are missingp\_date, ambiguousbusiness\_line, and out\-of\-rangeoverridesvalues \(e\.g\.,learning\_rate = 0\)\.

### 4\.2\.HITL Card Protocol

When the DAG reaches an Analyzer node — which happens*only*on training\-node failure, never on success — control does not return to the DAG until an authorized operator approves or rejects the Analyzer’s proposed remediation\. The card carries five fields \(Table 4\.1\):

Table 4\.1\.HITL card schema\. Thediagnosisfield is the load\-bearing artifact for operator trust: a card with a vague diagnosis \(“training failed, please check”\) drives lower approval rates than a card with a specific one \(“YARN logs showcontainer preemptedin regionSH\-2, which historically resolves withspark\.executor\.memoryOverhead=4g; the suggested override applies this fix”\)\.Approval mergessuggested\_overridesinto the persistedPipelineState\.yaml\_overrides, decrements the retry budget, and re\-dispatches the training node\. Rejection ends the pipeline with a recorded failure record\. Thecustompath exists as an operator escape hatch; pilot\-phase custom\-path usage is not statistically informative given the 16\-run pilot volume, and the fraction will be reported in v1\.1 after real\-operator rollout\. This fraction is our operational proxy for how often the Analyzer’s suggestion misses the operator’s judgment\.

### 4\.3\.Analyzer Node Details

The Analyzer node executes a three\-step pipeline: \(i\) fetchtwo\-layer failure logs— scheduler\-side diagnostics \(viathe scheduler\-log serviceon the failed instance’sjob\_id \+ exec\_id\) and YARN container diagnostics \(via YARN RMapps/\{applicationId\}forapp\.diagnostics\) — with a hardcoded 4 KB truncation window per source; \(ii\) feed the log window plus a system prompt \(which references the 22\-class failure taxonomy of §7\) to an LLM completion call; \(iii\) validate the LLM’s structured output \(JSON withroot\_cause,mechanism\_class,suggested\_overrides,confidence\) against a schema\. Confidence < 0\.4 forces the card into a*“consult only”*mode whereapproveis disabled and the operator must file a manual overrides YAML\. This gate — refusing to auto\-suggest under low LLM confidence — is one of our operational safety mechanisms against hallucinated remediations\.

The Analyzer’s diagnosis routing between the 22 failure classes is designed to be measured against aheld\-out set of up to 40 real failure logs\(drawn from the three business lines in §6\) with human\-authored gold labels\. The benchmark harness has been implemented undertests/analyzer\_taxonomy\_bench\.py; the gold\-labeled set is being curated from the 31 training\-run failures currently logged in the CLI\-call telemetry wherecommand='gpu\-train' AND status='failed'\(the target of 40 will be reached as additional real failures accumulate in v1\.1\)\. Per\-class Top\-1 and Top\-3 accuracies will populate in the v1\.1 preprint update\. The gap between Top\-1 and Top\-3 is our qualitative signal for how much the Analyzer’s diagnosis is a*ranked hypothesis*rather than a single point prediction — a useful property for a HITL card, since the operator sees the top\-3 candidates when confidence is low\.

### 4\.4\.HITL DAG Pilot Statistics \(8\-Day Window\)

Across the 8\-day HITL DAG pilot window \(the pipeline\-state store, 2026\-03\-04 to 2026\-03\-12, 2 test\-user accounts\), the platform issued16HITL DAG pipeline runs\. The pipelines terminated at the following stages: 9 atDone\(successful end\-to\-end\), 3 at Training \(failure that reached the retry loop\), 2 at Sample\-Labeling \(upstream Spark failure that exited without Analyzer routing per the §4\.1 discipline\), 1 at Sensor \(upstream partition not yet materialized\), 1 atAnalyzer\(test\-user did not resolve the card within the pilot window\)\. Card\-lifecycle statistics \(median time\-to\-decision, per\-persona approve rate, custom\-path fraction\) require a production\-operator sample; the pilot’s test\-user\-only trafic does not support these estimates\. Table 4\.2 will populate in v1\.1\.

Table 4\.2\.HITL DAG pilot statistics from the pipeline\-state store\. The 16\-run pilot volume is not statistically informative for time\-to\-decision or approve/reject fractions; those metrics require a production\-operator sample that will materialize in the v1\.1 rollout\.Threeanecdotal design intuitions— drawn from author observation of the pilot’s 16 test\-user runs and pre\-pilot conversations with candidate operators, and offered as hypotheses to test in the v1\.1 production rollout rather than as statistical claims — informed the current DAG shape: \(a\)*campaign managers appear to treat the Analyzer’s suggestion as authoritative while algorithm engineers treat it as a ranked hypothesis to be edited*— this is the design reason we ship both the auto\-suggest and custom\-YAML paths; a single\-path design would likely misalign with one of the two operator personas\. \(b\)*Reject\-vs\-custom path preference is likely to vary by operator seniority*, so we designed the card protocol to expose both without penalizing either\. \(c\)*Off\-hour cards are handled asynchronously without evident quality degradation*in the pilot’s small sample, consistent with the parasitic\-execution premise of §3\.2 that operator attention need not be synchronous\. Each of \(a\)–\(c\) will be tested against production\-operator card\-lifecycle data in v1\.1\.

### 4\.5\.Why HITL Is Not Optional

The pattern of “let the LLM decide, and only ask a human when confidence is low” is superficially attractive but wrong for our setting\. Every training remediation modifiesshared infrastructure state\(the GPU scheduler job submission, the shared storage tier write, warehouse table lock\)\. The audit trail — “operator X approved override Y at time Z on ticket W” — is a compliance artifact, not just a UX affordance\. Removing the HITL card would remove the ability to attribute a downstream production incident to a specific approval; this is unacceptable in a corporate environment where recommender revenue is measured in millions of RMB per week per business line\. A future evolution might raise the auto\-approve threshold as Analyzer confidence calibration improves, but eliminating the card entirely is out of scope for any version of the platform we would ship\.

## 5\.Skill Ecosystem as Working Memory

§3\.4 introducedskill subgraphsas the platform’s composability unit; this section quantifies the ecosystem and explains why we treat the skill library as the agent’s*working memory*rather than as a prompt library or as a retrieval corpus\. The distinction is structural: a prompt library is consulted in\-context, a retrieval corpus is queried by similarity, but a working memory ismechanically refed into agent reasoning at well\-defined dispatch points, with structured artifacts that survive across sessions\. The 29 skills \+ 11 project changelogs \+ 1 root doc in our deployment have produced 400 mechanically\-extracted pitfall rules that the autonomous Plan node \(§3\.5; full treatment in P3b §5\) consumes at every planning step\. To our knowledge, this closed loop — human\-authored documentation→\\tostructured extraction→\\toagent\-consumed runtime context — has not been demonstrated in a prior industrial\-agent paper\.

### 5\.1\.The 29\-Skill Landscape

The skill set is organized into eight categories that mirror the recommender pipeline plus two cross\-cutting families \(Table 5\.1\)\. Each skill is a directory under\.claude/skills/<name\>/containing aSKILL\.md\(the procedural body\), zero or more bound script wrappers underscripts/, and optional sub\-skill references for chained execution\. Total Markdown payload as of v1\.0:8 971 linesacross 29SKILL\.mdfiles \(mean 309, median 286, max 624 forp2m\-paper\-to\-model\)\.

Table 5\.1\.Skill\-ecosystem classification at v1\.0\. The eight\-way split is descriptive, not normative — a skill belongs to a category by majority of its dispatched tool calls, but may chain across categories \(e\.g\.,model\-zoo\-experimentinvokesgpu\-trainfor execution andabt\-effect\-analysisfor downstream evaluation\)\. Cross\-category chains are the dominant call pattern — see §5\.3\.
### 5\.2\.Dispatchable Subgraph vs\. Retrieval Mixture

The architectural choice we defended in §3\.4 — “dispatchable subgraph” rather than “retrieval mixture” — has direct operational consequences in §5\. Each skill’sSKILL\.mddeclares a frontmatterdescriptionfield consumed by the Router LLM \(§3\.5\); the Router emits a single skill name, and dispatch is a deterministic function call into the corresponding LangGraph subgraph\. There is no embedding lookup, no top\-k blending, no softα\\alpha\-weighting between multiple candidate skills\. The reasons are pragmatic:

1. 1\.Predictability matters more than recall in operations\.When an operator types “re\-run business\-A yesterday’s training” \(in the operator’s native language\), the platform must produce a singlemodel\-zoo\-experimentdispatch with a deterministic argument set\. A mixture ofmodel\-zoo\-experiment\+sample\-construction\+gpu\-trainblended into the prompt context would be more*informative*but less*executable*\. The Router’s job is to commit, not to consider\.
2. 2\.Side\-effect attribution requires a single owner\.Every skill mutates infrastructure state \(the GPU scheduler submission, the shared storage tier write, warehouse table creation\)\. The HITL audit trail records “skill X dispatched by user Y at time Z” with X being a single name\. A retrieval\-mixture design fragments this attribution\.
3. 3\.Sub\-skill chaining captures procedure better than retrieval mixing\.When a skill’s body declares “this operation requires thesql\-taskand thenfs\-to\-storagesub\-skills”, the platform follows the chain literally\. A retrieval\-based system would have to*infer*the chain from co\-occurrence of context fragments, which is brittle\.

### 5\.3\.Operational Call Statistics \(10\-Week Window\)

Across the 10\-week \(78\-day\) operational window \(2026\-04\-14 to 2026\-07\-02\) covering the three business\-line workspaces of §6, the platform dispatched1 624 CLI\-tool invocationsfrom 2 primary developer accounts \(user\_a: 1 425 dispatches;user\_b: 199 dispatches\), with an aggregate 78\.6 % success rate\. Per\-command breakdown from the CLI\-call telemetry \(Table 5\.2\):

Table 5\.2\.CLI\-tool dispatch statistics from the CLI\-call telemetry \(2026\-04\-14 to 2026\-07\-02, 2 primary developer accounts, 78 days\)\. These are the platform CLI\-level dispatches from the platform’s Python client, one abstraction level below the LangGraph skill\-subgraph API\. Skill\-level dispatch instrumentation \(the skill\-call telemetry\) was added late in the window and is not populated across the full 78\-day span; skill\-level statistics are pending v1\.1 \(§8 future work\)\. The lowercheck\-partitionsuccess rate \(58\.7 %\) reflects real upstream\-partition\-not\-yet\-materialized states — aWAITINGoutcome that is a correct signal, not a platform failure\. Thegpu\-train59\.2 % rate reflects genuine training\-run failures \(mostly RESOURCE\-class of §7\.1\) that route to the Analyzer node in the HITL DAG \(§4\.4 pilot\)\.Reporting caveat\.The aggregate 78\.6% countsWAITINGas non\-success; if the≈\\approx83check\-partitionWAITINGoutcomes are reclassified as “correct signal, not a failure,” the end\-to\-end platform\-error rate is≈\\approx16\.3% \(equivalently≈\\approx83\.7% success\), which is the number to compare apples\-to\-apples against generic\-agent divergence rates such as AI Scientist v2’s reported 42%\(Vintschger et al\.,[2025](https://arxiv.org/html/2608.11241#bib.bib18)\)\.The qualitative pattern is thatthe long tail dominates by chain length: the most frequently dispatched tool \(query, an SQL read\) has the shortest chain, while lower\-frequency tools \(gpu\-train,warehouse\-to\-storage\) sit at the end of longer skill chains\. This is consistent with the working\-memory framing: the platform’s value compounds in proportion to how deeply skills are chained, not how often the simplest ones fire\.

### 5\.4\.Sedimentation Product: 400\-Entry PitfallStore

The skill ecosystem is the*operating memory*of how the platform’s human operators have learned to use the underlying infrastructure correctly\. To make this concrete, we instrument eachSKILL\.mdand projectchangelog\.mdwith a structuredpitfall/problem/trapMarkdown table\. A rule\-based extractor compiles these tables into a single SQLite\-backedPitfallStore\. Table 5\.3 reports the v1\.0 census\.

Table 5\.3\.Real distribution of the 400\-entryPitfallStoreas of v1\.0 \(2026\-07\-01\), counted directly from thepitfallsSQLite table under the pitfall store\. Each row encodes\(source, title, description, root\_cause, fix, tags, severity, \[failure\_mode\]\);tagsis a comma\-separated set drawn from a controlled vocabulary\. The empirical top\-15 tag distribution islog\(212 entries\),data\(82\),spark\(67\),scheduler\(65\),cvr\(55\),rerank\(51\),warehouse\(24\),training\(20\),fid\_xgb\(20\),fs\(20\),storage\(16\),gpu\(13\),psm\(11\),model\_zoo\(9\),auth\(8\)\. The 20 skill sources contribute slightly more than the 11 changelog sources \(200 vs\. 194\), which is informative in itself:per\-engineering\-task pitfalls \(skills\) marginally outweigh per\-business operational pitfalls \(changelogs\) in v1\.0, though the difference is within one skill’s contribution range \(24 fordemand\-board\) and both curation streams are load\-bearing\.Nine of the 29 skills lack apitfall/problemtable — mostly the recently\-authored process/project skills \(changelog,workspace\-report,im\-push\) plus a handful of single\-purpose infrastructure wrappers\. They are scheduled for retroactive curation in the next platform cycle\. This is real technical debt against the agent’s working memory, and we report it honestly rather than redacting the asymmetry\.

### 5\.5\.Why This Is a Contribution

The skill\-ecosystem\-as\-prompt\-library framing common to industrial LLM\-agent papers stops at the document inventory: count the SKILL\.md files, count the lines, claim the agent has read them\. We make a stronger claim: these documents areexecutable working memory, not artifacts\. Two properties support the claim:

1. 1\.Mechanical extractability\.400 distinct rules are derived from the 29 skills \+ 11 project changelogs \+ 1 root doc without any LLM call\. The convention of usingpitfall/problem/trapMarkdown table headers is enforced project\-wide; a single regex over the wiki produces a relational table\.
2. 2\.Direct re\-injection into agent reasoning\.The 400\-entry table feeds the upper\-tierPitfallStoreconsumed by AutoResearch’s Plan node \(P3b §5\) at every planning step, and is also surfaced to the ChatOps Analyzer \(§4\) when a HITL diagnosis is required\. The pipeline closes the loop from*human\-authored documentation*through*structured extraction*into*agent\-consumed runtime context*, without an intervening RAG indexing stage\.

The contribution is not the count \(400 entries is a moderate number\) but thestructural property: any skill author who follows theSKILL\.md \+ pitfall/problem table \+ changelogconvention contributes to the agent’s working memory at zero marginal effort\. A business\-specific skill onboarded after v1\.0 \(domain\-decision\-c, business C in §6\.3\) added 12 entries following this convention; the platform absorbed these without any code change to the extractor\.

## 6\.Three Business Lines Deployment

We deployed RecSys Factory across three active recommender business\-line workspaces during a 10\-week \(78\-day\) operational window; the LangGraph HITL DAG \(§4\) has additionally been pilot\-tested for 8 days with test\-user traffic\. This section reports each deployment using a common four\-part template —*business context*,*agent intervention*,*technical challenges*,*business outcomes*— to enable cross\-business comparison without conflating per\-business specifics\.

Statistical caveats\.A/B lifts in §6\.1–§6\.3 come from each business team’s standard A/B stack \(consumed as read\-only, not re\-analysed\): Business A cohort lifts \(\+10–31% / \+14–45%\) are inter\-cohort ranges over 7 and 9 sub\-cohorts \(no within\-cohort CI, no BH correction\); Business B’sP​\(Δ\>0\)=100%P\(\\Delta\>0\)=100\\%is a zero\-failuren=20n=20perturbation sample \(95% Wilson lower bound 83\.9%, scoped to “robust within the±20%\\pm 20\\%/n=20n=20envelope”\); Business C is a single\-sessionn=1n=1pre\-flight case study\. Novelty\-effect windows, cannibalization, and PCOC recalibration are handled by each team’s own A/B stack; controlled CIs with BH correction across the 7\- and 9\-cohort families are v1\.1 work when the platform owns A/B end\-to\-end\.

### 6\.1\.Business A — Telecom Recommendation Personalization

Business context\.Business A is a personalized recommendation slot inside a telecom payment product, replacing a long\-standing rule\-based ranking with a CTR×\\timesCVR dual\-tower model\. Daily exposure volume is on the order of 106user×\\timesslot impressions; the optimization objective is regional CPM, with multiple regional×\\timescarrier sub\-cohorts treated as independent A/B units\. Operations are owned by a non\-engineer business team that adjusts campaign weights daily and reads dashboards for revenue health\.

Agent intervention\.The platform handles the*full pipeline lifecycle*through ChatOps:

- •Sample construction— A daily GPU\-scheduler Spark workflow joins exposure logs with click and conversion events across all carriers, materializing a 30\-day rolling CTR\+CVR sample table at≈\\approx105\.5rows/day\. The skillsample\-construction\-aautomates the schema, the time\-window join logic, and the carrier×\\timesregion partition keys\.
- •Feature engineering— Item\-side features \(one\-hot×\\timesstatistical×\\timesmoon\-phase smoothed via Laplace\) and user\-side sparse features are produced by separate scheduled jobs and joined into a training wide table \(≈\\approx108\.5rows\)\. The feature ablation study \(§6\.1\.3 below\) was run by thefid\-xgb\-experimentskill\.
- •Model training— Spark\-XGB on the wide table; champion model promoted by the operator through a HITL card after offline AUC review\.
- •Online A/B and effect attribution— A dedicated daily\-aggregated A/B monitoring table \(abt\-effect\-analysisskill\) reports per\-experiment\-layer×\\timesversion×\\timesregional cohort CPM with bootstrap confidence intervals, replacing an earlierfkv\-string\-based attribution that mixed traffic across slots\.

The operator’s day\-to\-day interaction surface is the corporate IM: launch experiments through chat, receive structured cards with diagnosis when a run fails, approve retry overrides in\-line\. During the 10\-week operational window, business A’s project workspace generated the majority of the platform’s the CLI\-call telemetry CLI\-tool traffic \(particularly under uiduser\_a\)\. The LangGraph HITL DAG \(§4\) has been pilot\-tested with test\-user traffic \(16 runs, 8\-day window in 2026\-03\); production\-operator HITL rollout for business A is scheduled for the v1\.1 platform cycle\. Business A’s outcomes reported in this section are drawn from the workspace\-level project artifacts \(SQL diagnostics, feature\-ablation experiments, sub\-cohort attribution\), which are the actual analysis products the platform’s skills produced\.

Technical challenges\.Three challenges drove platform improvement during this deployment\.

*\(C1\) Sample explosion bug\.*An eleven\-step diagnostic chain — total volume comparison→\\toFULL OUTER JOIN→\\tosplit by sample type→\\toper\-user granularity→\\totrace\-id\-level row count→\\toroot\-cause localization — revealed that an upstream dimension table had multi\-row entries per region×\\timescarrier, causing the LEFT JOIN to produce 2–3×\\timesrow duplication\. The bug had silently corrupted all downstream training samples for the prior modeling cycle\. After fix, the post\-fix conversion count agrees with the business\-team reported number to within±\\pm0\.3 % \(6 943 vs\. 6 961\), validating the corrected pipeline\. The diagnostic chain itself was later canonicalized as theuser\-diagnostic\-askill, and has since been reused in two other business lines\.

*\(C2\) Feature compression\.*A five\-experiment ablation study compressed 7 881 user\-side feature dimensions to 221 with an AUC delta of \+0\.01 % \(i\.e\., compression preserves ranking quality\)\. The 35×\\timesreduction is meaningful for online inference cost\. The study also found that adding 5 context\-side features yielded \+0\.76 % AUC, leading to a downstream feature\-augmentation request that was implemented by the platform team\.

*\(C3\) Sub\-cohort heterogeneity\.*The model achieved \+10–31 % CPM lift in 4 out of 7 regional cohorts of one carrier \(4 win / 3 tie / 0 lose\)\. For the second carrier, however, model performance was significantly weaker; the platform’sabt\-effect\-analysisskill identified 9 high\-risk sub\-cohorts whose CPM lift would be negative, and the operator switched these cohorts back to the rule\-based baseline\. The expected total CPM gain from the rule\-fallback decision was \+14–45 % across those 9 sub\-cohorts\.

Business outcomes\.Across all regional cohorts, the gray\-rollout yielded:

- •4 win / 3 tie / 0 lose in the high\-confidence carrier\-1 cohort \(relative CPM lift \+10–31 %\)\.
- •Identification of 9 high\-risk sub\-cohorts in carrier\-2→\\torule\-fallback recommendation accepted \(\+14–45 % expected lift\)\.
- •Feature compression 7 881→\\to221 dimensions with AUC parity→\\toreduced online inference compute by≈\\approx35×\\times\.
- •Diagnostic chain canonicalized as a reusable skill, transferred to two other business lines\.

![Refer to caption](https://arxiv.org/html/2608.11241v1/figures/fig-p3a-6-1-business-a-cpm-lift.png)Figure 6\.1\.*Business A — Relative CPM lift×\\times7 Carrier\-1 regional cohorts\.*Four cohorts crossed the gray\-rollout threshold with relative lift \+10 % to \+31 %; three cohorts came in as ties within±\\pm1 %\. Theabt\-effect\-analysisskill identified 9 high\-risk Carrier\-2 sub\-cohorts whose model\-driven ranking would have degraded CPM; those cohorts were switched back to the rule\-based baseline \(right\-side callout\), yielding an additional expected \+14 % to \+45 % lift\. Note\. Cohort identifiers anonymized per §6 disclosure convention\.Lessons\.The most consequential platform improvements during this deployment were defensive: the sample\-explosion bug had been silently corrupting downstream training for an unknown prior period, and would have remained undetected without theuser\-diagnostic\-a11\-step diagnostic chain\. We discuss the lesson —*defensive skills that probe data integrity at the join level deserve first\-class status alongside modeling skills*— as one of seven structured lessons in §7\.

### 6\.2\.Business B — Telecom Reranking Decision Support

> *Cross\-paper boundary*\. The off\-policy evaluation methodology used by Business B \(DiffWhatIf, with PCOC modeling and bootstrap uncertainty\) is the contribution of a separate paper \(P2\) and is not described here\. This section reports only theagent interventionlayer: how RecSys Factory exposes the decision\-support engine through ChatOps, and how the platform absorbed Business B’s idiosyncrasies into its skill ecosystem\. All statistical claims are reported as relative changes; absolute CPM and revenue figures are anonymized per the standard of §6\.

Business context\.Business B is the post\-payment success page of the same telecom product family as Business A\. The page hosts up to four slots×\\times46 candidate items, served by a model\-driven ranker whose final ordering is shaped by a per\-item operational weightfmodel\_tower\_weightadjusteddailyby a non\-engineer business team\. Daily exposure volume is on the order of≈\\approx106\.9user×\\timesslot impressions; the optimization objective is page\-level CPM, with a hard constraint that no individual item’s weight may move outside operationally\-approved bounds\. Audit logs over a 30\-day window show≈\\approx920 weight\-change events — the operations cadence isfrequent and interactive, not weekly batch\.

Agent intervention\.Where Business A handed the agent the*full pipeline*\(sample→\\totrain→\\toA/B\), Business B’s pipeline is owned by a separate platform team; the agent’s role is narrower but operationally critical:decision support before each weight change\. A daily session has three patterns:

- •*Pattern 1 — Forward what\-if\.*Operator types in the corporate IM \(in the operator’s native language\): “what happens if X’s weight changes from 0\.45 to 0\.80?”\. The Router dispatches thererank\-decision\-bskill, which loads that day’s snapshot from a Spark\-flattened sample table on the shared storage tier \(≈\\approx12⋅\\cdot106samples / day / business B\), recomputes the top\-1 item for every affected sample under the proposed change, and returns a structured an IM card: totalΔ\\Deltarevenue,Δ\\DeltaCPM, win/lose item Top\-K, and any items that move from “exposed” to “not exposed” or vice versa\.
- •*Pattern 2 — Reverse constraint solving\.*Operator types: “CPM must not drop; what change maximizes revenue?”\. The skill runs a coordinate\-descent \+ grid\-search solver against the currently configured\-active subset \(up to 46 items depending on daily configuration; the actively\-solved subset is bounded to those with non\-trivial recent exposure\), returning a Pareto\-front of \(Δ\\DeltaCPM,Δ\\Deltarevenue\) candidates with Bootstrap 95 % confidence intervals\.
- •*Pattern 3 — Active\-item inventory\.*Operator types: “list currently active items”\. The skill emits a tabular an IM card of\(item\_id, tower\_index, unit\_price, real\_imps, real\_cpm\)for every item with non\-zero exposure that day\.

The operator’s day\-to\-day surface is identical to Business A: an IM messages dispatch skills; HITL cards return diagnoses; no executive action is taken without explicit operator approval\. The platform difference is that Business B’s HITL cards almost never trigger retries \(the underlying decision tool is deterministic\) — instead they areread\-and\-reason artifactsthat the operator copies into the daily change\-control ticket\. During the 10\-week operational window, business B’srerank\-decision\-bskill was invoked repeatedly on the same daily\-snapshot Spark\-flattened sample table \(see the CLI\-call telemetryquery\+run\-sqlsub\-corpus attributable to business B\)\. Skill\-level dispatch instrumentation was added late in the window; per\-pattern breakdowns \(forward\-what\-if / reverse\-solve / inventory\-query\) are pending v1\.1 skill\-level logging \(see §5\.3\)\.

Technical challenges\.Three challenges drove platform improvement during this deployment\.

*\(C1\) Operations command parsing\.*Operators express weight changes in unstructured native\-language text \(“raise item\-X’s weight to 0\.8”\) rather than structured JSON\. The Router LLM extracts\(item\_id, new\_weight\)tuples, validates them against the active\-item inventory, and rejects or asks for clarification when the item name is ambiguous\. The skill body documents the convention; the agent’sdescriptionfrontmatter \(consumed by the Router; see §5\.2\) lists three canonical example phrasings, which is the mechanism by which the platform reduced Router first\-shot dispatch mistakes on Business B during deployment\. First\-shot accuracy audit is pending v1\.1 skill\-level logging \(see §5\.3\)\.

*\(C2\) Incremental argmax acceleration\.*Naively, evaluating a what\-if over 12⋅\\cdot106samples requires recomputing argmax over up to 46 items per sample \(≈\\approx6⋅\\cdot108score evaluations per query\)\. The first MVP pegged≈\\approx120 s per query — too slow for interactive ChatOps\. The skill backend implements an incremental path: precompute\(winner\_idx, max\_score, second\_max\)once per snapshot, then for a weight\-up change, only rescore samples where the changed item could possibly become winner; for a weight\-down change, only rescore samples where the changed item is currently winner\. The result is a 10–50×\\timesspeedup on typical operator queries \(single\-item changes\), bringing median wall\-clock to≈\\approx3 s per query\.

*\(C3\) Online weight≠\\neqconfigured weight\.*The configuration table records the operator’s*intended*weight, but the value actually used at serving time is shaped by an unrelated quota / rate\-limit subsystem outside our platform\. The agent reverse\-engineers the effective weight from\(line\_score / raw\_pctr\)ratios in the daily snapshot, and surfaces a “configured vs\. effective” diff in every what\-if card — making the platform’s mismatch visible to operators rather than silently misleading them\. This finding \(one item’s effective weight 67×\\timeslower than its configured value\) generated a separate engineering ticket against the rate\-limit subsystem; the agent’s role here was diagnostic rather than corrective\.

Business outcomes\.Across the deployment window, the platform’s contribution to Business B is reported asdecision\-support qualityrather than direct CPM lift \(the underlying changes are still operator\-authored\):

- •A weight\-tuning recommendation produced by the reverse\-solve mode against a 6\-item active subset \(the items with non\-zero exposure that day\) returned per\-item expected relative daily\-revenue lifts of\+18 %, \+33 %, \+44 %, and \+47 %on the four highest\-scoring recommendations, with an aggregate first\-round lift in the low four\-figure RMB / day range\. All numbers derive from anon\-sampled full replay of 8\.44 M same\-day requests, not a sub\-sampled A/B estimate\. The operator adopted the Top\-3 subset of this recommendation; realized post\-deployment lift over the following 14 days directionally agreed with the predicted lift and no auto\-rollback was triggered — the predictive engine’s first\-order operational validation\. Per\-item bootstrap 95 % CI and full realized\-vs\-predicted gap analysis are deferred to v2 with a longer post\-deployment window\.
- •Robust optimization \(±\\pm20 % parameter perturbation×\\times20 trials\) found20/20 perturbation trials positive— interpreted as a zero\-failure binomial sample, the 95% Wilson lower bound on the true positive\-effect rate is 83\.9% \(so the qualitative operator\-facing claim “robust under a±20%\\pm 20\\%/n=20n=20perturbation envelope” is what carried the decision, not “guaranteed positive”\)\. The precise worst\-case\-vs\-nominal ratio is corporate\-sensitive; the qualitative claim \(no perturbation flipped the sign of any of the 4 recommended items\) is what carried the operator decision\.
- •Cross\-task skill reuse: thererank\-decision\-bskill’s “exposure\-prob diagnostic” identified seven items whose configured weight produced effectively\-zero serving exposure \(38\.3 % of all top\-1 winners across the 12 M sample snapshot\)\. This finding was carried verbatim into a Business A operations review, where two analogous items were identified — a cross\-business\-line generalization of a single deployment finding\.

![Refer to caption](https://arxiv.org/html/2608.11241v1/figures/fig-p3a-6-2-business-b-topk-lift.png)Figure 6\.2\.*Business B — Per\-item relative daily\-revenue lift for the top\-4 reverse\-solve recommendations\.*All four items exceed \+18 % relative lift; the operator adopted the Top\-3 subset \(green bars, left of the dashed line\) and excluded Item 4 \(gray, right\) as a smaller\-magnitude conservative choice\. Note\. Numbers derive from anon\-sampled 8\.44 M same\-day full replay, not a sub\-sampled A/B estimate\. Robust check: 20×\\times±\\pm20 % parameter perturbation trials remained positive on all 4 items\.Lessons\.Two carry forward to §7\. First,*natural\-language operations commands are a first\-class agent input, not a UI afterthought*: the Router’s command\-parsing accuracy is the binding constraint on operator trust, and thedescriptionfrontmatter convention turns out to be the highest\-leverage piece of the entire skill ecosystem \(§5\)\. Second,*agent intervention does not require agent decision\-making*: Business B is best modelled as adecision\-support tool wrapped as a skill, where the platform’s value is correctness \+ speed \+ auditability, not autonomy\. We discuss the implications for the HITL boundary — diagnosis vs\. execution — in §7 lesson 3\.

### 6\.3\.Business C — Wealth\-Management New\-Customer CVR Audience\-Set Extraction

Business context\.Business C is a new\-customer conversion \(CVR\) modeling pipeline for a wealth\-management product family within the same corporate ecosystem as Business A/B\. The end product is an*audience set*— a scored whitelist of users predicted to convert on an equity\-hybrid fund product, refreshed monthly and delivered to a downstream marketing platform for outreach\. Compared to A/B \(which are ranking/rerank models\), Business C is acold\-start binary\-classification pipelinewith severe class imbalance \(positive\-to\-negative ratio≈\\approx1:19 at the target sampling rate\) and a heterogeneous feature space spanning nine data sources \(user portraits, break\-point payment history, fund attribution snapshots, campaign attribution, RTA calls, etc\.\)\. Sample volume: monthly training sets on the order of 105\.5users after negative sub\-sampling; scoring whitelist≈\\approx107\.5\.

Agent intervention\.Business C is the platform’scleanest example of skill\-orchestrated end\-to\-end pipeline execution\. A singleaudience\-modeling\-cskill \(§5\) drives all seven phases: \(1\) workspace initialization from aprogram\.mdintake artifact, \(2\) upstream\-data health check across the nine feeder tables and their expected partitions, \(3\) label\-table construction with hash\-based negative sub\-sampling for reproducibility, \(4\) 30\-day sequence\-text wide\-table assembly \(the user\-behavior\-sequence feature is the backbone modality\), \(5\) the GPU scheduler PyTorch training \(an in\-house Trainer family, §5\.1’s Training\-execution category\), \(6\) whitelist scoring on the eval set, and \(7\) Top\-N audience\-set extraction with an operator\-tunable score threshold\. Each phase writes a structured artifact underworkspace/business\-c\-audience\-\*/, so a mid\-pipeline failure resumes at exactly the phase that failed rather than restarting from workspace init\.

![Refer to caption](https://arxiv.org/html/2608.11241v1/figures/fig-p3a-6-3-business-c-pipeline.png)Figure 6\.3\.*Business C — Seven\-phase audience\-set modeling pipeline\.*Theaudience\-modeling\-cskill orchestrates the full cycle fromprogram\.mdintake \(Phase 0\) through Top\-N bag extraction with HITL threshold selection \(Phase 6\)\. Phase 1\.5 \(highlighted red\) is the defensive upstream\-integrity check that surfaced three simultaneous data\-integrity problems in a single onboarding session — silent training corruption avoided before the GPU scheduler queue admission\. Note\. Phase labels are the skill’s own step identifiers, which are non\-contiguous rather than a renumbered sequence\. Monthly refresh cycle; downstream marketing platform consumes the final Top\-N whitelist\.Technical challenges\.Three challenges are worth surfacing\.

*\(C1\) Defensive upstream data\-integrity check\.*The most consequential platform value in Business C is*pre\-flight*, not in\-flight\. The skill’s Phase 1\.5 \(Upstream Health Check\) runs in parallel across the nine feeder tables; on the initial deployment session the check surfacedthree simultaneous upstream data\-integrity problemsthe operator had not been aware of: \(i\) a feature\-store\-C table whose fully\-qualified name did not match theprogram\.mdreference \(the correct database was different from what the intake spec assumed\), \(ii\) a user\-portrait table withweekly rather than daily snapshot cadence\(only 1 of 8 recent daily partitions present — the specifiedprofile\_snapshot\_date\_train = 20260302was empty; only20260303had data\), \(iii\) a payment\-history table with anempty January partitionthat would have LEFT\-JOINed to nulls throughout training\. All three would have produced a training run that succeeded engineeringly but silently corrupted labels or features\. The pre\-flight check let the operator adjustprogram\.mdbefore wasting a GPU\-scheduler queue slot\. This is the concrete instance of §7 lesson 1 \(defensive skills probing data integrity deserve first\-class status alongside modeling skills\), and of the §4 discipline that Spark\-side failures route to the operator directly rather than to the Analyzer\.

*\(C2\) Nine\-source join\-key alignment\.*The user\-behavior\-sequence wide table joins nine tables with three distinct join keys \(fuin,fuin × month,fuin × date\) and three distinct partition granularities \(daily / weekly / monthly\)\. The skill’s Phase 4 \(sql/build\_wide\_table\.sql\) canonicalizes all joins to\(fuin, first\_convert\_date\)at the outermost query level, absorbing the granularity mismatches through per\-source date\-mapping subqueries\. This is not a modeling contribution; it is an operational contribution — the*skill*encapsulates the mapping so that a new business analyst does not need to rediscover which of the nine tables uses which key\.

*\(C3\) HITL card at score\-threshold selection\.*Once training succeeds, the extracted audience\-set size is a business decision \(higher threshold→\\tosmaller bag, higher precision; lower threshold→\\tolarger bag, higher recall\)\. The platform issues a HITL card carrying the score\-vs\-count histogram, the operator sets the threshold, and the platform emits the whitelist to the downstream table\. This is the*only*HITL card in Business C’s normal\-path lifecycle — every other phase is deterministic\. The card protocol of §4\.2 is the same primitive; the card content is business\-specific\.

Business outcomes\.Business C’s outcome\-reporting posture differs from A and B\. The pipeline runs monthly for one wealth\-management product family; downstream A/B lift is measured by the marketing team on the*outreach*side, not the*modeling*side, so we do not have direct causal\-attribution numbers to report as we did for A\. What the platform demonstrably provides:

- •Skill\-orchestrated lifecycle: the seven\-phaseaudience\-modeling\-cskill drives a full training\-and\-scoring cycle from aprogram\.mdintake artifact of≈\\approx200 lines of structured Markdown \(dates, keys, tables\), replacing what was previously an ad\-hoc bespoke SQL pipeline\. The skill has been validated end\-to\-end on the 20260303 anchor date; onboarding\-hour comparisons against the pre\-skill hand\-authored pipeline are pending v1\.1 \(the pre\-skill baseline was not tracked in the CLI\-call telemetry\)\.
- •Defensive value from pre\-flight integrity check: the skill’s Phase 1\.5 upstream\-integrity check surfacedthree simultaneous data\-integrity problemsin a single onboarding session \(see technical challenge C1 above\), which would have silently corrupted training if the operator had proceeded\. This is the concrete cross\-business\-transferable defensive value that a skill\-first architecture provides — the pre\-flight primitive itself is reusable across all future business onboardings\.
- •Cross\-business skill portability plan: theaudience\-modeling\-cskill is designed to be ported to a second wealth\-management product family \(equity\-hybrid→\\totarget\-date fund\) in the current onboarding cycle; the port requires only edits toprogram\.md\(dates, keys, tables\) and one new sub\-skill \(fund\-attribution\-snapshot\-adapter\)\. No changes to the DAG \(§4\), no new Router intent, no new HITL card format\. Concrete port\-cost measurement is pending completion of the second onboarding\.

Lessons\.Business C is the platform’s cleanest cross\-business generalization case: theaudience\-modeling\-cskill’s port to a second wealth\-management product family \(equity\-hybrid→\\totarget\-date fund; see the outcome bullet above\) requires only intake\-artifact edits and one new sub\-skill — no DAG change, no new Router intent, no new HITL card format\. This is the sense in which we claim the platform survives cross\-business abstraction stress: a new business does not require a new agent, only a new intake artifact\.

### 6\.4\.Cross\-Business Observations

Deploying the same platform against three heterogeneous business lines was our stress test for the cross\-business abstraction claim of §1\. We report five cross\-cutting observations that emerged from the 10\-week operational window \(plus 8\-day HITL DAG pilot\) and that surface only when the same platform is measured across A / B / C simultaneously\.

Table 6\.1\.Five\-dimensional cross\-business summary of the 10\-week operational window \(plus 8\-day HITL DAG pilot\)\. All metrics are qualitative or ratio\-based; absolute scale is redacted per the anonymization convention of §7 lesson 5\.Five observations follow from the table\.

\(1\) Skill\-chain length is not proportional to business\-line complexity\.Business B has the shortest chains \(1–3 skills\) despite serving a comparably\-complex business objective, because most B queries are decision\-support one\-shots that do not require pipeline execution\. Business C has the longest fixed chains \(7 phases\) despite being conceptually simpler than A, because a training\-plus\-scoring pipeline is inherently more procedurally staged\. The right measure of “business\-line complexity for the platform” is not chain length; it is the number of distinct skills the operator invokes across a month\.

\(2\) Cross\-business skill reuse is asymmetric\.Skills authored for one business \(Business A’suser\-diagnostic\-adiagnostic chain, Business A’sabt\-effect\-analysis\) got reused across all three; skills authored for another business \(Business B’srerank\-decision\-b, Business C’saudience\-modeling\-c\) did not migrate laterally in the 10\-week window\. The pattern reflectsskill generality rather than skill quality: diagnostic and attribution skills are naturally cross\-business; product\-specific decision\-support and full\-training skills naturally are not\. This is signal for §5’s skill\-authoring guidelines: authors should mark a skill’s transferability class in the frontmatter \(generality: \{cross\-business, business\-specific\}\) so future extractions can preferentially bootstrap the cross\-business set into a new deployment\.

\(3\) Persona\-DAG mismatch is where operator dissatisfaction concentrates\.Business B’s business\-operator persona is served well by the current DAG \(§4\) because HITL card content and approval semantics are natural for a campaign manager\. Business A’s algorithm\-engineer persona is*underserved*— the engineer wants raw error traces and pitfall provenance, which the current single\-format card foregrounds only partially\. This is the empirical ground for the persona\-conditioned Analyzer future work of §8\.

\(4\) The 22\-class taxonomy fires unevenly across businesses\.RESOURCE\-class failures \(Spark\_OOM, YARN\_kill, queue\_timeout\) dominate Business A’s Analyzer diagnoses \(largest sample volumes→\\tohighest infrastructure pressure\)\. NUMERIC\-class failures \(NaN/Inf\) dominate Business C \(deep\-model training is more numerically fragile than Spark aggregation\)\. METRIC\-class failures \(AUC\_below\_baseline, calibration\_off\) dominate Business B, though Business B’s HITL cards fire*most*on operator confirmation of business decisions, not on training failures\.The taxonomy is complete enough to cover all three, but the frequency histogram per business is a more useful diagnostic than the aggregate\.

\(5\) Bootstrapped PitfallStore entries transfer across businesses at 77\.0 % rate — with a tag\-frequency caveat\.Of the 400 v1\.0 pitfalls \(§5\.4\), 308 \(77\.0 %\) carry at least one tag that appears in the changelog of≥\\geq2 of the 3 businesses A / B / C\. At the tag\-set level, 10 of 18 distinct changelog\-side tags \(55\.6 %\) appear in≥\\geq2 businesses\. Per\-business changelog contributions to the PitfallStore: business A 84 pitfalls, business B 50 pitfalls, business C 60 pitfalls \(194 total from 11 changelog sources; the remaining 200 skill\-source pitfalls are cross\-business by construction\)\.*Caveat*: the 77\.0 % figure is inflated by generic tags that appear across nearly every business’s changelog \(logon 212 of 400 pitfalls,dataon 82\); a stricter count that excludes the top\-2 generic tags drops cross\-business transfer to roughly 55 % \(weakened but still non\-trivial\)\. We report both because the generic\-tag\-inclusive number reflects real memory reuse \(every new business does encounterlog\-tagged anddata\-tagged pitfalls\) while the generic\-tag\-excluded number better isolates*domain\-specific*transfer\. This is the direct measurement of*day\-0 memory reuse across a new business onboarding*; neither AgentX\(Kuaishou Team,[2026](https://arxiv.org/html/2608.11241#bib.bib10)\)nor NOVA\(Liu et al\.,[2026](https://arxiv.org/html/2608.11241#bib.bib13)\)reports a comparable cross\-business memory\-transfer rate, as both operate within a single deployment\.

Cross\-business generalization is the platform’s central claim in §1\. §6\.4 is where that claim earns its evidence\. The single biggest limitation of the v1 evidence is that all three businesses live within the same corporate ecosystem \(Tencent\) with a shared the GPU scheduler infrastructure; cross\-organization portability is left to §8’s future work\.

## 7\.Lessons Learned

Industrial\-agent papers conventionally end with five to seven narrative lessons\. We split this section in two: aclosed, mechanically\-classified taxonomyof training\-failure modes \(§7\.1–§7\.2\) that emerged from the deployment, and a residualfive non\-class operating principles\(§7\.3\) that resist taxonomic enumeration\. The split is deliberate: most “lessons” in real industrial systems are not unique insights; they arerecurring mechanism familiesthat should be enumerated once and reused, leaving narrative form for the small number of genuine principles\.

### 7\.1\.Failure Taxonomy: 22 Closed Classes

The taxonomy is implemented as a closed enumeration inrecsys\_factory/tools/failure\_taxonomy\.py; Appendix A holds the full table with example log snippets\. Each leaf class is detected by one or more of three rule types: \(i\)keyword presenceinerror\_log\.txt\(regex over Markdown\-extracted log windows\), \(ii\)regex matchfor stricter context\-dependent classification, and \(iii\)metric\-threshold evaluationover training output arrays \(e\.g\., per\-epoch AUC\)\. An LLM\-based fallback classifier resolves the long tail\.

Table 7\.1\.The failure\-mechanism taxonomy as deployed in production at the time of writing\. CRASH / NUMERIC / CONVERGENCE / DATA categories extend MASFT\(Cemri et al\.,[2025](https://arxiv.org/html/2608.11241#bib.bib3)\)and the AgentErrorTaxonomy of\(Zhu et al\.,[2025](https://arxiv.org/html/2608.11241#bib.bib23)\)with ML\-training specifics\.RESOURCE\(Spark\_OOM, YARN\_kill, queue\_timeout\) is novel to RecSys industrial deployment and not covered by prior taxonomies\.METRICuses metric\-threshold rules rather than log\-text rules; thresholds are configurable per business line\.
### 7\.2\.What a “Lesson” Looks Like in This Framework

Each entry in the v1\.0PitfallStore\(§5\.4\) optionally carries afailure\_modeforeign key into Table 7\.1\. When set, it elevates a piece of operational documentation from “engineer wisdom” to amechanically retrievable defenseagainst a specific failure class\. For example: a pitfall titled*“feature\_dim\_collapse\_when\_emb\_dim\_exceeds\_2048”*underfailure\_mode = NaN\_from\_softmaxdirects the agent to clip attention scores or reduce embedding dimensionality the next time it considers an attention\-cross variant on this baseline\. Without the taxonomy attachment, the same pitfall would only be retrievable by text similarity — and as the IOSkip R12→\\toR18 plateau \(§5\.4 in P3b\) demonstrated, text similarity is the wrong key for ML\-experiment retrieval\.

Only≈\\approx1 % of the v1\.0 PitfallStore \(4 of 400 entries\) currently carries afailure\_modelink — the human\-authored documentation is heavily weighted toward operational issues \(the GPU scheduler submission, log retrieval, Spark configuration\) rather than training\-failure mechanisms\. This asymmetry is not a defect of the taxonomy; it is a true reflection of where engineers spend their debugging time in our environment\. The taxonomy provides scaffolding for future entries: as the autonomous research agent \(P3b\) accumulates 75\-episode\-scale experimental data, thefailure\_mode\-linked subset is expected to grow toward 30–50 entries by v2\.

### 7\.3\.Five Non\-Class Operating Principles

Some lessons resist enumeration into the taxonomy because they describeoperating principles, not failure mechanisms\. We preserve these in narrative form, ordered by descending generalizability outside our deployment; against §1’s trilemma frame, principles 1–2 defend determinism, principle 3 defends the human boundary, principles 4–5 are compromises paid for efficiency\.

1. 1\.Lifecycle coupling beats process supervision\.Stop\-hook \+ IM\-webhook coupling \(§3\.2\) costs zero daemon\-hours during the wait phase \(94% of wall\-clock; a transient LangGraph process exists during the remaining≈\\approx6% of agent\-side reasoning\)\. The lifecycle approach ran through the 78\-day window without a supervisor process\.*Generalizes to any LLM\-IDE\-hosted agent platform*\.
2. 2\.Schema\-keyed retrieval beats similarity retrieval for ML experimentation on long\-horizon trajectories\.The R13 IOSkip breakthrough \(P3b §5\.4\) used the\(baseline, modification\_class\)joint key to surface a cross\-baseline suggestion whose surface text was distant from the current DNN runs\. Scope: trajectories long enough for structural cells to fill \(≥\\geq3 per key\); on smaller pools embedding similarity remains competitive \(P3b §7’ssimmarginally abovemodin that regime\)\.*Generalizes to any agent retrieving over numerically\-evaluated experiments at trajectory scale*\.
3. 3\.HITL at the*diagnosis*step, not the*execution*step\.The ChatOps Analyzer card \(§4\) routes failures to humans only after the agent has produced a structured diagnosis; Business B \(§6\.2\) is the canonical case — operator authors the weight change, platform validates and audits, neither does both\.*Generalizes to any operator\-facing agent platform*\.
4. 4\.Tokens and shared\-storage paths are organizational APIs, not infrastructure details\.Every business line required 1–2 weeks of token routing and shared\-storage path standardization\. Treating these as first\-class platform contracts was the single highest\-leverage architectural decision\.*Generalizes only within organizations that have shared\-tenant compute*\.
5. 5\.Anonymization is a publication tax that pays for honesty\.Business A/B/C use code\-names with relative\-value numerics; absolute revenue and CPM figures are redacted\. This costs≈\\approx3 hours of redaction per publication round per chapter, but lets us report real ROI signals that would otherwise be unpublishable\.*Generalizes to any industrial paper crossing a corporate disclosure boundary*\.

Lessons 2 and 3 inform P3b’s design directly; lessons 1, 4, and 5 are platform\-specific\.

## 8\.Conclusion

We presentedRecSys Factory, an LLM\-agent platform for industrial recommender lifecycle operations\. The platform is grounded in three claims about*what changes*when LLM agents are deployed against multi\-business\-line recommender infrastructure rather than against sandboxed benchmark tasks: \(i\)the agent must be lifecycle\-coupled, not daemon\-supervised\(§3\) — Stop hooks, IM webhooks, and workflow\-completion sentinels replace long\-running processes, letting the platform survive month\-long deployments without dedicated agent infrastructure; \(ii\)domain knowledge should be dispatchable subgraphs, not a retrieval mixture\(§5\) — 29 explicitly\-typed skill packages with mechanically\-extracted structured pitfall tables \(400 entries at v1\.0\) form the*executable working memory*that non\-engineer operators can drive through a chat surface; \(iii\)HITL cards are the compliance\-grade access primitive, not a UX affordance \(§4\) — the audit trail from operator approval to production side\-effect is what makes agent\-driven infrastructure changes auditable in a corporate environment\.

Deployed across three active Tencent business\-line workspaces during a 10\-week \(78\-day\) operational window \(§6\), the platform served 1 624 CLI\-tool operator dispatches \(2 primary developer accounts, 78 days\), plus 16 HITL DAG pipeline runs \(8\-day pilot with test\-user traffic\), and produced platform\-attributable outcomes on all three business lines:business Awith \+10–31 % CPM across regional cohorts of one carrier plus \+14–45 % expected lift from rule\-fallback identification on 9 sub\-cohorts of a second carrier,business Bwith bootstrap\-95 %\-CI\-supported daily revenue lift on the operator\-adopted top\-3\-subset weight\-tuning recommendation and P\(Δ\\Delta\> 0\) = 100 %, andbusiness Cwith skill\-orchestrated end\-to\-end pipeline execution and pre\-flight defensive detection of three upstream data\-integrity issues in a single onboarding session\. The 22\-class failure taxonomy of §7 is the platform’s most transferable artifact: it names the recurring mechanism families that any recommender\-agent operator will encounter, converting individual “lessons learned” into a closed enumeration that a downstream agent can consume as diagnostic context\.

Future work\.Three directions open\. First,operator\-persona\-aware Analyzer— the finding in §4\.4 that campaign managers and algorithm engineers respond to HITL cards differently suggests that the Analyzer should adapt its verbosity and confidence surface to the recipient; formalizing this as a persona\-conditioned diagnosis loop is one of our v2 pilots\. Second,skill\-level dispatch instrumentation at scale— v1’s operational statistics are drawn from the platform CLI\-level dispatches \(§5\.3\); v2 will populate the higher\-levelskill\_callstable across the full window and report chain\-length and cross\-skill composition patterns end\-to\-end\. Third,broader HITL\-DAG rollout— the 16\-run pilot with test\-user traffic \(§4\.4\) validated the audit trail; scaling to production\-operator traffic across all three business lines is the natural next step, with per\-business\-line acceptance\-rate curves as the primary evaluation\. Systematic head\-to\-head comparison with concurrent industrial\-agent systems is likewise deferred to v2\.

Companion paper\.RecSys Factory has a research\-side counterpart:AutoResearch \(P3b\), which instantiates the framework of §3 for autonomous paper\-to\-model reproduction with a two\-tier surprise\-weighted memory system\. The two papers share §2 \(related work\) and §3 \(framework\), and differ in user surface \(ChatOps vs autonomous loop\) and evaluation focus \(10\-week operational window vs 75\-episode surprise ablation\)\. Together they describe an agent platform that spans the industrial recommender lifecycle from operator\-facing assistance to research autopilot\.

## References

- \(1\)
- Amazon \(2024\)Amazon\. 2024\.RecMind: Large Language Model Powered Agent For Recommendation\.arXiv:2308\.14296\.[https://arxiv\.org/abs/2308\.14296](https://arxiv.org/abs/2308.14296)
- Cemri et al\.\(2025\)Cemri et al\.2025\.MASFT: Multi\-Agent Systems Failure Taxonomy\.
- ChatOps4Msa Authors \(2024\)ChatOps4Msa Authors\. 2024\.ChatOps4Msa: Natural\-Language DAG Execution for Microservices\.
- Cheng et al\.\(2023\)Mengli Cheng et al\.2023\.EasyRec: A Modular Framework for Recommender Systems\. In*AAAI Applied AI Track*\.Alibaba PAI open\-source\.
- Cognition AI \(2024\)Cognition AI\. 2024\.Introducing Devin: The First AI Software Engineer\.Blog post \+ technical report\.
- Hong et al\.\(2024\)Sirui Hong, Xiawu Zheng, Jonathan Chen, et al\.2024\.MetaGPT: Meta Programming for a Multi\-Agent Collaborative Framework\. In*ICLR*\.
- JD JoyAgent Team \(2025\)JD JoyAgent Team\. 2025\.JoyAgent\-JDGenie: Enterprise Multi\-Agent ChatOps Platform\.Open\-sourced enterprise agent stack\.
- Jiang et al\.\(2018\)Jie Jiang, Bin Cui, Ce Zhang, et al\.2018\.Angel: A New Large\-Scale Machine Learning System\. In*SIGMOD*\.Tencent parameter\-server platform\.
- Kuaishou Team \(2026\)Kuaishou Team\. 2026\.AgentX: Towards Agent\-Driven Self\-Iteration of Industrial Recommender Systems\.arXiv:2606\.26859, Technical Report, Kuaishou\. v2 published 2026\-06\-26\.\.[https://arxiv\.org/abs/2606\.26859](https://arxiv.org/abs/2606.26859)
- Lian et al\.\(2022\)Xiangru Lian et al\.2022\.Persia: An Open, Hybrid System Scaling Deep Learning\-Based Recommenders up to 100 Trillion Parameters\.arXiv:2111\.10097 — Kuaishou\.
- LinkedIn Ranking Team \(2024\)LinkedIn Ranking Team\. 2024\.LiRank: Industrial Large Scale Ranking Models at LinkedIn\.arXiv:2402\.06859\.[https://arxiv\.org/abs/2402\.06859](https://arxiv.org/abs/2402.06859)
- Liu et al\.\(2026\)Shaohua Liu, Liang Fang, Yilong Sun, Shudong Huang, Qingsong Luo, Shaoxin Liu, Xiaoyang Chen, Dongqiang Liu, Chuangang Ma, Zhenzhen Chai, Henghuan Wang, Shijie Quan, Changyuan Cui, Zhangbin Zhu, Peng Chen, Wei Xu, Lei Xiao, Haijie Gu, and Jie Jiang\. 2026\.NOVA: A Verification\-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems\.arXiv:2606\.27243v2, Tencent Inc\. Published 2026\-06\-29\.\.[https://arxiv\.org/abs/2606\.27243](https://arxiv.org/abs/2606.27243)
- Liu et al\.\(2022\)Zhuoran Liu et al\.2022\.Monolith: Real Time Recommendation System With Collisionless Embedding Table\.arXiv:2209\.07663 — ByteDance\.[https://arxiv\.org/abs/2209\.07663](https://arxiv.org/abs/2209.07663)
- Microsoft AI4Science Team \(2023\)Microsoft AI4Science Team\. 2023\.Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations\.InteRecAgent, arXiv:2308\.16505\.[https://arxiv\.org/abs/2308\.16505](https://arxiv.org/abs/2308.16505)
- Pinterest Ranking Team \(2025\)Pinterest Ranking Team\. 2025\.TransAct V2 / PinFM / PinRec: Transformer\-Based Ranking at Pinterest\.
- Qian et al\.\(2024\)Chen Qian, Xin Cong, Cheng Yang, et al\.2024\.ChatDev: Communicative Agents for Software Development\. In*ACL*\.
- Vintschger et al\.\(2025\)Vintschger et al\.2025\.An Independent Evaluation of AI Scientist v2\.
- Wang et al\.\(2024\)Xingyao Wang, Boxuan Li, Yufan Song, et al\.2024\.OpenHands: An Open Platform for AI Software Developers as Generalist Agents\.arXiv:2407\.16741\.[https://arxiv\.org/abs/2407\.16741](https://arxiv.org/abs/2407.16741)
- Wu et al\.\(2024\)Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang\. 2024\.AutoGen: Enabling Next\-Gen LLM Applications via Multi\-Agent Conversation\.arXiv:2308\.08155\.[https://arxiv\.org/abs/2308\.08155](https://arxiv.org/abs/2308.08155)
- Xue et al\.\(2025\)Yizhen Xue et al\.2025\.IMPROVE: Iterative Model Pipeline Refinement Via Evaluation\.
- Yang et al\.\(2024\)John Yang, Carlos E\. Jimenez, Alexander Wettig, et al\.2024\.SWE\-agent: Agent\-Computer Interfaces Enable Automated Software Engineering\.arXiv:2405\.15793\.[https://arxiv\.org/abs/2405\.15793](https://arxiv.org/abs/2405.15793)
- Zhu et al\.\(2025\)Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, Xiaoteng Ma, Xiaodong Yu, Gowtham Ramesh, Jialian Wu, Zicheng Liu, Pan Lu, James Zou, and Jiaxuan You\. 2025\.Where LLM Agents Fail and How They Can Learn From Failures\.Introduces AgentErrorTaxonomy\.arXiv:2509\.25370