A developer discovers that a mandated AI rollout for contract review fails because the human process relied on a worker's informal relationship and context, leading to poor escalations; the fix involves keeping the human in the loop as an interpretation step.
Wasnt actually my idea tho. Q1 this year, the directive came from above, we're adding AI to the contract review workflow, figure out the implementation. Not a pilot neither experiment but decision The workflow on the paper looked straightfotward, contracts came in, they get reviewed against a checklist of terms, flagged items get escalated to the legal team. I'd done more complex automations this. Scoped it in within a week or so The person who had been running contract review for 3 years had basically built a second job, found out later, like a second job inside the official one. She wasn't just checking terms, she was the relationship layer between the vendors and the legal team. so she knew which flagged items were actually worth escalating and which ones were just noice from a particular vendor and more so but none of that was in any process doc I just found it when the agent started producing escelations that legal kept pushing back on. Not wrong like just missing the read that a human would have added. The volume went up the quality of the escellations went down, after a few weeks the legal team started routing around it. Theyd ask her directly and shed handle it the old way. The technical stack was the eeasy part for this. spend around a week on the document ingestion and the contracts came in as pdfs in all kinds of formats, tried docling and llamaparse before settling on something that handles the messier vendor templates and the extraction logic or OCR was clean. The model as surfacing the right clauses and that part worked pretty neat What i underbuilt was the handoff layer, the agent was producing outputs but had no way to carry the cotext that made those outputs usable. the fix i am testing now is keeping her in the loop as the interpretation step and agent flags and extracts, she adds the one line cobtext before anything goes to legal. Slower than original pitch byt its actually getting utilized. One thing tho, caught me off guard: the workflow had no social architecture inside it that you cant see from the outside, the AI mandate assumed the process was just the process but it actually wasn't. the person running it was the process Are others running into this on mandated rollouts vs ones where the team opted in?? feels like adoption curve is completely different and i dont see ppl talking about it very much
A developer reflects on an AI automation project that achieved 95% accuracy but still required full human review, leading to a redesign that routes uncertain outputs to a review queue and saves time overall.
A developer reflects on six months of using AI for code review, finding that vague prompts produce plausible but useless feedback. The fix is treating review as a gated pipeline with explicit context, scoped passes, validation checklists, and adversarial self-critique.
Ironclad integrated OpenAI's GPT-4 into its AI Assist™ contract review tool, enabling legal teams to automate contract editing while maintaining human oversight and data privacy. The integration has been live since GPT-4's launch in April 2023.
The article discusses how the AI engineering loop can be fully automated but argues that handing over the entire loop produces 'agent slop' due to imperfect evals. It recommends automating certain steps while keeping human judgment for nuance.
Anthropic warns that human review is becoming a bottleneck as AI generates code faster than humans can review, raising concerns about agency and safety.