@sadhikesaven: Today, LLMs are no longer built from human data alone. They rely on other LLMs to generate training data, filter corpor…
Summary
ModSleuth is a new tool that traces the dependencies of modern LLMs, revealing that models like OLMo 3 and Nemotron 3 rely on hundreds of other models and datasets, highlighting the shift from human-only to AI-generated training data.
View Cached Full Text
Cached at: 06/12/26, 02:51 AM
Today, LLMs are no longer built from human data alone. They rely on other LLMs to generate training data, filter corpora, evaluate outputs, provide rewards, and guide development decisions. So how many models and datasets is a modern LLM built on?
• OLMo 3 → 89 model + 183 dataset dependencies • Nemotron 3 → 273 model + 560 dataset dependencies How did we find it out? We built ModSleuth.
Model-to-model influence is now so diverse, complex, and recursive that it far outpaces humans’ ability to trace.
So we built ModSleuth: an agentic system that automatically reconstructs a model’s dependency graph.
It reads papers, model cards, dataset cards, code, configs, and upstream artifacts, then pieces together a model’s “family tree.”
Some dependency chains go 8 hops deep.
One surprising lesson:
With Claude Code (which ModSleuth is built on), information extraction is no longer the main bottleneck.
The hard part is semantic and representational: • What actually counts as a dependency? • When do different names refer to the same artifact? • How do you reconcile versions, model families, development stages, and repositories?
The challenge is no longer finding information—it’s making sense of it.
Check out the paper to see how we tackle these problems.
Across 4 open-source releases, ModSleuth recovers 1,060 source-verified dependencies, with chains up to 8 hops deep. This graph also surfaces findings that are hard to find manually: • License-relevant multi-hop paths • Train-evaluation coupling • Mismatches between papers, cards, and code
Demo: https://modsleuth.cal-data-audit.org Code: https://github.com/cal-data-audit/modsleuth… Paper: https://arxiv.org/abs/2606.12385
with amazing collaborators @CoderBak @sewon__min !!
Similar Articles
Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs
Introduces ModSleuth, an agentic system that recursively reconstructs large-scale dependency graphs for LLM development by analyzing public artifacts, revealing multi-hop license obligations and documentation inconsistencies.
We’ve been analyzing how people are using LLMs for legal and compliance tasks (GDPR, AI Act, etc.).
Analysis of LLM usage in legal and compliance tasks reveals that models often produce confident but unverifiable citations, raising questions about reliable legal grounding for AI outputs.
@neural_avb: If you think about it, LLM training in 2026 is really a 3-step loop : - train it on some data - dogfood it/run categori…
The tweet outlines a 3-step loop for LLM training in 2026: train on data, run evals, and add synthetic data for underperforming tasks. It emphasizes the accessibility of legal distillation via open source models and cheap APIs, noting that training on reasoning traces alone can achieve high scores.
LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs
PropMe is a propensity-aware framework for evaluating LLM memorization, distinguishing between forced reproduction capabilities and natural propensity using SimpleTrace for deterministic attribution across open models and datasets.
Making a vintage LLM from scratch
The author documents their journey of building a 340M parameter LLM from scratch, trained exclusively on pre-1900 texts, including custom datasets, training scripts, and open-sourcing the model and code.