Automating LLM Fine-Tuning with Fireworks Agent

From Context Window to Weights

Andrej Karpathy (@karpathy) recently described the personal LLM Wiki as a kind of pre-AGI memory aid, a curated repo of notes about papers, tools, and ideas you read into context when you want a model to reason over them.

In his viral post, Karpathy flagged the obvious next move: “As the repo grows, the natural desire is to also think about synthetic data generation + finetuning to have your LLM ‘know’ the data in its weights instead of just context windows.”

Building LLM Knowledge Bases or LLM Wikis is already possible with agents like Claude Code or Codex, but this approach can quickly get inefficient and expensive as you try to scale them. Fine-tuning LLMs to maintain your knowledge bases is often a more efficient path forward.

This post takes that next step by putting the wiki’s output style into the weights. In under ten minutes of GPU time and a couple of cents of compute, a small open-weight model writes summaries of new papers in the exact format the wiki uses, with no system-prompt gymnastics, no few-shot exemplars, and no router logic. Once deployed, the summary comes back in a single fast call, fast enough to use inline inside a larger agent loop rather than as a batch job. The harder version (parametric knowledge injection of the wiki’s contents) is the natural follow-up to Karpathy’s framing, and I treat it as future work at the end.

The interesting part is not the model itself, but that one @FireworksAI_HQ Agent session did the entire pipeline (dataset inspection, hyperparameter sweep, full training, deployment, and a working inference endpoint). Fireworks Agent is the autonomous orchestration layer for fine-tuning runs, where you give it a natural-language goal, and it plans, executes, and surfaces decision gates back to you. The whole flow can be driven from a coding agent you already use (Claude Code, Codex, or similar), which is how I ran it.

The bigger picture this points to is self-improving LLMs and agents. Once training is a callable step inside an agent loop, the same coding agent that drives your workflow can also kick off fine-tuning runs to bake recurring patterns (a wiki’s voice, a coding style, a triage policy) into the model itself, closing the loop between using a model and improving it.

The rest of this post is the full walkthrough.

All resources from this run are available in a companion repo, including the training and validation splits (train.jsonl, val.jsonl, wiki-sft-2026.jsonl), the data-build scripts (parse_2026.py, fetch_abstracts.py, build_jsonl.py), the pilot-agent.md slash command, the smoke-test script (test_new_deployment.py), and the baseline-vs-fine-tuned comparison code (before_after.py). Grab it at github.com/dair-ai/wiki-sft, clone it, point it at your own corpus, and reproduce the run end to end.

Why Output Style Is the Right First SFT Target

For a personal wiki, the high-leverage thing is consistency. Readers recognize a summary by its shape, which is a one-paragraph lede that names the authors’ affiliation and the core contribution, followed by three to five bulleted takeaways with bolded short labels. A capable base model can be coaxed into this format with a careful system prompt, but the failure modes are familiar. It reverts to title-case headers, drops the affiliation line, varies bullet count, and sneaks in marketing language.

Supervised fine-tuning (SFT) fixes this at the parameter level. Once the format is in the weights, every generation conforms by default, and the system prompt collapses to a single sentence (or drops out entirely). The cost stays small when the dataset stays small, and a clean stylistic dataset of 50 to 100 examples is usually enough to get started.

Handing the Work to an Agent

Most fine-tuning tutorials walk you through ten distinct steps. You format your data, upload it, choose a base model, decide on LoRA rank and learning rate, launch a job, parse logs, pick a winner, retrain on full data, deploy, and smoke test. Each step is its own surface to mess up, and you end up playing the role of a tuning agent yourself.

Fireworks Agent inverts this. The interface is firectl session create -n “”, where firectl is the Fireworks CLI. After that, you watch events stream and respond to gates when the agent surfaces a decision, such as the proposed plan or the hyperparameter (HP) sweep results.

Fireworks also ships a Claude Code slash command (or you can format it as an agent skill), pilot-agent.md (previously known as Pilot Agent), that wraps the firectl commands and handles event streaming, gate detection, and resume-from-last-timestamp logic.

Full Walkthrough

Step 0: Setup

Install the Fireworks CLI and confirm your account.

bashbrew install fw-ai/firectl/firectl firectl signin firectl account get

In the Fireworks dashboard, create a service account that has the permissions Training Agent needs (the role that lets it launch training jobs and deployments on your behalf), then generate an API key tied to that service account. Also, create a separate user-level API key for inference and deployment inspection. Drop both into a .env file next to the project.

plaintextPI_API_KEY= FIREWORKS_API_KEY=

Step 1: Build the Dataset

The training data I use consists of chat-format records derived from the DAIR.AI Top AI Papers of the Week wiki, drawn from the top 5 papers per week in 2026 and paired with their arXiv abstracts. Three small Python scripts handle the pipeline, namely parse_2026.py (wiki to structured entries), fetch_abstracts.py (arXiv abstract lookup), and build_jsonl.py (chat-format assembly). The chat schema is the standard Fireworks shape:

json{ “messages”: [ {“role”: “system”, “content”: “You write concise, structured summaries of AI research papers in the style of DAIR.AI’s Top AI Papers of the Week. Open with a one-paragraph lede that names the authors’ affiliation and the core contribution, then follow with three to five bulleted takeaways. Each bullet begins with a bolded short label, a colon, then a crisp explanation. Keep the tone analytical, specific, and free of hype.”}, {“role”: “user”, “content”: “Paper: