Fine-tuning AI agents via projection of solutions on the evaluated environment

tl;dr: If you've a verifiably correct solution to a task that an AI is meant to undertake - a useful approach to train the AI to perform it and others like it, is is to take this solution, and *project* it onto a map that the agent is able to traverse during inference. This allows you, the researcher, to create training data which more realistically mimic what an AI will do at inference, and not broken, teleportation, data. I have achieved surprisingly good results, with an additional accidental control case which I elaborate on below, with only \~50 training tasks. \--- I recently tried to fine-tune a small model (Gemma-31B) on a cybersecurity benchmark I own, to test whether the benchmark carries enough signal to not only *evaluate* the model, but to also *train* it to perform better on it. That might have been a trivial question to test. Let me share my approach to solving it. My goal was to not use: A larger model to distill. Reinforcing the model's solutions. Manually labeling solutions. Simply because I wanted to take *any* model, and have it improved on things that *it can't currently do*. The approach I landed on was using something I called "projections". I'm sure there exists a known technical term for it, that I am simply unaware of. In short, the benchmark consists of interactive, vulnerable, web apps, which aim to evaluate a model's ability to solve them. The benchmark also utilizes a proxy, such that every tool call, text out, and reasoning tokens, are logged and stored for research. For each of those labs, in order to make sure they are exploitable, I also automatically build a *solver*, which takes a deterministic path along the lab to traverse it, retrieve the flag, and submit it. Learning from failed attempts of simply using this solver in SFT - I *projected* it to a site-map of the lab. Meaning, I crawled the live app, then took my *solver* and built the actual traversable path that an agent may plausibly take in the path to its goal. The outcome was a success with an accidental control. In three out of four cybersecurity techniques (about 5 labs per technique, ran 20 times each), the model displayed improvement in its ability to solve held-out labs (which cover the same techniques). In the last technique, it showed a slight regression. Researching the cause of the regression, I learned that the seed for the training was unbalanced, causing the regressed technique's training examples to only show up in \~3% of the corpus. I attribute the regression to this cause. To fully prove it, I'll need to build a few more labs that require that technique, and use them for training. I did not do this yet, so take this interpretation with a grain of salt.