I had Gemini train its own replacement for $9

# I had Gemini train its own replacement for $9 Source: [https://www.petervijeh.com/projects/reddit-ner](https://www.petervijeh.com/projects/reddit-ner) I like to cook, and somewhere along the way that turned into an obsession with high\-end chef's knives\. So I scrape the Reddit threads where people argue about them and pull out every brand, model and steel they mention, to see what is getting bought and argued about\. Picking product names out of text is a job called named\-entity recognition, and small models have done it for a decade\. I was doing it with Gemini 3\.1 Pro, one paid API call per comment\. Overkill, but it worked: from "picked up a Mazaki in white \#2, way better than my old Fibrox" it returned Mazaki as a brand, Fibrox as a model and white \#2 as a steel, and nothing else\. But the scraper pulls every new comment, so the bill grew with how much people posted, and the only way to cap it was to skip comments\. The obvious replacement, an open NER model called GLiNER run zero\-shot, cut the cost to nothing and the accuracy to about 0\.65 F1 against Gemini's answers\. That gap is what the rest of this is about: could Gemini label 4,290 comments once and teach GLiNER to close it? - **What:**Fine\-tuned GLiNER large v2\.5 \(459M\) to tag brands, models and materials in Reddit comments, on labels Gemini 3\.1 Pro wrote once\. - **Why:**Zero\-shot GLiNER scored about 0\.65 F1 \(est\.\)\. Gemini scored well and billed every comment for as long as the scraper ran\. - **Approach:**Ask Gemini for strings, not offsets\. Compute offsets in code\. Add comments with no products in them as negatives\. Lock a 225\-comment validation set before the second run\. - **Problems:**Five of ten runs produced no usable model\. Three failed on configuration\. Two failed on a tensor called words\_mask that I filled the way you fill an attention mask\. - **Result:**0\.83 F1 against Gemini's labels after 24 minutes on a Tesla T4\. $9 of labels, about $2\.50 of GPU time, and days of debugging\. ## What I set out to do The plan had three steps\. Have Gemini label a few thousand Reddit comments once, marking every brand, model and steel\. Train GLiNER on those labels\. Then run GLiNER on my own machine for every comment after that, and stop calling Gemini\. Gemini labeled 4,290 comments for $9, or $0\.0021 a comment\. That means the trained model pays for itself at roughly comment 4,291, as long as later comments are about the same length and it runs on a GPU I already own\. The test of success was simple: on 225 comments the model had never seen, how often does it tag the same words Gemini tagged? One catch to keep in mind for every score in this article\. Nobody checked Gemini's labels by hand, so the model is graded against Gemini, not against the truth\. Where Gemini was wrong, the model gets marked right for copying the mistake and wrong for fixing it\. ## The approach Gemini labeled the comments through OpenRouter at temperature 0 in 25 minutes\. The prompt decision that mattered most was to never ask the model for character offsets\. It counts characters badly and returns spans off by two or three positions\. The prompt asks for the exact substring and a label, and TypeScript finds the offsets\. If the string is not in the comment, the entity is dropped and logged\. ``` // The model returns strings. Code computes the offsets. { "entities": [ { "text": "Benchmade", "label": "knife brand" }, { "text": "940", "label": "knife model" }, { "text": "S30V", "label": "knife steel" } ] } ``` Product names are full of punctuation a generic tokenizer splits, so a regex keeps VG\-10, CPM\-154 and 1\.4116 whole and emits every other non\-space character as its own token\. Spans that still miss a token boundary are dropped rather than guessed\. About 30% of the training set is comments that contain a known false\-positive trigger \(gyuto, carbon, handle, patina\) and no product, labeled as empty\. Before the second run I set aside 225 comments as a validation set and never touched them again\. Training ran on a Tesla T4 on Modal with the HF Trainer\. ``` per_device_train_batch_size = 2 gradient_accumulation_steps = 8 learning_rate = 1e-5 threshold = 0.45 ``` What the model trained onWhat the model trained on4,290 forum comments annotated by an LLM for $91,575 positive \(69\.6%\)675 negative2,250 training examples2,029 train225 valsplit \(validation set locked from run 10 on\)3,907 entity spansbrand1,720product model1,345material spec842Roughly 30% of examples deliberately contain no entities at all\. ## What went wrong For five runs the model learned nothing\. The first three failed on configuration, and anyone using the HF Trainer with GLiNER will hit them in an afternoon\. RunWhat went wrongFix1GLiNER's default max\_steps=10000 overrode num\_train\_epochs=3; trained 39 epochsSet max\_steps explicitly2load\_best\_model\_at\_end without eval\_strategy throwsSet eval\_strategy="steps"3Trainer saved state\-dict keys without the "model\." prefix GLiNER's loader expectsPut the prefix back on save4ner\_labels missing on negative examplesSet the label list on every example4–5words\_mask built as binary; loss flat at 70–130Emit incremental word indices Runs 4 and 5 were the expensive ones\. GLiNER's tokenize\_inputs crashed on broken Reddit emoji, so I had patched it, and the patch has to fill a tensor called words\_mask\. It sits next to attention\_mask, has the same shape, and every attention mask I have ever built is ones for real tokens and zeros for padding\. I built it that way\. Nothing about the name or the shape says it is anything other than an attention mask\. Training ran to completion\. Loss started around 130, drifted to about 70 and stayed there\. No crash, no warning, no NaN, gradients of ordinary size, checkpoints saved on schedule, eval F1 near zero\. I blamed the label list first, because run 4 also had negatives with no labels set\. Fixing that and rerunning gave the same flat loss\. The only thing wrong with run 5 was words\_mask, a tensor I had never looked at\. Ten training runsTen training runsThe first five produced no usable model — all plumbing, no modeling0\.00\.20\.40\.60\.81\.0failedscoredbest F1in production✕R1step cap✕R2config✕R3checkpoint✕R4word mask✕R5word masknotscoredR6first success0\.800R7tuned 209M0\.879R8459M0\.799R9too many negs0\.832R10locked valRuns 1–5 failed outright\. The dashed line tracks overall F1 after the word\-mask fix\. ## How I fixed it I read GLiNER's training loop instead of its docstrings\. words\_mask is not a mask\. It is a word index: 0 for special, prompt and padding tokens, then 1, 2, 3 for the first sub\-token of each real word\. The span\-scoring head uses it to pool sub\-tokens back into words\. Filled with ones it says the whole comment is a single word, so the model is asked to find brand and material spans inside one enormous token\. It cannot, so the loss stays flat, and a flat loss does not show which input is wrong\. ``` # what I wrote # what GLiNER expects words_mask = [1,1,1,1,1] words_mask = [0,1,2,2,3] # [CLS] Mazaki wh ##ite #2 ``` With the index fixed, run 6 learned on the first try\. The rest was tuning against the locked set\. The 209M medium model reached 0\.800; the 459M large model, which fits a T4 only with gradient accumulation, reached 0\.83\. Ten times more adversarial negatives \(510 instead of 51\) dropped F1 to 0\.799, so run 10 went back to 51\. One threshold per class instead of a global cutoff took material recall from 0\.787 to 0\.911, because steel names like MagnaCut, S35VN and HAP40 score lower confidence than brands and a single cutoff dropped them\. Every large run bottoms out at epoch 2 and overfits after; with 2,000 examples that is a dataset\-size problem, and early stopping is the fix\. F1 by entity classF1 by entity classZero\-shot baseline vs\. fine\-tuned checkpoints, same held\-out dataZero\-shot \(est\.\)Fine\-tuned 209MFine\-tuned 459M0\.00\.20\.40\.60\.81\.0~0\.650\.8000\.879Overalln/a0\.8580\.904Brandn/a0\.7750\.877Productn/a0\.7120\.829SpecBigger encoder helps most where the vocabulary is purely domain\-specific\. ## What I learned It worked\. The model runs locally, matches Gemini's labels at 0\.83 F1 on comments it never saw, and knows that "carbon steel" is a category rather than a steel and that PM2 sometimes means the Spyderco Paramilitary 2 and sometimes is just letters\. One earlier run scored 0\.879 on a random split\. I do not count it as the result: on random splits, two drops in F1 that I had blamed on my changes turned out to come from which comments landed in the validation set\. On paper the project cost less than lunch: $9 of labels, $2\.50 of GPU\. What it cost me was the days spent on a tensor that passed every check the code had and was still wrong\. I think the lost days are the normal case for small fine\-tuning jobs\. The model and the data are rarely the problem; the code between them fails, and a flat loss from a wrong input tensor looks the same as a flat loss from hard data\. If I had to choose between a better label set and an assertion on every tensor I hand\-build, I would take the assertion\. This model powers New Knife Day, which tracks what knife people on Reddit are buying and arguing about\. The knife\-side write\-up there has the full run log\. Both are linked below\. ## At a glance ProblemFind the brand, model and material names in Reddit comments, and skip the generic words around them, without paying an LLM per commentApproachHave Gemini label the comments once, then fine\-tune GLiNER large v2\.5 \(a DeBERTa\-v3\-large encoder\) on those labels and run it locallyResult0\.83 F1 on a 225\-comment validation set fixed before training; material recall 0\.911 with a per\-class thresholdCost$9 in Gemini labels plus about $2\.50 of T4 time across ten runsStackTypeScript and MongoDB for the scraper and labels, Python and PyTorch for training on Modal, FastAPI to serve the model