Jeff: Compressing a Jev-like Decision Model to 0.8B, Running Locally

Author: Mathias Strasser (Scissero)

Let’s start by clarifying what this is—and what it isn’t.

Jeff is a fine-tuned version of Qwen3.5 and Gemma 4, designed for zero-shot classification. Think of it as a small decision model you embed into your code, with the same request format as Jev: you describe a situation and list the options in plain language, and it returns a calibrated probability for each option in a single forward pass.

No generated text, no parsing required.

And it runs locally: about 22ms per decision on an RTX PRO 6000, and 28ms on an Apple M4 Max (using MLX).

“Zero-shot” means the options can be anything: customer service queues, user intents, moderation labels, voice commands, game actions. Your categories don’t need to be in the training data—you describe them, and Jeff chooses.

But it’s not a replacement for Jev. The author states this plainly:

These are very small models. They make extremely fast, well-calibrated judgments between options, and are easy to embed in your local code. On benchmarks, they approach—and sometimes surpass—Jev; but at this size, their reasoning abilities cannot match Jev, which runs on a much larger model.

If zero-shot accuracy isn’t sufficient for your needs, a short fine-tuning run with your own samples can take you much further: our voice navigation fine-tuning improved held-out accuracy from 31.7% to 95.8% in under half an hour on a single GPU.

I. Benchmarks: 0.8B Matches Jev’s Overall Score

4,599 questions from five public benchmarks, plus JevBench’s public hard split (105 questions, scored separately).

A few ways to read this:

1. In overall score, Jeff-2B (83.1) nearly matches Jev (83.0), despite having only 2 billion parameters. However, the author immediately notes: the published Jev numbers were tested on different samples from the same benchmarks.
2. On classification and grounding tasks, the smaller models lead: Financial PhraseBank 96.4 vs 77.0, RAGTruth 86.1 vs 77.3. The author explains: “Jeff’s overall score comes from classification and grounding, where it matches or exceeds the larger model.”
3. On reasoning-intensive benchmarks, it clearly lags: BBH 64.0 vs 94.3, JevBench Hard Split 47.6 vs 73.3. The author says, “As you’d expect at this size”—no sugarcoating.

II. Zero-Shot Test: Jeff Plays Three Games

The author’s testing method is instructive: games aren’t ideal zero-shot tests because game states aren’t typical unstructured data; but they’re a common and interesting way to test System 1 models.

Each turn, the code describes the situation and legal moves in text; the model picks one. The options explain what each action leads to (Frogger: “You will be hit by a car and lose a life”; Doom: “The nearest monster is slightly to your left”), but never which action is correct.

Each result is over 20 games, seed 1234.

The most telling comparison:

Jev’s published Doom score is 6.55—but it was told the aiming rules. Without them, it scored −0.60.

Jeff-0.8B scored 6.55 without being told any rules. It equals the hand-coded bot’s performance.

Speed comparison: Jeff-0.8B runs at 29–49ms per action on M4 Max; Jev’s published Doom run took 114–212ms per call (including network). The author notes: “These two times were not measured on the same hardware.”

III. Fine-Tuning Example: Chess

When zero-shot isn’t enough, fine-tune. They used 600k Lichess positions labeled by Stockfish, training on a single GPU for about 3.5 hours.

On 1,000 held-out puzzles:

• Qwen3.5-0.8B (untrained): 6.2%
• Jeff-0.8B zero-shot (never trained on chess): 15.5%
• Jeff-0.8B-Chess: 55.8%

The author’s qualitative assessment of this model is honest:

It’s not a strong player: ~1000 Elo without search, and it loses to Stockfish’s weakest setting. The point is speed. Each move is a tens-of-milliseconds forward pass, so one GPU can keep up with about 600 human blitz games simultaneously.

They included a video of 100 simultaneous games—where the only win was a nine-move checkmate.

IV. Speed and Size

Note the 0.8B’s 1.7 GB and 28ms on M4 Max—this is small enough to fit into a laptop.

V. Five Usage Recommendations (I Think These Are the Most Valuable)

1. Use code for reasoning, use Jeff for decisions. It’s a classifier, not a planner. Spell out what each option leads to (“This move will get you hit by a car”); ask it to predict (“The car will arrive in two moves”), and it performs no better than random.
2. Wording matters hugely. Describe options consistently: giving the Frogger target option the same phrasing as all other forward options increased river crossings per game from 15 to 23.
3. Use short option keys + descriptive text: {"1": "Engagement letter"}. Avoid long IDs—they waste time and help no one.
4. Group independent questions into a single request.
5. Fine-tune if zero-shot isn’t enough. A voice navigation fine-tune with ~11k domain-specific samples took about half an hour on a single GPU, improving held-out accuracy from 31.7% to 95.8%, with decisions taking ~40ms on M4 Max.
6. Pick size by task. For fast option picking, 0.8B is the sweet spot: 2B is more cautious and actually plays games worse, despite higher benchmark scores.

VI. Six Caveats (All Preserved)

The author lists these, and I think they’re more valuable than the benchmark numbers:

1. Max 26 options per query, for now. Options are encoded A–Z, then AA, AB. But the biggest issue in training was 19 options, so the released model never learned to pick double-letter codes: options 27 and beyond are effectively never selected, regardless of what it outputs. The server rejects queries with more than 26 options; longer lists should be filtered down first. (This came from an issue report—#1 by puhuk—I verified it, and it’s still open.)
2. Small models don’t reason. What you can expect are fast, calibrated choices between the options you describe, not multi-step inference. This holds true for every model at the 0.8B–2B size, not just Jeff.
3. Jeff-2B’s game performance is worse than Jeff-0.8B’s. The untrained 2B already seemed more risk-averse than the untrained 0.8B, and our training appears to have made this worse. This needs more investigation.
4. Benchmark scores don’t predict game performance. Untrained Gemma 4 E2B beat untrained Qwen models on benchmarks but played games the worst: correct most of the time, but unreliable.
5. Prompts matter. Jev’s own Doom prompt (raw bearing numbers plus one aiming rule) didn’t work for any of our models; options that explained consequences did.
6. Only supports English and text.

VII. Sources and Origins

Code is MIT, model weights are Apache 2.0.

Jeff started as a fork of AutoJev—a public recipe from Denis Yarats (Perplexity) that fine-tunes Qwen3.8-27B to return Jev-style decisions. They kept its core design (one forward pass per decision, trained answer readout, fitted-temperature calibration), then built on it: small “student” models (0.8B/2B Qwen, Gemma 4 E2B), a local synthetic data pipeline with leakage filtering, domain-fine-tuned prompt layouts, MLX serving on Apple chips, game testing, and training dashboards.

The training environment is worth noting:

Built entirely on local hardware. Trained on an RTX PRO 6000 workstation GPU (~2 hours for 0.8B, ~3.5 hours for 2B). All synthetic training data was written by open models (Qwen3.8-Flash-Next) on two DGX Spark systems, and testing was done on a MacBook. No cloud GPUs, no closed-model outputs in the training data; closed models were only used to spot-check a subset of synthetic data quality.

And that necessary disclaimer:

Independent project. Jeff uses the same request format as Jev but is not affiliated with or endorsed by TypeSafe (Jev’s developer).

VIII. My Assessment

This is the most tangible open-source result to date on the Jev line.

Three reasons:

1. The size breakthrough is real. 0.8B / 1.7 GB / 28ms (M4 Max)—this scale means decision models can fit into anything: local apps, edge devices, batch processing on a laptop. And its overall benchmark score (79.1) is close to Jev’s published 83.0.
2. That Doom comparison is sharp. Jev published 6.55 when “told the aiming rules”; without them, it scored −0.60. Jeff-0.8B scored 6.55 without being told any rules. This shows that with a “describe consequences in natural language” prompting style, small models can match large ones—and that prompting style is exactly what you’d use in practice.
3. The author listed six caveats, and each one is specific. Especially that 26-option limit—it was discovered via a user-submitted issue, and the author didn’t hide that “our training made 2B’s game performance worse.”

The honest boundaries:

• Published Jev and AutoJev numbers were tested on different samples from the same benchmarks—not a controlled comparison.
• Jeff clearly lags on reasoning-intensive benchmarks (BBH, JudgeBench, JevBench)—this is dictated by size, and the author doesn’t avoid it.
• Only supports English and text.
• The author is an independent developer (Mathias Strasser, Scissero), the repo was created on 2026-09-28, has 918 stars but only one day of activity.
• Training data isn’t released (weights and code are), and some sources are CC BY-SA share-alike.

In short: If you want “tens of milliseconds, local, calibrated judgment between options you describe,” Jeff is ready to use now; if you want reasoning, it can’t provide it, and it doesn’t pretend it can.

Links

Repo: https://github.com/firelex/jeff

Jeff-Qwen3.5-0.8B: https://huggingface.co/mstrasser/Jeff-Qwen3.5-0.8B

Jeff-Qwen3.5-2B: https://huggingface.co/mstrasser/Jeff-Qwen3.5-2B

Chess Fine-tune: https://huggingface.co/mstrasser/Jeff-Qwen3.5-0.8B-Chess

Upstream AutoJev Recipe: https://github.com/denis-pplx/autojev

#Jev #LocalAI #OpenSource