Typed Decision-Making Across Modalities: Jev-Omni

Author: akhilaaa3 (Independent Researcher; Apache-2.0, based on google/gemma-4-12B-it)

Another piece from the Jev line has arrived, and it’s the most solid in recent months: the first open-weight model to bring typed decision-making into the multimodal domain — queryable with text, images, audio, and video. Answers are probabilities per option, with no generated explanations.

Link: https://huggingface.co/akhilaaa3/Jev-Omni

I. What It Does

Feed it a state, a question, and options, and it returns probabilities for each option. Not a generated explanation, nor JSON output — just a direct probability distribution.

classifier = load_jev_omni()

result = classifier.predict(
    state="The meeting starts at 10 AM. It is now 9 AM.",
    question="Has the meeting started?",
    options=["Yes", "No"],
)

For media inputs, add parameters: media points to the file, modality is set to image, audio, or video. Audio is capped at 30 seconds, video uses 16 frames.

The backbone is Gemma 4 12B IT, fine-tuned once on 30,000 questions. All three primitives are present: noul (yes/no), choice, and score.

II. Results

Calibration is its strongest point: the ECE on the Medium set is only 0.0400. This number is even better than the original Jev (a comparison follows).

III. Speed (H200, post-warmup median)

IV. Two Limitations It Admits

First, the number of options is best kept under 20. While the header accepts up to 256, it explicitly states “quality with over 20 options is unverified” — a refreshingly honest note, not pretending 256 is a practical limit.

Second, it requires a CUDA GPU. The FP32 weights are ~50GB; inference uses BF16 autocast. This isn’t for a laptop.

V. Its Published Training Recipe (Rare Transparency)

The decision_config.json lays out the entire training recipe:

• Backbone: Gemma4UnifiedTextModel (hidden size 3840)
• Output classes: 256
• Fine-tuning data: 24,000 examples (1 epoch)
• LoRA: rank 512, alpha 512
• Learning rate: 1e-5 (same for head and LoRA)
• Effective batch size: 32 (microbatch 8, grad_accum 1)
• Steps: 750
• Warmup: 0.1 (75 steps)
• Seed: 3407
• Initialization: FP32 merged v1 + trained head + new LoRA
• Checkpoint path: /out/hard4k-20260919/rank128

Several details stand out: LoRA rank 512 is unusually high (typically 8-32); the “hard4k” in the checkpoint path suggests a set of challenging scenarios was used for training; and adapters_merged indicates it was built atop a previous rank-128 v1.

It also includes a verification.json for numerical validation:

• Argmax of merged model matches that of adapter: true
• Max probability difference between FP32 merged and adapter: 3.39e-05
• Probabilities are exactly identical after save and reload: true

Publishing that “merged weights are numerically equivalent to the adapter” is something few authors do.

VI. Comparison with Its Referenced Baselines

It compares against two other open-weight models, both real:

The gap is large, and it explicitly notes that “reference scores are from official reports by respective developers, and evaluation protocols may differ.” The models also cover different modality sets (none are fully four-modal), so this is not an apples-to-apples comparison, nor does the text imply it is.

VII. DecisionBench: A Dataset with a Self-Correction

This is my favorite part of the paper. The dataset page openly documents a correction to one of its charts:

In an earlier version, Jev 1.13 was run on a “per-state” basis, while Jev-Omni was run “per-question.” Batch calling amortized the cost of a state across its questions, making Jev appear roughly three times cheaper than it actually was on the medium set (average 3.7 questions/state) — and it was the only point on the cost chart measured differently from the adjacent models.

Corrected comparison (medium set):

Hard set:

This data is unfavorable for Jev: it trails leading models by ~9 points on medium and Gemini by 31 points on hard.

The author also calculates the magnitude of this gap: splitting a state into per-question calls multiplies input tokens by 2.82x (medium) / 3.09x (hard). If the three chat models were priced the same way, they would all shift right by the same factor on the chart. On a logarithmic axis, this is a uniform shift that doesn’t change the ranking and doesn’t close the ~100x gap between Jev and the frontier.

The author adds: “Please consider the absolute cost for the starred row as a lower bound.”

VIII. A Detail on the Calibration Plot

On the reliability diagram (confidence vs. actual accuracy), the author annotates Luna’s behavior:

Luna’s confidence on the medium set is self-reported to two decimal places — 227 out of 293 answers said “99%.” They all fell into the 80–100% bin, so Luna appears as a single point on the chart, not a curve.

This annotation is well-placed: it explains why the chart looks that way, preventing the reader from mistakenly assuming the model is perfect.

IX. Why This Post Stands Alone

Because it’s the first on the Jev line to publish a disclaimer of possible error.

Compare with previous items we’ve written:

• TypeSafe’s official Jev release had no technical paper.
• The author who claimed to have “done this a year ago” initially stated “what I built is the Jev architecture” (we assessed this as exaggerated).
• This release, however, publishes the full training recipe, numerical verification, the true boundary of the option limit, and even corrects a chart that made its own model look better.

It also clearly states: “Jev-Omni is an independent open model implementing a typed decision interface, with no affiliation, endorsement, or sponsorship from TypeSafe AI or its Jev model, not derived from it, and trained on none of its outputs.”

X. My Assessment

Technically, its value lies in proving that “typed decision-making” can scale to other modalities: 26ms for images, 31ms for audio — providing probabilities directly without generating anything. This is a new path for pipelines requiring “look at an image/video to make a decision.”

Its ECE of 0.0400 is also worth noting — slightly worse than Jev 1.13’s 0.0324 on medium, but much better than Sonnet 5’s 0.1116. In other words, on the open-source side, calibration levels can already match or exceed some closed-source frontier models.

But it doesn’t change the fundamental landscape. A 12B model versus 100x larger models shows a 30-point gap on the hard set — model scale remains a hard constraint. Its significance is that “this path works, and it can be open-sourced,” not that “open source has caught up to the frontier.”

Finally: This model has 69 likes, was released two days ago, and is the author’s only model. Using our earlier standard that “favorites ≠ quality” — its value is in methodological transparency, not hype.

Links

• Model Page: https://huggingface.co/akhilaaa3/Jev-Omni
• Benchmark Dataset: https://huggingface.co/datasets/akhilaaa3/decision-bench
• Base Model: https://huggingface.co/google/gemma-4-12B-it

#TypedDecisionMaking #Multimodal #OpenSourceModel