Jev-style logit read on a 4B open VLM, measured: http://glance.yohei.me vs the same model writing JSON: ~1/3 less time for a yes/no on a full photo up to ~85% less GPU cost with many Qs per image same accuracy on fresh photos: level with the best hosted models on pick-one (0.933 vs 0.937), ~2 pts behind Gemini on yes/no pip install glance-vlm http://github.com/yoheinakajima/glance…

Glance: reading typed visual judgements from a frozen open vision-language model

Source: https://glance.yohei.me/ glance-vlm · working paper · v0.3.1 · snapshot 8f36e06, 2026-09-21 · every experiment registered before it ran · results regenerate from the repository

Reading typed visual judgements from a frozen open vision-language model

Coarse recognition close to the best hosted models (level on pick-one, two points behind on yes/no) is already in a small open vision-language model, and it can be read without generating. What remains hard about quality ratings is where the rubric draws its lines. On geometric judgements (relative size, line direction) the best hosted models are perfect and the small open model, read this way, is far behind.

Yohei Nakajima· independent · built with AI assistance throughout (the notebook records who did what)

Summary·Use it·Method·Yes/no and pick-one·Where it stops·Read against write·Ratings·Scale and family·Cost and speed·What is not new·Limits and misses·References

The result in three regimes

Abstract

Atypedquestion is one whose legal answers form a closed set known before the model runs: yes or no, one of a list, a level on a rubric. We put such questions about images to a frozen open vision-language model (Qwen3-VL-4B, Apache-2.0, on a laptop) and read the answer from the logits of one forward pass; nothing is generated.

On photographs taken after every model’s release, labelled by people outside this project (three sets, 541 yes/no questions and 270 pick-one photographs put to every system), the open model is level with the best hosted models on pick-one (0.933 against 0.937 for Claude Opus 5) and about two points behind the best on yes/no (0.939 against 0.961 for Gemini 3.1 Flash-Lite; the paired difference excludes zero for both Gemini models). We detect no difference from Claude Opus 5 and GPT-5.6 on either, and it is ahead of Claude Haiku 4.5 and GPT-5.6 Luna on both. On the hardest of the three sets, look-alike insect orders, the hosted models spread over 23 points on pick-one (0.74 to 0.97) and the open model scores 0.96. These questions are easy and the labels are imperfect: about 3 to 5% of items are answered “wrongly” by all seven systems. Reading is as accurate as the same model writing JSON.

Ratings behave differently. On five image-quality scales, zero-shot, every system orders images correctly (the open model is within one level on 0.99 of images) and places the level boundaries wrongly, by a constant offset per rubric; a low-cost hosted model leads (0.763 against 0.669), and an 8B model is no better than a 4B one. Because a read answer is a vector of logits, it can be fitted: 16 unlabeled images of the rubric remove most of the offset (0.758) and 32 labels reach 0.857; the hosted models were not given examples, so this is a comparison of products, not of models. On a real image-quality benchmark (KADID-10k) the approach missed every target we registered, hand-built features and a small trained quality model do as well or better on low-level artefacts, and on rubrics that are not image quality (tilt, how much of a subject is cut off) even 300 labels reach only 0.55.

The open model’s cost is of the same order as the low-cost hosted models and one to two orders below the flagships; no image leaves the machine. The readout is shared with other training-free tools and is not claimed as new; what is offered is the measurement, registered before it was run, with its misses.

1Method: score the allowed answers in one forward pass

The image and a question go to a frozen model. The assistant turn is left empty (or forced to begin a JSON value), the model runs once, and only the logits of the allowed answer tokens at that position are kept. A softmax over that closed set, at temperature 1, gives the probabilities; nothing is sampled, generated or parsed.

image + typed questionyes/no, pick one, or rateone forward passfrozen open model, localanswer-position logitsallowed answers onlyprobabilitiesnothing generated

image + typed questionyes/no, pick one, or rateone forward passfrozen open model, localanswer-position logitsallowed answers onlyprobabilitiesnothing generated

Yes/noQuestion: …? Answer Yes or No\. Read: the log-sum of the logits of “Yes”, “ Yes”, “yes”, “ yes” against the same four for “No”; the probability is the sigmoid of the difference.Pick one of KQuestion: … Candidate answer: X\. Is this candidate the correct answer? Answer Yes or No\. Read: one such statement per option, all sharing one image prefix; the yes-minus-no logit of each option; a softmax over the K options.Rating, K levels"answer": How … is the image? Allowed: 0 = …; 1 = …; \(answer with the number\), with the assistant turn forced to begin\{"answer": Read: the logits of the digits 0 to K−1 at the next position; a softmax; the score is the expected level. One pass.**Table 1.**The three readouts. Images are resized to at most 768 visual tokens (28-pixel patches), which sets most of the latency. Several questions about one image reuse its key-value prefix and branch into independent suffixes, so they cannot influence each other; on 100 checked requests this changed no prediction. “Zero-shot” below means these raw probabilities, with nothing fitted.

Two fits are possible because the answer is a logit vector.Without labels: over a small pool of unlabeled images of the rubric, each level logit is centred and scaled by the pool’s mean and standard deviation before the softmax. This assumes the model’s ordering is right and removes a constant offset; it cannot move boundaries it has no evidence for.With labels: an affine map of the logits before the softmax (matrix scaling, Guo et al. 2017), L2-regularised, with its sharpness chosen on held-out folds; about 32 labels suffice. A research variant fits the same map on the model’s final hidden state (a linear probe). An earlier four-pass rating readout of ours (digits in both orders, with and without a magnified crop) is the better input for the labeled fit and is worse zero-shot; section 5 reports both.

decisive: a bold jumping spider, iNaturalist 402011619 (CC BY, antimatterbee)
  "Is the main subject a spider or other arachnid?"   P(yes) = 1.00     correct
  "What kind of organism is it?"  (10 options)        arachnid 1.00     correct

uncertain: a fungus on bark, iNaturalist 401937340 (CC BY, Марина Давлетшина)
  "What kind of organism is it?"  (10 options)        fungus 0.63, insect 0.26, arachnid 0.11     correct, and unsure
  "Is the main subject an insect?"                    P(yes) = 0.60     wrong, and close to the fence
  "Is the main subject a fungus?"                     P(yes) = 0.03     wrong, and confident

Two logged examples from the open 4B model. Most answers look like the first. The second shows both what the probabilities are for (0.63 and 0.60 are flags) and that they are not a guarantee (0.03 is a confident miss).

2Coarse yes/no and pick-one: level with the best hosted models on pick-one, two points behind on yes/no

The same items went to the open 4B model (read, and also writing its answer as JSON), to three hosted flagships and to each provider’s lowest-cost current vision model. Three tests, all zero-shot: yes/no, 541 questions about fresh photographs (three sets pooled); pick one, 270 fresh photographs (three sets pooled); rating, exact level of 4, 1,000 lab images (ratings are the subject of section 5).

yes/nopick-oneratingAccuracy · longer is betterGemini 3.1 Pro0.960.940.65Claude Opus 50.940.940.55GPT-5.60.930.900.60Claude Haiku 4.50.840.790.61GPT-5.6 Luna0.910.880.69Gemini 3.1 Flash-Lite0.960.930.76Qwen3-VL-2B, written0.580.840.06Qwen3-VL-2B + Glance0.900.910.49Qwen3-VL-4B, written0.940.930.67Qwen3-VL-4B + Glance0.940.930.67Qwen3-VL-8B + Glance0.930.930.64Seconds per answer · shorter is betterGemini 3.1 Pro2.74 s3.18 s4.02 sClaude Opus 52.64 s2.98 s2.42 sGPT-5.61.18 s1.34 s1.07 sClaude Haiku 4.50.87 s0.92 s0.72 sGPT-5.6 Luna1.22 s1.30 s1.10 sGemini 3.1 Flash-Lite1.63 s1.89 s1.62 sQwen3-VL-2B, written1.08 s0.94 s0.30 sQwen3-VL-2B + Glance0.66 s0.85 s0.22 sQwen3-VL-4B, written1.65 s1.87 s0.91 sQwen3-VL-4B + Glance1.08 s1.43 s0.45 sQwen3-VL-8B + Glance2.12 s2.65 s0.64 sUS dollars per 1,000 answers · shorter is betterGemini 3.1 Pro2\.623.856\.15Claude Opus 53.374\.48›18.70 est.GPT-5.67\.557.82›14\.96 est\.Claude Haiku 4\.51.751\.890.55GPT-5.6 Luna0\.380.400\.12Gemini 3\.1 Flash\-Lite0.310\.340.33Qwen3-VL-2B, written0\.200.170\.06Qwen3\-VL\-2B \+ Glance0.120\.160.04Qwen3-VL-4B, written0\.300.350\.17Qwen3\-VL\-4B \+ Glance0.200\.260.08Qwen3-VL-8B + Glance0\.390.49$0.12

yes/noAccuracy · longer is betterGemini 3.1 Pro0.96Claude Opus 50.94GPT-5.60.93Claude Haiku 4.50.84GPT-5.6 Luna0.91Gemini 3.1 Flash-Lite0.96Qwen3-VL-2B, written0.58Qwen3-VL-2B + Glance0.90Qwen3-VL-4B, written0.94Qwen3-VL-4B + Glance0.94Qwen3-VL-8B + Glance0.93Seconds per answer · shorter is betterGemini 3.1 Pro2.74 sClaude Opus 52.64 sGPT-5.61.18 sClaude Haiku 4.50.87 sGPT-5.6 Luna1.22 sGemini 3.1 Flash-Lite1.63 sQwen3-VL-2B, written1.08 sQwen3-VL-2B + Glance0.66 sQwen3-VL-4B, written1.65 sQwen3-VL-4B + Glance1.08 sQwen3-VL-8B + Glance2.12 sUS dollars per 1,000 answers · shorter is betterGemini 3.1 Pro2\.62Claude Opus 53.37GPT-5.67\.55Claude Haiku 4\.51.75GPT-5.6 Luna0\.38Gemini 3\.1 Flash\-Lite0.31Qwen3-VL-2B, written0\.20Qwen3\-VL\-2B \+ Glance0.12Qwen3-VL-4B, written0\.30Qwen3\-VL\-4B \+ Glance0.20Qwen3-VL-8B + Glance0\.39pick\-oneAccuracy · longer is betterGemini 3\.1 Pro0\.94Claude Opus 50\.94GPT\-5\.60\.90Claude Haiku 4\.50\.79GPT\-5\.6 Luna0\.88Gemini 3\.1 Flash\-Lite0\.93Qwen3\-VL\-2B, written0\.84Qwen3\-VL\-2B \+ Glance0\.91Qwen3\-VL\-4B, written0\.93Qwen3\-VL\-4B \+ Glance0\.93Qwen3\-VL\-8B \+ Glance0\.93Seconds per answer · shorter is betterGemini 3\.1 Pro3\.18 sClaude Opus 52\.98 sGPT\-5\.61\.34 sClaude Haiku 4\.50\.92 sGPT\-5\.6 Luna1\.30 sGemini 3\.1 Flash\-Lite1\.89 sQwen3\-VL\-2B, written0\.94 sQwen3\-VL\-2B \+ Glance0\.85 sQwen3\-VL\-4B, written1\.87 sQwen3\-VL\-4B \+ Glance1\.43 sQwen3\-VL\-8B \+ Glance2\.65 sUS dollars per 1,000 answers · shorter is betterGemini 3\.1 Pro3.85Claude Opus 54\.48GPT\-5\.67.82Claude Haiku 4.51\.89GPT\-5\.6 Luna0.40Gemini 3.1 Flash-Lite0\.34Qwen3\-VL\-2B, written0.17Qwen3-VL-2B + Glance0\.16Qwen3\-VL\-4B, written0.35Qwen3-VL-4B + Glance0\.26Qwen3\-VL\-8B \+ Glance0.49ratingAccuracy · longer is betterGemini 3.1 Pro0.65Claude Opus 50.55GPT-5.60.60Claude Haiku 4.50.61GPT-5.6 Luna0.69Gemini 3.1 Flash-Lite0.76Qwen3-VL-2B, written0.06Qwen3-VL-2B + Glance0.49Qwen3-VL-4B, written0.67Qwen3-VL-4B + Glance0.67Qwen3-VL-8B + Glance0.64Seconds per answer · shorter is betterGemini 3.1 Pro4.02 sClaude Opus 52.42 sGPT-5.61.07 sClaude Haiku 4.50.72 sGPT-5.6 Luna1.10 sGemini 3.1 Flash-Lite1.62 sQwen3-VL-2B, written0.30 sQwen3-VL-2B + Glance0.22 sQwen3-VL-4B, written0.91 sQwen3-VL-4B + Glance0.45 sQwen3-VL-8B + Glance0.64 sUS dollars per 1,000 answers · shorter is betterGemini 3.1 Pro6\.15Claude Opus 5›18.70 est.GPT-5.6›14\.96 est\.Claude Haiku 4\.50.55GPT-5.6 Luna0\.12Gemini 3\.1 Flash\-Lite0.33Qwen3-VL-2B, written0\.06Qwen3\-VL\-2B \+ Glance0.04Qwen3-VL-4B, written0\.17Qwen3\-VL\-4B \+ Glance0.08Qwen3-VL-8B + Glance$0.12

**Figure 1.**Accuracy, seconds and dollars for every system, by question type, on scales that start at zero, on the items every system answered (Table 2 gives the counts). Black bars are open models on a laptop read with Glance; the lighter bar above each is the same model writing JSON without it (2B and 4B; the 8B model’s written row was not collected). A written answer that does not parse or is not an allowed value counts as wrong, for every system; that rule costs the 2B model most of its written score (35% of its yes/no answers and 86% of its ratings are malformed, typically a missing brace). Scored leniently, taking the first allowed answer in its text, it reaches 0.91, 0.91 and 0.49: reading spares a small model the formatting; it does not make it smarter. An outlined bar is an estimated cost; an estimate beyond the measured range is cut short and marked ›.Within any one photo set every 95% interval overlaps every other, which says as much about sample size as about the systems. Pooled over the three photo sets and paired on the same items, differences appear. On yes/no (541 items) the open model scores 0.939 [0.917, 0.959]. We detect no difference from Claude Opus 5 (+0.2 points [-1.7, +2.1]) or GPT-5.6 (+1.1 points [-0.7, +3.0]); all of these intervals lie within three points, the closest this sample comes to showing equivalence. It is behind Gemini 3.1 Pro (2.0 points [0.2, 3.9]) and Gemini 3.1 Flash-Lite (2.2 points [0.6, 4.0]) and ahead of Claude Haiku 4.5 (10.0 points [7.3, 12.8]) and GPT-5.6 Luna (3.3 points [1.1, 5.7]). On pick-one (270 items) the open model scores 0.933 [0.904, 0.959]. We detect no difference from Claude Opus 5 (-0.4 points [-3.0, +2.2]), GPT-5.6 (+3.0 points [+0.0, +6.3]), Gemini 3.1 Pro (-0.4 points [-3.0, +2.2]) or Gemini 3.1 Flash-Lite (+0.0 points [-3.0, +3.0]); the intervals for Claude Opus 5, Gemini 3.1 Pro and Gemini 3.1 Flash-Lite lie within three points, the closest this sample comes to showing equivalence. It is ahead of Claude Haiku 4.5 (14.8 points [10.4, 19.3]) and GPT-5.6 Luna (5.2 points [1.5, 8.9]). Two cautions apply. The questions are coarse (is there a bridge; which of thirteen everyday things is this), so they measure a floor that all current systems clear. And the labels are not gold: on the Commons set, 4 of 131 yes/no items and 3 of 65 pick-one items are answered “wrongly” by all seven systems, which is more likely a wrong or ambiguous label than seven identical mistakes (a proxy; no human audit was done). On ratings the leader depends on what is allowed: with nothing fitted a low-cost hosted model leads; once the open model has seen a few images of the rubric it is level or ahead (section 5).

**Table 2.**The numbers behind Figure 1, on the items every system answered (yes/no and pick-one accuracy pooled over the three fresh photo sets, Table 3; seconds and dollars as measured on the Commons photographs, because hosted cost depends on image size). Accuracy with 95% bootstrap intervals; bold rows are open models (Qwen3-VL at 2B, 4B and 8B read with Glance; the rows marked “written” are the same model generating a JSON answer without it (2B and 4B)). Hosted speed is wall time per call from one laptop, network included; hosted cost is the provider’s bill where it was logged (“est.” is a list-price upper estimate). The open model was timed on the same photographs with the GPU otherwise idle; its cost is those seconds at an on-demand cloud GPU price. No few-shot prompt was tried for any written row.

**Table 3.**The three fresh photo sets pooled (Commons, iNaturalist ten groups, iNaturalist insect orders), on the items the hosted models were asked; a failed call counts as wrong. Differences are paired on the same items, with 95% intervals from a bootstrap stratified by photo set. This pooling was added after the per-set results had been seen; its rule (every set, every system, nothing dropped) was fixed before it was computed.

2.1 The photographs

Wikimedia Commons photographs taken after 15 August 2026, labelled by their uploaders’ structured “depicts” statements, and iNaturalist observations uploaded on the day of the test, labelled by community identification. No labels were made by us or by any model.

**Table 4.**The open model on all items of each photo set, hosted models on the test half; 95% bootstrap intervals, uncalibrated decisions, nothing fitted. One exception to “indistinguishable”: Claude Haiku 4.5 is below the open model on the iNaturalist set (0.830 against 0.945 on yes/no, 0.790 against 0.940 on pick-one; the intervals do not overlap). Every other hosted interval overlaps the open model’s on both sets.

0.750.800.850.900.951.00Qwen3-VL-4B + Glance · yes/no0.931Gemini 3.1 Pro · yes/no0.947Claude Opus 5 · yes/no0.924GPT-5.6 · yes/no0.893Qwen3-VL-4B + Glance · pick one of 130.885Gemini 3.1 Pro · pick one of 130.923Claude Opus 5 · pick one of 130.908GPT-5.6 · pick one of 130.892

**Figure 2.**The Commons rows of Table 4, drawn to one scale. Filled marks are the open 4B model; hollow marks are hosted frontier models. Every interval overlaps every other.On four older public benchmarks (POPE, a GQA yes/no subset, Oxford Pets, Caltech-101), which may sit in every model’s training data, the one hosted flagship we ran, Claude Opus 5, is ahead of the open model by 3.2 points on average [1.0, 5.4]. On photographs no model can have seen that gap is not there, so the fresh-photo result is not a contamination effect in the open model’s favour. Why the gap appears only on the older sets is not established: contamination in the hosted model’s favour and harder items are both possible.

2.2 A finer test

Insect orders on iNaturalist (beetle, true bug, fly, …; 210 photographs; every “no” question names a look-alike order) ask for finer distinctions than the ten-group test. This test does separate systems: on pick-one the hosted models range from 0.743 (Claude Haiku 4.5) to 0.971 (Gemini 3.1 Pro), and the open 4B model, at 0.962, is within 1 point of the best. The open dual encoder falls to 0.71 here.

**Table 5.**Seven insect orders, zero-shot, 95% bootstrap intervals. The open model answered all items (420 yes/no questions, 210 photographs); hosted models, where present, the test half (210 and 105).

3Where the small open model falls behind: geometric judgements

Six sets of 150 images drawn by program, with labels exact by construction, mark where coarse recognition ends for the open model read this way. It reads which of six look-alike words is printed on 1.00 of images and gets left, right, above and below right on 0.94. It counts one to five balls almost without error (0.99) and then degrades: 0.89 at six, 0.89 at seven, 0.56 at eight. It tells horizontal from vertical stripes perfectly (1.00) and cannot tell the two diagonal directions apart (0.30, below a coin flip: a mirror-image confusion). And it picks the largest of four like shapes on only 0.52 of images: 0.73 when the largest has twice the area of the others, 0.32 at 1.15 times (chance 0.25). We had predicted at least 0.90 on stripes and on the twofold size difference, and a steeper fall in counting; all three predictions were wrong.

These are not limits of vision-language models. On the same items Gemini 3.1 Pro and GPT-5.6 answer every question of all six sets correctly. Paired on the items every system answered, the best hosted model is ahead of the open model by 41 points [31, 52] on the largest shape and 43 points [32, 53] on stripe direction, and by 7 points [1, 13] on counting and 9 points [4, 16] on position; on the look-alike words every system is perfect. The confusion of the two diagonals is shared by Claude Opus 5 and Claude Haiku 4.5 and by no other hosted model. The low-cost hosted models fail where the open model fails: on the largest shape at the smallest ratio, and when counting seven or eight. We had predicted that most hosted models would share the diagonal confusion and that counting would be weak for every system; both predictions were wrong. One difference in the setup should be kept in mind: a hosted model writes its answer and may reason before it, while the open model gets one forward pass. Are the open model’s two large failures in the readout or in the model? A registered control says they differ in kind. On stripe direction the token read scores 0.65 and the same model writing its answer 0.75, yet a linear probe on its final hidden state, cross-validated on the same 150 images, reaches 0.99: the model separates the four directions, the two diagonals included, and no answer token gets it out. On the largest shape all three routes fail alike (0.52, 0.52, 0.48): at this size and image resolution the model does not make that judgement. The probe sees labels, so it shows that the information is present, not that it can be read without examples.

Five hundred synthetic interface screens (five kinds of page, invented content, rendered from generated HTML so every label is exact) ask what an agent would ask. Given a goal in words and six to eight numbered marks on the screen, the open model names the mark to click on 0.993 of 300 screens, and so does every hosted model (0.99 to 1.00): finding the element that serves a goal does not separate systems. One step of reasoning first (the cheaper plan, the item out of stock, the earliest date) does: the open model scores 0.90 on the shared screens and the best hosted model 1.00, 10 points [2, 20] ahead, as we had predicted. On state questions (is a dialog open, is an error shown, is the user signed in) the best hosted model is 6 points [3, 9] ahead. The hosted results also exposed three weaknesses of our test, which we report rather than remove. A disabled main button, which our screens draw as a pale tint, is reported by the open model on 0.07 of the screens that have it and by 5 of the 6 hosted models on fewer than 0.80 (only Claude Opus 5 sees it every time). On page type every hosted model scores 0.807 and the open model 0.780, because on the 67 of 300 screens where a dialog is open its backdrop hides the page; on the others every system is perfect, the open model included. And for one goal (“go to the next page of results”) the after-screen does not show that anything was done; without it the open model is at 0.92 on “is this goal already done” and the hosted models at 0.95 to 1.00. Table 6 keeps the registered numbers on all items.

**Table 6.**Exact-answer accuracy on images whose labels are exact by construction (drawn or rendered by program; no photographs, no people). Hosted models answered the test half of each set.

4Reading is as accurate as writing, and gives the same answer when the prompt is the same

The alternative to reading is to let the same model write a JSON answer. With ground truth and one question per request, the two agree. On yes/no and pick-one the accuracy is the same on both photo sets (differences of 0.0 to 0.5 points, every interval spanning zero; Table 4), and each item comes out the same way, right or wrong, on 95.4% to 99.5% of items; the few that differ reflect the prompts, which are not the same (a JSON request against one statement per option). On ratings, where the digits can be read at the very position where the written answer puts them, 98% of answers are identical: a written answer under greedy decoding is an argmax over the same logits, so this is expected.

They stop agreeing when the prompts differ. Our earlier four-pass rating readout uses different wording from the JSON prompt and scores 10 points lower zero-shot. And when one written JSON object carries 25 ratings, each field is conditioned on the fields already written, while 25 separate reads are independent: the two agree on only 59% of fields (that request has no ground truth, so this is a difference, not an error rate). Reading is therefore not a free substitute for any prompt; it is a way to take the same decision without generating it.

What reading changes is cost and form: about 1.5 times faster for one question about a full-size photograph (encoding the image dominates), 2 to 3 times on small images, more as questions per image grow (section 8 sets these ratios beside independent timings of a hosted text product of the same kind); and the answer is a probability vector, which can be thresholded, ranked and fitted.

**Table 7.**The open model writing against reading, cost per 1,000 images on a rented GPU assumed no faster than the laptop; self-hosted cost is GPU time, so the saving is the measured time saving. “Agree” is the share of answer fields on which the written and the read answer are the same. For comparison, the frontier calls on these tasks measured $0.10 to $7.64 per 1,000 answers.

5Ratings: models get the order right and the boundaries wrong; a few images of the rubric fix the offset

Scope.“Ratings” in this section means five synthetic, single-factor, four-level image-quality scales (blur, exposure, JPEG, noise, resolution) on 1,000 held-out images, the same for every system; it does not mean aesthetic judgement. The pattern in the title holds on these scales and does not hold outside them: section 5.1 has KADID-10k, rubrics that are not image quality, and the specialised tools that do as well or better.

**Order against boundaries.**Zero-shot the open model is exactly right on 0.56 of images and within one level on 0.987; 97% of its errors are one step, in a direction that is constant per rubric (half a level harsh on blur, never the worst level on JPEG). The hosted models score 0.55 to 0.76; each provider’s low-cost model beats its own flagship, and Gemini 3.1 Flash-Lite leads. Where a rubric’s author drew the lines is a convention that no model can know unseen.

0.500.600.700.800.901.00Claude Opus 5, written pick0.550GPT-5.6, written pick0.597Gemini 3.1 Pro, written pick0.650anthropic/claude-haiku-4-5, written pick0.609openai/gpt-5.6-luna, written pick0.686openrouter/google/gemini-3.1-flash-lite, written pick0.763Qwen3-VL-4B, written answer0.672Qwen3-VL-4B + Glance, one-pass read0.669… earlier four-pass read0.570… one-pass read + 16 unlabeled images0.758… four-pass read + 32 labels0.857

**Figure 3.**Exact-level accuracy on the same 1,000 images, chance 0.25. Upper group: zero-shot. Lower group: the open model after seeing images of the rubric, first unlabeled (zero labels, but not zero-shot), then 32 labeled.**Fitting the offset.**The two fits of section 1 act on this offset. The comparison is asymmetric by design: the hosted models stayed zero-shot, because a written pick offers nothing to fit and we did not give them few-shot examples. It shows what a local, fittable readout buys a user; it does not show that the open model sees better.

**Table 8.**Exact-level accuracy on the rating test of Table 2 (the last row uses the full test split and is a research result, not shipped). Unlabeled fitting roughly halves the calibration error (ECE 0.33 to about 0.2); only the labeled fit gives calibrated probabilities (ECE about 0.03). The fitted four-pass number appears on this page in four values that differ by item set and label draw, not by method: 0.857 (here) and 0.853 (Table 10) are two independent sets of twenty random 32-label draws on these 1,000 images; 0.846 (section 5.1) is the 32-label fit on the 1,500 images the outside systems were run on; 0.867 is the fit with 500 labels on the full test split.

5.1 Outside the quality scales

**A real image-quality benchmark.**On KADID-10k (23 distortion types, five levels, human scores) the fitted open model reached 0.527 exact and missed every target we had registered. Zero-shot it is at 0.35 with the one-pass read and 0.33 with the four-pass read; that check, together with the rubrics below, is what made the one-pass read the default for a rubric with nothing fitted, by a rule fixed in advance; the gain on KADID-10k was 1.9 points where we had predicted at least five.

**Specialised tools.**With plentiful labels, 29 hand-built features score 0.979 on the synthetic scales against 0.867 for the fitted model. A 0.9B model trained for image quality (Q-SiT-mini; Zhang, Wu, Jia, Lin and Zhai 2025), given the same 32-label fit on the same items, scores 0.853 against 0.846 for the 4B model: indistinguishable, at a quarter of the size. For low-level artefacts these remain the better tools; a general model read this way earns its place by answering any typed question with one set of frozen weights. The other outside system we ran, a 4B open model trained to score claims (openjev v2), is 9.7 points behind the 4B model under the same 32-label fit [7.9, 11.5]; on its own it scores 0.41 and with our fit 0.75, so the fit is a part that transfers to another system’s readout.

**Rubrics that are not image quality.**The pattern of this section does not extend to every rubric. On five synthetic rubrics that are not image quality (subject cut off by the frame, occlusion, tilt, caption legibility, watermark) the zero-shot read is exactly right on 0.38 of images and within one level on 0.74 (chance 0.25), tilt and cut-off sit at chance, and unlabeled images do not help (0.37). A labeled fit does not rescue them either: with 300 labels per rubric the four-pass read reaches 0.55 exact and 0.90 within one, where we had predicted 0.75 and 0.95. A fit removes an offset; it cannot supply a judgement the model does not make, and the weakest rubrics are the geometric ones (tilt, how much of the subject is cut off).

6Scale and family: quality belongs to the model, size above 4B buys time and cost but no accuracy

The same prompts and readouts, not a word changed, on other sizes of the same family and on a model from a different family (different vision tower, different language model).

**Table 9.**Yes/no and pick-one on the three fresh photo sets, all items, uncalibrated. Within the Qwen3-VL family the 4B and 8B models are level everywhere; the 2B model keeps up on the two easier sets and falls behind on the insect orders (pooled and paired, 3.5 points behind the 4B model on yes/no), which is why the headline chart carries all three sizes. The 2.2B model of another family is level with them on everyday photographs, trails on nature photographs (5 points on yes/no, 12 on pick-one) and falls well behind on the insect orders (0.75 and 0.65 against 0.95 and 0.96), so the comparison with hosted models in section 2 is a statement about this family, not about every small open model.

**Table 10.**Ratings, the same 1,000 images: exact-level accuracy. From 2B to 4B zero-shot accuracy rises sharply; from 4B to 8B it does not rise at all, and after 32 labels the three sizes are within 1.4 points. We had predicted a monotone rise and were wrong. “Within one” is for the four-pass read.

**Speed and cost by size.**Timed the same way at every size (one laptop, GPU otherwise idle), a read yes/no about a full-size photograph takes 0.66, 1.08, 2.12 s at 2B, 4B and 8B: each doubling of the model roughly doubles the time and the cost, and above 4B it buys no accuracy (Tables 9 and 10). Does reading save more as the model grows? It depends on what dominates. On a full-size photograph the image has to be encoded either way, so writing costs 1.6, 1.5, 1.4 times a read at the three sizes: a steady saving of about a third, not a growing one. On small images, where the answer tokens are most of the work, the saving grows with size (1.4, 2.0, 3.2 times for a rating), because every generated token costs a full pass of a larger model while a read stays one pass.

**Table 11.**Qwen3-VL writing a JSON answer against the same model read with Glance. Yes/no and pick-one on the full-size Commons photographs, ratings on 448-pixel lab images (one-pass read). Cost is the measured seconds at a rented-GPU price of $0.53 to $0.80 per hour, the same for every size; a larger model may need a dearer GPU, which is not modelled. In a 40-image benchmark the 2B model’s written answers were short and often invalid, which flatters its writing time.

7Cost and speed: same order as the low-cost hosted models, one to two orders below the flagships

The two sides of this comparison are not measured the same way, and the caveats come first. Hosted cost is the provider’s bill per call and includes nothing for an operator; hosted latency is wall time from one laptop and includes the network and the provider’s queue. Open-model cost is measured seconds on a laptop multiplied by an on-demand cloud GPU price, with no batching, no idle time and no operator counted; a different GPU price or image resolution moves it by more than the gap to the low-cost hosted models. Hosted cost also depends on image size (GPT-5.6: $7.55 per 1,000 on 1,280-pixel files, $1.86 on 500-pixel files).

With that said: a yes/no about a full-size photograph takes the open 4B model 1.1 s on a laptop and costs $0.16 to $0.24 per 1,000 on a rented GPU; the low-cost hosted models cost $0.31 and up, the flagships several dollars. The durable differences are not the cents: the image never leaves the machine, there is no per-call bill, it works offline, and the answer can be fitted.

yes/no0.750.881.000\.101102B written2B4B written4B8BGOPHLFpick\-one0\.750\.881\.000.101102B written2B4B written4B8BGOPHLFrating0.450.620.800\.101$102B written2B4B written4B8BGOPHLFUS dollars per 1,000 answers, log scale; vertical axis is accuracy

yes/no0.750.881.000\.101102B written2B4B written4B8BGOPHLFUS dollars per 1,000 answers, log scale; vertical axis is accuracypick\-one0\.750\.881\.000.101102B written2B4B written4B8BGOPHLFUS dollars per 1,000 answers, log scale; vertical axis is accuracyrating0.450.620.800\.101$102B written2B4B written4B8BGOPHLFUS dollars per 1,000 answers, log scale; vertical axis is accuracy

**Figure 4.**Accuracy against cost, one panel per question type, 95% intervals. Filled circles are Qwen3-VL read with Glance, labelled by size; the filled square is the 4B model writing JSON. Hosted models: G Gemini 3.1 Pro, O Claude Opus 5, P GPT-5.6, H Claude Haiku 4.5, L GPT-5.6 Luna, F Gemini 3.1 Flash-Lite.0.3 s1 s3 s0\.101$10Qwen3-VL-2B, writtenQwen3-VL-2B + GlanceQwen3-VL-4B, writtenQwen3-VL-4B + GlanceQwen3-VL-8B + GlanceGemini 3.1 ProClaude Opus 5GPT-5.6Claude Haiku 4.5GPT-5.6 LunaGemini 3.1 Flash-Litemedian seconds per answer, log scaleUS dollars per 1,000 answers, log scale

0.3 s1 s3 s0\.101$10Qwen3-VL-2B, writtenQwen3-VL-2B + GlanceQwen3-VL-4B, writtenQwen3-VL-4B + GlanceQwen3-VL-8B + GlanceGemini 3.1 ProClaude Opus 5GPT-5.6Claude Haiku 4.5GPT-5.6 LunaGemini 3.1 Flash-Litemedian seconds per answer, log scaleUS dollars per 1,000 answers, log scale

**Figure 5.**Cost against speed; down and left is better. Each system is a large mark at the centre (geometric mean) of three small ones, one per question type:yes/nopick-onerating. Filled marks are open models on a laptop.

8What is not new, and the work this sits in

Reading answer-token logits from a frozen generative model, with several questions sharing one image prefix, is what Simple Jev, jev-visual and LitJev also do. For yes/no and pick-one the forward pass here is not new, and the accuracy belongs to the open model. What this project adds is a harness on top: a fresh-photograph comparison with paid frontier calls, the same model writing against reading, self-calibration from unlabeled images and labeled fitting for rating levels (glance fit), and measured dollars and milliseconds. It trains no weights, unlike YOFO (Zhang et al. 2025), Laya Vision or OpenJev v2.

**Speed, in context.**Hosted Jev (TypeSafe 2026) is the trained product of this family; it takes text, not images. Its launch material quotes 40 to 200 times faster than frontier language models. Independent timings show what that figure is made of. TrueStandard (Agrahri 2026) timed one three-way classification of a support ticket at 477 ms of server time against 790 ms for Gemini 3.1 Flash Lite and 928 ms for Claude Haiku 4.5, 1.7 and 1.9 times, and reached 100 times only when one call replaced six sequential calls to a thinking model: “the multiple is a property of the comparison, not of the model”. Goedecke (2026) measured 2 to 3 times from having a small open model emit one constrained token instead of written structured output, and dorarep (2026) found that going from one question to a hundred per request cost Jev 1.5 times the latency where generating models paid 6 to 28 times. Our ratios sit in the same modest band, with the image as a fixed cost that text systems do not pay: 1.5 times against the same model writing JSON on a full-size photograph, 2.4 to 6.1 times on small images as questions per image grow, 1.5 times against Gemini 3.1 Flash-Lite, and slower than Claude Haiku 4.5 (1.08 s against 0.87 s). We did not run a sequential thinking-model workflow and claim nothing about one. None of the outside figures is our measurement, and raw milliseconds do not transfer between a hosted text model and a 4B vision model on a laptop.

The pieces are older than any of these tools. Scoring a closed set of candidate answers instead of generating is standard for language models (Kadavath et al. 2022) and its option-letter pitfalls are known (Zheng et al. 2024); VQAScore reads P(“Yes”) from a VQA model in one pass (Lin et al. 2024); a rubric score as a probability-weighted sum over rating tokens is G-Eval (Liu et al. 2023); level-token readouts for image quality are Q-Bench and Q-Align (Wu et al. 2024a, b), and frozen-CLIP quality scores are CLIP-IQA (Wang et al. 2023); the fits are matrix scaling and its relatives (Guo et al. 2017; Kull et al. 2019); the hidden-state variant is a linear probe (Alain and Bengio 2016); sharing an image prefix across questions is prefix caching (vLLM). Our additions are the contamination-controlled comparison, the read-against-write control, the unlabeled fit for ratings with the failed content-free prior as its contrast, and the registered misses.

9Limits and misses

• not supportedA content-free prior (blank and noise images) was expected to help zero-shot ratings. It took exact accuracy from 0.558 to 0.400: for an image rubric there is no content-free image.
• not supportedOur four-pass rating readout was expected to match the same model’s written answer zero-shot. It trailed it by ten points: a readout selected with a calibration in the loop is good to fit and poor zero-shot.supportedThe registered fix, one pass read at the JSON answer position, matches the written answer (0.669) and reaches 0.758 with 16 unlabeled images.
• not supportedWe expected each provider’s cheapest model to score at or below its flagship on ratings. Every one beats its flagship, and Gemini 3.1 Flash-Lite (0.763, $0.12 to $0.33 per 1,000 among the cheap models) is nine points ahead of the open model zero-shot. With 16 unlabeled images the open model reaches 0.758, 0.5 points short of it; with 32 labels it leads.
• not supportedOn KADID-10k (23 distortion types, five levels, human scores) every registered target was missed: 0.527 exact with labels; zero-shot 0.33 with the four-pass read and 0.35 with the one-pass read.
• supportedA fitted readout on the model’s hidden state reaches 0.965 from one pass, against 0.867 for the token readout: the model represents severity almost perfectly. It needs on the order of a hundred labels.
• not supportedWe expected a 0.9B model trained for image quality (Q-SiT-mini; Zhang, Wu, Jia, Lin and Zhai 2025), given our fit, to stay below the frozen 4B model on the five quality scales. On the same items with the same 32-label fit it scores 0.853 against 0.846 (difference +0.7 points [-1.1, +2.5]): indistinguishable, at a quarter of the size. A general model read this way earns its place by answering any typed question with one set of frozen weights, not by being the best quality meter.
• knownHand-built image features beat the VLM on low-level artifacts when labels are plentiful (0.979). A calibration fitted on one rubric does not transfer to another. Two model families and three sizes are measured (Tables 9 and 10); that is not “any model”.

10Reproduce

uv sync && uv run glance doctor
uv run python tools/fetch_fresh_inat.py          # 200 photographs, about ten API calls
uv run glance eval --suite inat_choice --suite inat_yesno --model vlm
uv run python tools/make_results_zeroshot.py     # every table on this page
uv run glance --model-id <any Hugging Face image-text model> --revision <commit> ask photo.jpg "Is there a dog?"   # checked on SmolVLM2-2.2B only

The lab notebook records each hypothesis before its experiment, each verdict after, and its errata. Frontier model outputs are never stored; only whether each answer was right.

11References

1. Agrahri (TrueStandard), 2026. Is Jev really 193x faster? We measured 1.7x and 100x: independent timings of hosted Jev against fast classifiers and against a thinking-model workflow.truestandard.ai/blog/is-jev-really-193x-faster
2. Alain and Bengio, 2016. Understanding intermediate layers using linear classifier probes.arxiv.org/abs/1610.01644
3. dorarep, 2026. Jev against small LLMs (in Japanese): latency as the number of questions per request grows from 1 to 100.zenn.dev/dorarep/articles/8f1efbf10e3e8c
4. Goedecke, 2026. Jev means structured output is interesting again: one constrained token against written structured output on a small open model.seangoedecke.com/jev-means-structured-output-is-
5. Guo, Pleiss, Sun and Weinberger, 2017. On calibration of modern neural networks (temperature, vector and matrix scaling).arxiv.org/abs/1706.04599
6. Kadavath et al., 2022. Language models (mostly) know what they know: reading P(True) for a proposed answer.arxiv.org/abs/2207.05221
7. Kull et al., 2019. Beyond temperature scaling: Dirichlet calibration.arxiv.org/abs/1910.12656
8. Lin et al., 2024. VQAScore: the probability of “Yes” from one forward pass of a VQA model.linzhiqiu.github.io/papers/vqascore/
9. Liu et al., 2023. G-Eval: a rubric score as the probability-weighted sum over rating tokens.arxiv.org/abs/2303.16634
10. TypeSafe, 2026. Introducing System One models and Jev (the vendor’s launch post; its figures are the vendor’s own).typesafe.ai/blog/introducing-system-one-models-a
11. Wang, Chan and Loy, 2023. CLIP-IQA: frozen CLIP with antonym prompts as an image-quality score.arxiv.org/abs/2207.12396
12. Wu et al., 2024a. Q-Bench: a softmax over “good” and “poor” logits as a zero-shot quality score.arxiv.org/abs/2309.14181
13. Wu et al., 2024b. Q-Align: fine-tuning an LMM on text-defined rating levels and reading the level tokens.proceedings.mlr.press/v235/wu24ah.html
14. Zhang et al., 2025. YOFO: fine-tuned Qwen-VL judging many yes/no requirements in one forward pass.arxiv.org/abs/2511.16600
15. Zhang, Wu, Jia, Lin and Zhai, 2025. Q-SiT: teaching LMMs for image quality scoring and interpreting (Q-SiT-mini, 0.9B, is the trained quality model we compared against).arxiv.org/abs/2503.09197
16. Zheng et al., 2024. Large language models are not robust multiple choice selectors (option-letter bias).arxiv.org/abs/2309.03882
17. vLLM project. Automatic prefix caching, including multimodal inputs.docs.vllm.ai/en/stable/design/prefix_caching/
18. Simple Jev. Any Hugging Face model as a classifier, by reading next-token logits (training-free).github.com/featherless-ai/simple-jev
19. jev-visual. The same readout on a Qwen vision-language model, with a shared multimodal prefix (training-free).github.com/hr98w/jev-visual
20. LitJev. A Jev-like typed-decision endpoint on an off-the-shelf Qwen model (training-free).github.com/zhengxuyu/litjev