A hands-on benchmark of TypeSafe's Jev 1.14 image input and new Box (bounding-box) output across 48 labeled COCO photos, finding strong yes/no accuracy (98.4%) but notable pitfalls: Box never answers "not found", confidence thresholds don't transfer between model versions, and wording strongly affects probabilities. The author recommends pairing every Box with a presence check, per-question thresholds, and pinning the model version.
TypeSafe just raised $870M for Jev, a model that returns typed answers with probabilities instead of text. With the 1.14 release it also takes images and has a new Box question that returns a bounding box. The docs call images "dreamer-ready", not production-ready, so I measured it. Setup 48 COCO val2017 photos (CC BY / CC BY-SA only), 6 targets x 8 photos: dog, cat, horse, bus, pizza, bicycle. Human-drawn label boxes. Per photo: 8 yes/no questions (the 6 targets plus person and car), one "main subject" pick, one size level, and on 1.14 a box for the target and a box for an object that is NOT in the photo. A separate request asks the subject question with picture-only options (reference crops instead of words). jev-1.13.0 and jev-1.14.0, two runs each. 408 calls total, about 0.88M input tokens, about $0.04. Results (1.13 / 1.14) 1.13 1.14 Yes/no accuracy at 0.5 98.4% Brier score 0.0105 Subject pick, word options 97.9% Subject pick, picture-only options 96.9% Size level correct 60.4% Box IoU, mean / median n/a On 1.14, 93.8% of boxes reached IoU 0.5 or better. What surprised me Box never says "not found". Ask for a cat in a dog photo and you still get a box, and about a quarter of those landed on the dog. Pairing each Box with a yes/no "is there a cat?" in the same request rejected all 96 absent-object boxes and dropped only one real one (a blurred pizza slice). Wording beats thresholds. "Is there a horse?" was uncertain on a photo with a lit horse-and-rider traffic signal. "Is there a living horse?" scored 0.03 and "Is there a sign of a horse?" scored 0.99. The one "wrong" subject pick was arguably right. The label says bus, but the biggest thing in frame is that traffic light with the horse on it. Jev said horse at 0.96. Picture-option misses were all low confidence (0.26 to 0.47), so a confidence floor catches them. Thresholds do not transfer between versions. In a simple accept / review / reject router at 0.95 and 0.05, 6 photos go to human review on 1.13 and 20 on 1.14. 1.14 is better calibrated overall but puts its true "no" answers higher. Pin the version and retune when the alias moves. Small run-to-run drift. The largest change between identical runs was 0.08, and one answer crossed 0.5 (a van read as a car on 1.14). Median latency was 848 ms / 950 ms from a laptop on home internet, so treat it as an upper bound. Takeaways if you build with it: one question per thing, pair every Box with a presence check, set thresholds per question from your own labeled set, and pin the model version. Full write-up, with an interactive explorer that replays every recorded answer (pick a photo, toggle model and question type, see Box overlays against the label box, drag a confidence threshold): https://aiengineeringfromscratch.com/blog/typesafe-870m-can-jev-see Happy to share the question wording or run other image sets if people have ideas.
Jev decision model now supports multimodal input, allowing images in both state and questions, plus a new Box type that returns YOLO-style bounding boxes for object localization. The update also supports parallel question answering for interactive vision applications, with images and Box still in experimental status.
PixelJev is introduced as a native-image decision interface using small open multimodal models to map images, instructions, and candidate sets to structured choices. The study demonstrates adaptation improves accuracy on benchmarks like Pets and highlights challenges in calibration and generalization.
ImageJevBench v0.1 is introduced as a benchmark for evaluating AI models on image decision tasks, ranking systems like Jev-Omni and decider-2b-vision based on performance, cost, and calibration metrics.
New model Wity-1, a reasoning decision model positioned between non-reasoning decision models like Jev and fully reasoning LLMs, reportedly beats Jev on all 4 baselines and ranks #1 on the Image Jev Bench, and is multi-modal. It reasons just enough until internal LLM probabilities indicate the answer is reached.
A self-run evaluation of two small confidence-scoring classification models — TypeSafe AI's hosted Jev and the open-source Laya — finds that label descriptions phrased around input features, consistent time windows, and calibration (ECE) matter more than model scale for downstream thresholding performance.