The article describes testing the Qwen3.8-27B AI model with different quantizations and settings to recreate images as SVG, aiming to develop a benchmark resistant to benchmaxxing. Preliminary results indicate that high reasoning effort and specific cache configurations optimize performance.
The Artist: Qwen3.8-27B-UD-Q3\K_XL, q8_0 caches, xhigh, temp 1.0, image-min-tokens 1024, froggeric template) I was screwing around with different Qwen3.8-27B quants and thought of this very simplistic but seemingly bechmaxxing resistant combined SVG and vision test. Just let the model recreate any given image as SVG with this prompt: Recreate as SVG. Pelicans can be easily benchmaxxed, recreating random photos seems a lot harder to train for. I tried a shitload of very complex prompts but the above one does the job best in my opinion. I furthermore tried different --image-min-tokens from 512 to 4096, different reasoning levels from no reasoning to xhigh, different temperatures and different kv-caches. Preliminary results are, that --image-min-tokens 1024 and --reasoning-effort xhigh with --temperature 1.0 and --cache-type-k bf16 and --cache-type-v bf16 give the best results. Non-reasoning results are, at least with the quants (Q3 and Q4) I can run, more than creepy... I also have the suspicion, that the chat template influences the output quality – please check if you are bored. Interestingly kv-caches at q8_0 gave "good" results as well but q4_0 completely destroyed the output quality (insect legs everywhere... oh the horrors I have seen), which is a great, visually impressive reminder, to never ever use q4_0 caches! Would love to see how Q6 to BF16 model quants perform with this task. If you have enough VRAM, you know what to do! ;) Used quants: - Qwen3.8-27B-UD-Q3_K_XL (V2) - Qwen3.8-27B-UD-Q4_K_XL (V2) Used templates: - built in - qwen3.8-froggeric-v22.3.1 Other prompts I tried: - Analyze thoroughly and be very detailed about perspective, composition, proportions, colors etc. Recreate as simplified but true to the original SVG - Analyze perspective, composition, colors and detail. Copy as simplified but true to the original SVG - recreate as svg. simplify but make it recognizable - Make a SVG copy - Copy as SVG - Recreate as simplified but true to the original SVG
The article benchmarks local LLMs running on 128GB Apple Silicon for generating SVG pelicans, comparing Qwen3.8 variants and DeepSeek V4 with quantizations by Unsloth.
The article evaluates how different quantizations of the Qwen3.6-27B model affect output quality using KL divergence and top-1 token accuracy, as well as visual examples like SVG drawings.
Community testers evaluate quantized versions of Qwen3.6, ZAYA1, and other models for SVG chessboard generation accuracy using local inference frameworks like MLX.
Simon Willison compares Qwen3.6-35B-A3B running locally on a MacBook Pro against Claude Opus 4.7, finding that Qwen produces better SVG illustrations of pelicans riding bicycles and flamingos on unicycles, though he notes this narrow benchmark doesn't reflect broader model capabilities.
A new benchmark compares the performance of Qwen3.8-27b, Sol 5.6, and Qwen3.6-35B models on generating an SVG of a horse on a bicycle in the desert with a camel background, using a typo-included prompt.