@FeitengLi: OpenBMB open-sources MiniCPM-V 4.6, 1.3B parameters (SigLIP2-400M + Qwen3.5-0.8B), 262k context, visual encoding FLOPs 50%+ less than previous generation. Token cost for the same task is lower than Qwen3.5-0…

OpenBMB has open-sourced MiniCPM-V 4.6, 1.3B parameters (SigLIP2-400M + Qwen3.5-0.8B), 262k context, visual encoding FLOPs 50%+ less than the previous generation. Token cost on the same tasks is 19x lower than Qwen3.5-0.8B, and 43x lower with thinking mode enabled. OpenCompass / OCRBench / HallusionBench / MUIRBench all reach the level of Qwen3.5-2B. OCRBench 876, MMMU-Pro 38%. Edge adaptation code for iPhone 17 Pro Max, Redmi K70, and Huawei nova 14 is fully open-sourced, with real device screen recordings. Inference fully supported on vLLM / SGLang / llama.cpp / Ollama / transformers. https://huggingface.co/openbmb/MiniCPM-V-4.6… — # openbmb/MiniCPM-V-4.6 · Hugging Face Source: https://huggingface.co/openbmb/MiniCPM-V-4.6 A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone GitHub (https://github.com/OpenBMB/MiniCPM-o)|CookBook (https://github.com/OpenSQZ/MiniCPM-V-CookBook)|Demo (https://huggingface.co/spaces/openbmb/MiniCPM-V-4.6-Demo)|Feishu (Lark) (https://raw.githubusercontent.com/openbmb/MiniCPM-V/main/assets/feishu_qrcode.png) MiniCPM-V 4.6is our most edge-deployment-friendly model to date. The model is built based on SigLIP2-400M and the Qwen3.5-0.8B LLM. It inherits the strong single-image, multi-image, and video understanding capabilities of MiniCPM-V family, while significantly improving computation efficiency. It also introduces mixed 4x/16x visual token compression. Notable features of MiniCPM-V 4.6 include: - 🔥Leading Foundation Capability.MiniCPM-V 4.6 scores 13 on the Artificial Analysis Intelligence Index benchmark, outperforming Qwen3.5-0.8B’s score of 10 with 19x fewer token cost, and Qwen3.5-0.8B-Thinking’s score of 11 with 43x fewer token cost. It also surpasses the larger Ministral 3 3B (score of 11). - 💪Strong Multimodal Capability.MiniCPM-V 4.6 outperforms Qwen3.5-0.8B on most vision-language understanding tasks, and reaches Qwen3.5 2B-level capability on many benchmarks including OpenCompass, RefCOCO, HallusionBench, MUIRBench, and OCRBench. - 🚀Ultra-Efficient Architecture.Based on the latest technique inLLaVA-UHD v4 (https://github.com/THUMAI-Lab/LLaVA-UHD-v4), MiniCPM-V 4.6 reduces the visual encoding computation FLOPs by more than 50%. It enables MiniCPM-V 4.6 to achieve better efficiency to even smaller models, achieving ~1.5x token throughput compared to Qwen3.5-0.8B. It also supports mixed 4x/16x visual token compression rate, allowing flexible switching between accuracy and speed. - 📱Broad Mobile Platform Coverage.MiniCPM-V 4.6 can be deployed across all three mainstream mobile platforms — iOS, Android, and HarmonyOS. With every edge adaptation code open-sourced, developers can reproduce the on-device experience injust a few steps (https://huggingface.co/openbmb/MiniCPM-V-4.6#deploy-minicpm-v-46-on-ios-android-and-harmonyos-platforms). - 🛠️Developer Friendly.MiniCPM-V 4.6 is adapted toinference frameworks (https://huggingface.co/openbmb/MiniCPM-V-4.6#inference-and-training)such as vLLM, SGLang, llama.cpp, Ollama, and supportsfine-tuning ecosystems (https://huggingface.co/openbmb/MiniCPM-V-4.6#inference-and-training)such as SWIFT and LLaMA-Factory. Developers can quickly customize models for new domains and tasks on consumer-grade GPUs. We provide multiple quantized variants across GGUF, BNB, AWQ, and GPTQ formats. ### https://huggingface.co/openbmb/MiniCPM-V-4.6#evaluation-Evaluation Overall Performance (Instruct) Click to view MiniCPM-V 4.6-Thinking performance. Click to view MiniCPM-V 4.6 inference efficiency results.High-Concurrency Throughput Single Request TTFT (ms) ### https://huggingface.co/openbmb/MiniCPM-V-4.6#examples-Examples #### https://huggingface.co/openbmb/MiniCPM-V-4.6#overallOverall MiniCPM-V 4.6 can be deployed across three mainstream end-side platforms —iOS, Android and HarmonyOS. The clips below are raw screen recordings on phone devices without edition. ### https://huggingface.co/openbmb/MiniCPM-V-4.6#usagesUsages #### https://huggingface.co/openbmb/MiniCPM-V-4.6#inference-with-transformers-Inference with Transformers ##### https://huggingface.co/openbmb/MiniCPM-V-4.6#installation-Installation pip install "transformers[torch]>=5.7.0" torchvision torchcodec > Note on CUDA compatibility:torchcodec(used for video decoding) may have compatibility issues with certain CUDA versions. For example,torch\>=2\.11bundles CUDA 13.1 by default, while environments with CUDA 12.x may encounter errors such asRuntimeError: Could not load libtorchcodec. Two workarounds: 1. ReplacetorchcodecwithPyAV— supports both image and video inference without CUDA version constraints:pip install "transformers[torch]>=5.7.0" torchvision av 2. Pin the CUDA versionwhen installing torch to match your environment (e.g. CUDA 12.8):pip install "transformers>=5.7.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128 ##### https://huggingface.co/openbmb/MiniCPM-V-4.6#load-model-Load Model from transformers import AutoModelForImageTextToText, AutoProcessor model_id = "openbmb/MiniCPM-V-4.6" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForImageTextToText.from_pretrained( model_id, torch_dtype="auto", device_map="auto" ) # Flash Attention 2 is recommended for better acceleration and memory saving, # especially in multi-image and video scenarios. # model = AutoModelForImageTextToText.from_pretrained( # model_id, # torch_dtype=torch.bfloat16, # attn_implementation="flash_attention_2", # device_map="auto", # ) ##### https://huggingface.co/openbmb/MiniCPM-V-4.6#image-inference-Image Inference messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}, {"type": "text", "text": "What causes this phenomenon?"}, ], } ] downsample_mode = "16x" # Using `downsample_mode="4x"` for Finer Detail inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", downsample_mode=downsample_mode, max_slice_nums=36, ).to(model.device) generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=512) generated_ids_trimmed = [ out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) ] output_text = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False ) print(output_text[0]) ##### https://huggingface.co/openbmb/MiniCPM-V-4.6#video-inference-Video Inference messages = [ { "role": "user", "content": [ {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"}, {"type": "text", "text": "Describe this video in detail. Follow the timeline and focus on on-screen text, interface changes, main actions, and scene changes."}, ], } ] downsample_mode = "16x" # Using `downsample_mode="4x"` for Finer Detail inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", downsample_mode=downsample_mode, max_num_frames=128, stack_frames=1, max_slice_nums=1, use_image_id=False, ).to(model.device) generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048) generated_ids_trimmed = [ out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) ] output_text = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False ) print(output_text[0]) ##### https://huggingface.co/openbmb/MiniCPM-V-4.6#advanced-parameters-Advanced Parameters You can customize image/video processing by passing additional parameters toapply\_chat\_template: ParameterDefaultApplies toDescriptiondownsample\_mode``"16x"Image & VideoVisual token downsampling."16x"merges tokens for efficiency;"4x"keeps 4× more tokens for finer detail. Must also be passed togenerate\(\).max\_slice\_nums``9Image & VideoMaximum number of slices when splitting a high-resolution image. Higher values preserve more detail for large images. Recommended:36for image,1for video.max\_num\_frames``128Video onlyThemax\_num\_framesparameter dynamically controls the temporal context length and prevents VRAM overflow: Short Videos(duration ≤max\_num\_framessec): The processor defaults to1 FPS, capturing second-by-second details without hitting the upper limit. Long Videos(duration >max\_num\_framessec): The processor automatically switches touniform sampling, selecting exactlymax\_num\_framesevenly spaced across the entire timeline.stack\_frames``1Video onlyTotal sample points per second.1= main frame only (no stacking).N(N>1) = 1 main frame + N−1 sub-frames per second; the sub-frames are composited into a grid image and interleaved with main frames. Recommended setting is1for short videos, and3or5for long videos.use\_image\_id``TrueImage & VideoWhether to prependNtags before each image/frame placeholder. SetTruefor image,Falsefor video. > Note:downsample\_modemust be passed tobothapply\_chat\_template(for correct placeholder count) andgenerate(for the vision encoder). All other parameters only need to be passed toapply\_chat\_template. ##### https://huggingface.co/openbmb/MiniCPM-V-4.6#serving-with-transformers-serve-Serving withtransformers serve Hugging Face Transformers includes a lightweight OpenAI-compatible server for quick testing and moderate-load deployment. pip install "transformers[serving]>=5.7.0" Start the server: transformers serve openbmb/MiniCPM-V-4.6 --port 8000 --host 0.0.0.0 --continuous-batching Send a request: curl -s http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "openbmb/MiniCPM-V-4.6", "messages": [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}}, {"type": "text", "text": "What causes this phenomenon?"} ] }] }' Tool calling example: curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "openbmb/MiniCPM-V-4.6", "messages": [{"role": "user", "content": [ {"type": "text", "text": "the weather of Beijing"} ]}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a given location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"} }, "required": ["location"] } } }] }' The model returns a natural-language explanation followed by a structured block embedded in the content field. Note that a dedicated tool call parser for this format has not yet been added to the transformers library, so the tool calls need to be extracted manually via regex for now. { "id": "f4f09c7d-8045-4cb1-ade9-07aa5dee637d", "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "I need to check the current weather for Beijing, so I will call the get_weather function.\n\n\n\n\nBeijing\n\n\n", "role": "assistant" } } ], "created": 1778748859, "model": "openbmb/MiniCPM-V-4.6@main", "object": "chat.completion", "usage": { "completion_tokens": 47, "prompt_tokens": 283, "total_tokens": 330 } } #### https://huggingface.co/openbmb/MiniCPM-V-4.6#handling-escaped-newlines-in-model-outputs-Handling Escaped Newlines in Model Outputs In some cases, the model might output escaped newline characters\\nas string literals instead of actual newlines. To render the text correctly, especially in UI layers, you can use the following utility function. This function carefully replaces literal\\nwith real newlines while protecting scenarios where\\nhas specific semantic meaning. Utility Function: import re _PATTERN = re.compile( r'([\s\S]*?' # fenced code blocks r'|`[^`]+`' # inline code r'|\$\$[\s\S]*?\$\$' # display math r'|\$[^$]+\$' # inline math r'|\\\([\s\S]*?\\\)' # \(...\) r'|\\\[[\s\S]*?\\\]' # \[...\] r')' r'|(? str: """ Lightweight post-processing: Converts literal '\\n' to actual newlines, while protecting code blocks, inline code, and LaTeX commands. """ if not isinstance(text, str) or "\\" not in text: return text return _PATTERN.sub(lambda m: m.group(1) or '\n', text) #### https://huggingface.co/openbmb/MiniCPM-V-4.6#deploy-minicpm-v-46-on-ios-android-and-harmonyos-platforms-Deploy MiniCPM-V 4.6 on iOS, Android, and HarmonyOS Platforms We have adapted MiniCPM-V 4.6 for deployment oniOS, Android, and HarmonyOSplatforms, withall edge adaptation code fully open-sourced. Developers can reproduce the on-device experience in just a few steps. Visit ouredge deployment repository (https://github.com/OpenBMB/MiniCPM-V-edge-demo)for platform-specific build guides, or go to thedownload page (https://github.com/OpenBMB/MiniCPM-V-edge-demo/blob/main/DOWNLOAD.md)to try pre-built apps directly. #### https://huggingface.co/openbmb/MiniCPM-V-4.6#use-minicpm-v-46-in-other-inference-and-training-frameworks-Use MiniCPM-V 4.6 in Other Inference and Training Frameworks MiniCPM-V 4.6 supports multiple inference and training frameworks. Below are quick-start commands for each. For full details, see ourCookbook (https://github.com/OpenSQZ/MiniCPM-V-CookBook). vLLM—Full Guide (https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/deployment/vllm/minicpm-v4_6_vllm.md)vllm serve openbmb/MiniCPM-V-4.6 \ --port 8000 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --default-chat-template-kwargs '{"enable_thinking": false}' > Note:\-\-enable\-auto\-tool\-choiceand\-\-tool\-call\-parser qwen3\_coderenable tool/function calling support. If you don’t need tool use, you can omit these flags and simply runvllm serve openbmb/MiniCPM\-V\-4\.6. curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "openbmb/MiniCPM-V-4.6", "messages": [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}}, {"type": "text", "text": "What causes this phenomenon?"} ]}] }' Tool calling example: curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "openbmb/MiniCPM-V-4.6", "messages": [{"role": "user", "content": [ {"type": "text", "text": "北京的天气"} ]}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a given location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"} }, "required": ["location"] } } }] }' SGLang—Full Guide (https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/deployment/sglang/minicpm-v4_6_sglang.md)python -m sglang.launch_server --model openbmb/MiniCPM-V-4.6 --port 30000 curl -s http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "openbmb/MiniCPM-V-4.6", "messages": [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}}, {"type": "text", "text": "What causes this phenomenon?"} ]}] }' llama.cpp—Full Guide (https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/deployment/llama.cpp/minicpm-v4_6_llamacpp.md)llama-server -m MiniCPM-V-4.6-Q4_K_M.gguf --port 8080 curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "MiniCPM-V-4.6", "messages": [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}}, {"type": "text", "text": "What causes this phenomenon?"} ]}] }' Ollama—Full Guide (https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/deployment/ollama/minicpm-v4_6_ollama.m