Visual prompt engineering for video models

Hugging Face Daily Papers Papers
visual-prompt-engineering video-models prompt-engineering foundation-models visual-reasoning

Summary

This paper introduces Visual Prompt Engineering (VIPE), a method that automatically modifies task images to improve video model performance, showing it can be more effective than text-based prompt engineering or test-time scaling.

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models. Example videos on our project page at https://visual-prompt-engineering.github.io/.
Original Article
View Cached Full Text

Cached at: 07/29/26, 03:50 AM

Paper page - Visual prompt engineering for video models

Source: https://huggingface.co/papers/2607.25537

Abstract

Intheageoffoundationmodels,amodelisonlyasgoodasitsprompt.Forthisreason,promptengineeringhasbecomeanessentialtechniqueforimprovinglanguagemodelperformance.Sincevideomodelsarecurrentlybecomingfoundationmodelsforvisualtasks(e.g.,visualreasoning),wehereaskwhethertheysimilarlybenefitfromvisualpromptengineering:automaticallymodifyingthetaskimagetoimprovemodelperformance.Forexample,foravisualphysicsreasoningtask(“Wheredoestheballland,afterpassingasetofobstacles?”),anabstractsketch-likescenecanbeturnedintoaphotorealisticversionwithasimplecalltoanimageeditingmodel.Wefindthatvisualpromptengineering,orVIPEforshort,improvesvideoreasoningperformanceacrosstasks.Infact,forvideomodels,visualpromptengineeringcanbeevenmoreeffectivethanclassictext-basedpromptengineeringortest-timescaling.Ultimately,justastext-basedpromptengineeringsystematicallyimproveslanguagemodelperformance,visualpromptengineeringcanserveasasimple,compute-efficientapproachtoelicitbettervisualreasoningperformancefromvideomodels.Examplevideosonourprojectpageathttps://visual-prompt-engineering.github.io/.

View arXiv pageView PDFProject pageAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.25537 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.25537 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.25537 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Visual Prompts in Video Models (8 minute read)

TLDR AI

Visual prompt engineering (VIPE) automatically modifies task images to improve video model reasoning performance, often more effective than text-based prompting or test-time scaling.

When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

arXiv cs.AI

This paper introduces Visualized Task Semantics (VTS), a controlled intervention to study when multimodal LLMs are asked questions in the image rather than text. It reveals a consistent accuracy drop across all tested models and tasks, and proposes prompt-region grounding to recover the clean task representation, improving VTS accuracy from 58.0 to 66.3.

Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning

Hugging Face Daily Papers

The paper introduces Internalized Visual Thinking (IVT), a post-training framework that trains multimodal models to predict future frame embeddings, enabling direct answer generation without synthesizing intermediate images, thereby reducing latency over 5x for proactive video reasoning.