Visual Prompts in Video Models (8 minute read)

# Visual Prompt Engineering for Video Models Source: [https://visual-prompt-engineering.github.io/](https://visual-prompt-engineering.github.io/) [](https://deepmind.google/) \*Core team **TL;DR:**Just as text\-based prompt engineering systematically improves language model performance, visual prompt engineering \(VIPE\) serves as a simple, compute\-efficient approach to elicit better visual reasoning performance from video models by transforming the input task image \(e\.g\., making abstract sketches photorealistic\)\.       Physics VPCT Conjunctive Search Connect the Dots Sort 3 Numbers Maze RushHour       Physics VPCT Conjunctive Search Connect the Dots Sort 3 Numbers Maze RushHour ## Abstract In the age of foundation models, a model is only as good as its prompt\. For this reason, prompt engineering has become an essential technique for improving language model performance\. Since video models are currently becoming foundation models for visual tasks \(e\.g\., visual reasoning\), we here ask whether they similarly benefit from*visual*prompt engineering: automatically modifying the task image to improve model performance\. For example, for a visual physics reasoning task \("Where does the ball land, after passing a set of obstacles?"\), an abstract sketch\-like scene can be turned into a photorealistic version with a simple call to an image editing model\. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks\. In fact, for video models, visual prompt engineering can be even more effective than classic text\-based prompt engineering or test\-time scaling\. Ultimately, just as text\-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute\-efficient approach to elicit better visual reasoning performance from video models\. ## Podcast Listen to the NotebookLM\-generated podcast overview of our paper: Your browser does not support the audio element\. ## Task Examples Below, we show examples for each task and split\. We show a solution attempt on the original image \(which often fails\), a solution attempt on the most successful VIPE variant for the task and split, and other VIPE variants obtained by freeform VIPE or atomic concept edit \(ACE\) VIPE\. ### [Physics VPCT](https://huggingface.co/datasets/camelCase12/vpct-1) Predict which bucket the ball will land in after rolling down the ramps\. **Text Prompt:**"The ball moves down in a physically plausible way, sliding down the obstacles, and ends up in one of the three containers at the bottom\. \[\.\.\.\]" Original Solution AttemptFailed  Industrial factory chutes Success  #### Other Variants  Ancient stone ruins  Baking kitchen prep  Billiard trick shot  Candy sweets landscape  Claymation stop motion  Construction site steel  Garage rube goldberg  Macro forest nature  Marble run toy  Mini golf course  Neon cyberpunk arcade  Office desk supplies  Retro pachinko machine  Science chemistry lab  Steampunk brass mechanism  Water park slides  Winter ice slopes  Wooden physics puzzle  Zen garden bamboo ### Conjunctive Search Locate the target object defined by a conjunction of features \(e\.g\., color and shape\)\. **Text Prompt:**"The two blue balls begin to glow\." Original Solution AttemptFailed  Globe lightbulbs case Success  #### Other Variants \(Freeform\)  Arcade joystick panel  Billiard table felt  Candy assortment countertop  Deep sea bioluminescence  Dj mixer interface  Fantasy crystal orbs  Gemstone jewelry velvet  Gym equipment wood floor  Holiday ornaments snow  Lab plasma balls  Macro marbles and dice  Neon mini golf  Night pool led floats  Photorealistic 3d shapes  Pinball arcade bumpers  Sci fi energy cores  Synthwave neon grid  Twilight led bocce  Vintage toys hardwood #### Other Variants \(ACE\)  REPLACE flat color of the two blue circles with a 3D glossy material  REPLACE flat rendering of blue circles with 3D glossy volumetric material  SET border outline of the two blue balls as a thick white stroke  REPLACE flat color of blue balls with 3D glossy material  REPLACE flat material of blue circles with glossy 3D rendering  SET material of the two blue circles as glossy 3D glass  SET rendering material of the blue circles as 3D glossy  SET material of the blue balls as glossy 3D spheres  SET materiality of the two blue circles as 3D glossy spheres  REPLACE size of the blue circles with a significantly larger scale  SET size of the blue circles as significantly enlarged  SET material of the blue circles as glossy 3D  SET color saturation of non\-target shapes as completely desaturated grayscale  REMOVE color saturation from all non\-target objects  REPLACE colors of all red circles and blue squares with grayscale  SET material of the blue circles as glossy 3D spheres  SET the material of the blue circles as glossy 3D spheres  REPLACE flat shading of the two blue circles with a glossy 3D material  SET material of the blue circles as glossy 3D ### Connect the Dots Connect same\-colored dots\. **Text Prompt:**"Connect each pair of same\-colored circles with a line\." Original Solution AttemptFailed  Thick black stroke Success  #### Other Variants \(Freeform\)  Aerial flower beds  Building block baseplate  Chalk street art  Corkboard pushpins  Embroidery hoop  Frosted cookies baking sheet  Glowing neon nodes  Gym floor vinyl spots  Mobile puzzle ui  Pcb terminal nodes  Retro arcade crt  Round stickers cardboard  Sewing buttons fabric  Synth patchbay jacks  Wax seals parchment  Wet acrylic dollops  Whiteboard magnets  Wooden board game tokens  Yarn pegboard  Zen garden stones #### Other Variants \(ACE\)  REPLACE flat 2D shapes with volumetric 3D spheres  REPLACE standard hues with highly saturated neon colors  SET border of each circle as thick outline  REPLACE solid white background with dark black background  SET outer border of the circles as a thick black stroke  REPLACE current background color with a high\-contrast dark gray  SET internal content of the circles as matching text labels  REPLACE bright white background with a dark slate background  SET border of each circle as a thick black stroke  SET border of each circle as a thick white outline  SET color saturation of the circles as ultra\-vibrant  SET color saturation of the circles as maximally saturated neon  SET size of the circles as significantly larger  REPLACE current rendering style with glossy 3D spheres  REPLACE current background color with solid black  REPLACE current circle sizes with 40% enlarged versions  SET color saturation of the circles as highly increased  SET size of all circles as 30 percent larger  REPLACE plain white background with a solid black background ### Sort 3 Numbers Sort three numbers in ascending order\. **Text Prompt:**"The numbers pop and disappear one at a time, in numeric order, starting from the smallest one\." Original Solution AttemptFailed  Retro arcade sprites Success  #### Other Variants \(Freeform\)  Ar notification badges  Bubblegum bubbles  Chalk shooting targets  Dry ice smoke bubbles  Giant bubble wrap  Glass lightbulbs  Glossy candy blobs  Glowing fireworks  Holographic data nodes  Inflatable pool floats  Latte foam bubbles  Numbered balloons  Papier mache pinatas  Pop art comic splats  Popping boba pearls  Soap bubbles  Squishy jelly cubes  Vr aim trainer targets  Water balloons #### Other Variants \(ACE\)  SET rendering of the numbers as 3D embossed  REPLACE current plain rendering with a neon sign aesthetic  REPLACE current background color with a high\-contrast dark solid  REPLACE plain white background with a solid high\-contrast dark color  REPLACE sterile environment with a cohesive chalkboard texture  SET color of each number as a unique vivid hue  SET color of each number as a distinct saturated hue  REPLACE 2D flat typography with 3D volumetric geometry  REPLACE current flat typography with 3D metallic rendered text  REPLACE uniform black text color with distinct saturated colors  SET spatial boundaries of each number as distinct geometric boxes  SET overall scale of the numbers as uniformly enlarged  REPLACE white background with a dark high\-contrast background  REPLACE current font weight with an ultra\-bold heavy weight  REPLACE current uniform black text color with distinct vibrant colors  SET rendering of the numbers as subtle volumetric 3D geometry  REPLACE current white background with a dark solid color  SET surface texture of the scene as a green chalkboard  SET dimensionality of the numbers as subtly embossed 3D ### Maze Solving Find a path from start to finish in various maze sizes\. **Text Prompt:**"Create a 2D animation based on the provided image of a maze\. The red circle slides smoothly along the white path, stopping perfectly on the green circle\. \[\.\.\.\]" #### Split: 5×5 Maze Original Solution AttemptFailed  Classic 2d arcade game Success  ##### Other Variants  Algorithmic pathfinding visualization  Animated transit map infographic  Classic flash web animation  Dark mode gps tracking ui  Dark mode ui motion  Digital circuit routing  Digital rhythm game interface  Flat 2d physics sandbox  Flat felt stop motion  Glossy mobile puzzle interface  Industrial cam simulation  Layered 2d motion graphics  Matte papercraft 3d  Neon motion graphics  Precise vector math animation  Robot vacuum app map  Sleek 2d vector puzzle  Smooth arcade tabletop  Vintage celluloid animation #### Split: 7×7 Maze Original Solution AttemptFailed  Classic 2d arcade game Success  ##### Other Variants  Algorithmic pathfinding visualization  Animated transit map infographic  Classic flash web animation  Dark mode gps tracking ui  Dark mode ui motion  Digital circuit routing  Digital rhythm game interface  Flat 2d physics sandbox  Flat felt stop motion  Glossy mobile puzzle interface  Industrial cam simulation  Layered 2d motion graphics  Matte papercraft 3d  Neon motion graphics  Precise vector math animation  Robot vacuum app map  Sleek 2d vector puzzle  Smooth arcade tabletop  Vintage celluloid animation #### Split: 9×9 Maze Original Solution AttemptFailed  Neon motion graphics Success  ##### Other Variants  Algorithmic pathfinding visualization  Animated transit map infographic  Classic 2d arcade game  Classic flash web animation  Dark mode gps tracking ui  Dark mode ui motion  Digital circuit routing  Digital rhythm game interface  Flat 2d physics sandbox  Flat felt stop motion  Glossy mobile puzzle interface  Industrial cam simulation  Layered 2d motion graphics  Matte papercraft 3d  Precise vector math animation  Robot vacuum app map  Sleek 2d vector puzzle  Smooth arcade tabletop  Vintage celluloid animation #### Split: Irregular Maze Original Solution AttemptFailed  Digital rhythm game interface Success  ##### Other Variants  Algorithmic pathfinding visualization  Animated transit map infographic  Classic 2d arcade game  Classic flash web animation  Dark mode gps tracking ui  Dark mode ui motion  Digital circuit routing  Flat 2d physics sandbox  Flat felt stop motion  Glossy mobile puzzle interface  Industrial cam simulation  Layered 2d motion graphics  Matte papercraft 3d  Neon motion graphics  Precise vector math animation  Robot vacuum app map  Sleek 2d vector puzzle  Smooth arcade tabletop  Vintage celluloid animation ### [RushHour](https://jana-z.github.io/mentis-oculi/) Slide blocks to allow the red car to exit the grid\. **Text Prompt:**"\[\.\.\.\] Plan the minimal sequence of moves needed to free the red car and allow it to exit the parking lot\. Output: A video demonstrating the full solution to the puzzle, one move at a time\." #### Split: Level 2 Original Solution AttemptFailed  Anodized metal puzzle box Success  ##### Other Variants  3d game engine viewport  Automated mechanical parking garage  Carved wooden puzzle  Casual mobile game 3d  Drone asphalt parking lot  Macro pcb electronics  Magnetic whiteboard diagram  Minimalist 3d diorama  Neon holographic tactical map  Port shipping container yard  Retro pixel art arcade  Sci fi maglev depot  Screen recording web ui  Stop motion felt craft  Studded plastic brick toy  Tournament billiard table  Toy slot car diorama  Translucent ice blocks on frost  Vintage cardboard board game #### Split: Level 3 Original Solution AttemptFailed  Sci fi maglev depot Success  ##### Other Variants  3d game engine viewport  Anodized metal puzzle box  Automated mechanical parking garage  Carved wooden puzzle  Casual mobile game 3d  Drone asphalt parking lot  Macro pcb electronics  Magnetic whiteboard diagram  Minimalist 3d diorama  Neon holographic tactical map  Port shipping container yard  Retro pixel art arcade  Screen recording web ui  Stop motion felt craft  Studded plastic brick toy  Tournament billiard table  Toy slot car diorama  Translucent ice blocks on frost  Vintage cardboard board game ## Citation ``` @article{geirhos2026visual, title={Visual prompt engineering for video models}, author={Geirhos, Robert and Li, Yuxuan and Wiedemer, Thadd{\"a}us and Kalibhat, Neha and Wang, Zi and Malek, Mani and Tafjord, {\O}yvind and Swersky, Kevin and Kim, Been and Jaini, Priyank}, journal={arXiv preprint arXiv:2607.25537}, year={2026} } ```