@omarsar0: YT if you prefer https://youtu.be/_rIziQa48wQ?si=IsCBPvm07TDUIeu7…

YT if you prefer https://youtu.be/_rIziQa48wQ?si=IsCBPvm07TDUIeu7…

TL;DR

By injecting rich context into an AI agent via multimodal prompts (voice + screenshots + annotations + text), you can dramatically improve the speed and quality of solving complex tasks like web design fixes.

Why Multimodal Prompting?

Traditionally, when we want an AI agent to modify a web design, we have to manually take screenshots, draw arrows, switch apps to copy-paste text, and then describe each problem one by one. This process is fragmented, time-consuming, and prone to losing context. The speaker experimented with a much smoother approach in his own agent orchestrator: multimodal prompting — bundling voice, screenshots, real-time annotations, mouse trails, and text descriptions into a single request and throwing it at the agent all at once.

He notes: “As models improve, they’ll get better and better at understanding these rich prompts. What I’m doing is pushing the way we interact with agents.”

Demo Scenario: Fixing Web Design Flaws

The speaker is working on an open-source project (inspired by Anthropic’s Cloud Science), but the page has several typical issues:

• The artifacts list cannot scroll
• The sidebar and content panel alignment is messed up
• A section contains extra formatting that needs to be removed

He uses his own Chromium-based browser (for the orchestrator) to record his actions instead of manually taking screenshots.

Pain Points of the Traditional Approach

1. Multiple screenshot annotations: Taking screenshots, drawing circles, writing text descriptions manually each time.
2. Constant context switching: Jumping between browser, drawing tools, and agent chat interface.
3. Loss of information: Pure text descriptions cannot precisely convey visual alignment or scroll behavior.

How Multimodal Prompting Works

While the orchestrator runs a “recording task,” it simultaneously collects the following information:

• Voice: Transcribed in real-time via Eleven Labs, naturally expressing intent.
• Screen annotations: During recording, the speaker circles problem areas with the mouse; annotations are saved as part of the image.
• Mouse events: The system listens to mouse position and tracks cursor movement, accurately indicating “this is misaligned.”
• Screenshots: Automatically captures the entire page including annotations.

All these modalities are merged into a “rich prompt” and sent directly to the agent (defaulting to Claude 4.8). Although it consumes more tokens, the agent instantly understands the problem.

“Some problems are really hard to explain to an agent with just a text prompt. Multimodal prompts make collaborating with the agent super fun.”

Result: More Efficient One-Shot Fix

A few minutes later, the agent completed the modifications without any additional interaction:

• Scrolling works normally, artifacts list is navigable.
• Alignment is noticeably improved — the speaker exclaims, “I really like this alignment.”
• The extra tags have been completely removed.

The entire process required no manual prompt adjustments, no back-and-forth. The speaker emphasizes: “It’s not always perfect, but it lets me give Claude enough context to fix problems much more efficiently.”

Why Is This Better Than the Traditional Way?

Traditionally, to fix the same set of issues, you might have to:

1. Take a full-page screenshot
2. Draw arrows and circles in a drawing tool
3. Write “artifacts list can’t scroll, sidebar is misaligned, delete this text section”
4. Copy-paste into the agent chat
5. Wait for the agent to make changes, then iterate

With multimodal prompting, steps 1–4 are compressed into a single recording+annotation action, and the agent directly receives your full intent expressed through language, gestures, and notes.

Key advantage: fewer context switches, preserved natural interaction — especially helpful for visual alignment, layout tweaks, and other problems that are tricky for AI.

Future Outlook

The speaker believes multimodal prompting will become the trend. As models get better at understanding images, audio, and mouse trails, this interaction style will further lower the communication cost between human and agent. He plans to continue improving the orchestrator and explore more modality combinations (e.g., video, real-time cursor animation).

“It’s great — basically filling in the blanks, helping the agent fill in the blanks, and also helping yourself. There’s a kind of relationship formed in the interaction.”

Source
[Original Video] @omarsar0: YT if you prefer https://youtu.be/_rIziQa48wQ?si=IsCBPvm07TDUIeu7 (https://www.youtube.com/watch?v=_rIziQa48wQ)