@omarsar0: Multimodal prompting is clearly the future. I love experimenting with new ways to interact with agents. As a researcher…

Multimodal prompting is clearly the future. I love experimenting with new ways to interact with agents. As a researcher and engineer, I've found that the richer the inputs to the agent and the richer the outputs I consume, the better the overall results of the collaboration. In this little walkthrough, I go over what I mean by a multimodal prompt and when you might find it useful. It's more than simple text prompting, so I call it a "task" for lack of a better word. It helps me record my voice, annotate the screen, click/mouse actions, and more. Then all of that is preprocessed and passed to the agent to complete the task more efficiently. The agent has the high-level prompt, but it also has the raw transcriptions if needed. So naturally, I am also using this to build out multimodal skills that I reuse in workflows where agents tend to struggle. This has saved me hours of work. And even the older models are pretty great at understanding the tasks more clearly. Some noise is introduced in the process, but it doesn't seem to hurt the performance. I've also found that this new way of prompting has reduced the number of frustrating interactions I have with agents. This is something I have been thinking about for some time now because we are going to move more into multimodal AI models. And so the interactions are going to evolve with models being able to handle a variety of modalities natively. Currently, I process all the recorded tasks with another model in the background, but it's not crazy to think that all of it will just be naturally consumed by omnimodel in the future. All of these recorded tasks (which you can also think of as rich annotated datasets) are things I mine and recursively improve over time and, in some cases, package as reusable workflows/patterns/skills. This process has really elevated how I use coding agents for all kinds of work. I use multimodal prompting in things like web development, designing, artifact creation, prototyping, researching, reading, simulations, AI-assisted writing, and much more. So it's not just about prompting. It's going deeper into understanding and exploring the right level of detail that the agents need to make the right decision and to push/maximize their capabilities.