FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

Hugging Face Daily Papers Papers
real-time multimodal agents deployment optimization gpu latency-reduction

Summary

FlashRT is an agent harness that guides coding agents to automatically optimize and deploy real-time multimodal applications, achieving up to 70x latency reduction on NVIDIA B200 GPUs and 3.6x throughput improvement on AMD MI355X.

Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism. Existing serving systems and auto-parallelism compilers commit to limited transformations and fixed workload assumptions, so achieving high performance on a new application requires hand-crafting an efficient implementation. We present FlashRT, an agent harness that guides coding agents to lift simple developer-written reference implementations into optimized multi-GPU deployments that flexibly weigh target metrics like latency and throughput. Using a new chain-of-program paradigm, FlashRT directs a generic coding agent through a multi-pass transformation process where an agent transforms the reference into an intermediate representation (IR) to capture data dependencies and persistent-state scopes, validates this IR via a sequential interpreter, and performs static analyses to identify candidate transformations. Then, the agent iteratively implements, verifies, and benchmarks each candidate under a measurement-gated optimization loop to produce effective deployments that span different hardware budgets. Across various applications, including video world models and multimodal LLMs, FlashRT converts reference implementations into highly efficient deployments, delivering up to ~70x latency reduction and 2.8x throughput improvement on NVIDIA B200 GPUs. On AMD MI355X GPUs, FlashRT matches the peak latency reduction while increasing peak throughput improvement to 3.6x, demonstrating that agent-driven optimization can be more scalable on platforms with less mature expert optimization. In fact, for Qwen3-Omni text-to-audio inference, FlashRT reduces response latency by 65% compared to the expert vLLM-Omni implementation on AMD MI355X.
Original Article
View Cached Full Text

Cached at: 07/21/26, 06:35 AM

Paper page - FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

Source: https://huggingface.co/papers/2607.18171

Abstract

Real-timemultimodalapplications,includingvoiceagentsandinteractivevideogeneration,composeheterogeneousmodelsintopipelineswhoseefficientdeploymentrequiresapplication-specificdecisionsaboutplacement,streaming,andintra-modelparallelism.Existingservingsystemsandauto-parallelismcompilerscommittolimitedtransformationsandfixedworkloadassumptions,soachievinghighperformanceonanewapplicationrequireshand-craftinganefficientimplementation.WepresentFlashRT,anagentharnessthatguidescodingagentstoliftsimpledeveloper-writtenreferenceimplementationsintooptimizedmulti-GPUdeploymentsthatflexiblyweightargetmetricslikelatencyandthroughput.Usinganewchain-of-programparadigm,FlashRTdirectsagenericcodingagentthroughamulti-passtransformationprocesswhereanagenttransformsthereferenceintoanintermediaterepresentation(IR)tocapturedatadependenciesandpersistent-statescopes,validatesthisIRviaasequentialinterpreter,andperformsstaticanalysestoidentifycandidatetransformations.Then,theagentiterativelyimplements,verifies,andbenchmarkseachcandidateunderameasurement-gatedoptimizationlooptoproduceeffectivedeploymentsthatspandifferenthardwarebudgets.Acrossvariousapplications,includingvideoworldmodelsandmultimodalLLMs,FlashRTconvertsreferenceimplementationsintohighlyefficientdeployments,deliveringupto~70xlatencyreductionand2.8xthroughputimprovementonNVIDIAB200GPUs.OnAMDMI355XGPUs,FlashRTmatchesthepeaklatencyreductionwhileincreasingpeakthroughputimprovementto3.6x,demonstratingthatagent-drivenoptimizationcanbemorescalableonplatformswithlessmatureexpertoptimization.Infact,forQwen3-Omnitext-to-audioinference,FlashRTreducesresponselatencyby65%comparedtotheexpertvLLM-OmniimplementationonAMDMI355X.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2607\.18171

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.18171 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.18171 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.18171 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Step 3.7 Flash

Product Hunt

Step 3.7 Flash is a fast agents model designed to see and act in real time.

FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving

arXiv cs.AI

FlashDrive is an algorithm-system co-design framework that cuts the inference latency of vision-language-action models for autonomous driving by 4.7× (from 717 ms to 151 ms on a single GPU) using streaming KV-cache reuse, non-autoregressive diffusion drafting, and adaptive step caching, with negligible accuracy loss.

Harness design for long-running application development

Anthropic Engineering

Anthropic engineers detail a multi-agent harness design using generator and evaluator agents to improve Claude's ability to build complete, high-quality frontend applications autonomously over long durations.