multimodal-control

Surg-UniWorld is a unified surgical world model with multimodal control experts, enabling controllable generation of coherent instrument-tissue interaction videos using edge, depth, and optical-flow inputs. It introduces a new benchmark (Cholec80-SurgWAM) and outperforms existing controllable video generation methods.

Introduces appearance pointers, compact tokens that guide Diffusion Transformers to apply correct appearance cues at specified spatial locations, enabling modality-agnostic localized multimodal control without retraining the base model.