Tag
This paper introduces MMDG-Bench, a unified benchmark for multimodal domain generalization that reveals limited progress in current methods and significant robustness challenges across diverse tasks.
GIST is a multimodal knowledge extraction pipeline that transforms mobile point cloud data into semantically annotated navigation topologies for dense environments, enabling semantic search, localization, and natural language routing with 80% navigation success rates in real-world evaluation.
This paper introduces the one-sided conversation problem (1SC), addressing how to reconstruct missing dialogue and generate summaries when only one speaker's turns are available in real-world settings like telemedicine and call centers. The authors evaluate prompting and finetuned models on multiple datasets, finding that access to future context and utterance length information improves reconstruction, while high-quality summaries can be generated without full dialogue reconstruction.
SIMMER proposes a novel MLLM-based embedding approach for cross-modal food image-recipe retrieval, replacing traditional dual-encoder architectures with a unified encoder and achieving state-of-the-art results on the Recipe1M dataset with significant improvements over prior methods.
Switch-KD proposes a novel visual-switch knowledge distillation framework for efficiently compressing vision-language models by unifying multimodal knowledge transfer within a shared text-probability space. The method achieves 3.6-point average improvement across 10 multimodal benchmarks when distilling a 0.5B TinyLLaVA student from a 3B teacher model.
The article explains 'Vokenization,' a multimodal learning technique that bridges computer vision and natural language processing by using weak supervision to link visual data with language tokens. It contrasts this approach with text-only models like GPT-3 and BERT, highlighting how visual grounding can improve language understanding.
OpenAI discovers multimodal neurons in CLIP that respond to the same concept across different modalities (visual, symbolic, textual), mirroring biological neurons and explaining the model's robustness on challenging vision tasks. This interpretability research provides insights into how vision-language models organize and represent abstract concepts.