Tag
This paper proposes Scaffolding Minds, which learns a dedicated scaffolding encoder for optimized latent visual target representations and learns both mean and variance of the RL sampler, improving latent reasoning baselines by +9.5 points on Frozen Lake spatial planning and +5.6 points on average across nine visual reasoning benchmarks.
This paper identifies invisible metadata traces at the pixel level as shortcuts that vision encoders exploit, leading to performance degradation under metadata distribution shifts. Mitigation strategies during and after pretraining reduce sensitivity to both targeted and unseen metadata without sacrificing downstream performance.
Demonstrates deep dream effects on modern LLMs by optimizing images to maximize probability of a target caption; Gemma 12B (no vision encoder) stamps recognizable objects, while E4B (with vision encoder) drifts to texture.
SAGA framework uses frozen multimodal large language models to provide attribute-aware supervision for vision encoders via Group Relative Policy Optimization, improving zero-shot image retrieval by 3–6 points on fine-grained benchmarks.
Tuna-2 is a unified multimodal model that achieves state-of-the-art performance by processing visual understanding and generation directly from pixel embeddings, eliminating the need for pretrained vision encoders.