clip

This paper proposes using off-the-shelf CLIP-style multimodal encoders with a bimodal attribution method to predict gaze behavior in visual world experiments, successfully replicating a seminal study on human predictive processing without fine-tuning.

CT-Merging proposes a method to merge LoRA adapters by estimating consensus directions from task subspace projectors and assigning task-level RMS coefficient scales, achieving superior performance on the DC-Merge CLIP adapter benchmark.

This paper proposes modeling the CLIP latent space using Mixtures of von Mises–Fisher distributions on the unit hypersphere, capturing its directional and multimodal structure better than Gaussian assumptions. The model improves long-tailed and out-of-distribution detection and provides a semantic decomposition of CLIP embeddings.

MedPMC is an automated framework that transforms medical literature into high-fidelity multimodal data for foundation models, achieving significant improvements across multiple benchmarks and clinical settings.

Introduces Reward-Gated Test-Time Adaptation (RG-TTA), a reinforcement learning framework that selectively applies debiasing to CLIP models based on input bias sensitivity, resolving the fairness-utility trade-off.

ComMem proposes complementary memory systems inspired by biological memory to improve test-time adaptation of vision-language models, outperforming state-of-the-art on 15 benchmarks.

This paper presents VLM-Safe-RL, a framework that integrates frozen vision-language models into constrained MDP Lagrangian updates to provide anticipatory cost signals for safe reinforcement learning in high-speed visual control tasks. The method outperforms standard constraint-aware baselines on Safety-Gymnasium FormulaOne L2 and generalizes to held-out environments.

This paper introduces KODA (Kernel Optimization for Discrepancy Analysis), a kernel-based framework for comparing and aligning vision-language model representations by identifying sample subsets that are clustered differently across models like CLIP, SigLIP, and BLIP. The method uses contrastive embedding clustering and randomized low-dimensional approximations to scale to large datasets while providing interpretable structural differences between representations.

BRepCLIP introduces contrastive multimodal pretraining on boundary representation (BRep) primitives for CAD understanding, aligning BRep geometry with language and image embeddings to achieve state-of-the-art retrieval and zero-shot classification.

This paper investigates why CLIP struggles with concept binding, showing that while CLIP's binding function is high-complexity, controlled transformer models can learn low-complexity binding functions through multiplicative interactions that generalize better.

FAST-GOAL is a fine-tuning method that enhances CLIP's ability to align global and local semantics in images and lengthy text, introducing FLISM and TSL modules and the GLIT100k dataset. It achieves improvements on long caption datasets.

Visual Concept Fusion (VCF) enables dual conditioning on both an image and text prompt in diffusion models at inference time without retraining, using a lightweight aligner and fusion strategy.

SafeDiffusion-R1 introduces an online reinforcement learning framework using GRPO and a steering reward mechanism to improve safety in diffusion models without requiring supervised data or reward tuning, achieving state-of-the-art performance on multiple harm categories.

SAE-FT introduces a novel fine-tuning method for CLIP models that uses sparse autoencoder constraints to regularize visual representations, improving robustness against distribution shifts while maintaining performance and enabling interpretability.

FeatCal is a calibration method that reduces performance gaps in post-merging models by layer-wise weight updates without gradient descent, achieving superior results on CLIP and GLUE benchmarks with high sample efficiency.

TTL introduces a test-time textual learning framework for OOD detection using pretrained vision-language models like CLIP, which dynamically learns OOD semantics from unlabeled test streams without external OOD labels. The method uses pseudo-labeled samples and an OOD knowledge purification strategy to improve detection robustness across diverse and evolving OOD distributions.

OpenAI proposes a hierarchical two-stage model for text-conditional image generation using CLIP latents: a prior that generates CLIP image embeddings from text captions, and a diffusion-based decoder that generates images from embeddings. The approach improves image diversity and enables zero-shot language-guided image manipulations.

The article highlights the emerging scene of AI-generated art using OpenAI's CLIP model as a steering mechanism for generative models, showcasing various examples of text-to-image outputs.

OpenAI discovers multimodal neurons in CLIP that respond to the same concept across different modalities (visual, symbolic, textual), mirroring biological neurons and explaining the model's robustness on challenging vision tasks. This interpretability research provides insights into how vision-language models organize and represent abstract concepts.

Jina CLIP v2 is an improved multimodal embedding model supporting 89 languages, high-resolution images, and flexible embedding dimensions, with 3% better performance than v1 and state-of-the-art results on multilingual benchmarks.