Tag
Meta-CLIP introduces a metadata-conditioned contrastive learning framework that reweights cross-modal negatives using DICOM acquisition fields, improving zero-shot disease classification on CT-RATE and several CXR benchmarks over CT-CLIP.
这篇论文对持续学习场景下视觉-语言模型的环境感知OOD检测进行了理论分析,提出交叉环境最大概念匹配(MCM)分数,并证明了在环境转移为正且类匹配间隔有界条件下其假阳性率不劣于标准MCM,同时区分了可检测的软OOD与不可检测的硬OOD环境。
ShadowFPT 提出了一种针对联邦提示调优(FPT)的后门攻击方法,利用冻结骨干网络的表示空间植入影子触发器,在保持提示更新隐蔽的同时将攻击成功率从 19.81% 提升至 90.36%。研究表明仅依赖提示更新异常检测的防御措施不足以应对此类攻击。
The paper introduces SABER, a controlled benchmark exposing widespread attribute-object binding failures in multimodal retrievers, and shows that fine-tuning an MLLM-based embedder solely on synthetic attribute-binding text achieves SOTA on SABER while improving cross-modal and image-image retrieval without image supervision.
提出“隐喻电路”假说,解释 Vision Transformer 如何在缺乏直接指称证据时通过具体锚点概念(如 fire)间接接地抽象概念(如 angry),并利用 Transcoders 对 CLIP 和 DINO 的电路追踪与因果干预进行了验证。
This paper introduces Anchor Divergences, a method that leverages the interplay between contrastive learning, exponential families, and information geometry to define context-specific semantic geometries on fixed CLIP-like representations, allowing similarity to reflect task-relevant attributes (e.g., dog breed vs. background) instead of a single fixed cosine geometry. Retrieval experiments show the approach is both effective and efficient for specifying context-dependent semantic similarity.
该论文系统研究了混合模态检索器的模态偏好问题,发现无关文本比无关图像对检索性能的干扰更严重(称为 "Chaos in the Text"),并提出 Trident 方法,通过 Multi-Positive View InfoNCE 联合优化文本、图像和融合视图来缓解该偏差。
SCM (Screen Memories) is a local-first macOS app that adds AI-powered search over every photo and video frame in any folder, with scene-level video search, OCR, Whisper-based dialogue search, and opt-in local LLM chat — all running offline on-device. It is open-source, installable via Homebrew, and built with Bun, Electron, and a vision model (CLIP by default).
The paper investigates whether text conditioning actually steers neural PDE surrogates using OperatorCLIP, a CLIP-style contrastively aligned FNO with FiLM control, finding that constant-prompt controls often match or beat task-text conditioning and that single-description training makes text alignment unidentifiable.
This paper proposes UCF-Net, a cascaded fusion network that combines CLIP and DINO for improved deepfake image detection with better cross-domain generalization.
The paper investigates adapting CLIP for regional geolocalization, finding that encoder adaptation improves accuracy by leveraging intact scene configuration rather than coarse geographic cues.
This paper proposes a YOLO- and CLIP-based vision-language framework to classify mosquito flight frames for Dengue virus detection, achieving 98.54% accuracy and 99.91% sensitivity at frame level, with complete video-level performance after temporal aggregation.
This paper proposes using off-the-shelf CLIP-style multimodal encoders with a bimodal attribution method to predict gaze behavior in visual world experiments, successfully replicating a seminal study on human predictive processing without fine-tuning.
CT-Merging proposes a method to merge LoRA adapters by estimating consensus directions from task subspace projectors and assigning task-level RMS coefficient scales, achieving superior performance on the DC-Merge CLIP adapter benchmark.
This paper proposes modeling the CLIP latent space using Mixtures of von Mises–Fisher distributions on the unit hypersphere, capturing its directional and multimodal structure better than Gaussian assumptions. The model improves long-tailed and out-of-distribution detection and provides a semantic decomposition of CLIP embeddings.
MedPMC is an automated framework that transforms medical literature into high-fidelity multimodal data for foundation models, achieving significant improvements across multiple benchmarks and clinical settings.
Introduces Reward-Gated Test-Time Adaptation (RG-TTA), a reinforcement learning framework that selectively applies debiasing to CLIP models based on input bias sensitivity, resolving the fairness-utility trade-off.
ComMem proposes complementary memory systems inspired by biological memory to improve test-time adaptation of vision-language models, outperforming state-of-the-art on 15 benchmarks.
This paper presents VLM-Safe-RL, a framework that integrates frozen vision-language models into constrained MDP Lagrangian updates to provide anticipatory cost signals for safe reinforcement learning in high-speed visual control tasks. The method outperforms standard constraint-aware baselines on Safety-Gymnasium FormulaOne L2 and generalizes to held-out environments.
This paper introduces KODA (Kernel Optimization for Discrepancy Analysis), a kernel-based framework for comparing and aligning vision-language model representations by identifying sample subsets that are clustered differently across models like CLIP, SigLIP, and BLIP. The method uses contrastive embedding clustering and randomized low-dimensional approximations to scale to large datasets while providing interpretable structural differences between representations.