TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models

# TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models

Source: https://arxiv.org/html/2604.15756

Jinlun Ye1,3,5, Jiang Liao2, Runhe Lai1,3,5, Xinhua Lu1,3,5, Jiaxin Zhuang4, Zhiyong Gan2, Ruixuan Wang1,3,5†

1 Sun Yat-sen University
2 China United Network Communications Corporation Limited Guangdong Branch
3 Peng Cheng Laboratory
4 Hong Kong University of Science and Technology
5 Key Laboratory of Machine Intelligence and Advanced Computing, MOE

[email protected], [email protected]

###### Abstract

Vision-language models (VLMs) such as CLIP exhibit strong out-of-distribution (OOD) detection capabilities by aligning visual and textual representations. Recent CLIP-based test-time adaptation methods further improve detection performance by incorporating external OOD labels. However, such labels are finite and fixed, while the real OOD semantic space is inherently open-ended. Consequently, fixed labels fail to represent the diverse and evolving OOD semantics encountered in test streams. To address this limitation, we introduce Test-time Textual Learning (TTL), a framework that dynamically learns OOD textual semantics from unlabeled test streams, without relying on external OOD labels. TTL updates learnable prompts using pseudo-labeled test samples to capture emerging OOD knowledge. To suppress noise introduced by pseudo-labels, we introduce an OOD knowledge purification strategy that selects reliable OOD samples for adaptation while suppressing noise. In addition, TTL maintains an OOD Textual Knowledge Bank that stores high-quality textual features, providing stable score calibration across batches. Extensive experiments on two standard benchmarks with nine OOD datasets demonstrate that TTL consistently achieves state-of-the-art performance, highlighting the value of textual adaptation for robust test-time OOD detection. Our code is available at https://github.com/figec/TTL.

## 1 Introduction

![Figure 1: Comparison with existing OOD adaptation methods. (a) Existing methods adapt visual features within a fixed text space, limiting adaptation to samples that fall outside these predefined semantic scopes. (b) Our proposed TTL framework performs test-time textual adaptation by learning and purifying new text knowledge from unlabeled test streams. By aligning these learnable OOD prompts with OOD pseudo-labeled images, it captures clearer OOD knowledge across batches and enhances OOD detection performance. (c) Feature space visualization showing how TTL's adaptive text representations better align with image features compared to existing methods.](figure1.png)

Deep learning demonstrates remarkable performance in closed-set scenarios where training and testing data follow identical distributions. However, when deployed in open-world environments, models inevitably encounter OOD data from unknown classes. Critically, models often misclassify such OOD samples as high-confidence in-distribution (ID) classes, posing severe safety risks in critical applications such as autonomous driving and medical diagnosis. Therefore, accurately detecting OOD data is essential to ensure the reliability and safety of AI systems in real-world deployments.

Traditional OOD detection methods rely on pre-trained ID classifiers but are limited to the visual modality. The emergence of VLMs like CLIP has enabled leveraging multi-modal information for enhanced OOD detection. Recent VLM-based methods have been dedicated to learning OOD knowledge through external images or text labels, or extracting such knowledge solely from ID training data, such as background regions or randomly cropped images. However, OOD features derived from specific datasets inherently fail to capture the infinite OOD distributions encountered in the real world. In order to obtain more practical OOD knowledge, recent methods employ test-time adaptation to adapt VLMs to true OOD distributions. Some intuitive methods train the OOD detector using pseudo-labeled test samples. However, these batch-wise parameter update methods can easily lead to catastrophic forgetting and unstable OOD detection performance. To mitigate catastrophic forgetting and detection fluctuation, while OODD only stores visual features during testing, AdaNeg additionally leverages the textual modality by aligning external OOD text semantics with actual test distributions, achieving stronger OOD detection performance. Nevertheless, AdaNeg depends on a finite and fixed set of OOD labels to represent the open-ended OOD semantic space, which is inherently insufficient. As a result, it struggles to adapt to OOD samples that fall outside these predefined semantic scopes (as shown in Figure 1c), leading to limited performance.

Inspired by prompt learning, fine-tuning prompts enables text features to better align with actual data distributions. We thus pose a natural question: Would directly learning OOD textual semantics from the test stream—rather than aligning specific labels with OOD distributions—yield better adaptation results?

To this end, we propose Test-time Textual Learning (TTL), a novel framework that dynamically learns OOD textual semantics from unlabeled test streams and eliminates the dependence on external OOD labels. Our key insight lies in learning a set of OOD prompts that contain diverse and clearer OOD knowledge, ultimately improving OOD detection performance. Specifically, we introduce a learnable OOD prompt for each ID class. Our TTL can acquire valuable OOD textual semantics by amplifying the semantic similarity between test samples assigned OOD pseudo-labels and the learnable OOD prompts. Moreover, we propose an OOD knowledge purification strategy to suppress noise from pseudo-labels by decreasing the semantic similarity between OOD prompts and ID boundary samples (i.e., low-confidence pseudo-OOD samples). In this way, the OOD prompts can learn clearer OOD knowledge across test streams and facilitate advanced discernment between ID and OOD data. Furthermore, to ensure stable detection and provide broader semantic coverage through diverse distributions, TTL maintains an OOD textual knowledge bank that maintains a dynamic repository of high-quality OOD text features. During inference, the OOD textual knowledge bank is used to calibrate the final OOD detection score.

Our key contributions can be summarized as follows:

- We propose a Test-time Textual Learning (TTL) framework that dynamically learns OOD textual semantics from test streams without relying on external OOD labels.
- We present a novel purification strategy to reduce pseudo-label noise and obtain clearer OOD knowledge.
- Comprehensive experiments across multiple benchmarks demonstrate that TTL achieves consistent and significant improvements over state-of-the-art methods, including average gains of 12.67% FPR95 and 3.94% AUROC.

## 2 Related Work

**OOD Detection.** Traditional OOD detection methods focus on single-modal image analysis, falling into two categories. The first designs scoring functions using model outputs (e.g., logits, features, layer statistics). The second explores ID-OOD decision boundaries via various training strategies. Though achieving satisfactory results, they overlook textual modalities' rich semantic information, leading to sub-optimal performance. To leverage textual knowledge, recent research has focused on employing vision-language models like CLIP with powerful multimodal understanding capabilities. These VLM-based methods can be categorized into three main strategies.

Concept matching methods such as MCM and CMA leverage CLIP's image-text alignment to generate OOD scores based on category names or auxiliary concepts. GL-MCM extends MCM to multi-object scenarios. ID-enhanced methods improve discrimination by exploiting additional information from ID data. For example, FA uses ID prompts as references for learnable prompt optimization; LoCoOp and SCT employ entropy maximization to reduce background sensitivity; OSPCoOp, IDLike, Negprompt, and Local-Prompt generate pseudo-OOD samples through background extraction, image cropping, or assuming the relationship between OOD distribution and ID distribution. External knowledge-based methods improve OOD detection using OOD-related information. For example, Neglabel, CSP, and NegRefine collect potential OOD labels from large-scale corpora, while CLIPN learns negative prompts from massive datasets. APT employs entropy maximization by introducing external OOD data. However, such methods prove impractical due to the inherent diversity and infinite nature of real-world OOD distribution.

In order to obtain more practical OOD knowledge, there has been growing interest in leveraging information from real-time testing scenarios to assist OOD detection. AUTO updates the parameters of all batch normalization layers and the final feature block in the model by reducing the prediction confidence of potential OOD samples. Unlike updating the original model, AdaND trains an additional noise detector. On the other hand, OODD maintains a priority queue to accumulate more representative OOD image features, which are used to calibrate detector outputs for test samples. AdaNeg further exploits external textual labels to guide the selection and storage of visual features, achieving stronger performance. Despite these advances, existing test-time methods focus primarily on visual-side adaptation with fixed text labels. In contrast, our method actively learns discriminative OOD textual knowledge during testing, directly exploiting the adaptive potential of the text modality.

**Prompt Learning.** Originated in NLP as a replacement for manual prompt engineering, prompt learning has been adapted to vision-language scenarios with VLMs like CLIP serving as strong baselines. Methods such as CoOp use learnable vectors for template words, boosting CLIP's performance across tasks. While applied to OOD detection, existing methods operate at training-time with labeled ID data. Ours is the first to apply it to test-time OOD detection, adaptively learning textual knowledge aligned with real-world OOD distributions—enabling dynamic adaptation without pre-defined OOD categories or external datasets.

## 3 Methodology

### 3.1 Preliminaries

![Figure 2: Overview of the proposed TTL framework. (a) Adaptation with TTL: During test time adaptation, pseudo labels produced by a base OOD detector are used to optimize the learnable OOD prompts, allowing the model to gradually acquire OOD textual knowledge. To reduce noise in pseudo labels, an OOD knowledge purification strategy is introduced to distinguish reliable OOD samples from ID boundary samples. The learned OOD textual features are then updated in the OOD textual knowledge bank. (b) Calibration of Prediction: During inference, the base detector's predictions are further calibrated using the OOD textual knowledge bank.](figure2.png)

**OOD Detection.** The goal of OOD detection is to distinguish ID samples from OOD ones. For a test sample **x**, this can be formulated as a binary classification task. The decision function D(**x**) can be defined as

$$D(\mathbf{x})=\begin{cases}
1~(\text{ID}),&\text{if }S(\mathbf{x})\geq\lambda\\
0~(\text{OOD}),&\text{if }S(\mathbf{x})<\lambda
\end{cases}\,,$$

where the score function S(·) is obtained from a base OOD detector, and λ is the decision threshold.

**CLIP and Threshold Determination.** CLIP-based OOD detection leverages both visual and textual modalities for improved discrimination. CLIP contains two encoders. The image encoder f(·) converts an input image **x** into a feature vector **z** = f(**x**) ∈ ℝ^d. The text encoder g(·) converts the manual template **u**_c (i.e. "a photo of a {class label}") into a feature vector **t**_c = g(**u**_c) ∈ ℝ^d, c ∈ {1, ⋯, N}, where N represents the number of ID classes. We refer to these templates **u**_c as ID prompts. The cosine similarity cos(**z**, **t**_c) measures the degree of image-text alignment. For CLIP-based OOD detection, following prior study, the decision threshold λ is adaptively determined by minimizing intra-class variance based on the bimodal distribution of OOD detection scores as follows:

$$\min_{\lambda}\frac{1}{N_{\text{id}}}\sum_{S(\mathbf{x}_i)>\lambda}{(S(\mathbf{x}_i)-\mu_{\text{id}})^2}+\frac{1}{N_{\text{ood}}}\sum_{S(\mathbf{x}_j)\leq\lambda}{(S(\mathbf{x}_j)-\mu_{\text{ood}})^2},$$

where $\mu_{\text{id}}=\frac{1}{N_{\text{id}}}\sum_{S(\mathbf{x}_i)>\lambda}S(\mathbf{x}_i)$ and $\mu_{\text{ood}}=\frac{1}{N_{\text{ood}}}\sum_{S(\mathbf{x}_j)\leq\lambda}S(\mathbf{x}_j)$.