Labeled-Data-Free Meta-Learning: Efficient Task Generation Using Pre-trained Models and Unlabeled Data

# Labeled-Data-Free Meta-Learning: Efficient Task Generation Using Pre-trained Models and Unlabeled Data
Source: [https://arxiv.org/html/2607.02850](https://arxiv.org/html/2607.02850)
Lei Sun Division of Information Science Nara Institute of Science and Technology Nara, Japan sun\.lei\.sn5@is\.naist\.jp&Yusuke Tanaka NTT Communication Science Laboratories, NTT, Inc\. Kyoto, Japan RIKEN Center for Advanced Intelligence Project RIKEN, Saitama, Japan ysk\.tanaka@ntt\.com&Tomoharu Iwata NTT Communication Science Laboratories, NTT, Inc\. Kyoto, Japan tomoharu\.iwata@ntt\.com

###### Abstract

Meta\-learning without labeled data is crucial for real\-world applications, where obtaining labeled datasets can be expensive or restricted due to privacy concerns\. Data\-Free Meta\-Learning \(DFML\) addresses this challenge by leveraging pre\-trained models without access to training data\. However, existing DFML methods rely on model inversion to generate training data, a process that is generally difficult and computationally expensive due to the need of generating high\-dimensional data matching the original distribution\. To address this limitation, we propose a novel meta\-learning setting that avoids model inversion by jointly leveraging pre\-trained models and unlabeled data\. Our method generates meta\-training tasks by assigning soft labels from pre\-trained models to unlabeled data\. Since the quality of these tasks can vary, we introduce a task\-weighting mechanism based on task confidence and class distribution balance to ensure effective meta\-learning\. Extensive experiments demonstrate that our approach substantially reduces computational cost and improves generalization, achieving up to 104×\\timesspeedup and 8\.4%–36\.4% improvements in few\-shot classification accuracy compared to state\-of\-the\-art DFML methods\.

## 1Introduction

Meta\-learning, also known as “learning to learn”, enables models to leverage prior experience from solving diverse tasks, thereby facilitating the acquisition of inductive bias for the efficient learning of unseen but related tasks\(Finnet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib2); Snellet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib1); Liet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib3); Iwata and Kumagai,[2020](https://arxiv.org/html/2607.02850#bib.bib23); Sunet al\.,[2024](https://arxiv.org/html/2607.02850#bib.bib15); Iwata and Kumagai,[2022](https://arxiv.org/html/2607.02850#bib.bib24)\)\. Traditional meta\-learning approaches typically rely on a collection of tasks with labeled datasets\. However, in many real\-world scenarios, labeled data are often difficult or impossible to obtain due to concerns such as data privacy, security risks, and usage restrictions\(Chenet al\.,[2019](https://arxiv.org/html/2607.02850#bib.bib5); Truonget al\.,[2021](https://arxiv.org/html/2607.02850#bib.bib6)\)\. Indeed, numerous individuals and institutions release task\-specific pre\-trained models on platforms like GitHub or Hugging Face, yet rarely provide access to their original training data\.

To address this restriction, researchers have proposed Data\-Free Meta\-Learning \(DFML\)\(Huet al\.,[2023a](https://arxiv.org/html/2607.02850#bib.bib7); Weiet al\.,[2024](https://arxiv.org/html/2607.02850#bib.bib8); Huet al\.,[2023b](https://arxiv.org/html/2607.02850#bib.bib9); Wanget al\.,[2022](https://arxiv.org/html/2607.02850#bib.bib10)\)\. DFML aims to extract knowledge directly from a collection of pre\-trained models without requiring access to their original training data, enabling adaptation to unseen tasks\. However, existing DFML approaches rely on model inversion to reconstruct training distributions, which requires high computational costs and results in degraded predictive accuracy\. In particular, these methods typically train generators through model inversion\(Frikhaet al\.,[2023](https://arxiv.org/html/2607.02850#bib.bib30); Patelet al\.,[2023](https://arxiv.org/html/2607.02850#bib.bib31)\)to synthesize images that approximate the original training distribution of each pre\-trained model\. Because this process involves generating high\-dimensional data, it requires hundreds of iterative generate–forward–backward steps, resulting in an extremely time\-consuming recovery process\. Moreover, recovering the original training distribution through model inversion is generally difficult, as it cannot always generate high\-quality data that matches the original distributions\. Meta\-learning with low\-quality or mismatched data leads to error accumulation and degrades the performance of DFML methods\.

To overcome this limitation, we propose a new problem setting that avoids model inversion by leveraging both pre\-trained models and unlabeled data during the meta\-training phase\. Figure[1](https://arxiv.org/html/2607.02850#S1.F1)shows the proposed problem setting\. Under this setting, we develop a meta\-learning method that constructs meta\-training tasks by assigning soft labels \(i\.e\., probability distributions over classes\), generated from multiple pre\-trained models, to the unlabeled dataset\.

In the proposed setting, we assume that pre\-trained models can be evaluated on the unlabeled instances to generate soft labels\. We consider pre\-trained models trained on classification tasks involving the same input modality and the same or related application domains as the target tasks, although their original training data may differ from the data used in the target tasks\. The pre\-trained models may also be heterogeneous in architecture, scale, and convergence levels\.

Due to potential variations in the training domains and convergence levels of pre\-trained models, the quality of the generated soft labels may differ\. Consequently, directly utilizing tasks generated by these models may introduce noise into the meta\-training process\. Therefore, it is necessary to quantify the quality of each meta\-training task\. To address this issue, we propose a task\-weighting strategy that combines two factors: \(i\) the average negative entropy of soft labels for all samples within a task, reflecting the overall task confidence; and \(ii\) the entropy of the mean soft label across samples in the task, indicating how evenly samples are distributed across classes\. Specifically, higher values of factor \(i\) reflect greater task confidence, thereby providing clearer supervisory signals and enhancing stability during meta\-training\. However, excessively high negative entropy could indicate a concentration of samples within only one or a few classes, potentially causing class bias and negatively impacting model generalization\. Therefore, we introduce factor \(ii\) to quantify class distribution balance, where higher values indicate a more uniform distribution across classes\.

The proposed method avoids model inversion, thereby reducing computational costs, and introduces a task\-weighting strategy that evaluates the usefulness of tasks generated by different pre\-trained models, enabling more effective meta\-training\. Instead, the meta\-training phase requires unlabeled data that are related, but not necessarily identical, to the target tasks\. Here, “related” means that the unlabeled data come from the same input modality and application domain as the target tasks, although they may differ from the data used in the target tasks\. This requirement is often practical because unlabeled data are generally easier to collect than labeled data\. For example, in medical diagnosis, the target task may involve diagnosing specific diseases from only a few labeled examples, whereas hospitals can often collect large amounts of unlabeled medical images, such as X\-rays or CT scans, through routine clinical practice\. Although these unlabeled data may involve disease categories or patient populations different from those in the target tasks, they can still be considered related because they come from the same medical imaging domain\.

The main contributions of this paper are as follows\. \(i\) We propose a meta\-learning setting that enables the joint utilization of pre\-trained models and unlabeled data, aligning more closely with real\-world scenarios\. \(ii\) We propose a task construction method that avoids data recovery by leveraging soft labels generated by pre\-trained models from an unlabeled dataset\. \(iii\) We propose a task\-weighting mechanism that discerns the usefulness of tasks generated by various pre\-trained models, resulting in effective meta\-training algorithms\. \(iv\) Extensive experiments validate the superiority of our method across multiple benchmarks\.

![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_1.png)Figure 1:Illustration of the proposed setting\. During the meta\-training phase, only pre\-trained models and unlabeled data related to the target tasks are available, while the original training datasets of the pre\-trained models remain inaccessible\. In the meta\-testing phase, the model is adapted to target tasks using a few labeled instances\.Table 1:Comparison of the proposed setting with related problem settings\. We compared the settings using four items\. The first item \(A\) indicates whether learning can be performed without labeled training data during the corresponding training or adaptation phase\. The second item \(B\) indicates whether unlabeled data can be used\. The third item \(C\) indicates whether pre\-trained models are used as available information sources\. The last item \(D\) indicates whether the setting supports adaptation to unseen target tasks\.
## 2Related works

Many meta\-learning methods have been proposed\(Finnet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib2); Snellet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib1); Liet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib3); Iwata and Kumagai,[2020](https://arxiv.org/html/2607.02850#bib.bib23); Sunet al\.,[2024](https://arxiv.org/html/2607.02850#bib.bib15); Iwata and Kumagai,[2022](https://arxiv.org/html/2607.02850#bib.bib24)\), but they typically rely on a collection of tasks with labeled datasets\. In practice, labeled data are often costly or infeasible to obtain due to high costs and privacy concerns\. Data\-Free Meta\-Learning \(DFML\)\(Huet al\.,[2023a](https://arxiv.org/html/2607.02850#bib.bib7); Weiet al\.,[2024](https://arxiv.org/html/2607.02850#bib.bib8); Huet al\.,[2023b](https://arxiv.org/html/2607.02850#bib.bib9); Wanget al\.,[2022](https://arxiv.org/html/2607.02850#bib.bib10)\)addresses this issue by leveraging multiple pre\-trained models to enable adaptation to unseen tasks without accessing original training data\. However, existing DFML approaches rely on model inversion\-based data recovery, which is computationally expensive and fails to effectively utilize unlabeled data often available in real\-world scenarios\.

Unsupervised Meta\-Learning \(UML\)\(Hsuet al\.,[2019](https://arxiv.org/html/2607.02850#bib.bib25); Khodadadehet al\.,[2019](https://arxiv.org/html/2607.02850#bib.bib26); Janget al\.,[2023](https://arxiv.org/html/2607.02850#bib.bib17)\)constructs synthetic tasks from unlabeled data to acquire inductive bias for learning of unseen tasks\. These methods typically require a large\-scale, unlabeled dataset, which is difficult to obtain in practice, especially in sensitive domains like healthcare and finance, where cross\-institutional data sharing is heavily restricted\. Moreover, UML approaches do not make use of existing pre\-trained models\.

Source\-Free Domain Adaptation \(SFDA\)\(Mitsuzumiet al\.,[2024](https://arxiv.org/html/2607.02850#bib.bib27); Karimet al\.,[2023](https://arxiv.org/html/2607.02850#bib.bib28); Leeet al\.,[2022](https://arxiv.org/html/2607.02850#bib.bib29)\)leverages a pre\-trained model from the source domain and unlabeled data from the target domain to achieve domain adaptation\. Although both SFDA and the proposed method leverage unlabeled data and pre\-trained models to improve performance, existing SFDA methods are only applicable when the target task is predefined and cannot be directly extended to meta\-learning frameworks\.

Knowledge distillation \(KD\)\(Gouet al\.,[2021](https://arxiv.org/html/2607.02850#bib.bib19); Wang and Yoon,[2021](https://arxiv.org/html/2607.02850#bib.bib35); Moslemiet al\.,[2024](https://arxiv.org/html/2607.02850#bib.bib36)\)trains a student model using the predictions of a pre\-trained teacher model\. Although both KD and our proposed method exploit soft labels generated by pre\-trained models, standard KD generally aims to train the student to match the teacher’s predictions on a predefined task, rather than to enable adaptation to unseen tasks\. Some studies have incorporated KD into meta\-learning frameworks, such as KD\-MAML\(Zhanget al\.,[2020](https://arxiv.org/html/2607.02850#bib.bib18)\), which relies on labeled data to construct meta\-training tasks\. By contrast, our method constructs meta\-training tasks from unlabeled data and multiple pre\-trained models, without requiring access to the data originally used to train those models\. Table[1](https://arxiv.org/html/2607.02850#S1.T1)summarizes the assumptions of each problem setting\.

![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_2.png)Figure 2:Illustration of the proposed method\. At the meta\-training phase, we generate meta\-training datasets by assigning soft labels to the unlabeled dataset using pre\-trained models\. Task weights are then computed based on these soft labels, and the model is trained by minimizing the weighted meta\-training loss\. At the meta\-testing phase, the trained model predicts the label of an input instance using a few labeled data\.
## 3Problem formulation

At the meta\-training phase, we are given a collection of pre\-trained models, denoted byℳ=\{Mt\}t=1T\\mathcal\{M\}=\\\{M\_\{t\}\\\}\_\{t=1\}^\{T\}, where each pre\-trained modelMt:𝒳→ΔCt−1M\_\{t\}:\\mathcal\{X\}\\to\\Delta^\{C\_\{t\}\-1\}performs classification over input space𝒳\\mathcal\{X\}by producing a probability distribution overCtC\_\{t\}classes\. Here,ΔCt−1\\Delta^\{C\_\{t\}\-1\}denotes the set of non\-negativeCtC\_\{t\}\-dimensional vectors that sum to one\. Note that the original training data for these models are not accessible and may differ from the data used in the target tasks\. Additionally, we are given an unlabeled dataset𝒰=\{𝐱nu\}n=1N\\mathcal\{U\}=\\\{\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\\\}\_\{n=1\}^\{N\}, where each𝐱nu∈𝒳\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\\in\\mathcal\{X\}denotes thenn\-th input instance, andNNdenotes the number of instances\. This unlabeled dataset may differ from the data used in the target tasks, but it is assumed to come from the same input modality and application domain as the target tasks\.

At the meta\-testing phase, we are given a dataset on a target task, denoted as the support set𝒮=\{\(𝐱ns,yns\)\}n=1N𝒮\\mathcal\{S\}=\\\{\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{s\}\},y\_\{n\}^\{\\mathrm\{s\}\}\)\\\}\_\{n=1\}^\{N\_\{\\mathcal\{S\}\}\}, where each𝐱ns∈𝒳\\mathbf\{x\}\_\{n\}^\{\\mathrm\{s\}\}\\in\\mathcal\{X\}denotes thenn\-th input instance,ynsy\_\{n\}^\{\\mathrm\{s\}\}represents the label corresponding to𝐱ns\\mathbf\{x\}\_\{n\}^\{\\mathrm\{s\}\}\. Here, the number of instancesN𝒮N\_\{\\mathcal\{S\}\}is small\. Our aim is to improve the test label prediction performance in target tasks\.

Overall, the original training data of the pre\-trained models, the unlabeled data, and the data used in the target tasks are assumed to share the same input modality and to be from the same or related application domains, although they may differ from one another\.

## 4Proposed method

We propose a meta\-learning method that learns from both pre\-trained models and unlabeled data\. Our method aims to construct a set of meta\-training tasks by utilizing unlabeled data and pre\-trained models\. To achieve this, we generate meta\-training datasets by assigning soft labels to unlabeled dataset𝒰\\mathcal\{U\}using pre\-trained modelsℳ\\mathcal\{M\}\. The datasets for meta\-training tasks are denoted by𝒰^=\{𝒰^t\}t=1T\\hat\{\\mathcal\{U\}\}=\\\{\\hat\{\\mathcal\{U\}\}\_\{t\}\\\}\_\{t=1\}^\{T\}, where

𝒰^t=\{\(𝐱nu,Mt​\(𝐱nu\)\)\}n=1N,\\hat\{\\mathcal\{U\}\}\_\{t\}=\\left\\\{\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\},M\_\{t\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)\)\\right\\\}\_\{n=1\}^\{N\},\(1\)is thett\-th meta\-training dataset, andMt​\(𝐱nu\)M\_\{t\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)denotes the soft label generated by thett\-th pre\-trained model for input𝐱nu\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}and is defined as

Mt​\(𝐱nu\)=\[Mt1​\(𝐱nu\),Mt2​\(𝐱nu\),⋯,MtCt​\(𝐱nu\)\]\.M\_\{t\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)=\\left\[M\_\{t\}^\{1\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\),M\_\{t\}^\{2\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\),\\cdots,M\_\{t\}^\{C\_\{t\}\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)\\right\]\.\(2\)Here,Mtc​\(𝐱nu\)M\_\{t\}^\{c\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)denotes the predicted probability that input𝐱nu\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}belongs to classccaccording to thett\-th pre\-trained model, with∑c∈CtMtc​\(𝐱nu\)=1\\sum\_\{c\\in C\_\{t\}\}M\_\{t\}^\{c\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)=1andMtc​\(𝐱nu\)≥0M\_\{t\}^\{c\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)\\geq 0for allcc\.

Tasks generated by multiple pre\-trained models vary in usefulness for meta\-training\. To quantify this, we introduce the following task weights based on the soft labels:

wt=exp⁡\(−α​1N​∑n=1NH​\(Mt​\(𝐱nu\)\)\+β​H​\(1N​∑n=1NMt​\(𝐱nu\)\)\),\{w\}\_\{t\}=\\exp\\left\(\-\\alpha\\frac\{1\}\{N\}\\sum\\limits\_\{n=1\}^\{N\}H\(M\_\{t\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)\)\+\\beta H\(\\frac\{1\}\{N\}\\sum\\limits\_\{n=1\}^\{N\}M\_\{t\}\(\\mathbf\{x\}\_\{n\}^\{\\mathrm\{u\}\}\)\)\\right\),\(3\)whereH​\(⋅\)H\(\\cdot\)is the entropy,α\\alphaandβ\\betaare hyperparameters that balance the contributions of two entropy\-based components in the weight computation, andα,β∈ℝ≥0\\alpha,\\beta\\in\\mathbb\{R\}\_\{\\geq 0\}\. Our task\-weighting strategy computes each task weight by combining two factors: \(i\) the negative entropy of soft labels averaged over all samples, which measures predictive uncertainty; and \(ii\) the entropy of the mean soft label across all samples in the task, which measures how uniformly the unlabeled samples are distributed across classes\. Specifically, for \(i\), a higher average negative entropy indicates lower predictive uncertainty; thus, pre\-trained models that are more relevant to the target task are expected to produce lower uncertainty predictions on unlabeled data\. For \(ii\), the entropy of the mean soft label quantifies how uniformly samples are distributed across classes\. Under the assumption that the target task maintains a balanced class distribution, this term reflects class\-distribution alignment\. Based on these two factors, we assign higher weights to tasks exhibiting both lower predictive uncertainty and a more uniform predicted class distribution, as these tasks are more likely to align with the target task\.

We consider modelf​\(⋅;ψ,θ\)f\(\\cdot;\\psi,\\theta\)with task\-shared parametersθ\\thetaand task\-specific parametersψ\\psi, which is trained by a bi\-level optimization framework\. Task\-shared parametersθ\\thetaare optimized during the outer optimization, task\-specific parametersψ\\psiare adapted through the inner optimization\. Modelf​\(𝐱;ψ,θ\)f\(\\mathbf\{x\};\\psi,\\theta\)outputs theCtC\_\{t\}\-dimensional probability distribution over classes for input𝐱\\mathbf\{x\}\. Once the meta\-training tasks and their corresponding weights are determined, the objective to be minimized is the following expected weighted test error:

minθ𝔼𝒰^t∼𝒰^\[wt𝔼\(𝒮,𝒬\)∼𝒰^tLouter\(𝒬,f\(⋅,ψ𝒮,θ\)\)\],s\.t\.ψ𝒮=argminψLinner\(𝒮,f\(⋅,ψ,θ\)\),\\min\_\{\\theta\}\\mathbb\{E\}\_\{\\hat\{\\mathcal\{U\}\}\_\{t\}\\sim\\hat\{\\mathcal\{U\}\}\}\\left\[\{w\}\_\{t\}\\mathbb\{E\}\_\{\(\\mathcal\{S\},\\mathcal\{Q\}\)\\sim\\hat\{\\mathcal\{U\}\}\_\{t\}\}L^\{\\mathrm\{outer\}\}\(\\mathcal\{Q\},f\(\\cdot,\\psi\_\{\\mathcal\{S\}\},\\theta\)\)\\right\],\\quad\{\\rm s\.t\.\}\\quad\\psi\_\{\\mathcal\{S\}\}=\\arg\\min\_\{\\psi\}L^\{\\mathrm\{inner\}\}\(\\mathcal\{S\},f\(\\cdot,\\psi,\\theta\)\),\(4\)where𝔼\\mathbb\{E\}denotes the expectation\.Linner​\(𝒮,f​\(⋅,ψ,θ\)\)L^\{\\mathrm\{inner\}\}\(\\mathcal\{S\},f\(\\cdot,\\psi,\\theta\)\)is the inner loss used to adaptψ\\psion support set𝒮\\mathcal\{S\}, whileLouter​\(𝒬,f​\(⋅,ψ,θ\)\)L^\{\\mathrm\{outer\}\}\(\\mathcal\{Q\},f\(\\cdot,\\psi,\\theta\)\)is the outer loss used to evaluate the generalization performance ofθ\\thetaon query set𝒬\\mathcal\{Q\}\. The support and query sets are subsets of𝒰^t\\hat\{\\mathcal\{U\}\}\_\{t\}and are disjoint, i\.e\.,𝒮∩𝒬=∅\\mathcal\{S\}\\cap\\mathcal\{Q\}=\\emptyset\. Each set contains pairs of input instances and soft labels, which are randomly sampled from𝒰^t\\hat\{\\mathcal\{U\}\}\_\{t\}\. When the meta\-training tasks constructed from unlabeled data and pre\-trained models are related to the target tasks, training the task\-shared parameters using bi\-level optimization on these tasks can enhance the model’s performance on the target tasks\. However, due to variations among pre\-trained models, the quality of soft labels in the generated meta\-learning tasks may be inconsistent\. To address this, our proposed task\-weighting mechanism can assign weights to each task based on task confidence and class balance, enabling the model to discern the relative usefulness of different tasks\. By minimizing the weighted meta\-test error defined in Eq\. \([4](https://arxiv.org/html/2607.02850#S4.E4)\), our approach achieves more effective meta\-training\.

In outer optimization, we use the following loss:

Louter​\(𝒬,f​\(⋅;ψ𝒮,θ\)\)=1NQ​∑\(𝐱,Mt​\(𝐱\)\)∈𝒬𝖪𝖫​\(Mt​\(𝐱\),f​\(𝐱;ψ𝒮,θ\)\),L^\{\\mathrm\{outer\}\}\(\\mathcal\{Q\},f\(\\cdot;\\psi\_\{\\mathcal\{S\}\},\\theta\)\)=\\frac\{1\}\{N\_\{\\mathrm\{Q\}\}\}\\sum\\limits\_\{\(\\mathbf\{x\},M\_\{t\}\(\\mathbf\{x\}\)\)\\in\\mathcal\{Q\}\}\\mathsf\{KL\}\\left\(M\_\{t\}\(\\mathbf\{x\}\),f\(\\mathbf\{x\};\\psi\_\{\\mathcal\{S\}\},\\theta\)\\right\),\(5\)where𝖪𝖫​\(⋅\)\\mathsf\{KL\}\(\\cdot\)is the Kullback\-Leibler divergence\. By formulating the outer objective as the discrepancy between the predictions generated by the pre\-trained models and those produced by modelff, task\-shared knowledge can be effectively transferred from pre\-trained models to modelff\.

In inner optimization, we find task\-specific parameters adapted to the support set with soft labels\. As soft labels carry more informative content than hard labels, their use in inner optimization enables the learning of more effective task\-specific parameters\. The overall meta\-training procedure of our model is summarized in Algorithm[1](https://arxiv.org/html/2607.02850#alg1)\. In practice, the model parameters can be trained using various meta\-learning algorithms\(Vinyalset al\.,[2016](https://arxiv.org/html/2607.02850#bib.bib12); Bertinettoet al\.,[2019](https://arxiv.org/html/2607.02850#bib.bib32)\)\. In the following paragraph, we use Prototypical Networks\(Snellet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib1)\)as an illustrative example of the optimization process\. In the meta\-testing phase, the meta\-learned modelffis adapted to the target task through inner optimization with the support set𝒮\\mathcal\{S\}, after which the adapted model is employed to predict the labels of the query instances\. Figure[2](https://arxiv.org/html/2607.02850#S2.F2)illustrates the proposed method\.

Algorithm 1Meta\-learning procedures\.Input: Pre\-trained modelsℳ=\{Mt\}t=1T\\mathcal\{M\}=\\\{M\_\{t\}\\\}\_\{t=1\}^\{T\}, unlabeled dataset𝒰=\{𝐱n\}n=1N\\mathcal\{U\}=\\\{\\mathbf\{x\}\_\{n\}\\\}\_\{n=1\}^\{N\}, number of support instancesNSN\_\{\\mathrm\{S\}\}, number of query instancesNQN\_\{\\mathrm\{Q\}\}, batch sizeBB\.

Output: Trained model parametersθ\\theta

1:Randomly initialize

θ\\theta
2:Create meta\-training datasets

𝒰^=\{𝒰^t\}t=1T\\hat\{\\mathcal\{U\}\}=\\\{\\hat\{\\mathcal\{U\}\}\_\{t\}\\\}\_\{t=1\}^\{T\}by Eq\. \([1](https://arxiv.org/html/2607.02850#S4.E1)\) using

𝒰\\mathcal\{U\}and

ℳ\\mathcal\{M\}\.

3:Compute weight

\{wt\}t=1T\\\{\{w\}\_\{t\}\\\}\_\{t=1\}^\{T\}by Eq\. \([3](https://arxiv.org/html/2607.02850#S4.E3)\) using

𝒰^\\hat\{\\mathcal\{U\}\}
4:whilenot donedo

5:Initialize loss

J←0J\\leftarrow 0
6:Randomly sample a batch of tasks

ℬ\\mathcal\{B\}from

\{1,…,T\}\\\{1,\\ldots,T\\\}
7:for

t∈ℬt\\in\\mathcal\{B\}do

8:Randomly sample support set

𝒮\\mathcal\{S\}with size

NSN\_\{\\mathrm\{S\}\}from

𝒰^t\\hat\{\\mathcal\{U\}\}\_\{t\}
9:Randomly sample query set

𝒬\\mathcal\{Q\}with size

N𝒬N\_\{\\mathcal\{Q\}\}from

𝒰^t∖𝒮\\hat\{\\mathcal\{U\}\}\_\{t\}\\setminus\\mathcal\{S\}
10:Obtain task\-specific parameter

ψ𝒮\\psi\_\{\\mathcal\{S\}\}via inner optimization:

ψ𝒮=arg⁡minψ⁡Linner​\(𝒮,f​\(⋅;ψ,θ\)\)\\psi\_\{\\mathcal\{S\}\}=\\arg\\min\\limits\_\{\\psi\}L^\{\\mathrm\{inner\}\}\(\\mathcal\{S\},f\(\\cdot;\\psi,\\theta\)\)
11:Calculate weighted outer loss by Eq\. \([5](https://arxiv.org/html/2607.02850#S4.E5)\) and

wt\{w\}\_\{t\},

J←J\+wt​Louter​\(𝒬,f​\(⋅;ψ𝒮,θ\)\)J\\leftarrow J\+\{w\}\_\{t\}L^\{\\mathrm\{outer\}\}\(\\mathcal\{Q\},f\(\\cdot;\\psi\_\{\\mathcal\{S\}\},\\theta\)\)
12:endfor

13:Update model parameters

θ\\thetausing gradient of loss

J/BJ/B
14:endwhile

#### Illustrative example based on Prototypical Networks\.

In Prototypical Networks, the outer optimization aims to learn an embedding functionϕ​\(⋅;θ\)\\phi\(\\cdot;\\theta\)with task\-shared parametersθ\\theta, and task\-specific parametersψ=\{ψc\}c=1Ct\\psi=\\\{\\psi\_\{c\}\\\}\_\{c=1\}^\{C\_\{t\}\}are a set of prototype for each class\. Specifically, modelffis defined by

fc​\(𝐱;ψ𝒮,θ\)=exp⁡\(−12​‖\(ϕ​\(𝐱;θ\)−ψ𝒮c\)‖2\)∑c′exp⁡\(−12​‖\(ϕ​\(𝐱;θ\)−ψ𝒮c′\)‖2\)\.f^\{c\}\(\\mathbf\{x\};\\psi\_\{\\mathcal\{S\}\},\\theta\)=\\frac\{\\exp\\left\(\-\\frac\{1\}\{2\}\|\|\(\\phi\(\\mathbf\{x\};\\theta\)\-\\mathbf\{\\psi\}\_\{\\mathcal\{S\}\}^\{c\}\)\|\|^\{2\}\\right\)\}\{\\sum\\limits\_\{c^\{\\prime\}\}\\exp\\left\(\-\\frac\{1\}\{2\}\|\|\(\\phi\(\\mathbf\{x\};\\theta\)\-\\mathbf\{\\psi\}\_\{\\mathcal\{S\}\}^\{c^\{\\prime\}\}\)\|\|^\{2\}\\right\)\}\.\(6\)
Each prototypeψ𝒮c\\psi\_\{\\mathcal\{S\}\}^\{c\}adapted to the support set is computed as the weighted mean of the embedded support instances, where the weights correspond to the soft label probabilities of those instances belonging to classcc:

ψ𝒮c=arg⁡minψc⁡\[−∑𝐱∈𝒮Mtc​\(𝐱\)​log⁡𝒩​\(ϕ​\(𝐱;θ\);ψc,I\)\]=∑𝐱∈𝒮Mtc​\(𝐱\)​ϕ​\(𝐱;θ\)∑𝐱∈𝒮Mtc​\(𝐱\),\\psi\_\{\\mathcal\{S\}\}^\{c\}=\\arg\\min\_\{\\psi^\{c\}\}\\left\[\-\\sum\_\{\\mathbf\{x\}\\in\\mathcal\{S\}\}M\_\{t\}^\{c\}\(\\mathbf\{x\}\)\\log\\mathcal\{N\}\(\\phi\(\\mathbf\{x\};\\theta\);\\psi^\{c\},I\)\\right\]=\\frac\{\\sum\\limits\_\{\\mathbf\{x\}\\in\\mathcal\{S\}\}M\_\{t\}^\{c\}\(\\mathbf\{x\}\)\\phi\(\\mathbf\{x\};\\theta\)\}\{\\sum\\limits\_\{\\mathbf\{x\}\\in\\mathcal\{S\}\}M\_\{t\}^\{c\}\(\\mathbf\{x\}\)\},\(7\)where the weighted negative log likelihood with Gaussian is used as the inner loss, and𝒩​\(⋅;ψc,I\)\\mathcal\{N\}\(\\cdot;\\psi^\{c\},I\)is the probability density function of a multivariate Gaussian distribution with mean vectorψc\\psi^\{c\}and identity covariance matrixII\.

## 5Experiments

### 5\.1Datasets and pre\-trained models

We conducted experiments on three widely\-used few\-shot learning benchmark datasets: Omniglot\(Lakeet al\.,[2011](https://arxiv.org/html/2607.02850#bib.bib11)\), miniImageNet\(Vinyalset al\.,[2016](https://arxiv.org/html/2607.02850#bib.bib12)\), and tieredImageNet\(Renet al\.,[2018](https://arxiv.org/html/2607.02850#bib.bib13)\)\. Following standard splits\(Wertheimeret al\.,[2021](https://arxiv.org/html/2607.02850#bib.bib14)\), we split each dataset into the meta\-training, meta\-validation, and meta\-testing subsets with disjoint label spaces\. Here, we further divided the meta\-testing dataset into two subsets with disjoint label spaces: one for sampling unlabeled data used during meta\-training, and the other for evaluation during meta\-testing\. This ensured that the unlabeled data were drawn from tasks that are distinct from those used during meta\-testing\.

For the pre\-trained models, we followed the DFML assumption that the meta\-training data are inaccessible\. For miniImageNet and tieredImageNet, following prior work\(Huet al\.,[2023a](https://arxiv.org/html/2607.02850#bib.bib7); Weiet al\.,[2024](https://arxiv.org/html/2607.02850#bib.bib8); Huet al\.,[2023b](https://arxiv.org/html/2607.02850#bib.bib9); Wanget al\.,[2022](https://arxiv.org/html/2607.02850#bib.bib10)\), we collected 100 pre\-trained models, each trained on anNN\-way classification task sampled from the meta\-training subset\. For Omniglot, we first meta\-trained the models using MAML\(Finnet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib2)\)on the meta\-training subset and subsequently fine\-tuned them on N\-way classification tasks sampled from the same subset\. This results in 100 fine\-tuned pre\-trained models\. Through this procedure, we collected a pool of pre\-trained models covering a broad range of accuracies on each dataset\. The miniImageNet pool had an average accuracy of 78\.3%, with a minimum of 31\.0% and a maximum of 92\.4%, and included 3 models with accuracies below 50%\. The tieredImageNet pool averaged 72\.5% accuracy, ranging from 29\.2% to 89\.6%, and contained 7 models below 50%\. For Omniglot \(5\-way\), the pool had an average accuracy of 90\.7%, with accuracies between 59\.7% and 99\.1%, and 0 models fell below 50%\. For Omniglot \(20\-way\), the pool averaged 75\.3%, with a minimum of 47\.2% and a maximum of 94\.2%, and included one model with accuracies below 50%\.

In the multi\-domain scenario, we evaluated the proposed method on the CUB\(Wahet al\.,[2011](https://arxiv.org/html/2607.02850#bib.bib16)\)and CIFAR\-FS\(Bertinettoet al\.,[2019](https://arxiv.org/html/2607.02850#bib.bib32)\)datasets\. The data splits were kept consistent with the standard scenario\. The key difference was that, in the multi\-domain setting, we collected a total of 100 pre\-trained models, with 50 trained on miniImageNet and the other 50 trained on CUB or CIFAR\-FS\.

### 5\.2Implementation details

For the model architecture, we adopted Conv4\(Finnet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib2); Snellet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib1)\)as the architecture of the meta\-learner and the pre\-trained models for a fair comparison with existing works\. For hyperparameters, batch sizeBBwas set to 100, Adam\(Kingma and Ba,[2014](https://arxiv.org/html/2607.02850#bib.bib34)\)was used as the optimizer, with the learning rate set to 0\.001, and balance factorsα\\alphaandβ\\betawere set to 0\.5 and 0\.5\. The experimental results under varying hyperparameters are shown in the following sections\. Number of instancesNNin unlabeled dataset𝒰\\mathcal\{U\}was set to 100 for the 5\-way setting and 400 for the 20\-way setting\. In each meta\-training task constructed according to Eq\. \([1](https://arxiv.org/html/2607.02850#S4.E1)\), the support set contained 25 samples, and the query set contained 75 samples for the 5\-way setting, and 100 and 300 samples for the 20\-way setting, respectively\. The samples for the support and query sets were randomly selected from the dataset constructed according to Eq\. \([1](https://arxiv.org/html/2607.02850#S4.E1)\)\. Results are averaged over five independent runs with different unlabeled datasets and model initializations\. Each run is evaluated on 600 meta\-test tasks\. We report the mean ± standard error\. All methods are evaluated under the same meta\-test protocol and using the same support/query splits\. All training times were measured using an NVIDIA TITAN RTX GPU\.

### 5\.3Comparative methods

We compared the proposed method with the following baselines and competing methods\.\(i\) Random:model parameters were randomly initialized and optimized from scratch using only the support set of each meta\-test task\.\(ii\) K\-nearest neighbors \(embedded\):samples were embedded using the pre\-trained model with the highest validation accuracy on the original training task, and query labels were predicted by nearest\-neighbor classification based on Euclidean distances in the embedding space\.\(iii\) K\-nearest neighbors \(pixel\):query labels were predicted by nearest\-neighbor classification using Euclidean distances computed directly in the raw pixel space without embedding\.\(iv\) Best pre\-trained model:the pre\-trained model was selected according to its validation accuracy on the original training task, fine\-tuned on the support set of each meta\-test task, and then used to predict the query labels\.\(v\) Pseudo\-supervised contrastive learning \(PsCo\)\(Janget al\.,[2023](https://arxiv.org/html/2607.02850#bib.bib17)\):a state\-of\-the\-art unsupervised meta\-learning method based on contrastive learning with a momentum network\.\(vi\) Fast and better data\-free meta\-learning \(FREE\)\(Weiet al\.,[2024](https://arxiv.org/html/2607.02850#bib.bib8)\):a state\-of\-the\-art data\-free meta\-learning method that recovers meta\-training tasks from pre\-trained models and employs gradient alignment for meta\-training\.\(vii\) Knowledge distillation for model\-agnostic meta\-learning \(KD\-MAML\)\(Zhanget al\.,[2020](https://arxiv.org/html/2607.02850#bib.bib18)\):A meta\-learning method based on knowledge distillation\. Since the original method requires labeled data during the meta\-training phase, we adapted it to our setting by training it on the meta\-training tasks constructed via Eq\. \([1](https://arxiv.org/html/2607.02850#S4.E1)\)\. Under this task construction, KD\-MAML was treated as optimizing Eq\. \([4](https://arxiv.org/html/2607.02850#S4.E4)\) without task weighting, while the inner\-loop optimization was performed using hard pseudo\-labels\.

Methods \(i\)–\(iv\) did not involve meta\-training and used only the support set of each meta\-test task for task\-specific adaptation or prediction\. Method \(v\) used only the unlabeled dataset for meta\-training, while Method \(vi\) used only the pre\-trained models\. Method \(vii\) was trained using both the unlabeled dataset and the pre\-trained models\. The proposed method \(Ours\) was trained under the same setting as in Method \(vii\)\.

Table 2:Average accuracies \(%\) on the Omniglot dataset\. Results are averaged over five independent runs with different unlabeled datasets, with standard errors reported\. Methods without unlabeled data report only average accuracies\.Boldindicates the best and statistically comparable results according to the pairedtt\-test \(p=0\.05p=0\.05\)\.Table 3:Average accuracies \(%\) on the miniImageNet and tieredImageNet dataset\. Results are averaged over five independent runs with different unlabeled datasets, with standard errors reported\. Methods without unlabeled data report only average accuracies\.Boldindicates the best and statistically comparable results according to the pairedtt\-test \(p=0\.05p=0\.05\)\.Table 4:Training computational time in hours on miniImageNet under the 5\-way 1\-shot setting\.Table 5:Average accuracies \(%\) on CUB and CIFAR\-FS in the multi\-domain scenario\. Results are averaged over five independent runs with different unlabeled datasets, with standard errors reported\. Methods without unlabeled data report only average accuracies\.Boldindicates the best and statistically comparable results according to the pairedtt\-test \(p=0\.05p=0\.05\)\.Table 6:Average accuracies \(%\) on miniImageNet under cross\-architecture scenario\. Results are averaged over five independent runs with different unlabeled datasets, with standard errors reported\. Methods without unlabeled data report only average accuracies\.Boldindicates the best and statistically comparable results according to the pairedtt\-test \(p=0\.05p=0\.05\)\.Table 7:Ablation study\. Average accuracies and standard errors \(%\) under miniImageNet 5\-way 1\-shot setting\. We compare the proposed method \(ours\) with the following variants: removing factor \(i\) in task\-weighting \(α=0,β=1\\alpha=0,\\beta=1\), removing factor \(ii\) \(α=1,β=0\\alpha=1,\\beta=0\), removing both factors \(w/o weight\) and replace soft labels with hard labels in the support \(inner hard\), query \(outer hard\)\.![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_4.png)\(a\)Omniglot 5\-way
![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_5.png)\(b\)Omniglot 20\-way
![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_3.png)\(c\)miniImageNet
![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_6.png)\(d\)tieredImageNet

Figure 3:Analysis of the balance factorsα\\alphaandβ\\betain the proposed task\-weighting mechanism under the 1\-shot setting\. Meta\-testing accuracies are reported by varyingα\\alphawithβ=1−α\\beta=1\-\\alphaon \(a\) Omniglot 5\-way, \(b\) Omniglot 20\-way, \(c\) miniImageNet 5\-way, and \(d\) tieredImageNet 5\-way tasks\. Bars indicate the standard error\.![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_7.png)\(a\)Number of pre\-trained models
![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_8.png)\(b\)Number of low\-quality models

Figure 4:Effect of the number of pre\-trained models and the number of low\-quality models under the miniImageNet 5\-way 1\-shot setting\. \(a\) Meta\-testing accuracies with different numbers of pre\-trained models\. \(b\) Meta\-testing accuracies with different numbers of low\-quality pre\-trained models\. Bars indicate the standard error\.
### 5\.4Results

Tables[2](https://arxiv.org/html/2607.02850#S5.T2)and[3](https://arxiv.org/html/2607.02850#S5.T3)presented the average accuracies \(%\) of different methods under various task settings, along with the standard errors when using different unlabeled datasets; for methods that do not utilize unlabeled data, only the average accuracy is reported\. Bold numbers indicated results that are not statistically different from the best\-performing method on each dataset at the 5% level according to a paired t\-test\. The proposed method achieved the highest accuracy\. For most methods, accuracy increased reasonably as the support set size grew\. The proposed method outperformed non\-meta\-learning baselines \(Random, K\-nearest neighbors, and Best pre\-trained model\)\. This result indicated the importance of sharing knowledge across tasks, and it is difficult to perform well with only a small number of training samples\. The accuracy of the existing unsupervised meta\-learning method \(PsCo\) was lower than that of the proposed method\. This result implies that leveraging pre\-trained models to construct diverse and target\-relevant tasks is effective\. The data\-free meta\-learning method \(FREE\) performed worse than the proposed method\. Together with the training time results shown in Table[4](https://arxiv.org/html/2607.02850#S5.T4), this indicates that incorporating unlabeled data related to the target tasks not only improves performance but also avoids costly data recovery, thereby achieving substantial speed\-ups\. Although KD\-MAML also leverages pre\-trained models and unlabeled data for meta\-learning, it ignores the differences in soft label quality and relies on hard\-label updates in the inner loop, resulting in inferior performance compared to the proposed method\. This further demonstrated that the performance improvements of the proposed method are not only due to the newly introduced setting but also attributable to the effectiveness of the method itself\.

In practical applications, publicly available pre\-trained models often originate from domains that are not perfectly aligned with the target domain, due to resource diversity and selection constraints\. To reflect this realistic scenario, we conducted experiments under a more challenging multi\-domain setting\. In this setting, the pool of pre\-trained models included not only those trained on the target domain \(CUB or CIFAR\-FS\) but also models trained on a different domain \(miniImageNet\)\. This setting requires algorithms to extract useful knowledge while demonstrating robust generalization across multiple domains\. As shown in Table[5](https://arxiv.org/html/2607.02850#S5.T5), the proposed method achieved the highest accuracy\. The results showed that, even when pre\-trained models are drawn from multiple domains, our method can effectively leverage them for robust meta\-learning\. By incorporating the proposed task\-weighting mechanism, our method discerns the usefulness of tasks generated from different models, thereby enabling efficient meta\-learning across domains\.

In real\-world applications, the diversity of publicly available resources often leads to pre\-trained models that differ in both architecture and scale\. To evaluate this scenario, we further assessed our method in a cross\-architecture setting, where each pre\-trained model adopted a different architecture\. Specifically, the architecture of each model was randomly selected from Conv4, Conv5\(Finnet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib2); Snellet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib1)\), ResNet\-10, and ResNet\-18\(Heet al\.,[2016](https://arxiv.org/html/2607.02850#bib.bib33)\)\. As shown in Table[6](https://arxiv.org/html/2607.02850#S5.T6), our method achieved the best performance among all comparative approaches and could be directly applied to cross\-architecture scenarios without modification\. These results demonstrated that our method effectively learns across heterogeneous model architectures\. This robustness arises because our method makes no assumptions regarding the architecture or scale of the pre\-trained models\.

We also conducted ablation studies to further investigate how each key component contributed to the performance of the proposed method\. We first evaluated the effectiveness of the task\-weighting mechanism\. As illustrated in Table[7](https://arxiv.org/html/2607.02850#S5.T7)\(w/o weight\), removing this mechanism results in a decline in meta\-testing performance, demonstrating its capability to accurately identify the usefulness of tasks generated by various pre\-trained models, thereby enhancing performance on unseen tasks\. We also examined the contributions of two factors within the task\-weighting mechanism\. Table[7](https://arxiv.org/html/2607.02850#S5.T7)showed that removing either component \(α=0,β=1\\alpha=0,\\beta=1orα=1,β=0\\alpha=1,\\beta=0\) leads to reduced meta\-testing performance compared to the proposed method \(Ours\)\. This highlighted the significance of both components in effectively discerning the usefulness of various meta\-training tasks\. Next, we assessed the impact of modifying the inner optimization\. Specifically, soft labels in the support set were converted into hard labels via argmax, and prototypes were computed as the mean of sample embeddings for each class\. As shown in Table[7](https://arxiv.org/html/2607.02850#S5.T7)\(inner hard\), using hard labels leads to a decrease in accuracy\. This is because converting soft to hard labels may result in insufficient samples for some classes, and prototypes calculated by a few examples are typically less robust\. Moreover, since hard labels contain less information than soft labels, this may also lead to a degradation in performance\. Finally, we evaluated the effect of modifying the query loss \(Eq\. \([5](https://arxiv.org/html/2607.02850#S4.E5)\)\) by replacing soft labels in the query set with hard labels and computing the loss using cross\-entropy\. As shown in Table[7](https://arxiv.org/html/2607.02850#S5.T7)\(outer hard\), this change results in degraded performance\. The reason is that soft labels contain richer information than hard labels, a well\-established finding in the knowledge distillation\(Gouet al\.,[2021](https://arxiv.org/html/2607.02850#bib.bib19); Phuong and Lampert,[2019](https://arxiv.org/html/2607.02850#bib.bib20); Hintonet al\.,[2015](https://arxiv.org/html/2607.02850#bib.bib21)\)\.

![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_9.png)\(a\)\(α,β\)=\(0,1\)\(\\alpha,\\beta\)=\(0,1\), PCC: \-0\.15
![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_10.png)\(b\)\(α,β\)=\(1,0\)\(\\alpha,\\beta\)=\(1,0\), PCC: \-0\.33
![Refer to caption](https://arxiv.org/html/2607.02850v1/Figure_11.png)\(c\)\(α,β\)=\(0\.5,0\.5\)\(\\alpha,\\beta\)=\(0\.5,0\.5\), PCC: \-0\.56

Figure 5:\(a–c\) Relationship between task weights \(computed using Eq\. \([3](https://arxiv.org/html/2607.02850#S4.E3)\)\) and cross\-entropy with ground\-truth labels for meta\-training tasks constructed by Eq\. \([1](https://arxiv.org/html/2607.02850#S4.E1)\)\. Each point represents a single task\. Pearson correlation coefficients \(PCC\) are also reported\.Table 8:Average accuracy \(%\) and standard error on miniImageNet under 5\-way 1\-shot and 5\-way 5\-shot settings\. Ours\-ProtoNet and Ours\-MAML denote two instantiations of the proposed framework based on Prototypical Networks\(Snellet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib1)\)and MAML\(Finnet al\.,[2017](https://arxiv.org/html/2607.02850#bib.bib2)\), respectively\.We conducted experiments under the 1\-shot setting to analyze the sensitivity of the proposed task\-weighting mechanism to the balance factorsα\\alphaandβ\\beta, whereβ=1−α\\beta=1\-\\alpha\. As shown in Figure[3](https://arxiv.org/html/2607.02850#S5.F3), using both entropy\-based factors achieved better or comparable performance than using either factor alone, i\.e\.,α=0\\alpha=0orα=1\\alpha=1, in most cases across Omniglot, miniImageNet, and tieredImageNet\. These results indicate that both factors in Eq\. \([3](https://arxiv.org/html/2607.02850#S4.E3)\) are important for improving generalization to unseen tasks\.

To further analyze the effectiveness of the weighting mechanism, we conducted experiments on miniImageNet under the 5\-way setting\. Specifically, we randomly sample 200 meta\-training tasks from those constructed by Eq\.\([1](https://arxiv.org/html/2607.02850#S4.E1)\), compute the cross\-entropy between the soft labels and ground\-truth labels for each task, and calculate the corresponding task weight using Eq\. \([3](https://arxiv.org/html/2607.02850#S4.E3)\)\. A lower cross\-entropy loss indicates that the soft labels are more consistent with the ground\-truth labels\. Note that ground\-truth labels are not accessible during the meta\-training phase\. We then visualize the relationship between task weights and cross\-entropy and compute their correlation coefficient\. As shown in Figure[5](https://arxiv.org/html/2607.02850#S5.F5)\(c\), the task weight is negatively correlated with cross\-entropy, with a correlation coefficient of \-0\.56\. This suggests that tasks with higher weights tend to have soft labels more consistent with the ground\-truth labels\. These results validate the effectiveness of our weighting strategy: tasks with more accurate soft labels provide clearer supervision signals and more balanced class distributions, both of which are essential for the meta\-training process\. Additionally, we compare the same correlation underα=1,β=0\\alpha=1,\\beta=0andα=0,β=1\\alpha=0,\\beta=1\. Although correlation is observed in both cases, the absolute values of the correlation coefficients are substantially lower than those obtained withα=0\.5,β=0\.5\\alpha=0\.5,\\beta=0\.5, further highlighting the necessity of incorporating both factors in Eq\. \([3](https://arxiv.org/html/2607.02850#S4.E3)\)\.

We also evaluated the effect of varying the number of pre\-trained models, each independently trained on miniImageNet under the 5\-way 1\-shot setting\. As shown in Figure[4\(a\)](https://arxiv.org/html/2607.02850#S5.F4.sf1), meta\-testing accuracies improved progressively with the number of pre\-trained models, up to approximately 700\. This is because a larger pool of pre\-trained models enables the construction of more diverse and target\-relevant meta\-training tasks\. This helps the model acquire a more robust inductive bias, facilitating faster and more effective adaptation to unseen tasks\.

In practical applications, the diversity of publicly available resources often leads to the inclusion of low\-quality models\. To investigate the impact of low\-quality models, we evaluated the effect of varying their proportion in the pre\-trained model pool under the miniImageNet 5\-way 1\-shot setting\. As described in Section[5\.1](https://arxiv.org/html/2607.02850#S5.SS1), the miniImageNet pool already contained 3 low\-quality models \(with accuracies below 50%\)\. To further adjust the proportion of low\-quality models, we replaced a subset of the pool with models initialized with random parameters\. As Figure[4\(b\)](https://arxiv.org/html/2607.02850#S5.F4.sf2)showed, meta\-testing accuracy decreased as the proportion of low\-quality models increased\. Nevertheless, accuracy remained relatively stable at around 33–34% when the number of low\-quality models ranged between 3 and 30\. Once the number exceeded 60, performance began to decline\. As expected, accuracy dropped sharply when most of the pre\-trained models were low\-quality \(close to 100\), as reliable soft labels were difficult to obtain under such conditions\. These results demonstrated that, even with a substantial proportion of low\-quality models, the proposed task\-weighting mechanism can effectively identify the usefulness of tasks generated from pre\-trained models, thereby enabling efficient meta\-learning\.

We further evaluated whether the proposed framework is tied to a specific meta\-learning algorithm\. As shown in Table[8](https://arxiv.org/html/2607.02850#S5.T8), although Ours\-ProtoNet achieved slightly higher average accuracies than Ours\-MAML on miniImageNet under both the 5\-way 1\-shot and 5\-way 5\-shot settings, the differences were not statistically significant\. In addition, Ours\-MAML outperformed all baseline methods, indicating that the proposed framework can also improve a gradient\-based meta\-learning algorithm such as MAML\. Since Ours\-MAML and KD\-MAML used the same meta\-learning algorithm, the improvements of Ours\-MAML over KD\-MAML further demonstrated the effectiveness of the proposed soft\-label task construction and task\-weighting mechanism, rather than being attributable to ProtoNets alone\.

## 6Conclusion

We proposed a meta\-learning setting that leverages both pre\-trained models and unlabeled data\. In this setting, our method constructs meta\-training tasks by assigning soft labels from pre\-trained models to unlabeled data, thereby avoiding the high computational cost and difficulties of data recovery\. To further ensure effective meta\-learning, we introduce a task\-weighting mechanism based on confidence and class balance\. We believe this work is an important step for learning from diverse information sources, and future research may extend this framework to broader tasks such as regression and reinforcement learning\.

## Funding statement

This work was supported by JST SPRING, Grant Number JPMJSP2140, Japan\.

## References

- Meta\-learning with differentiable closed\-form solvers\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HyxnZh0ct7)Cited by:[§4](https://arxiv.org/html/2607.02850#S4.p5.2),[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p3.1)\.
- H\. Chen, Y\. Wang, C\. Xu, Z\. Yang, C\. Liu, B\. Shi, C\. Xu, C\. Xu, and Q\. Tian \(2019\)Data\-free learning of student networks\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 3514–3522\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p1.1)\.
- C\. Finn, P\. Abbeel, and S\. Levine \(2017\)Model\-agnostic meta\-learning for fast adaptation of deep networks\.InInternational conference on machine learning,pp\. 1126–1135\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p1.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p2.1),[§5\.2](https://arxiv.org/html/2607.02850#S5.SS2.p1.5),[§5\.4](https://arxiv.org/html/2607.02850#S5.SS4.p3.1),[Table 8](https://arxiv.org/html/2607.02850#S5.T8)\.
- A\. Frikha, H\. Chen, D\. Krompaß, T\. Runkler, and V\. Tresp \(2023\)Towards data\-free domain generalization\.InAsian Conference on Machine Learning,pp\. 327–342\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p2.1)\.
- J\. Gou, B\. Yu, S\. J\. Maybank, and D\. Tao \(2021\)Knowledge distillation: a survey\.International Journal of Computer Vision129\(6\),pp\. 1789–1819\.Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p4.1),[§5\.4](https://arxiv.org/html/2607.02850#S5.SS4.p4.2)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 770–778\.Cited by:[§5\.4](https://arxiv.org/html/2607.02850#S5.SS4.p3.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§5\.4](https://arxiv.org/html/2607.02850#S5.SS4.p4.2)\.
- K\. Hsu, S\. Levine, and C\. Finn \(2019\)Unsupervised learning via meta\-learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=r1My6sR9tX)Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p2.1)\.
- Z\. Hu, L\. Shen, Z\. Wang, T\. Liu, C\. Yuan, and D\. Tao \(2023a\)Architecture, dataset and model\-scale agnostic data\-free meta\-learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 7736–7745\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p2.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p2.1)\.
- Z\. Hu, L\. Shen, Z\. Wang, B\. Wu, C\. Yuan, and D\. Tao \(2023b\)Learning to learn from apis: black\-box data\-free meta\-learning\.InInternational Conference on Machine Learning,pp\. 13610–13627\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p2.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p2.1)\.
- T\. Iwata and A\. Kumagai \(2020\)Meta\-learning from tasks with heterogeneous attribute spaces\.Advances in Neural Information Processing Systems33,pp\. 6053–6063\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p1.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1)\.
- T\. Iwata and A\. Kumagai \(2022\)Sharing knowledge for meta\-learning with feature descriptions\.Advances in Neural Information Processing Systems35,pp\. 16637–16649\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p1.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1)\.
- H\. Jang, H\. Lee, and J\. Shin \(2023\)Unsupervised meta\-learning via few\-shot pseudo\-supervised contrastive learning\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TdTGGj7fYYJ)Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p2.1),[§5\.3](https://arxiv.org/html/2607.02850#S5.SS3.p1.1.5)\.
- N\. Karim, N\. C\. Mithun, A\. Rajvanshi, H\. Chiu, S\. Samarasekera, and N\. Rahnavard \(2023\)C\-sfda: a curriculum learning aided self\-training framework for efficient source free domain adaptation\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 24120–24131\.Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p3.1)\.
- S\. Khodadadeh, L\. Boloni, and M\. Shah \(2019\)Unsupervised meta\-learning for few\-shot image classification\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p2.1)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[§5\.2](https://arxiv.org/html/2607.02850#S5.SS2.p1.5)\.
- B\. M\. Lake, R\. Salakhutdinov, J\. Gross, and J\. B\. Tenenbaum \(2011\)One shot learning of simple visual concepts\.In33rd Annual Meeting of the Cognitive Science Society: Expanding the Space of Cognitive Science, CogSci 2011,pp\. 2568–2573\.Cited by:[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p1.1)\.
- J\. Lee, D\. Jung, J\. Yim, and S\. Yoon \(2022\)Confidence score for source\-free unsupervised domain adaptation\.InInternational conference on machine learning,pp\. 12365–12377\.Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p3.1)\.
- Z\. Li, F\. Zhou, F\. Chen, and H\. Li \(2017\)Meta\-sgd: learning to learn quickly for few\-shot learning\.arXiv preprint arXiv:1707\.09835\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p1.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1)\.
- Y\. Mitsuzumi, A\. Kimura, and H\. Kashima \(2024\)Understanding and improving source\-free domain adaptation from a theoretical perspective\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 28515–28524\.Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p3.1)\.
- A\. Moslemi, A\. Briskina, Z\. Dang, and J\. Li \(2024\)A survey on knowledge distillation: recent advancements\.Machine Learning with Applications18,pp\. 100605\.Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p4.1)\.
- G\. Patel, K\. R\. Mopuri, and Q\. Qiu \(2023\)Learning to retain while acquiring: combating distribution\-shift in adversarial data\-free knowledge distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 7786–7794\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p2.1)\.
- M\. Phuong and C\. Lampert \(2019\)Towards understanding knowledge distillation\.InInternational conference on machine learning,pp\. 5142–5151\.Cited by:[§5\.4](https://arxiv.org/html/2607.02850#S5.SS4.p4.2)\.
- M\. Ren, S\. Ravi, E\. Triantafillou, J\. Snell, K\. Swersky, J\. B\. Tenenbaum, H\. Larochelle, and R\. S\. Zemel \(2018\)Meta\-learning for semi\-supervised few\-shot classification\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HJcSzz-CZ)Cited by:[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p1.1)\.
- J\. Snell, K\. Swersky, and R\. Zemel \(2017\)Prototypical networks for few\-shot learning\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p1.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1),[§4](https://arxiv.org/html/2607.02850#S4.p5.2),[§5\.2](https://arxiv.org/html/2607.02850#S5.SS2.p1.5),[§5\.4](https://arxiv.org/html/2607.02850#S5.SS4.p3.1),[Table 8](https://arxiv.org/html/2607.02850#S5.T8)\.
- L\. Sun, Y\. Tanaka, and T\. Iwata \(2024\)Meta\-learning under task shift\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p1.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1)\.
- J\. Truong, P\. Maini, R\. J\. Walls, and N\. Papernot \(2021\)Data\-free model extraction\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 4771–4780\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p1.1)\.
- O\. Vinyals, C\. Blundell, T\. Lillicrap, D\. Wierstra,et al\.\(2016\)Matching networks for one shot learning\.Advances in neural information processing systems29\.Cited by:[§4](https://arxiv.org/html/2607.02850#S4.p5.2),[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p1.1)\.
- C\. Wah, S\. Branson, P\. Welinder, P\. Perona, and S\. Belongie \(2011\)The caltech\-ucsd birds\-200\-2011 dataset\.Technical reportTechnical ReportCNS\-TR\-2011\-001,California Institute of Technology\.Cited by:[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p3.1)\.
- L\. Wang and K\. Yoon \(2021\)Knowledge distillation and student\-teacher learning for visual intelligence: a review and new outlooks\.IEEE transactions on pattern analysis and machine intelligence44\(6\),pp\. 3048–3068\.Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p4.1)\.
- Z\. Wang, X\. Wang, L\. Shen, Q\. Suo, K\. Song, D\. Yu, Y\. Shen, and M\. Gao \(2022\)Meta\-learning without data via wasserstein distributionally\-robust model fusion\.InUncertainty in Artificial Intelligence,pp\. 2045–2055\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p2.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p2.1)\.
- Y\. Wei, Z\. Hu, Z\. Wang, L\. Shen, C\. Yuan, and D\. Tao \(2024\)Free: faster and better data\-free meta\-learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 23273–23282\.Cited by:[§1](https://arxiv.org/html/2607.02850#S1.p2.1),[§2](https://arxiv.org/html/2607.02850#S2.p1.1),[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p2.1),[§5\.3](https://arxiv.org/html/2607.02850#S5.SS3.p1.1.6)\.
- D\. Wertheimer, L\. Tang, and B\. Hariharan \(2021\)Few\-shot classification with feature map reconstruction networks\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 8012–8021\.Cited by:[§5\.1](https://arxiv.org/html/2607.02850#S5.SS1.p1.1)\.
- M\. Zhang, D\. Wang, and S\. Gai \(2020\)Knowledge distillation for model\-agnostic meta\-learning\.InECAI 2020,pp\. 1355–1362\.Cited by:[§2](https://arxiv.org/html/2607.02850#S2.p4.1),[§5\.3](https://arxiv.org/html/2607.02850#S5.SS3.p1.1.7)\.