Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields

# Breaking the Boundaries of Neural Tangent Kernels for Neural Fields
Source: [https://arxiv.org/html/2609.03117](https://arxiv.org/html/2609.03117)
## Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural FieldsThanks:Corresponding author A\. Mallak: \(mallak002@gmail\.com\)

PubID:pubid:© 2026 IEEE\. Personal use of this material is permitted\. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works\. DOI: 10\.1109/TPAMI\.2026\.3692624\.Alaa MaaloufAffiliation:Department of Computer Science, University of HaifaAffiliation:Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of TechnologyLior WolfAffiliation:School of Computer Science and AI, Tel Aviv UniversityDaniela Rus and Dan RosenbaumAffiliation:Department of Computer Science, University of HaifaAffiliation:Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology

###### Abstract

Neural fields \(NFs\) map continuous coordinates to signals such as color or density, but fast high\-quality reconstruction from sparse observations remains difficult\. Classical Neural Tangent Kernel \(NTK\) regression gives closed\-form fits, yet it is fundamentally linear and cannot accumulate reusable task priors\. We develop three algorithms that address these gaps\. NTK\-KIP learns a distilled support set of coordinates \(and optional labels\) so that a finite NTK can inpaint large missing regions from little observed data, yielding a compact non\-linear representation instead of a raw kernel solve\. MetaQuill meta\-learns a shared initialization for an INR so that new scenes can be adapted by updating only a small task\-specific weight offset, which provides true feature learning and a reusable prior\. Finally, MetaQuill\-KIP fuses both ideas: it seeds the task with a KIP\-style non\-linear warm start, then refines only that small offset around the meta\-learned initialization\. MetaQuill\-KIP achieves high\-PSNR reconstructions and semantically plausible inpainting under very sparse observations, while requiring only lightweight per\-instance adaptation, whereas diffusion\-style baselines typically depend on large pretrained generative priors and costly per\-image tuning\. This shows that NTK\-driven neural fields can be made both non\-linear and meta\-learnable, narrowing the gap between analytic kernels and practical few\-shot reconstruction\.

###### Index Terms:

Machine learning, Representations, data structures, and transforms, Vision and Scene Understanding, Computer vision, Knowledge retrieval, Neural nets\.

## IIntroduction

Neural field models \(NFs\), also known as implicit neural representations, are continuous functions that map input positional coordinates—such as spatial or temporal coordinates—directly to outputs representing properties of a field, such as color, density, or geometry\. Specifically, NFs are neural networks designed to learn this mapping\[[1](https://arxiv.org/html/2609.03117#bib.bib4),[2](https://arxiv.org/html/2609.03117#bib.bib1)\]\. These models are particularly useful for tasks such as image representation\[[2](https://arxiv.org/html/2609.03117#bib.bib1)\], 3D object reconstruction\[[3](https://arxiv.org/html/2609.03117#bib.bib5),[4](https://arxiv.org/html/2609.03117#bib.bib6)\], 3D scenes\[[5](https://arxiv.org/html/2609.03117#bib.bib2)\], audio\[[6](https://arxiv.org/html/2609.03117#bib.bib7)\], and more\[[7](https://arxiv.org/html/2609.03117#bib.bib8)\]\. Typically, the model does not use spatial coordinates directly as input; instead, it leveragespositional encodings\(PEs\) of these coordinates\[[5](https://arxiv.org/html/2609.03117#bib.bib2),[8](https://arxiv.org/html/2609.03117#bib.bib3)\], providing a continuous input space\. This approach enables the model to represent complex, high\-resolution fields in a memory\-efficient manner, without relying on the discretized grid representations typical in traditional models\.

Critical to the success of NFs is their ability to perform representation learning, enabling them to capture useful features for various downstream tasks\[[9](https://arxiv.org/html/2609.03117#bib.bib10),[10](https://arxiv.org/html/2609.03117#bib.bib11)\]\. This learning process is highly non\-linear and challenging to model\[[10](https://arxiv.org/html/2609.03117#bib.bib11)\]\. While NFs have demonstrated considerable potential, their use presents two main challenges: first, representing data as neural network weights is not inherently compatible with downstream tasks and restricts the network size; second, the training process must be repeated for each data sample, which impacts efficiency\.

Neural Tangent Kernels \(NTKs\)\.NTKs\[[11](https://arxiv.org/html/2609.03117#bib.bib9)\]aim to model the learning dynamics of neural networks, explaining their evolution during training via gradient descent through kernel methods\. The NTK approximates the dynamics of wide neural networks by their first\-order Taylor expansion around their initialization\. At infinite width \(under suitable parameter scaling\), this approximation is exact, and at finite widths, it has proven to be a useful model for network training, showing applications in enabling kernel\-based dataset distillation\[[12](https://arxiv.org/html/2609.03117#bib.bib12),[13](https://arxiv.org/html/2609.03117#bib.bib13),[14](https://arxiv.org/html/2609.03117#bib.bib14)\], incremental learning\[[15](https://arxiv.org/html/2609.03117#bib.bib16)\], regression\[[16](https://arxiv.org/html/2609.03117#bib.bib15)\], meta\-learning\[[17](https://arxiv.org/html/2609.03117#bib.bib45),[18](https://arxiv.org/html/2609.03117#bib.bib46)\], and more\[[19](https://arxiv.org/html/2609.03117#bib.bib19)\]\.

Formally, letN\>0\\mathit\{N\}\>0be the total number of samples in the dataset,ddbe the dimension of each sample, and letx,x′∈ℝdx,x^\{\\prime\}\\in\\mathbb\{R\}^\{d\}be a pair of input samples\. Letf⁡\(x,θ\):ℝd→ℝC\\mathit\{f\}\(\\mathit\{x\};\\theta\):\\mathbb\{R\}^\{d\}\\,\\rightarrow\\,\\mathbb\{R\}^\{C\}\\,be a fully connected neural network consisting of an arbitrary number of different\-sized layers with a set of parametersθ\\mathit\{\\theta\}, and letL⁡\(y^,y\)L\(\\hat\{y\},y\)be the loss function of predicted labelsy^\\hat\{y\}with respect to ground\-truth labelsyy\. The NTK derives that the evolution off\\mathit\{f\}outputs over time is represented by,

∇ft\(x′;θ\)=−1N∑i=1n𝒦\(x,x′;θ\)∇Lf\(f\(xi,θ\),yi\)\\displaystyle\\nabla\{\{\}\_\{t\}\}f\(\\mathit\{x^\{\\prime\}\};\\theta\)=\-\\frac\{1\}\{N\}\\sum\_\{\\mathit\{i\}=\\mathit\{1\}\}^\{\\mathit\{n\}\}\\mathcal\{K\}\(x,x^\{\\prime\};\\theta\)\\nabla\{\{\}\_\{f\}\}L\(f\(x\_\{i\},\\,\\theta\),\\,y\_\{i\}\)
Where,

𝒦⁡\(x,x′,θ\)=∇θf\(x;θ\)T∇θf\(x′;θ\)\|𝒦:ℝd×ℝd→ℝC×C,\\begin\{split\}\\mathcal\{K\}\(x,x^\{\\prime\};\\theta\)=&\\nabla\{\{\}\_\{\\theta\}\}f\(\\mathit\{x\}\\,;\\,\\theta\)^\{T\}\\ \\nabla\{\{\}\_\{\\theta\}\}f\(x^\{\\prime\};\\,\\theta\)\\ \\\\ &\|\\ \\mathcal\{K\}:\\mathbb\{R\}^\{d\}\\,\\times\\,\\mathbb\{R\}^\{d\}\\ \\rightarrow\{\}\\ \\mathbb\{R\}^\{\\mathit\{C\}\\times\\mathit\{C\}\},\\end\{split\}\(1\)
From infinite width neural network to kernel ridge regression\.For infinite\-width neural networks under the correct parameter scaling, the NTK

𝒦⁡\(x,x′,θ\)\\mathcal\{K\}\(\\mathit\{x\},x^\{\\prime\};\\theta\)remains constant over time\[[11](https://arxiv.org/html/2609.03117#bib.bib9)\], allowing the time\-dependent kernel to be replaced by a constant kernel

𝒦⁡\(x,x′\)\\mathcal\{K\}\(x,x^\{\\prime\}\)\. Furthermore, for multi\-class networks, the kernels are the same for each class and can be treated as

CCindependent kernels each with equal value\. Furthermore, if a network is trained under MSE loss with

NNinput data points, each with

CClabels, this leads to a closed\-form solution as Kernel\-Ridge Regression\.

We have the full kernel matrix

𝒦∈ℝN×N\\mathcal\{K\}\\in\\mathbb\{R\}^\{N\\times N\}, whose entries are the corresponding

𝒦⁡\(xi,xj\)\\mathcal\{K\}\(x\_\{i\},x\_\{j\}\)at positions

𝒦i,j\\mathcal\{K\}\_\{i,j\}with target

Y∈ℝN×CY\\in\\mathbb\{R\}^\{N\\times C\}\. We seek an optimal parameter

W\\mathit\{W\}that minimizes the empirical loss\. Formally,

W:=arg⁡minW′∈ℝN×C\|\|Y−KW′\|\|22W:=\\underset\{W^\{\{\}^\{\\prime\}\}\\in\\mathbb\{R\}^\{N\\times C\}\}\{\\arg\\min\}\|\|\\mathit\{Y\}\-\\mathit\{K\}W^\{\{\}^\{\\prime\}\}\|\|\_\{2\}^\{2\}\. Solving for the optimal

W′W^\{\{\}^\{\\prime\}\}:

𝒲=𝒦−1​𝒴\\mathcal\{W\}=\\mathcal\{K\}^\{\-1\}\\mathcal\{Y\}\(2\)
Predicting the output on a new data pointx′x^\{\\prime\}is found by compute the kernel with respect to the training dataset𝒦⁡\(x′,x\)∈ℝ1×N\\mathcal\{K\}\(x^\{\\prime\},x\)\\in\\mathbb\{R\}^\{1\\times N\}, with entries at positioniigiven by𝒦⁡\(x′,xi\)\\mathcal\{K\}\(x^\{\\prime\},x\_\{i\}\)\. The output is then given by

𝒴^=𝒦⁡\(x′,x\)​𝒲\\mathcal\{\\hat\{Y\}\}=\\mathcal\{K\}\(x^\{\\prime\},x\)\\mathcal\{W\}\(3\)
This approach is useful not only as a model of neural network training, but also shortcuts the need of training a model, which can be faster, particular for very large models, at the cost of inverting the kernel matrix\.

NTK for representation learning\.Motivated by the advantages of NTK, we investigate the usage of NTKs as a model for training Neural Fields\. Specifically, performing Representation Learning with NFs on a dataset requires training multiple NFs where each is a representation of a single data sample, thus, intuitively, these multi\-NF models could be replaced and presented via the NTK method as a single Kernel matrix\. Moreover, this allows NFs to leverage all of NTKs notable advantages, such as the use of infinite\-width layers \(providing a high\-dimensional parameter space and model flexibility\), closed\-form kernel regression \(making training and inference as straightforward as matrix inversion\), and utility in certain representation learning applications \(such as training multi\-neural network models\)\.

NTK NF Matrix Completion \(MC\)\.Neural Fields are often used for inpainting/matrix completion tasks\. In this case, the model is given incomplete data, for example, only a subset of pixels of an image, and the model has to inpaint the missing pixels\. For MC\[[20](https://arxiv.org/html/2609.03117#bib.bib44)\]tasks, the algorithm for reconstructing the target image is slightly different due to Kernel matrix calculation for missing pixels\. Denoting the target image’s observed \(known\) pixels byoo, and assuming the ground\-truth \(GT\) image’s shape is\(h,w,C\)\(h,w,C\)\. For calculating theW∈ℝo×CW\\in\\mathbb\{R\}^\{o\\times C\}vector,

𝒲=𝒦t​r​a​i​n−1𝒴\|𝒦t​r​a​i​n∈ℝo×o,𝒴∈ℝo×C\\mathcal\{W\}=\\mathcal\{K\}\_\{train\}^\{\-1\}\\mathcal\{Y\}\\quad\|\\,\\mathcal\{K\}\_\{train\}\\in\\mathbb\{R\}^\{o\\times o\},\\,\\mathcal\{Y\}\\in\\mathbb\{R\}^\{o\\times C\}\(4\)
Where𝒦t​r​a​i​n\\mathcal\{K\}\_\{train\}takes as input the model’s weights and the NF observed PEs \(only the PEs that correspond to the observed target image’s pixels\)\. For reconstructing the GT image,

𝒴^=𝒦t​e​s​t𝒲\|𝒦t​e​s​t∈ℝ\(h⋅w\)×o,𝒴^∈ℝ\(h⋅w\)×C\\mathcal\{\\hat\{Y\}\}=\\mathcal\{K\}\_\{test\}\\mathcal\{W\}\\quad\|\\,\\mathcal\{K\}\_\{test\}\\in\\mathbb\{R\}^\{\(h\\cdot w\)\\times o\},\\,\\mathcal\{\\hat\{Y\}\}\\in\\mathbb\{R\}^\{\(h\\cdot w\)\\times C\}\(5\)
Where𝒦t​e​s​t\\mathcal\{K\}\_\{test\}takes the model’s weights as input, along with the observed and complete PEs of the NF\.

Why improve NTK instead of abandoning it\.NTK\-style neural field solvers remain attractive because they provide analytic, closed\-form control of fitting a signal, extremely fast per\-instance adaptation, and transparent conditioning on observed coordinates, all without requiring a large pretrained generative prior\. This makes them appealing for settings such as on\-device personalization or single\-scene reconstruction, where we cannot assume access to a massive diffusion backbone or a text semantic prior\. The problem is that classical NTK pipelines are fundamentally limited: they are linear in function space \(so they cannot represent rich structure\), and they cannot meta\-learn reusable features across tasks\. The central objective of this work is therefore not to argue that NTK alone beats modern diffusion models, but to demonstrate that these two structural gaps in NTK can be closed in practice\. NTK\-KIP injects nonlinear representational capacity by distilling a compact set of task\-specific inducing points, and MetaQuill supplies true feature learning and meta\-initialization so that new tasks can adapt in a few steps\. Their fusion, MetaQuill\-KIP, unifies both effects and turns NTK\-style neural fields into a practical, fast\-adapting, per\-instance reconstructor\.

### I\-AOur Contributions

We study the practical use of Neural Tangent Kernels \(NTKs\) for representation learning with neural fields \(NFs\), and we ask whether NTK\-style methods can be made fast, learnable, and useful for single\-image tasks such as reconstruction and inpainting\. We begin by replacing slow NF training with NTK regression, which in principle provides a one\-step solution for fitting an \(effectively\) infinite\-width NF\. Our initial analysis reveals that the standard NTK pipeline, when applied directly to neural fields, suffers from fundamental shortcomings\. Our first contribution is a systematic study of these limitations:

- •NTK representational limitations: Because NTK fits a linear function in weight space around a fixed initialization, its predictions are structurally restricted\. This prevents reconstruction of high\-frequency structure and fine semantic detail in many real signals \(see Section[II\-A](https://arxiv.org/html/2609.03117#S2.SS1)\)\.
- •NTK lack of robustness: The explicit kernel regression solution is highly sensitive to noise and masking, and it struggles in partially observed settings like inpainting, where even modest missing regions degrade output smoothness and semantic plausibility \(see Section[II\-B](https://arxiv.org/html/2609.03117#S2.SS2)\)\.
- •NTK inefficiency: Finite\-width NTK on dense neural field grids requires forming or applying large Jacobian blocks\. This leads to high memory use and non\-trivial solve cost, which makes naive NTK unattractive for practical, iterative adaptation \(see Section[II\-C](https://arxiv.org/html/2609.03117#S2.SS3)\)\.

To address these issues, we develop and evaluate three NTK\-driven algorithms:

1. 1\.NTK\-KIP: Distilled nonlinear NTK representation\.We introduce Neural Tangent Kernels Kernel Inducing Points \(NTK\-KIP\), which converts NTK fitting from “solve on every pixel” into “learn a small support set and solve only there\.” NTK\-KIP learns a compact set of spatial support coordinates and \(optionally\) their associated target values, by optimizing them through the NTK\. This amounts to distilling the signal and its supervision into a handful of inducing points\. As a result, NTK is no longer forced to behave as a strictly linear regressor over the raw grid: instead, the learned support set itself becomes nonlinear and task\-adaptive\. In experiments, NTK\-KIP achieves aggressive compression while preserving reconstruction quality\. For example, using Fourier positional encodings, we reconstruct RGB Flowers images with≈20%\\approx 20\\%of the pixels observed atPSNR≈31\\text\{PSNR\}\\approx 31dB, and with only≈5%\\approx 5\\%observed atPSNR≈20\\text\{PSNR\}\\approx 20dB\. With raw positional encodings, we see≈80%\\approx 80\\%compression atPSNR\>36\\text\{PSNR\}\>36dB and≈50%\\approx 50\\%atPSNR\>23\\text\{PSNR\}\>23dB \(See Fig\.[11](https://arxiv.org/html/2609.03117#S3.F11)\)\. NTK\-KIP therefore addresses two core NTK weaknesses: it injects nonlinearity via learned supports, and it reduces kernel cost by collapsing the solve to a tiny subset\. However, NTK\-KIP by itself does not provide task\-transfer or shared feature learning across images\.
2. 2\.MetaQuill: NTK\-based meta\-learning with feature learning and fast adaptation\.We propose MetaQuill, a meta\-learning procedure for neural fields that learns a reusable shared initializationθS\\theta\_\{S\}across a small training set of tasks, together with a per\-task low\-rank updateΔ​θi\\Delta\\theta\_\{i\}\. Unlike classical MAML, MetaQuill does not require an expensive inner\-loop optimization during meta\-training\. Instead, MetaQuill directly learns bothθS\\theta\_\{S\}and the structure of the task\-specific updates in a way that is compatible with NTK linearization\. At test time, a new task can be adapted in tens of gradient steps \(or even a single step in some settings\), rather than training a fresh NF from scratch\. MetaQuill enables two properties that standard NTK lacks: \(i\) reusable features that transfer across tasks, and \(ii\) data\-efficient per\-task adaptation\. On MNIST\-style digit fields, MetaQuill reconstructs new digits atPSNR≈20\\text\{PSNR\}\\approx 20dB after only≈50\\approx 50update steps on the task\-specificΔ​θi\\Delta\\theta\_\{i\}\(See Table[II](https://arxiv.org/html/2609.03117#S4.T2)\), while a naive “Single\-INR” trained from scratch requires similar or more optimization effort just to approach that quality\. MetaQuill, however, is still limited by NTK linearity during inference, and its tangent\-only adaptation can struggle with large missing regions \(e\.g\., hole inpainting\)\.
3. 3\.MetaQuill\-KIP: Fusing nonlinear NTK\-KIP priors with MetaQuill\-style fast adaptation\.Our final contribution is a new hybrid method that unifies the strengths of NTK\-KIP and MetaQuill\. MetaQuill\-KIP first runs a KIP\-style support optimization on a single target image, producing a distilled support set and an initial task\-specific update directionΔ​θi\\Delta\\theta\_\{i\}\. It then refines only that low\-dimensionalΔ​θi\\Delta\\theta\_\{i\}nonlinearly, while keeping the shared MetaQuill initializationθS\\theta\_\{S\}fixed\. This gives us \(i\) nonlinear, structure\-aware inpainting and reconstruction from NTK\-KIP, and \(ii\) the rapid, few\-step adaptation and reusable feature backbone inherited from MetaQuill\. In practice, MetaQuill\-KIP delivers high\-fidelity reconstructions and semantically plausible inpainting within a few hundred lightweight refinement steps, and does so without relying on a large pretrained diffusion prior\.

Finally, we validate all three methods, and we compare them both internally and against modern diffusion\-based inpainting systems\. On MNIST, MetaQuill\-KIP reaches over3232dB PSNR after only5050refinement steps, substantially improving over plain kernel regression \(about1111dB\) and over MetaQuill’s tangent\-only update \(about1818–1919dB\) \(See T\.[II](https://arxiv.org/html/2609.03117#S4.T2)\)\. On RGB Flowers images, MetaQuill\-KIP produces visually coherent hole fills from sparse masks, outperforming naive single\-image baselines in both PSNR and perceptual quality while requiring only a few seconds of adaptation for the new task\. We also benchmark against Stable Diffusion \(SD\) inpainting, StrDiffusion\[[21](https://arxiv.org/html/2609.03117#bib.bib47)\], Denoising Diffusion Probabilistic Models \(DDPM\), and a global structure\-guided diffusion model \(GSDM\)\[[22](https://arxiv.org/html/2609.03117#bib.bib48)\]\. Although diffusion models achieve strong PSNR when given large observed regions and pretrained semantic priors, our NTK\-driven pipelines operate in a fundamentally different and complementary regime: they adapt from scratch to a single image in seconds, using only per\-image supervision, without external text conditioning or massive pretraining\. This establishes MetaQuill\-KIP as an effective bridge between theoretical NTK structure and practical single\-image neural field reconstruction and inpainting\.Notations\.For readability, we provide a consolidated list of acronyms and abbreviations in the Appendix \(List of Acronyms and Abbreviations\) \- See Table[VII](https://arxiv.org/html/2609.03117#A1.T7)\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/kernel_representational_nature/Intro/NF_Architecture_with_FPE.png)Fig\. 1:Neural Field architecture used throughout the paper\. Input is a 2D coordinate\(a,b\)∈\[−1,1\]2\(a,b\)\\in\[\-1,1\]^\{2\}, encoded either by raw positional encoding \(RPE\) \(Green\) or Fourier positional encoding \(FPE\) \(Orange\) withkkfrequency bands\. The encoded vector \(Red\) is passed through a 5 layer MLP of width 256 with nonlinear activations \(Purple\) to produce an RGB output inℝ3\\mathbb\{R\}^\{3\}\(or a scalar output inℝ\\mathbb\{R\}for single channel signals\) \(Yellow\)\.

## IIInvestigating the use of NTK in NFs

We start by investigating the direct use of NTKs to release the long learning process in NFs\.

The setting\.Inspired by Functa\[[9](https://arxiv.org/html/2609.03117#bib.bib10)\], throughout the experiments, we use the NF architecture described in Fig\.[1](https://arxiv.org/html/2609.03117#S1.F1), which consists of 2D spatial PEs, or alternatively Fourier positional encodings \(FPEs\),5\\mathit\{5\}hidden layers \(Each of width256\\mathit\{256\}neurons\), and one RGB or a Single\-value output layer\. Unless otherwise specified, we use FPE with frequency bands set tokb​a​n​d​s=20k\_\{bands\}=20as the default\. This value was determined from an experiment set to find the optimal FPE band; Figs\.[23](https://arxiv.org/html/2609.03117#A2.F23)and[22](https://arxiv.org/html/2609.03117#A2.F22)in the appendix illustrates the effect of different frequency bandsk\\mathit\{k\}\. See Appendix[B](https://arxiv.org/html/2609.03117#A2)for more details\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/kernel_representational_nature/NF_vs_NTK_Inpainting/Inpainting_NF_NTK.png)Fig\. 2:Inpainting experiment\. From left to right: ground truth image, masked image, NF at iterations00,100100,300300,10001000\(PSNR≈17\.06\\approx 17\.06dB, masked PSNR≈9\.03\\approx 9\.03dB\), and Infinite NTK reconstruction \(PSNR≈20\.21\\approx 20\.21dB, masked PSNR≈12\.07\\approx 12\.07dB\)\.### II\-AExploring NTK\-based NFs in Representation Rigidity and Inpainting

To investigate the capabilities and limitations of NTK\-based NFs, we delve into how NTK performs in capturing and representing complex data structures\. We compare traditional finite NF to an NTK\-based NF\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/kernel_representational_nature/NF_vs_NTK_RR/RR_NF_NTK.png)Fig\. 3:Representation rigidity experiment\. Row 1: Neural Field training snapshots \(ground truth, then reconstructions at iterations00,500500,10001000\), PSNR=32\.1232\.12dB\. Row 2, left to right: finite time NTKWW, finite time NTK reconstruction \(PSNR=18\.9318\.93dB\), infinite NTKWW, infinite NTK reconstruction \(PSNR=101\.79101\.79dB\)\.Representation Rigidity: NF vs NTK\-based NF\.We begin by examining NTK’s representational rigidity \(”inpainting” a fully known image\)\. With all labels known, the Kernel can accurately capture pixel correlations\. Thus, we test the infinite NTK method and compare it to standard neural network training\. Specifically, we train the NF model in Fig\.[1](https://arxiv.org/html/2609.03117#S1.F1)for1000\\mathit\{1000\}steps with a learning rate of10−3\\mathit\{10\}^\{\-3\}\. For the NTK comparison, we use the NTK reconstruction method while the observed pixels here are the whole image pixels\(h×w\)\(h\\times w\)\. Here, the PE vector and the target image are fully known\. Formally, letN=h⋅w\\mathit\{N\}=\\mathit\{h\}\\cdot\\mathit\{w\}be the total number of pixels,a,b∈\[−1,1\]a,\\,b\\in\[\-1,\\,1\]be a pair of values representing a spatial 2D coordinate \(NF input\), and letx,y,z∈\[0,255\]x,y,z\\in\[0,\\,255\]be the target image’s RGB values accordingly\. The PE vector is then represented by\(\(ai,bi\)\)i=1N\\big\(\(\\mathit\{a\}\_\{i\},\\mathit\{b\}\_\{i\}\)\\big\)^\{N\}\_\{i=1\}, and the target image,\(\(,,,,,\)\)i=1N\\big\(\(\\mathit\{x\}\_\{i\},\\mathit\{y\}\_\{i\},\\mathit\{z\}\_\{i\}\)\\big\)^\{N\}\_\{i=1\}\.

We chose the single flower image for this experiment because it represents high\-frequency semantic information, edges, and texture, while lacking predictable spatial patterns\.

Reported results \(representation rigidity\)\.Figure[3](https://arxiv.org/html/2609.03117#S2.F3)presents the results\. In the first row, from left to right, we show the ground truth image and the reconstructed image at iterations 0, 500, and 1000\. In the second row, we present, left to right: the Finite\-NTK reconstructed image, the Finite\-NTKW\\mathit\{W\}vector, the Infinite\-NTK reconstructed image, and the Infinite\-NTKW\\mathit\{W\}vector\. Here, Finite\-NTK refers to the Neural Tangent Kernel applied within finite\-width neural networks, representing finite\-time training\. The PSNR values indicate the performance differences: NF model at 32\.12, infinite NTK\-based NF at 101\.79, and finite NTK\-based NF at 18\.93\. With all labels known, NTK excels in reconstruction due to its kernel correlation calculation, allowing nearly perfect reconstruction when all pixel relationships in the target image are known\. Visually, there is a minimal difference, thus,𝑃𝑆𝑁𝑅≈30\\mathit\{PSNR\}\\approx 30, suggesting visual saturation\. Additionally, theW\\mathit\{W\}vectors differ between Finite and Infinite NTK: the infinite\-time NTK captures more details, as expected since Finite\-time NTK illustrates training for only a limited time period \(here, 1000 iterations\) without full convergence\.

NF vs NTK Inpainting\.In this experiment, we aim to study the NTK’s capabilities in a more complex task: inpainting, where some labels are masked, and the Kernel’s pixel correlations can be partially disrupted\. We apply the infinite \(width and time\) NTK reconstruction method to inpaint a masked image, comparing it to the standard NFs training approach, where we train the same NF model \(Fig\.[1](https://arxiv.org/html/2609.03117#S1.F1)\) for 1000 steps with a learning rate of10−310^\{\-3\}\. For NTK, we use the reconstruction method for MC task \(Eqs\.[4](https://arxiv.org/html/2609.03117#S1.E4)and[5](https://arxiv.org/html/2609.03117#S1.E5)\)\. Here, the PE vector and the target image are masked in a circular ’hole’ centered shape\. Definer∈ℝr\\in\\mathbb\{R\}to be the centric hole masking radius, and set

ℒ2r=\{i∣i∈\{1,⋯,N\},‖\(ai,bi\)‖22≥r\}\\mathcal\{L\}\_\{2\}^\{r\}=\\\{i\\mid i\\in\\\{1,\\cdots,N\\\},\\left\\lVert\(\\mathit\{a\}\_\{i\},\\mathit\{b\}\_\{i\}\)\\right\\rVert\_\{2\}^\{2\}\\geq\\mathit\{r\}\\\}to be the set of unmasked coordinate pairs\. Then, the NF PE vector is\(\(ai,bi\)\)i∈ℒ2r\\big\(\(\\mathit\{a\}\_\{i\},\\mathit\{b\}\_\{i\}\)\\big\)\_\{i\\in\\mathcal\{L\}\_\{2\}^\{r\}\}, and the target image\(\(,,,,,\)\)i∈ℒ2r\\big\(\(\\mathit\{x\}\_\{i\},\\mathit\{y\}\_\{i\},\\mathit\{z\}\_\{i\}\)\\big\)\_\{i\\in\\mathcal\{L\}\_\{2\}^\{r\}\}\.

Reported results \(inpainting\)\.The results are shown in Fig\.[2](https://arxiv.org/html/2609.03117#S2.F2)\. The NF training results are presented from left to right: the GT image, the masked GT image, and reconstructed images at iterations 0, 100, 300, and 1000\. The Infinite\-NTK reconstruction result is the last image on the right\.

The NF\-trained model and NTK method yield relatively close results \(PSNR≈17\.06\\text\{PSNR\}\\approx 17\.06vs\.PSNR≈20\.21\\text\{PSNR\}\\approx 20\.21\) despite only training the NF model for 1000 iterations\. In the masked pixel region, the scores are also comparable \(PSNR≈9\.03\\text\{PSNR\}\\approx 9\.03for NF vs\.PSNR≈12\.07\\text\{PSNR\}\\approx 12\.07for NTK\)\. However, several distinctions emerge: first, the NF model predicts flowers in the masked region—reasonable given the lack of a fixed spatial pattern—while NTK does not, unable to capture object shape connections\. Second, NTK’s reconstruction lacks semantic coherence, with edge transitions from observed to masked areas being rough and inconsistent\. The NTK method also fails to capture semantic details and texture of the flowers, unlike the NF model, which preserves semantic edges, details, and texture, yielding visually superior results\. This experiment highlights a few critical points\. Flower images often feature objects with unpredictable patterns and high\-frequency signals, making pixel prediction challenging\. Additionally, the hole\-shaped mask introduces hidden neighboring pixels, where NTK’s kernel matrix struggles to approximate correlations, resulting in blurred reconstructions\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/kernel_representational_nature/NF_vs_NTK_Inpainting/Inpainting_NF_NTK_Radii_PSNR_Graph.png)Fig\. 4:PSNR and masked PSNR versus hole radius\. Green: NTK based NF PSNR\. Blue: NF PSNR\. Red: NTK based NF masked PSNR\. Orange: NF masked PSNR\. Masked PSNR is computed only over the masked region\.The effect of missing pixels\.To analyze the impact of hole radius on the center hole inpainting task, we evaluate reconstruction PSNR as a function of the masked hole radius for both the NF model and the NTK method\. For each method, we report PSNR on the full image as well as masked PSNR computed only over the hole region\. The results are shown in Fig\.[4](https://arxiv.org/html/2609.03117#S2.F4)\.

Overall, NF and NTK exhibit similar full image PSNR and masked PSNR trends across radii\. However, NTK might be expected to better reconstruct small masked regions if local pixel correlations alone were sufficient\. Instead, we observe a clear gap between NTK performance in representational rigidity and inpainting, indicating that the fixed kernel correlations that enable near perfect reconstruction when all labels are observed do not transfer reliably to partially observed settings\.

These results suggest a key limitation of NTK in this context: its fixed kernel behaves primarily as a correlation structure rather than a feature\-adaptive representation\. As a result, NTK can reconstruct pixels in representation rigidity via kernel inversion, yet struggles in settings that require learning semantic structure and long range context, such as edge continuity, texture, and object level consistency\. The disparity between representation rigidity and inpainting therefore highlights the limitations of the NTK regime for tasks that benefit from adaptive feature learning\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/kernel_representational_nature/NTK_Representational_Linearity/NTK_W_Linearity.png)Fig\. 5:Ground truth image versus NTK coefficient representationWWon CelebA and MNIST\. First row: ground truth images\. Second row: correspondingWWrepresentations\.Mathematical derived explanation\.To better understand the results, we delve into the core Kernel representation and present theoretical findings and explanations\. From equation[1](https://arxiv.org/html/2609.03117#S1.E1), if to consider a specific cell\[\(i,j\)\|1≤i,j≤C\]\[\(i,j\)\\,\|\\,1\\leq i,j\\leq C\]in the NTK,

𝒦\(i,j\)\(x,x′;θ\)\\displaystyle\\mathcal\{K\}\_\{\(\\mathit\{i\},\\mathit\{j\}\)\}\(\\mathit\{x\},\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\,\\theta\)=∇fiθ\(x;θ\)T⋅∇fjθ\(x′;θ\)=\\displaystyle=\\nabla\{\{\}\_\{\\theta\}\}\\mathit\{f\}\_\{\\mathit\{i\}\}\(\\mathit\{x\}\\,;\\,\\theta\)^\{T\}\\ \\cdot\\ \\nabla\{\{\}\_\{\\theta\}\}\\mathit\{f\}\_\{\\mathit\{j\}\}\(\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\,\\theta\)==∑q=1Q∂fi​\(x,θ\)T∂θq⋅∂fj\(x′;θ\)∂θq,\\displaystyle=\\sum\_\{\\mathit\{q\}=\\mathit\{1\}\}^\{\\mathit\{Q\}\}\\frac\{\\partial\\mathit\{f\}\_\{\\mathit\{i\}\}\(\\mathit\{x\}\\,;\\,\\theta\)^\{\\mathit\{T\}\}\}\{\\partial\\theta\_\{\\mathit\{q\}\}\}\\cdot\\frac\{\\partial\\mathit\{f\}\_\{\\mathit\{j\}\}\(\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\,\\theta\)\}\{\\partial\\theta\_\{\\mathit\{q\}\}\},
whereQ\\mathit\{Q\}is the total number of parameters in the NNf\\mathit\{f\}\. See Fig\.[24](https://arxiv.org/html/2609.03117#A3.F24)in the Appendix for illustration\.

Calculating correlations through inner products of neural network gradients compresses meaningful information, neglecting dynamic architectural effects and resulting in a low\-dimensional projection rather than a robust representation\. While NTK offers advantages in training flexibility, it lacks effective feature extraction and true learning capability\. To explore these limitations, we propose the concept of ”NTK Representational Linearity”, which addresses the fundamental constraints in NTK’s approach\. This analysis contributes to understanding and mitigating limitations in the NTK theorem, highlighting its impact on representation learning\.

NTK representational linearity\.Recall the coefficient representation𝒲\\mathcal\{W\}in Eq\.[2](https://arxiv.org/html/2609.03117#S1.E2), obtained by solving a kernel regression system on the observed coordinates\. Denoting by𝒦train\\mathcal\{K\}\_\{\\mathrm\{train\}\}the kernel matrix over the observed coordinates, the solution satisfies𝒲=𝒦train−1​𝒴\\mathcal\{W\}=\\mathcal\{K\}\_\{\\mathrm\{train\}\}^\{\-1\}\\mathcal\{Y\}, hence𝒲\\mathcal\{W\}depends linearly on the labels once𝒦train\\mathcal\{K\}\_\{\\mathrm\{train\}\}is fixed\. Predictions at query coordinates are then reconstructed via kernel evaluations against the observed coordinates, i\.e\.,𝒴~=𝒦test​𝒲\\tilde\{\\mathcal\{Y\}\}=\\mathcal\{K\}\_\{\\mathrm\{test\}\}\\mathcal\{W\}, where𝒦test\\mathcal\{K\}\_\{\\mathrm\{test\}\}denotes the cross kernel between query and observed coordinates\. Therefore,𝒲\\mathcal\{W\}is best interpreted as a coefficient space solution to a fixed kernel regression problem, rather than as a task adaptive learned feature representation\. Empirically, Fig\.[5](https://arxiv.org/html/2609.03117#S2.F5)shows that𝒲\\mathcal\{W\}is highly correlated with the target image in our setting, consistent with this coefficient space interpretation\. In addition, the coefficient representation can preserve label noise and local artifacts, rather than encoding semantically organized features produced by nonlinear representation learning\.

In short\.NTK provides a powerful framework for analyzing neural network training dynamics and, in the infinite width regime, can approximate neural network evolution\. However, in the kernel regression form used here, the resulting coefficient representation remains a linear, non adaptive mapping with respect to the labels once the kernel is fixed\. This limits its ability to model richer task dependent feature transformations that are often important in neural field generation and structured reconstruction tasks\. In representation rigidity and inpainting settings, this limitation is reflected in weaker semantic continuity and texture recovery\. These observations motivate the development of NTK\-KIP, MetaQuill, and MetaQuill\-KIP\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/NTK_Noise_Sensitive/NTK_Robustness.png)Fig\. 6:NTK robustness analysis on CelebA \(top\) and CIFAR\-10 \(bottom\)\. Left: perturbing a single coefficient inWW\. Right: perturbing all coefficients inWW\. Within each block, we show: ground truth image \(top\-left\), originalWW\(bottom\-left\), Gaussian noise \(𝒩⁡\(0,10\)\\mathcal\{N\}\(0,10\)\), perturbedWW, reconstruction from perturbedWW\(top\-middle\), and reconstruction error heatmap \(top\-right\)\. Small perturbations inWWcause large, structured artifacts in pixel space, highlighting noise sensitivity of the explicit NTK coefficient representation\.
### II\-BExploring NTK’s Robustness to Noise

While classic NNs can learn and adapt to noisy or imperfect data, NTK struggles with such robustness, especially in very deep networks and complex, large\-scale data\. This limitation stems from NTK’s core theory\. For example, in Matrix Completion \(MC\) scenarios, added noise in the samples propagates through the kernel, significantly affecting reconstruction due to NTK’s linear matrix transformations and pixel\-space correlations\. Consequently, NTK’s performance suffers in real\-world tasks with noise, skewed samples, or varying data distributions\.

Image reconstruction underWWperturbations\. To demonstrate this claim, we examine image reconstruction under perturbations to theWWvector\. In this experiment, we compute theWWvector for a set of GT images and then add to it stochastic noise from a normal distribution \(φ∼N⁡\(μ,σ\)\\varphi\\sim N\(\\mu,\\sigma\)\)\. We conduct two experiments: first, perturb theWWvector at a single pixel, and second, across the whole vector\. Then use NTK to reconstruct the fully observed image\. Results for different perturbation levels are in Fig\.[6](https://arxiv.org/html/2609.03117#S2.F6), where we can notice:

1. 1\.A single noisy pixel inWWaffects not just the target pixel, but also impacts rows and columns at fixed frequency in the reconstructed image\.
2. 2\.When adding noise to the entire vector, the image structure remains somewhat visible, but the reconstruction is highly perturbed, with minimal preservation of the GT image’s structure\.

This leads to a significant vulnerability in the NTK method, specifically in such tasks and applications that rely on a highS​N​RSNRyielded results\.

In short\.While classic NNs handle noise, NTK faces challenges\. To test NTK’s robustness, we perturb its𝒲\\mathcal\{W\}vector with various noise distributions\. The reconstruction of labels via NTK’s𝒲\\mathcal\{W\}vector is as follows,𝒴^=𝒦T\(x,x′;θ\)×𝒲\|𝒦:ℝd×ℝd→ℝC×C\\hat\{\\mathcal\{Y\}\}=\\mathcal\{K\}\_\{\\mathit\{T\}\}\(\\mathit\{x\},\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\theta\)\\times\\mathcal\{W\}\\ \|\\ \\mathcal\{K\}:\\mathbb\{R\}^\{d\}\\ \\times\\ \\mathbb\{R\}^\{d\}\\ \\rightarrow\{\}\\ \\mathbb\{R\}^\{\\mathit\{C\}\\times\\mathit\{C\}\}\. We found that NTK’s representation is extremely noise\-sensitive, leading to a blurry reconstruction containing Kernel\-resulting frequencies\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/NTK_Representation_Efficiency/Functa_NF_NTK.png)Fig\. 7:Left: Classic Functa\. Right: Functa via NTK\. In Functa via NTK, each neural field in the classic Functa pipeline is replaced by its NTK coefficient representationWW\.![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/NTK_Representation_Efficiency/DM_Guided_Inverse.png)Fig\. 8:Guided diffusion reconstruction results for DM and DM\_W\. Columns from left to right: masked image, DM\_W inverse estimateW^\\hat\{W\}, reconstructed image viaW^\\hat\{W\}, and DM inverse\. Learning rates: left two rows use10−610^\{\-6\}\(top\) and10−510^\{\-5\}\(bottom\), right column uses10−210^\{\-2\}\.
### II\-CA step forward: NTK for Functa

Functa\.Deep learning often represents data on discrete grids \(e\.g\., pixels\), though signals are continuous\. NFs offer an alternative, predicting values at any spatial location,N​F:l→vNF:l\\rightarrow v\. Functa\[[9](https://arxiv.org/html/2609.03117#bib.bib10)\]explored using NFs as data, showing that datasets can be distilled into compact NF representations, reducing the size by∼0\.5%\\sim 0\.5\\%while preserving key features for accurate reconstruction, highlighting the importance of feature learning, which we benchmarked to evaluate NTK’s representation efficiency \(see Fig\.[7](https://arxiv.org/html/2609.03117#S2.F7)for an illustration\)\. We now experiment with NTK for representation learning via the Functa setting\. Our downstream chosen task is MC, and more specifically, Inpainting\.

The setting\.In the NTK Functa method, unlike classic Functa, where a Diffusion Model \(DM\) is being trained on a compact form of the NFs weights, we train the DM on infinite\-width NFs data\. And the way to make this feasible is via NTK kernel trick; see Fig\.[7](https://arxiv.org/html/2609.03117#S2.F7)\. We replace each NF with aWWvector \(via Eq\.[2](https://arxiv.org/html/2609.03117#S1.E2)\), and train the DM upon\. The calculation process is convenient due to the NTK Kernel being shared upon allWWs \(Same NF model and fixed image size, thus fixed PEs\)\. At inference, We’ll generate a novel𝒲^\\mathcal\{\\hat\{W\}\}vector via our trained DM, and predict the missing pixels via the pre\-calculated NTK Kernel \(Eq\.[3](https://arxiv.org/html/2609.03117#S1.E3)\)\. To use the DM for specific tasks such as inpainting, we use a guidance algorithm where we pass the observed labels as a guided input for the DM\. Formally,

xt−1=1αt​\(xt−1−αt1−αt¯⋅ϵ^t−1\)\+S⋅∇P​\(y\|x\)\+β¯t⋅zx\_\{t\-1\}=\\frac\{1\}\{\\sqrt\{\{\\alpha\_\{t\}\}\}\}\(x\_\{t\}\\,\-\\,\\frac\{1\-\\alpha\_\{t\}\}\{\\sqrt\{1\-\\overline\{\\alpha\_\{t\}\}\}\}\\cdot\\hat\{\\epsilon\}\_\{t\-1\}\)\\,\+\\,S\\cdot\\nabla P\(y\|x\)\\,\+\\,\\sqrt\{\\overline\{\\beta\}\_\{t\}\}\\cdot z
The experiment was conducted on the MNIST dataset\. In which, we trained two DMs\. First, directly on the image domain, and second, on theWWs dataset\. Example results of the calculatedWWs dataset, can be seen in Fig,[5](https://arxiv.org/html/2609.03117#S2.F5)\. For the full formulation and notation, see Appendix[D](https://arxiv.org/html/2609.03117#A4)\. We evaluate NTK’s representational performance by comparing the W vector representation with the original image domain in an MC inpainting task, using reconstructed images from Guided\-DMs trained on the Ws dataset and directly on GT images\. Results can be seen in Fig[8](https://arxiv.org/html/2609.03117#S2.F8)\.

Representational robustness\.Results in the GT image space are smoother and more refined \(Right\-most column\), whereas training in theWWvector space exhibits noise sensitivity \(Bottom\-left section \-22and00digits\) and inconsistent guidance \(Top\-left and bottom\-left results gap with respect to the guidance learning rate\) while showing poor results \(See Section[II\-B](https://arxiv.org/html/2609.03117#S2.SS2)\), resulting in blurry reconstructions\. The denoising process in the diffusion trajectory is complex, and added noise in learned representations further complicates the inversion\. This suggests that the NTK representation does not add valuable features and may even degrade results\. Even successful reconstructions with NTK exhibited noise issues\.

Run\-Time efficiency\.Considering inference runtime efficiency, we compared the representational size and computation of both methods \(Functa vs\. NTK\) using the basic NF in both algorithms\. In Functa, the base NF has5×2565\\times 256hidden layers and an RGB output \(1×31\\times 3\), allowing us to estimate the runtime complexity \(ignoring bias\)\. The results: With NF,

\(h⋅w\)×d⏞P​E⋅l⏟1s​t​h​i​d​d​e​n​l​a​y​e​r\+\(l×l×Nl\)⏟m​i​d​h​i​d​d​e​n​l​a​y​e​r​s\\displaystyle\\underbrace\{\\overbrace\{\(h\\cdot w\)\\times d\}^\{PE\}\\cdot l\}\_\{1^\{st\}\\,hidden\\,layer\}\\,\+\\,\\underbrace\{\(l\\times l\\times N\_\{l\}\)\}\_\{mid\\,hidden\\,layers\}\\,\+\(l×c\)⏟f​i​n​a​l​o​u​t​p​u​t​l​a​y​e​r\\displaystyle\+\\,\\underbrace\{\(l\\times c\)\}\_\{final\\,output\\,layer\}≅8×105=0\.8M\\displaystyle\\cong 8\\times 10^\{5\}=\\textbf\{0\.8M\}
In NTK \(Neglecting FPE and kernel mapping calculation\),

\(∗\)=\(h⋅w\)×d⏞P​E×\(d\+2⋅d⋅k⏞F​o​u​r​i​e​r​b​a​n​d​s\)⏟F​P​E\\displaystyle\(\*\)=\\underbrace\{\\overbrace\{\(h\\cdot w\)\\times d\}^\{PE\}\\times\(d\\,\+\\,2\\cdot d\\cdot\\overbrace\{k\}^\{Fourier\\,bands\}\)\}\_\{FPE\}\\,\+\(Kt​r​a​i​n−1×Y\)⏟Wv​e​c​t​o​r\\displaystyle\+\\,\\underbrace\{\(K\_\{train\}^\{\-1\}\\times Y\)\}\_\{W\_\{vector\}\}\\,\+Kt​e​s​t×W⏟R​e​c​o​n​s​t​r​u​c​t​Y^\\displaystyle\+\\,\\underbrace\{K\_\{test\}\\times W\}\_\{Reconstruct\\,\\hat\{Y\}\}
where,\(∗\)≥2G\(\*\)\\geq\\textbf\{2G\}\(For general matrix multiplication\), and\(∗\)≥6M\(\*\)\\geq\\textbf\{6M\}\(For matrix\-Vector multiplication\)\. For the full calculation, see Appendix[D](https://arxiv.org/html/2609.03117#A4)\.

In summary:NTK is unsuitable for practical representation tasks due to weaknesses in structure, representation, and runtime, posing limitations for applications like model scaling and edge AI \(energy\-efficient systems\)\. We observe three main weaknesses in the NTK method:

1. 1\.Representational structure efficiency:The NTK matrixWWhas a representational size ofℝ\(h⋅w\)×c\\mathbb\{R\}^\{\(h\\cdot w\)\\times c\}, matching the ground truth \(GT\) image size but lacking efficient structure\.
2. 2\.Representational effectiveness:AlthoughWWmatches the image size, it fails to achieve effective representation in terms of performance results\.
3. 3\.Time\-complexity:The NTK method exhibits high time complexity and computational cost, being at least one order of magnitude less efficient in runtime\.

## IIIMethod

The limitations outlined in the previous section indicate that the straightforward application of NTK as a substitute for training neural fields is inadequate for effective representation learning\. To address these issues, we propose two novel approaches that can be used independently or combined to create efficient NTK representation learning algorithms\. Each approach targets a unique, previously unresolved challenge within NTK, and their integration could overcome existing limitations\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/NTK_Novel_Algorithm/Single_Multi_NF_Shared.png)Fig\. 9:Top section: Training a Shared NF from a Single Image\. Iterations from left to right: 0, 100, 200, 300, 400, 500, 600\. Bottom section: Last iteration in training a Multi\-Shared NF \(From 100 images\)![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/NTK_Novel_Algorithm/NF_i_Reconstruction_image.png)Fig\. 10:Top section: Training aN​FiNF\_\{i\}fromN​FθSNF\_\{\\theta\_\{S\}\}\(Shared NF\) for 200 iterations, the iterations from left to right: 0, 100, 200\. Bottom section: Reconstructed Images viaWWVector andN​FθSNF\_\{\\theta\_\{S\}\}\. Bottom section, from top to bottom: GT image, W vector, Reconstructed image### III\-ANeural Field Kernel Inducing Points \(NTK\-KIP\)

As opposed to using the NTK as a direct mapping between dataYY\(e\.g\. images\) and representationWW\(creating a linear transformation representation\. See Section[II\-A](https://arxiv.org/html/2609.03117#S2.SS1)\), we propose to use the NTK as a mean to accelerate a different method of extracting useful representation, namely, finding inducing points in images\. This will yield distilled data, and also a non\-linear NTK representation \(WWvector\)\.

NTK\-KIP\.While the typical way to train a NF is to use all the image pixels as the training set, here we seek to find a smaller set of induced pixels that can serve as a compressed training set for the neural field\. In other words, we are optimizing the values of a small training set, such that training a NF on it will result in a good generalization of the real pixels in the image\. Notably, each update of the inducing point optimization requires full training of the NF, which renders this unrealistic\. To solve this, we propose to replace the NF training process with the NTK regression approximation and optimize the inducing points by differentiating through the NTK regression\. Formally, letNNbe the total number of images in our dataset\. Given imagei∈\[1,…,N\]i\\in\[1,\\,\.\.\.,\\,N\]of size\(h,w,c\)\(h,\\,w,\\,c\), letP=h⋅wP=h\\cdot wbe the total number of pixels inii,p∈\[1,…,P\]p\\in\[1,\\,\.\.\.,\\,P\]be a pixel inii,x∈ℝN×2x\\in\\mathbb\{R\}^\{N\\times 2\}be the PE vector,y∈ℝ\(P×c\)y\\in\\mathbb\{R\}^\{\(P\\times c\)\}be the image labels \(values\)\. We then definexip∈ℝ2x\_\{i\}^\{p\}\\in\\mathbb\{R\}^\{2\}to be the PE coordinate which corresponds to pixelppin imageii, andyipy\_\{i\}^\{p\}to be the value \(label\) of pixelppin imageii\. In the Raw PE \(RPE\) example, we optimizexipx\_\{i\}^\{p\}andyipy\_\{i\}^\{p\}to minimize the reconstruction error for the full image:

arg⁡minxip,yip​‖𝒦⁡\(x,xip\)​𝒦​\(xip,xip\)−1​yip−y‖\\displaystyle\\arg\\min\_\{x\_\{i\}^\{p\},y\_\{i\}^\{p\}\}\\left\\\|\\mathcal\{K\}\(x,x\_\{i\}^\{p\}\)\\;\\mathcal\{K\}\(x\_\{i\}^\{p\},x\_\{i\}^\{p\}\)^\{\-1\}\\;y\_\{i\}^\{p\}\-y\\right\\\|
This representation is therefore no longer a simple linear mapping of the signal, and can potentially lead to extracting more meaningful and useful information from the image\. Previous work has shown that this approach is effective for dataset distillation in classification problems\[[23](https://arxiv.org/html/2609.03117#bib.bib33),[13](https://arxiv.org/html/2609.03117#bib.bib13)\]\. Here we apply the same method on NF training, where our dataset consists of image coordinates and pixels, and therefore ‘dataset distillation‘ translates to finding inducing points for an image\. For the full algorithm elaboration and formulation, please refer to Appendix[E\-A1](https://arxiv.org/html/2609.03117#A5.SS1.SSS1)\.

### III\-BMetaQuill Algorithm

One of the limitations of replacing NF training with NTK separately for each image is that no global information at the level of the dataset is extracted\. In comparison, in the standard implementation of Functa\[[9](https://arxiv.org/html/2609.03117#bib.bib10)\]with finite NFs, a shared set of weights for the whole dataset is used, and the representation of each image consists of a latent vector that is used to modulate the shared network\. The advantage of this is that global information that is common to the whole dataset does not need to be stored in the representation of each image\.

MetaQuill\.We proposeMetaQuillAlgorithm\. An algorithm for Neural Field meta\-learning through NTK, which enhances the NTK method with feature\-learning capabilities in an efficient manner\. We propose to use a similar approach to Functa, using Model Agnostic meta\-learning \(MAML\)\[[24](https://arxiv.org/html/2609.03117#bib.bib34)\]and finite width NTK\[[25](https://arxiv.org/html/2609.03117#bib.bib35)\]\. Our method is similar to MAML, where the initial values of the network weights are optimized through the \(short\) training of the network for different images\. In order to accelerate the inner loop of MAML, we use finite\-width NTK, a method that approximates the training process of a finite network, by linearizing the weights updates around some initial value of the weights\. Formally, we approximate a few stepsttof gradient descent on the parametersθ\\thetaaround the parametersθτ\\theta\_\{\\tau\}by:

fθτ\+t​\(x\)−fθτ​\(x\)\\displaystyle f\_\{\\theta\_\{\\tau\+t\}\}\(x\)\-f\_\{\\theta\_\{\\tau\}\}\(x\)≈\(θτ\+t−θτ\)T​∇θτfθτ​\(x\)\\displaystyle\\approx\(\\theta\_\{\\tau\+t\}\-\\theta\_\{\\tau\}\)^\{T\}\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x\)=Δ​θT​∇θτfθτ​\(x\)\\displaystyle=\\Delta\\theta^\{T\}\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x\)
In our context,θτ\\theta\_\{\\tau\}is shared for all images and serves as a starting point for the above approximation and is trained simultaneously with separate values ofΔ​θ\\Delta\\thetacorresponding to each image in the dataset\. From the shared NF, we can calculate each specific NF model which corresponds to an imagei∈Ni\\in Nby,

N​Fi​\(x\)=△​θiT⋅∇θSfθS​\(x\)\\displaystyle NF\_\{i\}\(x\)=\\triangle\\theta\_\{i\}^\{T\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x\)
The values ofΔ​θ\\Delta\\thetacan then serve as the representation of the specific image, or mapped to the kernel coefficientWW\. Formally,

△​θi=∑Wip,xpWip⋅∇θSfθS​\(xp\),\\displaystyle\\triangle\\theta\_\{i\}=\\sum\_\{W\_\{i\}^\{p\},\\,x^\{p\}\}W\_\{i\}^\{p\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x^\{p\}\),such thatWi=KθS−1⋅YiW\_\{i\}=K\_\{\\theta\_\{S\}\}^\{\-1\}\\cdot Y\_\{i\}, whereYiY\_\{i\}is GT imageiiandppis an image pixel\. For the full algorithm development and the theoretical analysis, please refer to Appendix[E\-A2](https://arxiv.org/html/2609.03117#A5.SS1.SSS2)\. In Section[IV](https://arxiv.org/html/2609.03117#S4), Fig\.[9](https://arxiv.org/html/2609.03117#S3.F9)shows example of our method applied to a dataset of 100 images from MNIST, achieving shared NF mappingθτ\\theta\_\{\\tau\}\. And Fig\.[10](https://arxiv.org/html/2609.03117#S3.F10)shows results of both achieving△​θ\\triangle\\thetaand extracting theWWvector \(via finite\-NTK\) from the shared NF model\.

### III\-CMetaQuill\-KIP Hybrid Adaptation

NTK\-KIP and MetaQuill address two different failure modes of applying NTK regression to neural fields\. NTK\-KIP learns a compact, task\-specific support set of synthetic coordinates and labels that makes NTK fitting nonlinear, efficient, and robust to sparse observations\. MetaQuill learns a reusable shared initializationθS\\theta\_\{S\}across tasks and adapts new images by optimizing only a lightweight residualΔ​θ\\Delta\\theta, rather than re\-training a full network\. However, each method alone still retains a weakness: NTK\-KIP does not transfer shared structure across tasks, and MetaQuill remains locally linear aroundθS\\theta\_\{S\}and can struggle to hallucinate large missing regions\. We therefore propose a hybrid procedure,*MetaQuill\-KIP*, which fuses these strengths\.

##### Setup\.

Letfθ:ℝd→ℝcf\_\{\\theta\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{c\}be the neural field \(for example, an MLP mapping positional encodings to RGB\)\. MetaQuill meta\-learns a shared parameter vectorθS\\theta\_\{S\}across a pool of training tasks\. For a new target task𝒯\\mathcal\{T\}\(a single image\), we seek a small per\-task updateΔ​θ𝒯\\Delta\\theta\_\{\\mathcal\{T\}\}such that

fθS\+Δ​θ𝒯​\(x\)≈y⁡\(x\)f\_\{\\theta\_\{S\}\+\\Delta\\theta\_\{\\mathcal\{T\}\}\}\(x\)\\approx y\(x\)\(6\)for all spatial coordinatesxxin the image, including coordinates that are never observed during inpainting\.

##### Step 1: Shared backbone from MetaQuill\.

MetaQuill providesθS\\theta\_\{S\}by minimizing, over a set of training tasks\{𝒯j\}\\\{\\mathcal\{T\}\_\{j\}\\\},

min⁡∑jθS⁡ℒ⁡\(fθS\+Δ​θj​\(x\),yj​\(x\)\)\.\\min\_\{\\theta\_\{S\}\}\\sum\_\{j\}\\mathcal\{L\}\\Big\(f\_\{\\theta\_\{S\}\+\\Delta\\theta\_\{j\}\}\(x\),\\,y\_\{j\}\(x\)\\Big\)\.\(7\)HereΔ​θj\\Delta\\theta\_\{j\}is the task\-specific residual obtained from a short adaptation starting atθS\\theta\_\{S\}, approximated via a finite\-width NTK linearization so that no expensive full inner loop retraining is required\. In practice, MetaQuill approximates the inner adaptation with a finite\-width NTK linearization aroundθS\\theta\_\{S\}, so that each task learns only its residualΔ​θj\\Delta\\theta\_\{j\}without re\-training the full network\. This makesθS\\theta\_\{S\}a meta\-learned feature extractor that already encodes global structure \(for example, digit strokes in MNIST or petal texture in Flowers\)\.

##### Step 2: KIP\-style distilled initialization\.

For a new task𝒯\\mathcal\{T\}with observed pixels\{\(xi,yi\)\}i∈𝒪\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i\\in\\mathcal\{O\}\}and possibly large missing regions, we first run a Kernel Inducing Point \(KIP\) procedure to produce a compact distilled support set

𝒮𝒯=\{\(x~m,y~m\)\}m=1M,\\mathcal\{S\}\_\{\\mathcal\{T\}\}=\\\{\(\\tilde\{x\}\_\{m\},\\tilde\{y\}\_\{m\}\)\\\}\_\{m=1\}^\{M\},\(8\)whereM≪\|𝒪\|M\\ll\|\\mathcal\{O\}\|\. Thesex~m\\tilde\{x\}\_\{m\}are synthetic coordinates \(learned positional encodings\), andy~m\\tilde\{y\}\_\{m\}are synthetic labels, optimized so that solving an NTK\-style regression on𝒮𝒯\\mathcal\{S\}\_\{\\mathcal\{T\}\}alone already approximates the full image\. Concretely, we optimize𝒮𝒯\\mathcal\{S\}\_\{\\mathcal\{T\}\}to minimize

ℒKIP=‖𝒦θS​\(X,X~\)​\(𝒦θS​\(X~,X~\)\+λ​I\)−1​Y~−Y‖2,\\mathcal\{L\}\_\{\\mathrm\{KIP\}\}=\\left\\\|\\mathcal\{K\}\_\{\\theta\_\{S\}\}\(X,\\tilde\{X\}\)\\Big\(\\mathcal\{K\}\_\{\\theta\_\{S\}\}\(\\tilde\{X\},\\tilde\{X\}\)\+\\lambda I\\Big\)^\{\-1\}\\tilde\{Y\}\-Y\\right\\\|^\{2\},\(9\)whereXXandYYare the coordinates and pixel values of either all known pixels or just the observed \(unmasked\) pixels,X~\\tilde\{X\}andY~\\tilde\{Y\}stack the distilled supports\(x~m,y~m\)\(\\tilde\{x\}\_\{m\},\\tilde\{y\}\_\{m\}\), and𝒦θS\\mathcal\{K\}\_\{\\theta\_\{S\}\}is the finite\-width NTK computed aroundθS\\theta\_\{S\}\. The regularizerλ​I\\lambda Iis the usual ridge term\. This step gives us two things: \(i\) an NTK predictor conditioned on a tiny distilled set, and \(ii\) an implicit task\-specific direction in parameter space\.

##### Step 3: Lifting the KIP solution into parameter space\.

Because𝒦θS\\mathcal\{K\}\_\{\\theta\_\{S\}\}is the tangent kernel offθf\_\{\\theta\}atθS\\theta\_\{S\}, its solution corresponds to a first\-order update of the weights\. In other words, there exists a residualΔ​θKIP\\Delta\\theta\_\{\\mathrm\{KIP\}\}such that, to first order,

fθS\+Δ​θKIP​\(x\)≈𝒦θS​\(x,X~\)​\(𝒦θS​\(X~,X~\)\+λ​I\)−1​Y~\.f\_\{\\theta\_\{S\}\+\\Delta\\theta\_\{\\mathrm\{KIP\}\}\}\(x\)\\approx\\mathcal\{K\}\_\{\\theta\_\{S\}\}\(x,\\tilde\{X\}\)\\Big\(\\mathcal\{K\}\_\{\\theta\_\{S\}\}\(\\tilde\{X\},\\tilde\{X\}\)\+\\lambda I\\Big\)^\{\-1\}\\tilde\{Y\}\.\(10\)We take thisΔ​θKIP\\Delta\\theta\_\{\\mathrm\{KIP\}\}as the*initialization*for our per\-task residual, instead of starting fromΔ​θ=0\\Delta\\theta=0as in vanilla MetaQuill\.

##### Step 4: Fast nonlinear residual refinement\.

Finally, we refine only the residualΔ​θ\\Delta\\thetaby standard gradient descent for a short budget of stepsTT:

Δ​θ\(t\+1\)=Δ​θ\(t\)−η​∇Δ​θℒ​\(fθS\+Δ​θ\(t\)​\(xi\),yi\)i∈𝒪\.\\Delta\\theta^\{\(t\+1\)\}=\\Delta\\theta^\{\(t\)\}\-\\eta\\,\\nabla\_\{\\Delta\\theta\}\\mathcal\{L\}\\Big\(f\_\{\\theta\_\{S\}\+\\Delta\\theta^\{\(t\)\}\}\(x\_\{i\}\),\\,y\_\{i\}\\Big\)\_\{i\\in\\mathcal\{O\}\}\.\(11\)Heret=0,1,…,T−1t=0,1,\\ldots,T\-1,η\\etais a small learning rate, and𝒪\\mathcal\{O\}indexes only the observed \(unmasked\) pixels\. withΔ​θ\(0\)=Δ​θKIP\\Delta\\theta^\{\(0\)\}=\\Delta\\theta\_\{\\mathrm\{KIP\}\}andθS\\theta\_\{S\}frozen\. BecauseΔ​θ\\Delta\\thetais typically orders of magnitude smaller thanθS\\theta\_\{S\}, this refinement is extremely fast in wall\-clock time \(on the order of a few seconds forT∈\[50,500\]T\\in\[50,500\]in our experiments\)\. Importantly, this last stage is*nonlinear*: we are no longer constrained to the pure tangent regime of MetaQuill’s linear approximation\. This lets the model hallucinate plausible structure in large missing regions\.

##### Why this hybrid matters\.

MetaQuill\-KIP combines: \(i\) a meta\-learned backboneθS\\theta\_\{S\}that captures globally reusable structure across tasks, \(ii\) a distilled, task\-specific inducing set from KIP that encodes the identity of the new instance using only a handful of synthetic coordinates and values, and \(iii\) a fast, nonlinear residual refinement that updates onlyΔ​θ\\Delta\\thetainstead of re\-training the full network\. Empirically, on MNIST, MetaQuill\-KIP reaches over3232dB PSNR after only5050residual\-update steps, compared to about1111dB for kernel ridge regression alone and under1919dB for a tangent\-only MetaQuill\-style adaptation with no KIP initialization \(See Table[II](https://arxiv.org/html/2609.03117#S4.T2)\)\. On a Flowers inpainting task with large missing regions, MetaQuill\-KIP reconstructs semantically consistent petal and texture structure while adapting in seconds per image, despite observing as little as1%1\\%to10%10\\%of the pixels\. Competing diffusion models in our study either require heavy pretrained generative priors \(Stable Diffusion, StrDiffusion\) or significantly longer per\-image adaptation \(for example, full diffusion or many\-step score refinement\), and they cannot reach comparable reconstruction quality under the same lightweight adaptation budget\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/NF_KIP/fig11_ntk_kip_grid_90.png)Fig\. 11:NTK\-KIP experiment\. Left section: Raw PEs\. Right section: FPEs\. In each section, columns from left to right: ground truth image, distilled image, initial PSNR, final PSNR\. Left section distillation percentages \(top to bottom: 20%, 50%, 80%, 90%\)\. Right section distillation percentages \(top to bottom: 80%, 90%, 95%, 99%\)\.

## IVResults

We present the results of our three algorithms:NTK\-KIP,MetaQuill, andMetaQuillKIP\.

### IV\-ANTK\-KIP Results

The setting\.We train an NF in such a manner where we seek to find a small, distilled set of induced pixel’s coordinates that can serve as a compressed training set for the NF\. In other words, we are optimizing the values of a small training set, such that training an NF on it will result in a good generalization of the real pixels in the image\. We show two experiments: in the first, the pixel position is optimized, and in the second, the Fourier positional feature is optimized\. For Fig\.[11](https://arxiv.org/html/2609.03117#S3.F11)and Table[I](https://arxiv.org/html/2609.03117#S4.T1)we report a qualitative slice atKbands=20K\_\{\\mathrm\{bands\}\}=20\. For the hyperparameter study in Fig\.[12](https://arxiv.org/html/2609.03117#S4.F12)and Fig\.[13](https://arxiv.org/html/2609.03117#S4.F13), we sweepKbandsK\_\{\\mathrm\{bands\}\}and distillation percentage while keeping the remaining settings fixed\.

Distilled representation\.Fig\.[11](https://arxiv.org/html/2609.03117#S3.F11)shows the reconstruction results of our method using different number of inducing points\. Fig\.[11](https://arxiv.org/html/2609.03117#S3.F11)shows representative reconstructions across distillation levels for Raw PE and FPE atKbands=20K\_\{\\mathrm\{bands\}\}=20\. To summarize the quantitative trend across distillation percentages and encoding types, see Table[I](https://arxiv.org/html/2609.03117#S4.T1)\. AtKbands=20K\_\{\\mathrm\{bands\}\}=20, Raw PE reaches about33\.3533\.35dB at20%20\\%distillation and about19\.9519\.95dB at50%50\\%, while FPE maintains about20\.4320\.43dB at95%95\\%and about17\.0717\.07dB at99%99\\%\.

Optimizing the Fourier positional encoding improves reconstruction quality at high distillation rates and increases representational capacity throughKbandsK\_\{\\mathrm\{bands\}\}\. Fig\.[11](https://arxiv.org/html/2609.03117#S3.F11)shows qualitative reconstructions at a fixedKbandsK\_\{\\mathrm\{bands\}\}, while Fig\.[12](https://arxiv.org/html/2609.03117#S4.F12)and Fig\.[13](https://arxiv.org/html/2609.03117#S4.F13)summarize the corresponding hyperparameter sweep overKbandsK\_\{\\mathrm\{bands\}\}and distillation\.

3D effect\.To study sensitivity to hyperparameters, we sweep FourierKbandsK\_\{\\mathrm\{bands\}\}and distillation percentage and record the final PSNR\. Fig\.[12](https://arxiv.org/html/2609.03117#S4.F12)visualizes a continuous PSNR surface estimated from the sweep using a quintic radial basis function interpolation\. To isolate each factor, Fig\.[13](https://arxiv.org/html/2609.03117#S4.F13)reports two 2D slices, one varyingKbandsK\_\{\\mathrm\{bands\}\}at a fixed distillation level and one varying distillation at a fixedKbandsK\_\{\\mathrm\{bands\}\}\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/NF_KIP/fig12_kip_surface_fourier.png)Fig\. 12:NTK\-KIP PSNR surface as a function of FourierKbandsK\_\{\\text\{bands\}\}\(horizontal axis\) and observed fraction / distillation percentage \(depth axis\)\. The vertical axis is PSNR \[dB\]\. This surface shows a clear ridge: performance improves sharply as we \(i\) raiseKbandsK\_\{\\text\{bands\}\}to allow richer Fourier positional encoding and \(ii\) allow a modest increase in observed fraction, then saturates\. The high\-PSNR band corresponds to the sweet spot where NTK\-KIP has both expressive positional features and enough distilled samples to remain numerically stable\.PE\\Distillation20%20\\%50%50\\%80%80\\%90%90\\%95%95\\%99%99\\%Raw\_PE33\.3533\.3519\.9519\.9524\.2724\.2721\.5021\.5019\.8119\.8117\.2117\.21FPE54\.6754\.6747\.4747\.4730\.3330\.3323\.7523\.7520\.4320\.4317\.0717\.07

TABLE I:PSNR \[dB\] as a function of observed fraction \(distillation percentage\) and positional encoding type \(Raw PE and FPE\) for NTK\-KIP\.![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/NF_KIP/fig14_kip_slices.png)Fig\. 13:Left: NTK\-KIP PSNR \[dB\] versus FourierKbandsK\_\{\\mathrm\{bands\}\}at distillation50%50\\%, with best PSNR≈48\.40\\approx 48\.40atKbands=30K\_\{\\mathrm\{bands\}\}=30\. Right: NTK\-KIP PSNR \[dB\] versus distillation percentage atKbands=30K\_\{\\mathrm\{bands\}\}=30, with best PSNR≈54\.24\\approx 54\.24at distillation20%20\\%\.Across the sweep, PSNR increases withKbandsK\_\{\\mathrm\{bands\}\}and then exhibits diminishing returns once the Fourier features are sufficiently expressive\. This saturation behavior is consistent with the elbow style ablation in Appendix[B](https://arxiv.org/html/2609.03117#A2), Fig\.[22](https://arxiv.org/html/2609.03117#A2.F22)\. We therefore useKbands=20K\_\{\\mathrm\{bands\}\}=20as a practical operating point in the main experiments, since it lies beyond the initial steep gain region while avoiding the higher computational cost of very largeKbandsK\_\{\\mathrm\{bands\}\}\.

NTK\-KIP summary\.To address NTK’sWWrepresentational linearity and enabling extracting useful information, we developed NTK\-KIP, achieving a95%95\\%distillation rate \(two orders of magnitude\) withP​S​N​R\>20PSNR\>20via FPE\. While it provides a more distilled, non\-linear representation, it remains limited by NTK’s inherent lack of learning and feature extraction\.

### IV\-BMetaQuill Algorithm Results

The setting\.We train a NF based on MetaQuill Algorithm on a small sub\-set of the MNIST dataset to achieve a shared\-base model\. This model can then be used as a generalized, well\-established starting point for efficiently training and adapting a new NF model to unseen data sample\. Our model consists of33total layers\. Two of width128128, and a single output layer\. For the shared NF section, we trained the single\-image NF for600600steps with learning rate of3​e−33e^\{\-3\}, and the Multi\-image NF \(Multi\-NF\) for50005000steps with a learning rate of1​e−31e^\{\-3\}\. The training results can be seen in Fig\.[9](https://arxiv.org/html/2609.03117#S3.F9)\.

Results explained\.Fig\.[9](https://arxiv.org/html/2609.03117#S3.F9)reports the results\. In its top section, we visualize the training process of the shared NF \(we show a single MNIST image for digit 5\)\. The bottom section shows the last iteration of training the shared Multi\-NF on a subset of the training dataset \(100 images in our example\)\. The shared NF \(N​FθSNF\_\{\\theta\_\{S\}\}\) is learning bothθS\\theta\_\{S\}and△​θi\\triangle\\theta\_\{i\}simultaneously\. After achieving the shared NF base parameters\(θS\)\(\\theta\_\{S\}\)\. We train a new NF from the shared obtainedN​FSNF\_\{S\}model, creatingN​FiNF\_\{i\}\(For imageii\) to obtain△​θi\\triangle\\theta\_\{i\}\(See Fig\.[10](https://arxiv.org/html/2609.03117#S3.F10), top section \- digit 7\)\.

After calculatingKθSK\_\{\\theta\_\{S\}\}, we calculate theWiW\_\{i\}vector \(With correspondence toYiY\_\{i\}sample image\) via Finite\-NTK\. Then, we calculate the reconstructed image \(As in Eq\.[5](https://arxiv.org/html/2609.03117#S1.E5)\)\. The results can be seen in Fig\.[10](https://arxiv.org/html/2609.03117#S3.F10), bottom section\. Unlike the infinite\-width NTK, where theWiW\_\{i\}vector has no feature learning and ignores model parameters, the Finite\-NTK with feature learning loses representation power\. We found this occurs because the kernel matrixKθSK\_\{\\theta\_\{S\}\}from Finite\-NTK ofN​FθSNF\_\{\\theta\_\{S\}\}is ill\-conditioned, making it nearly singular and prone to numerical errors, resulting in blurry image reconstructions\.

Discussion\.Our experiments showed that while our proposed algorithm achieved efficient feature learning, NTK’s limited representational nature, even with Finite\-Feature Learning, led to an ill\-conditioned kernel matrix, causing blurry reconstructions\. This indicates NTK lacks the ability to effectively distill learned features and capture the dataset distribution needed for diverse tasks due to its representational linearity \(See Section[II\-A](https://arxiv.org/html/2609.03117#S2.SS1),NTK Representational linearity\)\. The NTK\-KIP algorithm \(See Section[III\-A](https://arxiv.org/html/2609.03117#S3.SS1)\) addresses this issue\. We believe combining NTK\-KIP and MetaQuill could enable NTK to achieve non\-linear representation mapping, learning capabilities, efficiency, and meta\-learning\.

MetaQuillSummary\.The MetaQuill Algorithm enhances NTK by introducing feature learning and an efficient meta\-learning approach akin to MAML\[[24](https://arxiv.org/html/2609.03117#bib.bib34)\], but without the inner loop\. This enables training a shared NF model on a small data subset and adapting to new samples with a single gradient update\. Despite these improvements, it remains limited by NTK’s representational linearity \(addressed byNTK\-KIP\) and an ill\-conditioned matrix, leading to blurry reconstructions\. These findings aim to advance NTK\-based representation learning and inspire stronger theoretical frameworks and methods to overcome these limitations\.

Method / RegimePSNR \[dB\]Train secNotesSingle\-INR18\.3718\.370\.570\.57trains full network from scratchθi\\theta\_\{i\}tangent\-only18\.3218\.320\.750\.75MetaQuill\-style linearizedΔ​θ\\Delta\\thetaKRR\-only10\.7710\.7712\.6112\.61closed\-form kernel solve \(no GD steps\)KIP\(init\)11\.3111\.317\.497\.49distilled support solve, no nonlinear refineCombined\(KRR→\\rightarrownonlin\)26\.6526\.650\.450\.45refineΔ​θ\\Delta\\thetaafter KRR\-initCombined\(KIP→\\rightarrownonlin\)32\.490\.46refineΔ​θ\\Delta\\thetaafter KIP\-init

TABLE II:MNIST adaptation results\. PSNR is measured on a held\-out digit after per\-task adaptation\. “Train sec” is per\-digit wall\-clock adaptation time\. All MetaQuill\-style entries adapt only a small task\-specificΔ​θ\\Delta\\thetawhile freezing a sharedθS\\theta\_\{S\}, rather than fully retraining the network\. Combined\(KIP→\\rightarrownonlin\) reaches32\+32\{\+\}dB in well under a second of compute for that digit\.Method / RegimePSNR \[dB\]Train secNotesSingle\-INR13\.5713\.570\.970\.97full net, scratch on 1 imageθi\\theta\_\{i\}tangent\-only15\.0615\.061\.051\.05MetaQuill\-style linearizedΔ​θ\\Delta\\thetaKIP\(init\)9\.379\.3796\.3796\.37distilled support solve, no refineCombined\(KIP→\\rightarrownonlin\)26\.293\.12refineΔ​θ\\Delta\\thetaaroundθS\\theta\_\{S\}

TABLE III:Flowers reconstruction results\. PSNR is computed on the full180×180180\{\\times\}180RGB image\. Reported “Train sec” is the total wall\-clock adaptation budget for that single task\. Single\-INR trains the entire network from scratch on that image and reaches about13\.613\.6dB in about one second\. KIP\(init\) solves via a large distilled support but remains slow \(tens of seconds\) and under1010dB without nonlinear refinement\. MetaQuill\-KIP updates only a compact task\-specificΔ​θ\\Delta\\thetaaround a reusableθS\\theta\_\{S\}and reaches26\+26\{\+\}dB in about three seconds\.![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/mnist/mnist_combined_quad.png)Fig\. 14:MNIST qualitative comparison\. Columns: \(i\) ground truth digit, \(ii\) KIP\(init\) \(11\.3111\.31dB\), \(iii\) KRR\-only \(10\.7710\.77dB\), \(iv\)θi\\theta\_\{i\}tangent\-only \(18\.3218\.32dB\), \(v\) Combined\(KRR→\\rightarrownonlin\) \(26\.6526\.65dB\), \(vi\)Combined\(KIP→\\rightarrownonlin\)\(32\.4932\.49dB\)\. All reconstructions target the same held\-out digit\. The final column shows that MetaQuill\-KIP recovers clean digit structure with sharp strokes that neither pure kernel regression nor tangent\-only linearization could capture\.
### IV\-CMetaQuill\-KIP: Combined Results and Comparative Evaluation

In this subsection we report quantitative and qualitative results for the proposed MetaQuill\-KIP pipeline described in Section[III\-C](https://arxiv.org/html/2609.03117#S3.SS3)\. We evaluate three axes: \(i\) fast per\-task adaptation and reconstruction quality on MNIST, \(ii\) inpainting and hole filling on the Flowers image under extreme masking, and \(iii\) runtime and PSNR comparisons against both our internal baselines \(Single\-INR, KRR\-only, KIP alone, tangent\-only MetaQuill\-style adaptation\) and external diffusion\-style inpainting systems \(Stable Diffusion, StrDiffusion, DDPM, and GSDM\)\. All timing numbers are wall\-clock seconds on a single GPU\. Prediction cost \(forward pass\) is effectively negligible compared to adaptation, so we report adaptation/training time as the relevant budget\.

#### IV\-C1Fast per\-task adaptation on MNIST

Table[II](https://arxiv.org/html/2609.03117#S4.T2)summarizes reconstruction quality and wall\-clock adaptation time for MNIST digits using several baselines and our MetaQuill\-KIP method\. All MNIST experiments use a28×2828\{\\times\}28grayscale neural field\. We cap per\-task adaptation to at most5050gradient steps for the MetaQuill\-style methods and report the corresponding PSNR\. We also include KIP\(init\), which is a KIP\-style distilled support solve without nonlinear refinement, and KRR\-only, which is a pure kernel ridge regression solve using the finite NTK\. The MetaQuill\-KIP entry “Combined\(KIP→\\rightarrownonlin\)” initializes from a distilled KIP support and then refines only a low\-rankΔ​θ\\Delta\\thetafor5050steps, keeping the sharedθS\\theta\_\{S\}fixed\.

Several trends are clear\. First, pure kernel baselines \(KRR\-only, KIP\(init\)\) achieve low PSNR because they either solve a linearized kernel regression \(KRR\-only\) or distill inducing points but do not perform nonlinear refinement \(KIP\(init\)\)\. Second, the classic Single\-INR baseline eventually reaches high PSNR if trained long enough on that single image, but doing so requires optimizing*all*weights from scratch and scales poorly to a stream of new tasks\. By contrast, MetaQuill\-style tangent\-only adaptation reaches a similar PSNR in a comparable budget, but it is still limited by the linearized tangent approximation of the NTK\. Most importantly, MetaQuill\-KIP, shown asCombined\(KIP→\\rightarrownonlin\), reaches32\.4932\.49dB PSNR in about0\.460\.46seconds of adaptation, after only5050refinement steps onΔ​θ\\Delta\\theta, not on the full network\. This indicates that seedingΔ​θ\\Delta\\thetawith a KIP\-style distilled support set and then refining it nonlinearly around a meta\-learnedθS\\theta\_\{S\}closes both gaps at once: we get nonlinear expressivity \(missing from pure NTK\-style tangent updates\) and we get a strong warm start \(missing from naive single\-image INR\)\. Figure[14](https://arxiv.org/html/2609.03117#S4.F14)shows qualitative reconstructions for the same held\-out digit, including KIP init, KRR\-only, tangent\-only, and the final MetaQuill\-KIP refinement\. We also visualize the learned shared multi\-task initializationθS\\theta\_\{S\}as a grid of reconstructions for100100Flowers images in Figure[15](https://arxiv.org/html/2609.03117#S4.F15), demonstrating thatθS\\theta\_\{S\}encodes reusable image structure \(shared color statistics, edges, and texture primitives\) before any task\-specificΔ​θ\\Delta\\thetaadaptation\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/flowers/shared_thetaS_grid_10x10.jpg)Fig\. 15:Shared initializationθS\\theta\_\{S\}learned by MetaQuill\. Each tile shows the reconstruction of one of 100 Flowers training images using the shared initializationθS\\theta\_\{S\}together with its task specific offsetΔ​θi\\Delta\\theta\_\{i\}at the end of meta training\. The grid illustrates thatθS\\theta\_\{S\}encodes reusable structure across tasks, providing a strong starting point for rapid per task refinement in MetaQuill KIP\.
#### IV\-C2Flowers: inpainting, hole filling, and long\-horizon refinement

We next test on a180×180180\{\\times\}180RGB flower image\. We evaluate \(i\) unconstrained reconstruction \(full supervision\) and \(ii\) inpainting under severe random or circular masks, where only a fractionϕ\\phiof pixels is observed\. We report PSNR on the full image \(psnr\) and also PSNR restricted to thehole\(psnr\_hole\) when applicable\. Table[III](https://arxiv.org/html/2609.03117#S4.T3)summarizes full\-image reconstruction and adaptation cost\. Single\-INR after about 1 second of full\-network training from scratch \(50 steps\), is still below 14 dB\. MetaQuill\-style tangent\-only reaches15\.0615\.06dB after5050steps in about1\.051\.05seconds\. KIP\(init\) \(a distilled support solve with no nonlinear refinement\) produces only9\.379\.37dB even after tens of seconds because it lacks any task\-specific nonlinear correction\. In contrast, MetaQuill\-KIPCombined\(KIP→\\rightarrownonlin\)reaches26\.2926\.29dB in roughly3\.123\.12seconds of refinement \(small\-step updates toΔ​θ\\Delta\\thetaaround the frozenθS\\theta\_\{S\}\)\. That is an order\-of\-magnitude PSNR boost over raw KIP\(init\) and a dramatic improvement over Single\-INR in only less than an extra second\.

Figure[16](https://arxiv.org/html/2609.03117#S4.F16)provides a side\-by\-side qualitative comparison on the same flower image: ground truth, KIP init \(9\.379\.37dB\), tangent\-only \(15\.0615\.06dB\), and the final MetaQuill\-KIP result \(26\.2926\.29dB\)\. The MetaQuill\-KIP reconstruction preserves thin petal edges and textured background leaves that are completely missing in earlier baselines\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/flowers/combined_quad.png)Fig\. 16:Flowers qualitative comparison on the same masked image\. Columns follow the titles in the figure: ground truth, KIP\-init \(PSNR9\.379\.37dB\),θi\\theta\_\{i\}tangent only \(PSNR15\.0615\.06dB\), and Combined\(KIP→\\rightarrownonlin\) \(PSNR26\.2926\.29dB\)\. Hereθi=θS\+Δ​θi\\theta\_\{i\}=\\theta\_\{S\}\+\\Delta\\theta\_\{i\}with shared initializationθS\\theta\_\{S\}and task specific offsetΔ​θi\\Delta\\theta\_\{i\}\. The tangent only variant adaptsΔ​θi\\Delta\\theta\_\{i\}in the linearized NTK regime, while Combined\(KIP→\\rightarrownonlin\) refinesΔ​θi\\Delta\\theta\_\{i\}with nonlinear updates aroundθS\\theta\_\{S\}, recovering fine petal structure and realistic leaf texture rather than a blurry global color field\.To illustrate temporal refinement, Figure[17](https://arxiv.org/html/2609.03117#S4.F17)shows the evolution of the MetaQuill\-KIP prediction over500500adaptation steps onΔ​θ\\Delta\\theta: from the KIP initialization \(9\.379\.37dB\) through intermediate checkpoints \(16\.7316\.73dB att=100t=100,21\.7721\.77dB att=200t=200,24\.3124\.31dB att=300t=300,25\.3925\.39dB att=400t=400\) to the final26\.2926\.29dB att=500t=500\. For comparison, Figure[18](https://arxiv.org/html/2609.03117#S4.F18)shows the purely tangent\-only MetaQuill adaptation over its first5050steps, saturating near1515dB\. Together, these strips emphasize that nonlinearΔ​θ\\Delta\\thetarefinement is critical for recovering high\-frequency semantics such as veins and shadows on the petals\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/flowers/combined_kip_strip.png)Fig\. 17:MetaQuill\-KIP refinement strip on Flowers\. Each tile shows the reconstruction aftert∈\{0,100,200,300,400,500\}t\\in\\\{0,100,200,300,400,500\\\}refinement steps, updating only the task specific offsetΔ​θi\\Delta\\theta\_\{i\}while keeping the shared initializationθS\\theta\_\{S\}fixed, withθi=θS\+Δ​θi\\theta\_\{i\}=\\theta\_\{S\}\+\\Delta\\theta\_\{i\}\. The corresponding PSNR values are \(9\.379\.37,16\.7316\.73,21\.7721\.77,24\.3124\.31,25\.3925\.39,26\.2926\.29dB\), illustrating progressive recovery of petal curvature and background texture\.![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/flowers/adapt_theta_i_tangent_strip.png)Fig\. 18:Tangent\-only MetaQuill adaptation strip on Flowers\. Reconstructions aftert∈\{0,10,20,30,40,50\}t\\in\\\{0,10,20,30,40,50\\\}steps of linearizedΔ​θ\\Delta\\thetaadaptation, with PSNR \(7\.877\.87,11\.2911\.29,12\.5712\.57,12\.6112\.61,14\.6914\.69,15\.0615\.06dB\)\. This variant improves quickly in the first few steps but cannot fully recover fine detail, reflecting the expressive ceiling of a purely linearized NTK update\.For inpainting, we mask out most of the flower and reveal only a fractionϕ\\phiof pixels at random or in a center circle\. We then ask each method to hallucinate the missing region\. Table[IV](https://arxiv.org/html/2609.03117#S4.T4)reports full\-image PSNR and hole PSNR across several observed fractionsϕ∈\{0\.01,0\.05,0\.1,0\.2,0\.5,0\.8\}\\phi\\in\\\{0\.01,0\.05,0\.1,0\.2,0\.5,0\.8\\\}\. We compare three variants: Single\-INR\(masked\), KIP\-inpaint \(distilled support \+ kernel solve\), and MetaQuill\-KIPCombined\(KIP→\\rightarrownl\)which refinesΔ​θ\\Delta\\thetaaroundθS\\theta\_\{S\}under the mask\. Single\-INR\(masked\) trains for5050steps on only the visible pixels; KIP\-inpaint optimizes its support set using only observed pixels; MetaQuill\-KIP performs a masked nonlinear refinement ofΔ​θ\\Delta\\thetafor500500steps\.

Obs\. fracϕ\\phiRegimePSNR \[dB\]PSNR hole \[dB\]Train sec0\.010\.01Single\-INR\(masked\)13\.6013\.6013\.5613\.560\.340\.34KIP\-inpaint11\.9611\.9611\.9611\.9693\.2593\.25Combined\(KIP→\\rightarrownl\)9\.639\.639\.629\.622\.732\.730\.050\.05Single\-INR\(masked\)14\.3814\.3814\.2414\.240\.340\.34KIP\-inpaint12\.4712\.4712\.4712\.4792\.9892\.98Combined\(KIP→\\rightarrownl\)8\.998\.998\.978\.972\.552\.550\.100\.10Single\-INR\(masked\)14\.7414\.7414\.4914\.490\.330\.33KIP\-inpaint12\.8612\.8612\.8712\.8793\.9493\.94Combined\(KIP→\\rightarrownl\)11\.9711\.9711\.8811\.882\.552\.550\.200\.20Single\-INR\(masked\)14\.7014\.7014\.5214\.520\.340\.34KIP\-inpaint12\.2012\.2012\.3512\.3593\.2993\.29Combined\(KIP→\\rightarrownl\)13\.5913\.5913\.4013\.402\.572\.570\.500\.50Single\-INR\(masked\)14\.4114\.4114\.7814\.780\.340\.34KIP\-inpaint12\.8712\.8712\.5712\.5793\.2493\.24Combined\(KIP→\\rightarrownl\)18\.8918\.8917\.7017\.702\.732\.730\.800\.80Single\-INR\(masked\)14\.0014\.0014\.4314\.430\.340\.34KIP\-inpaint10\.8810\.889\.679\.6793\.2793\.27Combined\(KIP→\\rightarrownl\)21\.7121\.7119\.4419\.442\.782\.78

TABLE IV:Flowers inpainting under random masks at various observed fractionsϕ\\phi\. “PSNR hole” measures PSNR only inside the missing region\. At high coverage \(ϕ∈\{0\.5,0\.8\}\\phi\\in\\\{0\.5,0\.8\\\}\), MetaQuill\-KIP \(*Combined\(KIP→\\rightarrownl\)*\) surpasses both masked Single\-INR and KIP\-inpaint in full\-image PSNR and produces visually coherent fills while adapting only a task\-specificΔ​θ\\Delta\\theta\. At very low coverage \(ϕ≤0\.20\\phi\\leq 0\.20\), PSNR alone can be misleading: MetaQuill\-KIP sometimes reports lower global PSNR than Single\-INR, yet qualitative results show it can synthesize more globally structured petals stems and background foliage rather than just diffusing color noise from the visible pixels\. See Figures[20](https://arxiv.org/html/2609.03117#S4.F20)and[21](https://arxiv.org/html/2609.03117#S4.F21)for qualitative examples atϕ=0\.10\\phi\{=\}0\.10andϕ=0\.20\\phi\{=\}0\.20, and Figure[19](https://arxiv.org/html/2609.03117#S4.F19)for circular\-hole completion\.At extremely sparse coverage \(ϕ≤0\.20\\phi\\leq 0\.20\), MetaQuill\-KIP does not always maximize global PSNR\. For example, atϕ=0\.10\\phi\{=\}0\.10, Single\-INR\(masked\) reaches14\.7414\.74dB while Combined\(KIP→\\rightarrownl\) reports11\.9711\.97dB, and atϕ=0\.20\\phi\{=\}0\.20the numbers are14\.7014\.70dB vs13\.5913\.59dB\. However, the qualitative behavior is very different\. Figures[20](https://arxiv.org/html/2609.03117#S4.F20)and[21](https://arxiv.org/html/2609.03117#S4.F21)show that Combined\(KIP→\\rightarrownl\) already reconstructs petal contours, stem structure, and plausible leaf\-colored background inside the missing region, whereas Single\-INR\(masked\) mostly smooths or repeats noisy color fragments from the observed pixels\. In other words, under severe sparsity, PSNR alone understates the semantic plausibility of the MetaQuill\-KIP hallucination\.

Figure[19](https://arxiv.org/html/2609.03117#S4.F19)shows a representative circular\-hole inpainting case \(ϕ≈0\.90\\phi\\approx 0\.90observed outside the hole\)\. Columns include: ground truth, the masked input, Single\-INR\(masked\), KIP\-inpaint, and MetaQuill\-KIPCombined\(KIP→\\rightarrownl\)\. Although the latter’s global PSNR may be similar to Single\-INR in that setting \(both around1313–1414dB full\-image\), the MetaQuill\-KIP fill inside the hole reproduces realistic autumn\-leaf texture and the correct petal colors without obvious seams, rather than simply averaging surrounding colors\. This qualitative coherence is exactly the “semantic plausibility” gap that standard kernel regression and short Single\-INR training fail to bridge\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/flowers/circle_r32_inpaint_quad.png)Fig\. 19:Circular\-hole inpainting on the Flowers image\. Columns: \(i\) ground truth, \(ii\) masked input \(large center hole\), \(iii\) Single\-INR\(masked\) \(13\.6813\.68dB full\-image\), \(iv\) KIP\-inpaint \(12\.5012\.50dB\), \(v\) Combined\(KIP→\\rightarrownl\) \(13\.6713\.67dB full\-image\)\. MetaQuill\-KIP reconstructs the missing region with leaf\-like texture and petal color continuity that blends into the context with minimal boundary artifacts, even though no pixels inside the hole were observed\.
#### IV\-C3Comparison to diffusion\-based inpainting baselines

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/flowers/rand_10_inpaint_quad.png)Fig\. 20:Random\-mask inpainting on the Flowers image atϕ=0\.10\\phi\{=\}0\.10observed fraction\. Columns: \(i\) ground truth, \(ii\) masked input, \(iii\) Single\-INR\(masked\) \(14\.7414\.74dB full\-image\), \(iv\) KIP\-inpaint \(12\.8612\.86dB\), and \(v\) Combined\(KIP→\\rightarrownl\) after maskedΔ​θ\\Delta\\thetarefinement \(11\.9711\.97dB\)\. Although Combined\(KIP→\\rightarrownl\) has lower global PSNR at this extreme sparsity, it already begins to synthesize coherent petal structure, stem\-like geometry, and plausible background leaf color, while Single\-INR largely preserves a noisy, low\-detail interpolation of the visible pixels\. This suggests that at very low coverage, PSNR alone can underestimate semantic plausibility\.![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/flowers/rand_20_inpaint_quad.png)Fig\. 21:Random\-mask inpainting on the Flowers image atϕ=0\.20\\phi\{=\}0\.20observed fraction\. Columns: \(i\) ground truth, \(ii\) masked input, \(iii\) Single\-INR\(masked\) \(14\.7014\.70dB full\-image\), \(iv\) KIP\-inpaint \(12\.2012\.20dB\), and \(v\) Combined\(KIP→\\rightarrownl\) after maskedΔ​θ\\Delta\\thetarefinement \(13\.5913\.59dB\)\. At this moderate sparsity, Combined\(KIP→\\rightarrownl\) reconstructs petal boundaries and stem coloration with consistent global structure, while Single\-INR remains dominated by noisy color blobs\. Even when its PSNR is close to or below Single\-INR, the Combined\(KIP→\\rightarrownl\) output exhibits more globally consistent object semantics\.Finally, we situate MetaQuill\-KIP against modern diffusion\-style inpainting systems on the Flowers image, and against lightweight diffusion baselines on MNIST\. For Flowers, we include: Stable Diffusion Inpainting \(SD\) in zero\-shot mode \(SD\-ZS\) and with per\-image finetuning \(SD\-IA\), StrDiffusion \(StrDiff\-ZS\), a strong pretrained inpainting prior, and our MetaQuill\-KIPCombined\(KIP→\\rightarrownl\)from Table[IV](https://arxiv.org/html/2609.03117#S4.T4)\. All diffusion models are large pretrained generative priors with strong scene priors\. MetaQuill\-KIP, in contrast, starts from a task\-agnosticθS\\theta\_\{S\}and adapts only a tinyΔ​θ\\Delta\\thetaon*the same single test image*without any external dataset\. Table[V](https://arxiv.org/html/2609.03117#S4.T5)reports PSNR for several observed fractionsϕ\\phi, along with wall\-clock time\. For SD\-ZS and StrDiff\-ZS we report zero\-shot inpainting with no task\-specific gradient steps\. For SD\-IA we report PSNR after per\-image finetuning \(150 steps, roughly 30–35 seconds\)\. For MetaQuill\-KIP we reuse the masked refinement numbers from Table[IV](https://arxiv.org/html/2609.03117#S4.T4)\(500 steps,≈\\approx2–3 seconds\)\. Although StrDiffusion can yield high PSNR at highϕ\\phi\(for example30\.630\.6dB atϕ=0\.8\\phi\{=\}0\.8\), this comes from a massive model pretrained on broad natural images\. MetaQuill\-KIP, by contrast, is compact, ledgered by an explicit neural field parameterization, and can be adapted on\-device to a completely novel scene without any external prior\. At intermediate coverage \(ϕ=0\.5\\phi\{=\}0\.5andϕ=0\.8\\phi\{=\}0\.8\), MetaQuill\-KIP reaches18\.8918\.89dB and21\.7121\.71dB in only a few seconds, compared to SD\-IA’s17\.5617\.56dB and24\.2024\.20dB in about half a minute and StrDiffusion’s24\.5724\.57dB and30\.6030\.60dB in under a second zero\-shot\.

Obs\. fracϕ\\phiMethodPSNR \[dB\]Hole PSNR \[dB\]Time \[sec\]Mode0\.010\.01SD\-ZS11\.2811\.2816\.0016\.00∼1\.9\\sim 1\.9zero\-shotSD\-IA11\.0011\.0015\.7215\.72∼33\\sim 33per\-image finetuneStrDiff\-ZS14\.2614\.2618\.9818\.98∼0\.6\\sim 0\.6zero\-shotMetaQuill\-KIP9\.639\.639\.629\.62∼2\.7\\sim 2\.7per\-imageΔ​θ\\Delta\\thetarefine0\.100\.10SD\-ZS12\.1012\.1016\.4116\.41∼1\.9\\sim 1\.9zero\-shotSD\-IA12\.2612\.2616\.5716\.57∼35\\sim 35per\-image finetuneStrDiff\-ZS18\.2618\.2622\.5722\.57∼0\.6\\sim 0\.6zero\-shotMetaQuill\-KIP11\.9711\.9711\.8811\.88∼2\.6\\sim 2\.6per\-imageΔ​θ\\Delta\\thetarefine0\.500\.50SD\-ZS16\.8816\.8818\.6518\.65∼2\.0\\sim 2\.0zero\-shotSD\-IA17\.5617\.5619\.3319\.33∼35\\sim 35per\-image finetuneStrDiff\-ZS24\.5724\.5726\.3426\.34∼0\.6\\sim 0\.6zero\-shotMetaQuill\-KIP18\.8918\.8917\.7017\.70∼2\.7\\sim 2\.7per\-imageΔ​θ\\Delta\\thetarefine0\.800\.80SD\-ZS23\.9323\.9321\.7321\.73∼1\.9\\sim 1\.9zero\-shotSD\-IA24\.2024\.2022\.0022\.00∼35\\sim 35per\-image finetuneStrDiff\-ZS30\.6030\.6028\.4028\.40∼0\.6\\sim 0\.6zero\-shotMetaQuill\-KIP21\.7121\.7119\.4419\.44∼2\.8\\sim 2\.8per\-imageΔ​θ\\Delta\\thetarefine

TABLE V:Flowers inpainting: diffusion models vs MetaQuill\-KIP\. Here SD denotes Stable Diffusion, StrDiff denotes the structure guided diffusion model, ZS denotes zero shot inpainting, and IA denotes per image adaptation\. Time is average wall\-clock per masked input\. SD\-ZS and StrDiff\-ZS use large pretrained diffusion priors with zero\-shot inpainting\. SD\-IA performs an expensive per\-image finetune \(about 150 gradient steps, 30–35 seconds\)\. MetaQuill\-KIP performs a lightweight, fully localΔ​θ\\Delta\\thetarefinement \(2–3 seconds, no external prior\) around a meta\-learnedθS\\theta\_\{S\}\. High PSNR at largeϕ\\phifor StrDiffusion reflects access to a powerful global semantic prior rather than rapid on\-device adaptation\. MetaQuill\-KIP instead targets rapid, self\-contained personalization to a single novel scene\.On MNIST we also compare against diffusion\-style baselines trained or adapted under small budgets\. We include a Denoising Diffusion Probabilistic Model \(DDPM\) trained from scratch on the target digit and then adapted for up to5050steps, and a Global Structure Guided Diffusion Model \(GSDM\) using a pretrained restoration module adapted for the same budget\. Table[VI](https://arxiv.org/html/2609.03117#S4.T6)summarizes PSNR and adaptation time\. After5050steps, DDPM reaches about16\.0916\.09dB in roughly0\.160\.16seconds per digit, while GSDM reaches about6\.706\.70dB\. MetaQuill\-KIP, by comparison, reaches32\.4932\.49dB PSNR in about0\.460\.46seconds for5050steps ofΔ​θ\\Delta\\thetarefinement \(Table[II](https://arxiv.org/html/2609.03117#S4.T2)\)\. This is a\>15\>15dB absolute gain over DDPM at similar or modestly higher wall\-clock cost, without training a full generative model or sampling a diffusion trajectory\.

ModelStepsPSNR \[dB\]Time \[sec\]DDPM \(from scratch\)09\.419\.411\.841\.84DDPM \(from scratch\)2513\.5313\.530\.190\.19DDPM \(from scratch\)5016\.0916\.090\.160\.16GSDM \(pretrained RM\)05\.725\.720\.060\.06GSDM \(pretrained RM\)256\.566\.560\.0260\.026GSDM \(pretrained RM\)506\.706\.700\.0270\.027MetaQuill\-KIP5032\.490\.46

TABLE VI:MNIST reconstruction: diffusion\-style baselines vs MetaQuill\-KIP\. DDPM is trained or adapted directly on the target digit via diffusion steps\. GSDM adapts a pretrained restoration module \(RM\) for the digit\. MetaQuill\-KIP corresponds to the Combined\(KIP→\\rightarrownonlin\) entry in Table[II](https://arxiv.org/html/2609.03117#S4.T2)\. Despite a comparable adaptation budget \(tens of steps and well under a second of wall\-clock time\), MetaQuill\-KIP reaches over3030dB PSNR, whereas DDPM and GSDM remain under1717dB and77dB respectively\.Interpreting these comparisons\.The goal of MetaQuill\-KIP is not to replace large pretrained diffusion models as a universal inpainting prior, but to show that Neural Tangent Kernel style neural field fitting can be made both nonlinear and rapidly adaptable in practice, in a way that was not previously demonstrated\. Classical NTK pipelines suffer from two structural limitations: they are linear in function space, and they cannot meta\-learn reusable task structure\. MetaQuill\-KIP addresses both by \(i\) distilling a compact KIP\-style support, which injects nonlinear structure, and \(ii\) adapting only a task\-specific offsetΔ​θ\\Delta\\thetaaround a sharedθS\\theta\_\{S\}, which provides true meta\-learned reuse\. The diffusion baselines \(Stable Diffusion, StrDiffusion, etc\.\) operate under a very different regime: they rely on massive pretrained generative priors with broad semantic knowledge and, in some cases, long per\-image finetuning\. MetaQuill\-KIP instead adapts locally to a*single*novel image in a few seconds, without any external dataset or text guidance, while achieving competitive PSNR in moderate\- and high\-observation masks \(e\.g\.,ϕ∈\{0\.5,0\.8\}\\phi\\in\\\{0\.5,0\.8\\\}in Tables[IV](https://arxiv.org/html/2609.03117#S4.T4)and[V](https://arxiv.org/html/2609.03117#S4.T5)\) and dramatically outperforming purely kernel\-based or tangent\-only NTK baselines\. These experiments are therefore not intended as a head\-to\-head SOTA inpainting challenge, but as evidence that NTK\-based neural fields can be upgraded into a fast, self\-contained, nonlinear, meta\-learned reconstruction pipeline\. Additional robustness sweeps where the observed pixels are corrupted \(Gaussian and salt and pepper\) are reported in Appendix[E\-A3](https://arxiv.org/html/2609.03117#A5.SS1.SSS3), and summarized in Table[VIII](https://arxiv.org/html/2609.03117#A5.T8)\.

##### Summary\.

Across both MNIST and Flowers, MetaQuill\-KIP delivers three concrete advantages\. First, it achieves large PSNR gains \(for example32\.4932\.49dB on MNIST,26\.2926\.29dB on Flowers\) at sub\-second to few\-second adaptation times that scale gently with resolution, because only a compactΔ​θ\\Delta\\thetais updated\. Second, it produces semantically plausible inpainting inside masked regions even under extreme sparsity, reconstructing fine\-grained texture and object boundaries that are not recovered by kernel\-only fits or shallow tangent updates\. See Figure[19](https://arxiv.org/html/2609.03117#S4.F19)\. Third, unlike heavy pretrained diffusion priors, MetaQuill\-KIP can adapt from scratch to a*single*novel scene or digit without relying on a large text\- or scene\-conditioned generative model\. It therefore fills a gap in the design space: fast, per\-instance, self\-contained neural field reconstruction with learned structure and true nonlinearity, extending what NTK\-style pipelines were thought to be capable of\.

## VRelated Work

The Neural Tangent Kernel \(NTK\) theory\.Introduced by\[[11](https://arxiv.org/html/2609.03117#bib.bib9)\], has provided a valuable framework for understanding the training dynamics of infinitely wide neural networks, showing that, in the infinite\-width limit, they behave like kernel methods and evolve under gradient descent according to a deterministic kernel function\. This has enabled a more analytical approach to understanding deep networks, bridging neural networks and classical kernel methods\. A key insight is that networks in this regime exhibit near\-linear behavior during training, aiding both theoretical analysis and empirical predictions\[[11](https://arxiv.org/html/2609.03117#bib.bib9),[26](https://arxiv.org/html/2609.03117#bib.bib20)\]\. Notably, the theoretical advantage of NTK allowed a wide range of applications such as, dataset distillation\[[12](https://arxiv.org/html/2609.03117#bib.bib12),[14](https://arxiv.org/html/2609.03117#bib.bib14),[27](https://arxiv.org/html/2609.03117#bib.bib18)\], federated learning analysis\[[28](https://arxiv.org/html/2609.03117#bib.bib17)\], incremental learning\[[15](https://arxiv.org/html/2609.03117#bib.bib16)\], regression\[[16](https://arxiv.org/html/2609.03117#bib.bib15)\], meta\-learning\[[17](https://arxiv.org/html/2609.03117#bib.bib45)\], and more\[[19](https://arxiv.org/html/2609.03117#bib.bib19)\]\.

Kernel perspectives in quantum machine learning\.Kernel viewpoints have also been studied in quantum machine learning, including quantum feature map kernel methods for supervised learning and analyses of parametrized quantum models through quantum neural tangent kernels\[[29](https://arxiv.org/html/2609.03117#bib.bib49),[30](https://arxiv.org/html/2609.03117#bib.bib50),[31](https://arxiv.org/html/2609.03117#bib.bib51),[32](https://arxiv.org/html/2609.03117#bib.bib52)\]\. We include this brief connection to better contextualize Eq\.[1](https://arxiv.org/html/2609.03117#S1.E1): while our setting is classical neural field generation, the broader motivation of studying kernel induced learning behavior is shared across classical and quantum learning literature\.

NTK Application on various architectures\.Subsequent research has expanded the NTK framework to cover a wide range of architectures and settings\[[33](https://arxiv.org/html/2609.03117#bib.bib21)\]\. e\.g\.\[[34](https://arxiv.org/html/2609.03117#bib.bib23),[35](https://arxiv.org/html/2609.03117#bib.bib22)\]extended NTK theory to deep and convolutional neural networks \(CNNs\), showing that NTK approximations hold in these more complex architectures\. Additionally,\[[36](https://arxiv.org/html/2609.03117#bib.bib24)\]developed a formalism for studying NTK in recurrent neural networks \(RNNs\) and transformer, further broadening the applicability of NTK theory\. These works highlight the versatility of NTK in understanding the dynamics of various neural network architectures\.

NTK Limitations\.While NTK has become a powerful tool for theoretical analysis\[[37](https://arxiv.org/html/2609.03117#bib.bib36),[38](https://arxiv.org/html/2609.03117#bib.bib37),[39](https://arxiv.org/html/2609.03117#bib.bib38),[40](https://arxiv.org/html/2609.03117#bib.bib39)\], several studies have explored its limitations\[[41](https://arxiv.org/html/2609.03117#bib.bib40),[42](https://arxiv.org/html/2609.03117#bib.bib25),[25](https://arxiv.org/html/2609.03117#bib.bib35)\], particularly in real\-world settings where networks are finite in size\. A key limitation of NTK is its assumption of an infinite\-width regime, which is rarely seen in practice\. NTK approximations fail to capture the full generalization of real networks and struggle with tasks requiring advanced feature learning\[[42](https://arxiv.org/html/2609.03117#bib.bib25)\]\. In finite\-width networks, feature learning plays a crucial role\[[43](https://arxiv.org/html/2609.03117#bib.bib30),[44](https://arxiv.org/html/2609.03117#bib.bib31)\], and NTK’s inability to capture this non\-linear phenomenon limits its applicability\. Moreover, NTK describes a ”lazy training” regime with minimal parameter changes, as shown by\[[45](https://arxiv.org/html/2609.03117#bib.bib26)\], which contradicts the non\-linear nature of real\-world NN training, where parameters and features evolve significantly\.

NTK representational struggle\.Another important criticism of NTK is its failure to capture neural networks’ ability to learn complex, hierarchical representations of data\[[44](https://arxiv.org/html/2609.03117#bib.bib31)\]\. In the NTK regime, the network essentially acts as a fixed kernel\[[11](https://arxiv.org/html/2609.03117#bib.bib9)\], limiting the model’s ability to dynamically learn representations during training\[[46](https://arxiv.org/html/2609.03117#bib.bib27)\]\. This contrasts with the empirical success of neural networks, which owe much of their generalization capabilities to representational learning\. Studies by\[[47](https://arxiv.org/html/2609.03117#bib.bib28)\]and\[[48](https://arxiv.org/html/2609.03117#bib.bib29)\]argue that real\-world neural networks continually refine their feature representations throughout training, a phenomenon that NTK theory does not fully explain\. Furthermore, empirical evidence suggests that feature learning plays a crucial role in many tasks, particularly those involving structured data like images and natural language\[[47](https://arxiv.org/html/2609.03117#bib.bib28),[48](https://arxiv.org/html/2609.03117#bib.bib29),[42](https://arxiv.org/html/2609.03117#bib.bib25)\]\.

Representation Learning\.Unlike NTK, representation learning focuses on how deep networks extract and refine features throughout training\[[49](https://arxiv.org/html/2609.03117#bib.bib42),[50](https://arxiv.org/html/2609.03117#bib.bib43)\]\. Representation learning centers on the idea that deep networks, particularly in their intermediate layers, build hierarchical representations that capture progressively abstract features of input data\[[43](https://arxiv.org/html/2609.03117#bib.bib30),[44](https://arxiv.org/html/2609.03117#bib.bib31),[51](https://arxiv.org/html/2609.03117#bib.bib41)\], enabling their success in image classification, object detection, and NLP\. Unlike NTK, representation learning captures how neural networks adapt their internal representations to complex data distributions\[[52](https://arxiv.org/html/2609.03117#bib.bib32)\]\. This is especially important in networks with finite widths, where feature learning is key to high performance on real\-world tasks\.

Transferable representations in modern adaptation and embodied systems\.The need for compact, transferable, and deployment\-aware representations also appears beyond neural fields\. Dataset distillation under domain shift seeks compact training sets that remain useful across target distributions\[[53](https://arxiv.org/html/2609.03117#bib.bib53)\], while deployment\-specific subset selection studies how data choice affects specialized performance\[[54](https://arxiv.org/html/2609.03117#bib.bib63)\]\. Test\-time fine\-tuning adapts language models to individual queries through selective retrieval and lightweight updates\[[55](https://arxiv.org/html/2609.03117#bib.bib54),[56](https://arxiv.org/html/2609.03117#bib.bib55)\]\. In computer vision and robotics, pretrained multimodal representations support open\-set detection and following\[[57](https://arxiv.org/html/2609.03117#bib.bib56)\], generalizable end\-to\-end driving\[[58](https://arxiv.org/html/2609.03117#bib.bib57),[59](https://arxiv.org/html/2609.03117#bib.bib58)\], factorized analysis of out\-of\-distribution driving robustness\[[60](https://arxiv.org/html/2609.03117#bib.bib59)\], open\-set 3D mapping\[[61](https://arxiv.org/html/2609.03117#bib.bib60)\], and language\-conditioned or decentralized aerial navigation\[[62](https://arxiv.org/html/2609.03117#bib.bib62),[63](https://arxiv.org/html/2609.03117#bib.bib61)\]\. Related studies also evaluate multimodal language models as driving world models\[[64](https://arxiv.org/html/2609.03117#bib.bib64)\]and combine pretrained video\-language and language models for training\-free video summarization\[[65](https://arxiv.org/html/2609.03117#bib.bib65)\]\. Although these systems operate in different application domains, they share our interest in representations that transfer, adapt efficiently, and remain useful under distribution shift\.

## VIConclusions

This work begins from a limitation of standard NTK pipelines for neural fields: they adapt only via linearized weight updates around the current parameters, and they do not accumulate reusable structure that transfers across tasks\. We make this concrete by analyzing three methods, all introduced in this work\.*NTK\-KIP*learns, for each target scene or image, a compact “support set” of spatial coordinates \(and optionally pixel values\) so that finite\-width NTK kernel regression can reconstruct or inpaint the full signal\. This overcomes a core NTK weakness: fixed, non\-adaptive sampling, and enables strong inpainting with extremely sparse observations\. However, NTK\-KIP must be solved from scratch for every new instance, which limits responsiveness at test time and prevents true meta\-learning\.*MetaQuill*addresses that second point directly: it meta\-learns a shared initializationθS\\theta\_\{S\}across many tasks, and then adapts a new task by optimizing only a tiny task\-specificΔ​θ\\Delta\\theta, rather than retraining the whole network\. This turns a neural field into a fast, reusable prior with learned transferable features: we get sub\-second adaptation and a lightweight representation of task identity\. But because MetaQuill’s adaptation is effectively tangent\-space \(NTK\-like\) aroundθS\\theta\_\{S\}, it remains locally linear\. In heavily occluded or masked regions, that linearity caps expressiveness and hurts hole filling\.*MetaQuill\-KIP*is our answer to both limitations at once\. It first uses a KIP\-style non\-linear warm start that captures high\-level structure and is not restricted to a purely linear NTK response, and then continues with MetaQuill’s efficientΔ​θ\\Delta\\thetarefinement around the learnedθS\\theta\_\{S\}\. This provides a unified mechanism for non\-linearity, feature learning, and fast adaptation: non\-linear expressivity via the KIP warm start, learned transferable features via the shared initializationθS\\theta\_\{S\}, and fast per\-task refinement by updating onlyΔ​θ\\Delta\\thetarather than the full network\. In practice, this yields high\-PSNR reconstructions and semantically plausible inpainting from extreme sparsity \(down to∼\\sim1% observed pixels\) in only tens to hundreds of lightweight refinement steps, just a few seconds of wall\-clock, while strong diffusion baselines such as Stable Diffusion with per\-image finetuning or StrDiffusion require tens of seconds of sampling/adaptation to reach comparable fidelity\. In summary, NTK\-KIP solves NTK’s lack of non\-linear expressivity but is instance\-specific and slow; MetaQuill solves fast adaptation and reusability but is locally linear; MetaQuill\-KIP merges both\. The result is an NTK\-driven neural field pipeline that is \(a\) non\-linear where it matters, \(b\) meta\-learned and reusable across tasks, and \(c\) fast enough for practical few\-shot reconstruction and inpainting\.

Future work\.We view MetaQuill\-KIP as a first step toward a unified NTK framework that is both non\-linear and explicitly meta\-learned\. NTK\-KIP supplies distilled, task\-specific non\-linear support sets; MetaQuill supplies a fast, reusable meta\-initialization\. Their combination suggests an NTK pipeline with learnable features, non\-linear expressivity, and practical wall\-clock efficiency\. We hope this direction motivates further work on NTK methods that behave less like fixed kernels and more like adaptable representation learners\.

## References

- \[1\]K\. O\. Stanley\(2007\)Compositional pattern producing networks: a novel abstraction of development\.Genetic programming and evolvable machines8,pp\. 131–162\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p1.1)\.
- \[2\]D\. Ha\(2016\)Generating large images from latent vectors\.Note:Otoro\.netAvailable: https://blog\.otoro\.net/2016/04/01/generating\-large\-images\-from\-latent\-vectors/Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p1.1)\.
- \[3\]L\. Mescheder, M\. Oechsle, M\. Niemeyer, S\. Nowozin, and A\. Geiger\(2019\)Occupancy networks: learning 3d reconstruction in function space\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 4460–4470\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p1.1)\.
- \[4\]Z\. Chen and H\. Zhang\(2019\)Learning implicit fields for generative shape modeling\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 5939–5948\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p1.1)\.
- \[5\]B\. Mildenhall, P\. P\. Srinivasan, M\. Tancik, J\. T\. Barron, R\. Ramamoorthi, and R\. Ng\(2021\)Nerf: representing scenes as neural radiance fields for view synthesis\.Communications of the ACM65\(1\),pp\. 99–106\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p1.1)\.
- \[6\]V\. Sitzmann, J\. Martel, A\. Bergman, D\. Lindell, and G\. Wetzstein\(2020\)Implicit neural representations with periodic activation functions\.Advances in neural information processing systems33,pp\. 7462–7473\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p1.1)\.
- \[7\]E\. Dupont, Y\. W\. Teh, and A\. Doucet\(2022\)Generative models as distributions of functions\.InProceedings of the 25th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.151,pp\. 2989–3015\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p1.1)\.
- \[8\]M\. Tancik, P\. Srinivasan, B\. Mildenhall, S\. Fridovich\-Keil, N\. Raghavan, U\. Singhal, R\. Ramamoorthi, J\. Barron, and R\. Ng\(2020\)Fourier features let networks learn high frequency functions in low dimensional domains\.Advances in neural information processing systems33,pp\. 7537–7547\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p1.1)\.
- \[9\]E\. Dupont, H\. Kim, S\. M\. A\. Eslami, D\. J\. Rezende, and D\. Rosenbaum\(2022\)From data to functa: your data point is a function and you can treat it like one\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 5694–5725\.Cited by:[Appendix D](https://arxiv.org/html/2609.03117#A4.p3.1),[§I](https://arxiv.org/html/2609.03117#S1.p2.1),[§II\-C](https://arxiv.org/html/2609.03117#S2.SS3.p1.1),[§II](https://arxiv.org/html/2609.03117#S2.p2.1),[§III\-B](https://arxiv.org/html/2609.03117#S3.SS2.p1.1)\.
- \[10\]M\. Bauer, E\. Dupont, A\. Brock, D\. Rosenbaum, J\. R\. Schwarz, and H\. Kim\(2023\)Spatial functa: scaling functa to imagenet classification and generation\.arXiv preprint arXiv:2302\.03130\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p2.1)\.
- \[11\]A\. Jacot, F\. Gabriel, and C\. Hongler\(2018\)Neural tangent kernel: convergence and generalization in neural networks\.Advances in neural information processing systems31\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1),[§I](https://arxiv.org/html/2609.03117#S1.p6.1),[§V](https://arxiv.org/html/2609.03117#S5.p1.1),[§V](https://arxiv.org/html/2609.03117#S5.p5.1)\.
- \[12\]T\. Nguyen, R\. Novak, L\. Xiao, and J\. Lee\(2021\)Dataset distillation with infinitely wide convolutional networks\.Advances in Neural Information Processing Systems34,pp\. 5186–5198\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1),[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[13\]N\. Loo, R\. Hasani, A\. Amini, and D\. Rus\(2022\)Efficient dataset distillation using random feature approximation\.Advances in Neural Information Processing Systems35,pp\. 13877–13891\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1),[§III\-A](https://arxiv.org/html/2609.03117#S3.SS1.p3.1)\.
- \[14\]A\. Maalouf, M\. Tukan, N\. Loo, R\. Hasani, M\. Lechner, and D\. Rus\(2023\)On the size and approximation error of distilled datasets\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 61085–61102\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/c03a9ccdb3e95f2c2dcfc3f4bc16bf42-Paper-Conference.pdf)Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1),[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[15\]J\. Liu, Z\. Ji, Y\. Pang, and Y\. Yu\(2024\)NTK\-guided few\-shot class incremental learning\.IEEE Transactions on Image Processing33,pp\. 6029–6044\.External Links:[Document](https://dx.doi.org/10.1109/TIP.2024.3478854)Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1),[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[16\]S\. Qadeer, A\. Engel, A\. Howard, A\. Tsou, M\. Vargas, P\. Stinis, and T\. Chiang\(2023\)Efficient kernel surrogates for neural network\-based regression\.arXiv preprint arXiv:2310\.18612\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1),[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[17\]Y\. Zhou, Z\. Wang, J\. Xian, C\. Chen, and J\. Xu\(2021\)Meta\-learning with neural tangent kernels\.InInternational Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1),[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[18\]G\. Yang and E\. J\. Hu\(2021\)Tensor programs iv: feature learning in infinite\-width neural networks\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 11727–11737\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1)\.
- \[19\]Z\. Zhang, C\. Hensley, and Z\. Chen\(2024\)Improving node classification with neural tangent kernel: a graph neural network approach\.InProceedings of the International Conference on Machine Learning, Pattern Recognition and Automation Engineering,pp\. 93–97\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p3.1),[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[20\]A\. Radhakrishnan, G\. Stefanakis, M\. Belkin, and C\. Uhler\(2022\)Simple, fast, and flexible framework for matrix completion with infinite width neural networks\.Proceedings of the National Academy of Sciences119\(16\),pp\. e2115064119\.Cited by:[§I](https://arxiv.org/html/2609.03117#S1.p11.1)\.
- \[21\]H\. Liu, Y\. Wang, B\. Qian, M\. Wang, and Y\. Rui\(2024\)Structure matters: tackling the semantic discrepancy in diffusion models for image inpainting\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 8038–8047\.Cited by:[§I\-A](https://arxiv.org/html/2609.03117#S1.SS1.p1.3)\.
- \[22\]S\. Zhu, P\. Fang, C\. Zhu, Z\. Zhao, Q\. Xu, and H\. Xue\(2024\)Text image inpainting via global structure\-guided diffusion models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 7775–7783\.Cited by:[§I\-A](https://arxiv.org/html/2609.03117#S1.SS1.p1.3)\.
- \[23\]T\. Nguyen, Z\. Chen, and J\. Lee\(2021\)Dataset meta\-learning from kernel ridge\-regression\.InInternational Conference on Learning Representations,Cited by:[§E\-A1](https://arxiv.org/html/2609.03117#A5.SS1.SSS1.p6.1),[§III\-A](https://arxiv.org/html/2609.03117#S3.SS1.p3.1)\.
- \[24\]C\. Finn, P\. Abbeel, and S\. Levine\(2017\)Model\-agnostic meta\-learning for fast adaptation of deep networks\.InInternational conference on machine learning,pp\. 1126–1135\.Cited by:[§E\-A2](https://arxiv.org/html/2609.03117#A5.SS1.SSS2.Px2.p8.1),[§III\-B](https://arxiv.org/html/2609.03117#S3.SS2.p2.1),[§IV\-B](https://arxiv.org/html/2609.03117#S4.SS2.p5.1)\.
- \[25\]R\. Novak, J\. Sohl\-Dickstein, and S\. S\. Schoenholz\(2022\)Fast finite width neural tangent kernel\.InInternational Conference on Machine Learning,pp\. 17018–17044\.Cited by:[§III\-B](https://arxiv.org/html/2609.03117#S3.SS2.p2.1),[§V](https://arxiv.org/html/2609.03117#S5.p4.1)\.
- \[26\]J\. Lee, L\. Xiao, S\. Schoenholz, Y\. Bahri, R\. Novak, J\. Sohl\-Dickstein, and J\. Pennington\(2019\)Wide neural networks of any depth evolve as linear models under gradient descent\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 8570–8581\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[27\]M\. Tukan, A\. Maalouf, and M\. Osadchy\(2023\)Dataset distillation meets provable subset selection\.arXiv preprint arXiv:2307\.08086\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[28\]B\. Huang, X\. Li, Z\. Song, and X\. Yang\(2021\)Fl\-ntk: a neural tangent kernel\-based framework for federated learning analysis\.InInternational Conference on Machine Learning,pp\. 4423–4434\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p1.1)\.
- \[29\]V\. Havlíček, A\. D\. Córcoles, K\. Temme, A\. W\. Harrow, A\. Kandala, J\. M\. Chow, and J\. M\. Gambetta\(2019\)Supervised learning with quantum\-enhanced feature spaces\.Nature567\(7747\),pp\. 209–212\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p2.1)\.
- \[30\]M\. Schuld and N\. Killoran\(2019\)Quantum machine learning in feature hilbert spaces\.Physical review letters122\(4\),pp\. 040504\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p2.1)\.
- \[31\]J\. Liu, F\. Tacchino, J\. R\. Glick, L\. Jiang, and A\. Mezzacapo\(2022\)Representation learning via quantum neural tangent kernels\.PRX Quantum3\(3\),pp\. 030323\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p2.1)\.
- \[32\]M\. Incudini, M\. Grossi, A\. Mandarino, S\. Vallecorsa, A\. Di Pierro, and D\. Windridge\(2023\)The quantum path kernel: a generalized neural tangent kernel for deep quantum machine learning\.IEEE Transactions on Quantum Engineering4,pp\. 1–16\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p2.1)\.
- \[33\]J\. Mok, B\. Na, J\. Kim, D\. Han, and S\. Yoon\(2022\)Demystifying the neural tangent kernel from a practical perspective: can it be trusted for neural architecture search without training?\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11861–11870\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p3.1)\.
- \[34\]S\. Arora, S\. S\. Du, W\. Hu, Z\. Li, R\. R\. Salakhutdinov, and R\. Wang\(2019\)On exact computation with an infinitely wide neural net\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 8139–8148\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p3.1)\.
- \[35\]Z\. Li, R\. Wang, D\. Yu, S\. S\. Du, W\. Hu, R\. Salakhutdinov, and S\. Arora\(2019\)Enhanced convolutional neural tangent kernels\.arXiv preprint arXiv:1911\.00809\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p3.1)\.
- \[36\]G\. Yang\(2020\)Tensor programs ii: neural tangent kernel for any architecture\.arXiv preprint arXiv:2006\.14548\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p3.1)\.
- \[37\]J\. Lee, S\. Schoenholz, J\. Pennington, B\. Adlam, L\. Xiao, R\. Novak, and J\. Sohl\-Dickstein\(2020\)Finite versus infinite neural networks: an empirical study\.Advances in Neural Information Processing Systems33,pp\. 15156–15172\.Cited by:[§E\-A2](https://arxiv.org/html/2609.03117#A5.SS1.SSS2.Px1.p2.1),[§V](https://arxiv.org/html/2609.03117#S5.p4.1)\.
- \[38\]A\. Zandieh, I\. Han, H\. Avron, N\. Shoham, C\. Kim, and J\. Shin\(2021\)Scaling neural tangent kernels via sketching and random features\.Advances in Neural Information Processing Systems34,pp\. 1062–1073\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p4.1)\.
- \[39\]M\. Seleznova, D\. Weitzner, R\. Giryes, G\. Kutyniok, and H\. Chou\(2023\)Neural \(tangent kernel\) collapse\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 16240–16270\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p4.1)\.
- \[40\]E\. Nichani, Y\. Bai, and J\. D\. Lee\(2022\)Identifying good directions to escape the ntk regime and efficiently learn low\-degree plus sparse polynomials\.Advances in Neural Information Processing Systems35,pp\. 14568–14581\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p4.1)\.
- \[41\]M\. Seleznova and G\. Kutyniok\(2022\)Neural tangent kernel beyond the infinite\-width limit: effects of depth and initialization\.InInternational Conference on Machine Learning,pp\. 19522–19560\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p4.1)\.
- \[42\]N\. Vyas, Y\. Bansal, and P\. Nakkiran\(2023\)Limitations of the ntk for understanding generalization in deep learning\.InInternational Conference on Learning Representations,Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p4.1),[§V](https://arxiv.org/html/2609.03117#S5.p5.1)\.
- \[43\]P\. Wang, X\. Li, C\. Yaras, Z\. Zhu, L\. Balzano, W\. Hu, and Q\. Qu\(2025\)Understanding deep representation learning via layerwise feature compression and discrimination\.Journal of Machine Learning Research26\(220\),pp\. 1–61\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p4.1),[§V](https://arxiv.org/html/2609.03117#S5.p6.1)\.
- \[44\]M\. Chen, Y\. Bai, J\. D\. Lee, T\. Zhao, H\. Wang, C\. Xiong, and R\. Socher\(2020\)Towards understanding hierarchical learning: benefits of neural representations\.Advances in Neural Information Processing Systems33,pp\. 22134–22145\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p4.1),[§V](https://arxiv.org/html/2609.03117#S5.p5.1),[§V](https://arxiv.org/html/2609.03117#S5.p6.1)\.
- \[45\]L\. Chizat, E\. Oyallon, and F\. Bach\(2019\)On lazy training in differentiable programming\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 2933–2943\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p4.1)\.
- \[46\]M\. Geiger, S\. Spigler, A\. Jacot, and M\. Wyart\(2020\)Disentangling feature and lazy training in deep neural networks\.Journal of Statistical Mechanics: Theory and Experiment2020\(11\),pp\. 113301\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p5.1)\.
- \[47\]Y\. Sharon and Y\. Dar\(2024\)How does perfect fitting affect representation learning? on the training dynamics of representations in deep neural networks\.arXiv preprint arXiv:2405\.17377\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p5.1)\.
- \[48\]N\. Vyas, A\. Atanasov, B\. Bordelon, D\. Morwani, S\. Sainathan, and C\. Pehlevan\(2023\)Feature\-learning networks are consistent across widths at realistic scales\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 1036–1060\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p5.1)\.
- \[49\]P\. Vincent, H\. Larochelle, I\. Lajoie, Y\. Bengio, P\. Manzagol, and L\. Bottou\(2010\)Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion\.\.Journal of machine learning research11\(12\)\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p6.1)\.
- \[50\]X\. Chen, Y\. Duan, R\. Houthooft, J\. Schulman, I\. Sutskever, and P\. Abbeel\(2016\)InfoGAN: interpretable representation learning by information maximizing generative adversarial nets\.InAdvances in Neural Information Processing Systems,Vol\.29,pp\. 2172–2180\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p6.1)\.
- \[51\]A\. Coates and A\. Y\. Ng\(2012\)Learning feature representations with k\-means\.InNeural Networks: Tricks of the Trade: Second Edition,pp\. 561–580\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p6.1)\.
- \[52\]K\. Kawaguchi, L\. Zhang, and Z\. Deng\(2022\)Understanding dynamics of nonlinear representation learning and its application\.Neural computation34\(4\),pp\. 991–1018\.Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p6.1)\.
- \[53\]N\. Loo, A\. Maalouf, R\. Hasani, M\. Lechner, A\. Amini, and D\. Rus\(2024\)Large scale dataset distillation with domain shift\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 32759–32780\.External Links:[Link](https://proceedings.mlr.press/v235/loo24a.html)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[54\]N\. Hulkund, A\. Maalouf, L\. Cai, D\. Yang, T\. Wang, A\. O’Neil, T\. Haucke, S\. Mukherjee, V\. Ramaswamy, J\. H\. Shen, G\. Tseng, M\. Walmsley, D\. Rus, K\. Goldberg, H\. Kerner, I\. Chen, Y\. Girdhar, and S\. Beery\(2025\)DataS3: dataset subset selection for specialization\.External Links:2504\.16277,[Link](https://arxiv.org/abs/2504.16277)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[55\]A\. Khamis and A\. Maalouf\(2026\)Efficient test\-time finetuning of LLMs via convex reconstruction and gradient caching\.External Links:2605\.30337,[Link](https://arxiv.org/abs/2605.30337)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[56\]J\. Hübotter, S\. Bongni, I\. Hakimi, and A\. Krause\(2025\)Efficiently learning at test\-time: active fine\-tuning of LLMs\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NS1G1Uhny3)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[57\]A\. Maalouf, N\. Jadhav, K\. M\. Jatavallabhula, M\. Chahine, D\. M\. Vogt, R\. J\. Wood, A\. Torralba, and D\. Rus\(2024\)Follow anything: open\-set detection, tracking, and following in real\-time\.IEEE Robotics and Automation Letters9\(4\),pp\. 3283–3290\.External Links:[Document](https://dx.doi.org/10.1109/LRA.2024.3366013)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[58\]T\. Wang, A\. Maalouf, W\. Xiao, Y\. Ban, A\. Amini, G\. Rosman, S\. Karaman, and D\. Rus\(2024\)Drive anywhere: generalizable end\-to\-end autonomous driving with multi\-modal foundation models\.In2024 IEEE International Conference on Robotics and Automation,pp\. 6687–6694\.External Links:[Document](https://dx.doi.org/10.1109/ICRA57147.2024.10611590)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[59\]A\. Mallak, E\. Aasi, S\. Sreeram, T\. Wang, D\. Rus, and A\. Maalouf\(2026\)See less, drive better: generalizable end\-to\-end autonomous driving via foundation models stochastic patch selection\.External Links:2601\.10707,[Link](https://arxiv.org/abs/2601.10707)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[60\]A\. Mallak and A\. Maalouf\(2026\)Robustness is a function, not a number: a factorized comprehensive study of OOD robustness in vision\-based driving\.External Links:2602\.09018,[Link](https://arxiv.org/abs/2602.09018)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[61\]K\. M\. Jatavallabhula, A\. Kuwajerwala, Q\. Gu, M\. Omama, G\. Iyer, S\. Saryazdi, T\. Chen, A\. Maalouf, S\. Li, N\. Keetha, A\. Tewari, J\. B\. Tenenbaum, C\. M\. de Melo, M\. Krishna, L\. Paull, F\. Shkurti, and A\. Torralba\(2023\)ConceptFusion: open\-set multimodal 3d mapping\.InProceedings of Robotics: Science and Systems XIX,Daegu, Republic of Korea\.External Links:[Document](https://dx.doi.org/10.15607/RSS.2023.XIX.066)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[62\]M\. Chahine, A\. Quach, A\. Maalouf, T\. Wang, and D\. Rus\(2026\)FLeX: end\-to\-end text\-instructed visual navigation from foundation model features\.IEEE Robotics and Automation Letters11\(5\),pp\. 6480–6487\.External Links:[Document](https://dx.doi.org/10.1109/LRA.2026.3678454)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[63\]M\. Chahine, W\. Yang, A\. Maalouf, J\. Siriska, N\. Jadhav, D\. Vogt, S\. Gil, R\. Wood, and D\. Rus\(2025\)Decentralized vision\-based autonomous aerial wildlife monitoring\.External Links:2508\.15038,[Link](https://arxiv.org/abs/2508.15038)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[64\]S\. Sreeram, T\. Wang, A\. Maalouf, G\. Rosman, S\. Karaman, and D\. Rus\(2025\)Probing multimodal LLMs as world models for driving\.IEEE Robotics and Automation Letters10\(11\),pp\. 11403–11410\.External Links:[Document](https://dx.doi.org/10.1109/LRA.2025.3608656)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.
- \[65\]M\. Barbara and A\. Maalouf\(2025\)Prompts to summaries: zero\-shot language\-guided video summarization with large language and video models\.External Links:2506\.10807,[Link](https://arxiv.org/abs/2506.10807)Cited by:[§V](https://arxiv.org/html/2609.03117#S5.p7.1)\.

![[Uncaptioned image]](https://arxiv.org/html/2609.03117v1/Photos/Amir_Mallak_Photo.jpg)Amir Mallakis a Ph\.D\. researcher at the University of Haifa\. He holds a B\.Sc\. in Electrical and Computer Engineering from Ben Gurion University, and an M\.Sc\. in Computer Science from the University of Haifa\. His research focuses on generalizable, efficient, and reliable AI systems, spanning machine learning, computer vision and spatial intelligence, robotics, multimodal foundation models, algorithms, and machine learning theory\.![[Uncaptioned image]](https://arxiv.org/html/2609.03117v1/Photos/Alaa_Maalouf_Photo.png)Alaa Maaloufis the Neubauer Assistant Professor of Computer Science at the University of Haifa\. He is also a Research Affiliate at MIT CSAIL, an Associate at Harvard SEAS, and a Principal Investigator with Project CETI\. Before that, he was a postdoctoral researcher at MIT CSAIL and received his Ph\.D\. from the University of Haifa\. He received the Maof Scholarship for Excellent Young Researchers from the Council for Higher Education, a Neubauer Family Foundation Fellowship for excellent young faculty, and the 2019 NeurIPS Outstanding Paper Award Honorable Mention\. Maalouf’s research lies at the intersection of machine learning, computer vision, robotics, and large\-scale data\.![[Uncaptioned image]](https://arxiv.org/html/2609.03117v1/Photos/Lior_Wolf_Photo.png)Lior Wolfis the CEO at Mentee Robotics and a full professor in the School of Computer Science at Tel\-Aviv University, Israel\. He conducted postdoctoral research at prof\. Poggio’s lab at the Massachusetts Institute of Technology and received his PhD degree from the Hebrew University, under the supervision of Prof\. Shashua\. He is an ERC grantee and has won the ICCV 2001 and ICCV 2019 honorable mention, and the best paper awards at ECCV 2000 and ICANN 2016\. His research focuses on computer vision and deep learning\.![[Uncaptioned image]](https://arxiv.org/html/2609.03117v1/Photos/Daniela_Rus_Photo.jpg)Daniela Rusis the Andrew \(1956\) and Erna Viterbi Professor of Electrical Engineering and Computer Science and Director of the Computer Science and Artificial Intelligence Laboratory \(CSAIL\) at MIT\. Rus’s research interests are in robotics, mobile computing, and data science\. Rus is a Class of 2002 MacArthur Fellow, a fellow of ACM, AAAI and IEEE, and a member of the National Academy of Engineering, and the American Academy for Arts and Science\. She earned her PhD in Computer Science from Cornell University\.![[Uncaptioned image]](https://arxiv.org/html/2609.03117v1/Photos/Dan_Rosenbaum_Photo.jpg)Dan Rosenbaumis a Senior Lecturer in the Department of Computer Science at the University of Haifa\. He is working on machine learning and computer vision, and specifically on 3D scene understanding and generative approaches that model vision as an inverse problem\. Before joining the University of Haifa Dan completed his PhD in 2016 at the Hebrew University of Jerusalem, advised by Prof\. Yair Weiss, and then worked as a Research Scientist at DeepMind in London between 2016 and 2021\.## Appendix AList of Acronyms and Abbreviations

For readability, Table[VII](https://arxiv.org/html/2609.03117#A1.T7)consolidates the acronyms and abbreviations used throughout the main manuscript and appendix\.

AcronymMeaningAcronymMeaningNTKNeural Tangent KernelNFNeural FieldINRImplicit Neural RepresentationNNNeural NetworkMCMatrix completionMLPMulti Layer PerceptronKRRKernel Ridge RegressionKIPKernel Inducing PointsNTK\-KIPOur NTK based KIP method for neural fieldsMetaQuillMeta learned initialization and task adaptive tangent space refinement methodMetaQuill\-KIPCombined method that uses KIP style distilled support and MetaQuill style adaptationGTGround TruthPEPositional EncodingRPERaw Positional EncodingFPEFourier Positional EncodingGDGradient DescentMSEMean Squared ErrorPSNRPeak Signal to Noise RatioRGBRed Green BlueDMDiffusion ModelDM\_WDM trained on NTK coefficient representations \(W\) rather than directly on GT imagesDDPMDenoising Diffusion Probabilistic ModelGSDMGlobal Structure\-guided Diffusion ModelStrDiffusionStructure\-guided Diffusion ModelSDStable DiffusionMNISTModified National Institute of Standards and Technology datasetCIFAR\-10Canadian Institute For Advanced Research\-10 class datasetCelebALarge\-scale Celebrity faces Attributes datasetQMLQuantum Machine LearningSOTAState of the ArtTABLE VII:Consolidated list of acronyms and abbreviations used in the main manuscript and appendix\. We include common method names, kernel terms, optimization terms, and datasets to improve readability\.
## Appendix BDetermining the frequency bands k value

In our experiments, we use FPE with frequency bands ofk=20\\mathit\{k\}=20for the NF architecture’s input positional encoding\. This value was determined from an experiment to identify the optimal FPE band\. An illustration of the effect ofk\\mathit\{k\}is shown in Fig\.[23](https://arxiv.org/html/2609.03117#A2.F23)\.

Atk=20\\mathit\{k\}=20, thew\\mathit\{w\}vector achieves the highest accuracy with sufficient detail \(P​S​N​R≈105\.35PSNR\\approx 105\.35\), closely matching the target image’s representation through a near\-linear transformation\. To further illustrate the optimal Fourier frequencykb​a​n​d\\mathit\{k\}\_\{band\}selection in terms ofP​S​N​RPSNRaccuracy and feature efficiency, we present a graph of Fourierkb​a​n​d​s\\mathit\{k\}\_\{bands\}versusP​S​N​RPSNR\(for the reconstructed image\), with a cubic interpolation to reveal the continuous trend\.

In FPE, each positional encoding \(PE\) 2D coordinate is transformed intod\+2⋅d⋅kd\+2\\cdot d\\cdot kFourier features, whereddis the PE dimensionality, andkkis the number of Fourier bands \(e\.g\.,d=2d=2,k=20k=20\)\. Thus, the number ofkb​a​n​d​s\\mathit\{k\}\_\{bands\}impacts the Fourier feature space by approximately∼4×kb​a​n​d​s\\sim 4\\times\\mathit\{k\}\_\{bands\}\. For instance, increasingkb​a​n​d​s\\mathit\{k\}\_\{bands\}by 10 adds 40 positional input features\. The optimalkb​a​n​d​s\\mathit\{k\}\_\{bands\}yields an elbow point in the graph with minimal bands\. We calculate this elbow value across Fourier frequencies and include it in the graph, as shown in Fig\.[22](https://arxiv.org/html/2609.03117#A2.F22)\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/kernel_representational_nature/Ablation/Optimal_FF_Kband_Elbow.png)Fig\. 22:Optimal Fourier FrequencyKb​a​n​d\\mathit\{K\}\_\{band\}with Elbow ValueWe can notice that the maximumP​S​N​R=102\.63PSNR=102\.63is atkb​a​n​d=50\\mathit\{k\}\_\{band\}=50\. The Elbow and the optimal chosen Fourier Frequencykb​a​n​d\\mathit\{k\}\_\{band\}iskb​a​n​d=15\\mathit\{k\}\_\{band\}=15\. In this paper though, we ended up choosingkb​a​n​d=20\\mathit\{k\}\_\{band\}=20for the experiments\. It yields slightly higher𝑃𝑆𝑁𝑅\\mathit\{PSNR\}and has a bit more feature representation\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/kernel_representational_nature/Ablation/FF_K_bands_vs_PSNR.png)Fig\. 23:Optimal FPE Band according to theW\\mathit\{W\}vector
## Appendix CNTK Kernel Illustration

To better understand the NTK kernel representation, we delve into its core and present theoretical findings and explanations\. From equation[1](https://arxiv.org/html/2609.03117#S1.E1), if to consider a specific cell\[\(i,j\)\|1≤i,j≤C\]\[\(i,j\)\\,\|\\,1\\leq i,j\\leq C\]in the NTK,

𝒦\(i,j\)\(x,x′;θ\)\\displaystyle\\mathcal\{K\}\_\{\(\\mathit\{i\},\\mathit\{j\}\)\}\(\\mathit\{x\},\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\,\\theta\)=∇fiθ\(x;θ\)T⋅∇fjθ\(x′;θ\)=\\displaystyle=\\nabla\{\{\}\_\{\\theta\}\}\\mathit\{f\}\_\{\\mathit\{i\}\}\(\\mathit\{x\}\\,;\\,\\theta\)^\{T\}\\ \\cdot\\ \\nabla\{\{\}\_\{\\theta\}\}\\mathit\{f\}\_\{\\mathit\{j\}\}\(\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\,\\theta\)==∑q=1Q∂fi​\(x,θ\)T∂θq⋅∂fj\(x′;θ\)∂θq,\\displaystyle=\\sum\_\{\\mathit\{q\}=\\mathit\{1\}\}^\{\\mathit\{Q\}\}\\frac\{\\partial\\mathit\{f\}\_\{\\mathit\{i\}\}\(\\mathit\{x\}\\,;\\,\\theta\)^\{\\mathit\{T\}\}\}\{\\partial\\theta\_\{\\mathit\{q\}\}\}\\cdot\\frac\{\\partial\\mathit\{f\}\_\{\\mathit\{j\}\}\(\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\,\\theta\)\}\{\\partial\\theta\_\{\\mathit\{q\}\}\},
whereQ\\mathit\{Q\}is the total number of parameters in the NNf\\mathit\{f\}\. See Fig\.[24](https://arxiv.org/html/2609.03117#A3.F24)for illustration\.

Calculating correlations through inner products of neural network gradients compresses meaningful information, neglecting dynamic architectural effects and resulting in a low\-dimensional projection rather than a robust representation\. While NTK offers advantages in training flexibility, it lacks effective feature extraction and true learning capability\. To explore these limitations, we propose the concept of ”NTK Representational Linearity”, which addresses the fundamental constraints in NTK’s approach\. This analysis contributes to understanding and mitigating limitations in the NTK theorem, highlighting its impact on representation learning\.

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/kernel_representational_nature/Intro/NTK_Matrix_Illustrarion.png)Fig\. 24:Neural Tangent Kernel Matrix Illustration
## Appendix DFuncta via NTK

It is common practice in deep learning to represent a measurement of the world on a discrete grid, e\.g\. a 2D grid of pixels\. However, the underlying signal represented by these measurements is often continuous, e\.g\. the scene depicted in an image, or the transformed Fourier of an audio signal, etc\.

A powerful frequency\-continuous alternative is then to represent these measurements using an NF, a neural function trained to output the appropriate measurement value for any input spatial locationN​F:l→vNF:l\\rightarrow v\(a location to value transformation\)\.

In Functa paper\[[9](https://arxiv.org/html/2609.03117#bib.bib10)\], the authors take this idea to a higher approach \(Regarding Meta\-Learning\): what would it take to perform deep learning on these functions \(NF\) instead, treating them as data? In this context it’s referred to the data as Functa\.

In this experiment, in order to test NTK’s representation and efficiency, we decided to refer to the Functa paper as a comparison\. In this paper, the authors transform each data point in the dataset to a function \(NF Model\) and manage to distillate this function representation \(hence the dataset\) up to∼0\.5%\\sim 0\.5\\%from the original dataset size with great accuracy\. Meaning, they managed to distillate the data by more than two orders of magnitude while maintaining the interesting features to be able to reconstruct the distilled NF\.

This scale of feature learning has radical importance in representation learning, and one of the methods to be able to determine the quality and ability of the conducted method to adapt and actual include and learn new features\.

In order to test this, we conducted an NTK for representation learning via Functa experiment\. Our downstream chosen task is MC, and more specifically, Inpainting\.

For illustration of Functa, see Fig\.[7](https://arxiv.org/html/2609.03117#S2.F7)\.

In NTK Functa, the process in slightly different\. Instead of training our generative model \(Diffusion Model \- DM\) on the NFs new dataset\. We train the generative model \(DM\) on an infinite\-width NF\. And the way to make this feasible is via NTK kernel trick, Fig\.[7](https://arxiv.org/html/2609.03117#S2.F7)\. Thus, calculating𝒦t​r​a​i​n\\mathcal\{K\}\_\{train\}and𝒦t​e​s​t\\mathcal\{K\}\_\{test\}NTK Kernels for all the dataset \(shared for all the NFs\), and calculating and training the DM on theWWvectors \(derived from the kernel and labels, Eq\.[4](https://arxiv.org/html/2609.03117#S1.E4)\)\.

As stated above, after calculating the NTK kernels, we’ll calculate the𝒲\\mathcal\{W\}vectors \(as explained earlier via𝒦t​r​a​i​n\\mathcal\{K\}\_\{train\}, Eq\.[4](https://arxiv.org/html/2609.03117#S1.E4)\) as such:𝒲=𝒦t\(x,x′;θ\)−1𝒴\|𝒦:ℝd×ℝd→ℝC×C\\mathcal\{W\}=\\mathcal\{K\}\_\{\\mathit\{t\}\}\(\\mathit\{x\},\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\theta\)^\{\\mathit\{\-1\}\}\\mathcal\{Y\}\\ \|\\ \\mathcal\{K\}:\\mathbb\{R\}^\{d\}\\ \\times\\ \\mathbb\{R\}^\{d\}\\ \\rightarrow\{\}\\ \\mathbb\{R\}^\{\\mathit\{C\}\\times\\mathit\{C\}\}\(such that we’re taking the full PE vector \(This means, all of its elements representing all of the matrix coordinates\), And the𝒴\\mathcal\{Y\}vector as whole\. Namely, containing all the labels \(observed and non\-observed ones\)\)\. And replacing the learned NFs with these𝒲\\mathcal\{W\}vectors\.

Note, Though this is an Inpainting downstream task, we’re taking the whole image pixels \(and coordinates\) as observed at this stage \(creating the𝒲\\mathcal\{W\}vectors\), due to the nature of Functa application, where the INR model needs first to learn the distribution of the dataset before being tested and generate novel data points\.

After completing this, as the case with classic Functa, we’d actually achieve a whole new representation of the dataset\. Replacing each dataset sample with a𝒲\\mathcal\{W\}vector\. Turning the dataset of samples into a dataset of𝒲​s\\mathcal\{W\}s\.

After calculating the𝒲​s\\mathcal\{W\}srepresentations, we’ll train a generative\-diffusion model on those𝒲​s\\mathcal\{W\}sto be able to generate novel𝒲\\mathcal\{W\}vectors\. Thus, generating new INRs which can handle other real\-world tasks \(e\.g\. Super\-resolution, continuous\-unseen scenery samples, etc\.\)\.

By doing so, we’re training the DM \(diffusion model\) to fit the probability distribution of such𝒲​s\\mathcal\{W\}s\. Letting it learn how the probability of such𝒲​s\\mathcal\{W\}srepresentational vectors, for this specific trained task, supposed to look like\.

At inference, We’ll generate a novel𝒲^\\mathcal\{\\hat\{W\}\}vector via our DM, and predict the missing pixels via the pre\-calculated NTK \(𝒦t​e​s​t\\mathcal\{K\}\_\{test\}to be specific\) \- see Eq\.[5](https://arxiv.org/html/2609.03117#S1.E5)\(Yet, here, the observed pixels are the whole image\. Thus,𝒦t​e​s​t=𝒦t​r​a​i​n\\mathcal\{K\}\_\{test\}=\\mathcal\{K\}\_\{train\}\)\. Formally,

1. 1\.\{𝒲i\}i=0N−1=𝒦t​r​a​i​n−1×\{𝒴i\}i=0N−1∣N:\|D\|,D:Dataset\\\{\\mathcal\{W\}\_\{i\}\\\}\_\{i=0\}^\{N\-1\}=\\mathcal\{K\}\_\{train\}^\{\-1\}\\times\\\{\\mathcal\{Y\}\_\{i\}\\\}\_\{i=0\}^\{N\-1\}\\mid N:\|D\|,\\,D:Dataset
2. 2\.T​r​a​i​n​D​MW​o​n​\{𝒲i\}i=0N−1Train\\,DM\_\{W\}\\,on\\,\\\{\\mathcal\{W\}\_\{i\}\\\}\_\{i=0\}^\{N\-1\}
3. 3\.𝒲^=I⁡\[D​MW\]\|I:I​n​v​e​r​s​e​f​r​o​m​t​r​a​i​n​e​d​D​MW\\hat\{\\mathcal\{W\}\}=I\[DM\_\{W\}\]\\mid I:Inverse\\,from\\,trained\\,DM\_\{W\}
4. 4\.𝒴^=𝒦t​e​s​t​𝒲^\\hat\{\\mathcal\{Y\}\}=\\mathcal\{K\}\_\{test\}\\hat\{\\mathcal\{W\}\}

Configuring the Loss Function of the DM, Each Diffusion model block consists of a Unet NN architecture with attention\. We train the Unet model to identify and output the noise associated to its input image at each time step t \(t∈\[0,T−1\]t\\in\[0,T\-1\], whereTTif the total number of steps in the diffusion process\) accordingly, for various time steps\. Formally,

ℒ=‖ϵ−Uθ​\(xt,t\)‖22\\displaystyle\\mathcal\{L\}=\|\|\\epsilon\-U\_\{\\theta\}\(x\_\{t\},t\)\|\|\_\{2\}^\{2\}S\.T\.

ϵ∼N⁡\(0,1\)\\displaystyle\\epsilon\\sim N\(0,1\)xt=αt¯⋅x,\+1−αt¯⋅ϵ\\displaystyle x\_\{t\}=\\sqrt\{\{\\overline\{\\alpha\_\{t\}\}\}\}\\cdot x\_\{,\}\+\\,\\sqrt\{\{1\-\\overline\{\\alpha\_\{t\}\}\}\}\\cdot\\epsilonUθ​\(xt,t\)=ϵ^t−1\|ϵ^t−1:P​r​e​d​i​c​t​e​d​n​o​i​s​e​a​t​s​t​e​p​t−1\\displaystyle U\_\{\\theta\}\(x\_\{t\},t\)=\\hat\{\\epsilon\}\_\{t\-1\}\\,\|\\,\\hat\{\\epsilon\}\_\{t\-1\}:Predicted\\,noise\\,at\\,step\\,t\-1
The experiment was conducted on both MNIST and Celeb\-A datasets\. In each, we trained two DMs\. First, directly on the image doamin, and second, on theWWs dataset\. Example results of the calculated\{𝒲\}i\|i∈\{0,…,N−1\}\\\{\\mathcal\{W\}\\\}\_\{i\}\\mid i\\in\\\{0,\.\.\.,N\-1\\\}, can be seen in Fig,[5](https://arxiv.org/html/2609.03117#S2.F5)\.

The trained DM will generating a novel data distribution sample via its learnt Unet parameters\. But what if we want to use it for Inpainting or Matrix Completion \(MC\) tasks? In that case, we’ll have an incomplete\\\\backslashmasked matrix \(e\.g\. Image\), and we would want the DM to output the complete hidden\-pixels predicted image\. Similarly for completing\\\\backslashpredicting the corrupted pixels\\\\backslashvalues of a masked𝒲\\mathcal\{W\}vector\. Thus, given a corrupted image\\\\backslash𝒲\\mathcal\{W\}vector, we pass it through the trained DM \(D​M,D​MWDM,\\,DM\_\{W\}accordingly\) and use its observed pixels as guidance in every stept∈\[0,T−1\]t\\in\[0,T\-1\]of the Diffusion process \(For eachU​n​e​tθUnet\_\{\\theta\}\)\. Formally,

1. 1\.ϵ^t−1=Uθ​\(xt,t\)\\hat\{\\epsilon\}\_\{t\-1\}=U\_\{\\theta\}\(x\_\{t\},t\)
2. 2\.x^0​\(xt,t\)=1αt¯​\(xt−1−αt¯⋅ϵ^t\)\\hat\{x\}\_\{0\}\(x\_\{t\},t\)=\\frac\{1\}\{\\sqrt\{\\overline\{\\alpha\_\{t\}\}\}\}\(x\_\{t\}\\,\-\\,\\sqrt\{1\-\\overline\{\\alpha\_\{t\}\}\}\\cdot\\hat\{\\epsilon\}\_\{t\}\)
3. 3\.ℒ=∑o​b​s‖𝒴o​b​s−x^0‖22\|o​b​s:O​b​s​e​r​v​e​d​p​i​x​e​l​s\\mathcal\{L\}=\\sum\_\{obs\}\|\|\\mathcal\{Y\}\_\{obs\}\-\\hat\{x\}\_\{0\}\|\|\_\{2\}^\{2\}\\ \|\\,obs:Observed\\,pixels
4. 4\.∇P​\(y\|x\)=−∂ℒ∂xt\\nabla P\(y\|x\)=\-\\frac\{\\partial\{\\mathcal\{L\}\}\}\{\\partial\{x\_\{t\}\}\} Samplext−1x\_\{t\-1\}from the new probability distribution,
5. 5\.xt−1∼N⁡\(μt−1\+△​P​\(y\|x\),σt−1\)x\_\{t\-1\}\\sim N\(\\mu\_\{t\-1\}\+\\triangle P\(y\|x\),\\sigma\_\{t\-1\}\)S\.T\. - •μt−1​\(xt,t\)=1αt​\(xt−1−αt1−αt¯⋅ϵ^t−1\)\\mu\_\{t\-1\}\(x\_\{t\},t\)=\\frac\{1\}\{\\sqrt\{\{\\alpha\_\{t\}\}\}\}\(x\_\{t\}\\,\-\\,\\frac\{1\-\\alpha\_\{t\}\}\{\\sqrt\{1\-\\overline\{\\alpha\_\{t\}\}\}\}\\cdot\\hat\{\\epsilon\}\_\{t\-1\}\) - •σt−1=β¯t⋅z\|z\[T−1,2\]∼N\(0,1\),z1=0\\sigma\_\{t\-1\}=\\sqrt\{\\overline\{\\beta\}\_\{t\}\}\\cdot z\\ \|\\,z\_\{\[T\-1,2\]\}\\sim N\(0,1\),\\,z\_\{1\}=0 - •β¯t=βt−1⋅1−α^t−21−α^t\\overline\{\\beta\}\_\{t\}=\\beta\_\{t\-1\}\\cdot\\frac\{1\-\\hat\{\\alpha\}\_\{t\-2\}\}\{1\-\\hat\{\\alpha\}\_\{t\}\} - •αt=1−βt;α¯t=∏i=1Tαi;β=\[10−4,2⋅10−2\]T\\alpha\_\{t\}=1\-\\beta\_\{t\}\\ ;\\ \\overline\{\\alpha\}\_\{t\}=\\prod\_\{i=1\}^\{T\}\\alpha\_\{i\}\\ ;\\ \\beta=\[10^\{\-4\},2\\cdot 10^\{\-2\}\]\_\{T\} Thus, resulting in, xt−1=1αt​\(xt−1−αt1−αt¯⋅ϵ^t−1\)\+S⋅∇P​\(y\|x\)\+β¯t⋅zx\_\{t\-1\}=\\frac\{1\}\{\\sqrt\{\{\\alpha\_\{t\}\}\}\}\(x\_\{t\}\\,\-\\,\\frac\{1\-\\alpha\_\{t\}\}\{\\sqrt\{1\-\\overline\{\\alpha\_\{t\}\}\}\}\\cdot\\hat\{\\epsilon\}\_\{t\-1\}\)\\,\+\\,S\\cdot\\nabla P\(y\|x\)\\,\+\\,\\sqrt\{\\overline\{\\beta\}\_\{t\}\}\\cdot z

In order to compare the representational performance of the NTK method, let’s consider the comparison between the𝒲\\mathcal\{W\}vector representation and the original image\. For this task, we trained two Diffusion Models, one on the GT images and the other on the created NTK𝒲\\mathcal\{W\}vectors\. The results in Fig\.[8](https://arxiv.org/html/2609.03117#S2.F8)show the end downstream task of Inpainting and MC, inverse guided\-sample of the DM and the reconstructed image\.

We can notice the representational difference between training and performing directly in the image space and when the NTK space is considered as a representational mean\. Many conclusions can be drawn from these results\.

In the GT images space, the results are much smoother and more refined\. Also, in the bottom\-left \(22mnist digit\) GT image example, we can see that in the NTK representational space \(WWspace\), theWWvector is noise sensitive \(as stated and shown in Section[II\-B](https://arxiv.org/html/2609.03117#S2.SS2)\) and hence affecting the performance of the Diffusion Model\. The process of de\-noising in the diffusion trajectory is rather complex and an additional noise in the learned representations could only make it more difficult to inverse\. Furthermore, after the diffusion model has generated a new sample \(as in the bottom\-left \(22digit\) example\), if this sample is noisy \(which as stated before, theWWvector representation doesn’t make this easier\), the reconstruction image from thisWWvector will be radically blurry \(as can be seen in the reconstructed image part\)\. Again, due toWWvector noise sensitivity\.

Another conclusion which can be made, by examining the bottom\-right result\. We can see that when the diffusion model was trained on the original GT images \(image space\), the resulted guidance was much easier, without much of a hyperparameter effort, and with great accuracy in each result\. Yet, when it was trained on theWWvectors \(NTK space\), the guided regularization hyperparameter wasn’t consistent for all samples\. For some the DM could conduct a good inverse guidance, but for other samples, the guidance was disrupted and the inverse wasn’t an accurate one\. Leading to the observation that this NTK representation doesn’t bring any extra interested features or information in its core, but rather a degraded one\.

We can notice that even in the upper results, when the DM could reconstruct the guided novel sample via the NTK method, the reconstruction suffers from noise\.

If to consider Inference run\-time wise efficiency, Let’s compare the representational size and computation in both methods \(Functa vs NTK\), considering the basic NF which was used in both algorithms\. In Functa, the basic NF which was used in the paper, as shown above \(without the FPE\), consists of5×2565\\times 256hidden layers and an RGB output \(1×31\\times 3\)\. Thus, if to calculate the Inference run\-time complexity required for the algorithm \(Neglecting the bias term\),

\(h⋅w\)×d⏞P​E⋅l⏟1s​t​h​i​d​d​e​n​l​a​y​e​r\+\(l×l×Nl\)⏟m​i​d​h​i​d​d​e​n​l​a​y​e​r​s\+\(l×c\)⏟f​i​n​a​l​o​u​t​p​u​t​l​a​y​e​r=\\displaystyle\\underbrace\{\\overbrace\{\(h\\cdot w\)\\times d\}^\{PE\}\\cdot l\}\_\{1^\{st\}\\,hidden\\,layer\}\\,\+\\,\\underbrace\{\(l\\times l\\times N\_\{l\}\)\}\_\{mid\\,hidden\\,layers\}\\,\+\\,\\underbrace\{\(l\\times c\)\}\_\{final\\,output\\,layer\}==⏟i​n​o​u​r​c​a​s​e​\(32⋅32\)×2⋅256\+\(256×256×4\)\+\(256×3\)=\\displaystyle\\underbrace\{=\}\_\{in\\,our\\,case\}\(32\\cdot 32\)\\times 2\\cdot 256\\,\+\\,\(256\\times 256\\times 4\)\\,\+\\,\(256\\times 3\)=≅8×105=0\.8M\\displaystyle\\cong 8\\times 10^\{5\}=\\textbf\{0\.8M\}
In the NTK method, we’ll mention the whole process in Inference\. Yet, at the end, neglect the FPE and the kernel mapping calculation \(it differs depending on model architecture, input data, algorithm, etc\.\)\. The process will be to calculate the FPE, using it to perform Kernel calculation \(Calculating bothKt​r​a​i​nK\_\{train\}andKt​e​s​tK\_\{test\}\), usingKt​r​a​i​nK\_\{train\}to calculateWW, and then calculatingY^\\hat\{Y\}viaKt​e​s​tK\_\{test\}\(Considering the reconstruction complexity\)\. We get,

\(h⋅w\)×d⏞P​E×\(d\+2⋅d⋅k⏞F​o​u​r​i​e​r​b​a​n​d​s\)⏟F​P​E\+\(Kt​r​a​i​n−1×Y\)⏟Wv​e​c​t​o​r\+\\displaystyle\\underbrace\{\\overbrace\{\(h\\cdot w\)\\times d\}^\{PE\}\\times\(d\\,\+\\,2\\cdot d\\cdot\\overbrace\{k\}^\{Fourier\\,bands\}\)\}\_\{FPE\}\\,\+\\,\\underbrace\{\(K\_\{train\}^\{\-1\}\\times Y\)\}\_\{W\_\{vector\}\}\\,\+\\,Kt​e​s​t×W⏟R​e​c​o​n​s​t​r​u​c​t​Y^=\\displaystyle\\underbrace\{K\_\{test\}\\times W\}\_\{Reconstruct\\,\\hat\{Y\}\}==\(h⋅w\)×d⏞P​E×\(d\+2⋅d⋅k⏞F​o​u​r​i​e​r​b​a​n​d​s\)⏟F​P​E\\displaystyle=\\underbrace\{\\overbrace\{\(h\\cdot w\)\\times d\}^\{PE\}\\times\(d\\,\+\\,2\\cdot d\\cdot\\overbrace\{k\}^\{Fourier\\,bands\}\)\}\_\{FPE\}\+\(𝒪⁡\(o3\)⏞Kt​r​a​i​n​i​n​v​e​r​s​e​𝒪​\(n3\)×\(o⋅o\)⏞Kt​r​a​i​n​s​t​r​u​c​t​u​r​e×\(o×c\)⏞Y​s​t​r​u​c​t​u​r​e\)⏟Wv​e​c​t​o​r\\displaystyle\+\\,\\underbrace\{\(\\overbrace\{\\mathcal\{O\}\(o^\{3\}\)\}^\{K\_\{train\}\\,inverse\\,\\mathcal\{O\}\(n^\{3\}\)\}\\times\\overbrace\{\(o\\cdot o\)\}^\{K\_\{train\}\\,structure\}\\times\\overbrace\{\(o\\times c\)\}^\{Y\\,structure\}\)\}\_\{W\_\{vector\}\}\+\[\[\(h⋅w\)×o\]⏞Kt​e​s​t​s​t​r​u​c​t​u​r​e×\(o×c\)⏞Wv​e​c​t​o​r\]⏟R​e​c​o​n​s​t​r​u​c​t​Y^\\displaystyle\+\\,\\underbrace\{\[\\overbrace\{\[\(h\\cdot w\)\\times o\]\}^\{K\_\{test\}\\,structure\}\\times\\overbrace\{\(o\\times c\)\}^\{W\_\{vector\}\}\]\}\_\{Reconstruct\\,\\hat\{Y\}\}=⏟i​n​o​u​r​c​a​s​e​\(32⋅32\)×2⏞P​E×\(2\+2⋅2⋅20⏞F​o​u​r​i​e​r​b​a​n​d​s\)⏟F​P​E\\displaystyle\\underbrace\{=\}\_\{in\\,our\\,case\}\\underbrace\{\\overbrace\{\(32\\cdot 32\)\\times 2\}^\{PE\}\\times\(2\\,\+\\,2\\cdot 2\\cdot\\overbrace\{20\}^\{Fourier\\,bands\}\)\}\_\{FPE\}\+\(𝒪⁡\(\(32⋅32\)3\)⏞Kt​r​a​i​n​i​n​v​e​r​s​e​𝒪​\(n3\)×\(\(32⋅32\)×\(32⋅32\)\)⏞Kt​r​a​i​n​s​t​r​u​c​t​u​r​e×\(\(32⋅32\)×c\)⏞Y​s​t​r​u​c​t​u​r​e\)⏟Wv​e​c​t​o​r\\displaystyle\+\\,\\underbrace\{\(\\overbrace\{\\mathcal\{O\}\(\(32\\cdot 32\)^\{3\}\)\}^\{K\_\{train\}\\,inverse\\,\\mathcal\{O\}\(n^\{3\}\)\}\\times\\overbrace\{\(\(32\\cdot 32\)\\times\(32\\cdot 32\)\)\}^\{K\_\{train\}\\,structure\}\\times\\overbrace\{\(\(32\\cdot 32\)\\times c\)\}^\{Y\\,structure\}\)\}\_\{W\_\{vector\}\}\+\[\[\(32⋅32\)×\(32⋅32\)\]⏞Kt​e​s​t​s​t​r​u​c​t​u​r​e×\(\(32⋅32\)×c\)⏞Wv​e​c​t​o​r\]⏟R​e​c​o​n​s​t​r​u​c​t​Y^=\(∗\)\\displaystyle\+\\,\\underbrace\{\[\\overbrace\{\[\(32\\cdot 32\)\\times\(32\\cdot 32\)\]\}^\{K\_\{test\}\\,structure\}\\times\\overbrace\{\(\(32\\cdot 32\)\\times c\)\}^\{W\_\{vector\}\}\]\}\_\{Reconstruct\\,\\hat\{Y\}\}=\(\*\)
In the above calculation, we’ve calculated the Inference complexity term using the variablecc\. Here,ccis an indication for the last dimension of the matrixYY\(Which is used for the Training data phase \(CalculatingWW\)\)\. WhileYYis determined according to the downstream task which we’re solving\. e\.g\. In case we’re dealing with medical data for instance, and theYYmatrixccdimension of a size∼𝒪⁡\(n\)\|n=h⋅w\\sim\\mathcal\{O\}\(n\)\\,\|\\,n=h\\cdot w\. Then,

\(∗\)\\displaystyle\(\*\)≥⏞△3​105⏞F​P​E\+\[109\+109\]⏞Wv​e​c​t​o​r\+109⏞R​e​c​o​n​s​t​r​u​c​Y^\\displaystyle\\overbrace\{\\geq\}^\{\\triangle\_\{3\}\}\\overbrace\{10^\{5\}\}^\{FPE\}\\,\+\\,\\overbrace\{\[10^\{9\}\+10^\{9\}\]\}^\{W\_\{vector\}\}\\,\+\\,\\overbrace\{10^\{9\}\}^\{Reconstruc\\,\\hat\{Y\}\}≥⏞△4​2×109=2G\\displaystyle\\overbrace\{\\geq\}^\{\\triangle\_\{4\}\}2\\times 10^\{9\}=\\textbf\{2G\}And in caseYYmatrix is an RGB image\. Thus,c=3c=3\. Then,

\(∗\)\\displaystyle\(\*\)≥⏞△5​105⏞F​P​E\+\[109\+106⋅3\]⏞Wv​e​c​t​o​r\+\[106⋅3\]⏞R​e​c​o​n​s​t​r​u​c​Y^\\displaystyle\\overbrace\{\\geq\}^\{\\triangle\_\{5\}\}\\overbrace\{10^\{5\}\}^\{FPE\}\\,\+\\,\\overbrace\{\[10^\{9\}\+10^\{6\}\\cdot 3\]\}^\{W\_\{vector\}\}\\,\+\\,\\overbrace\{\[10^\{6\}\\cdot 3\]\}^\{Reconstruc\\,\\hat\{Y\}\}≥⏞△4​2⋅\[106⋅3\]=2⋅3×106=6M\\displaystyle\\overbrace\{\\geq\}^\{\\triangle\_\{4\}\}2\\cdot\[10^\{6\}\\cdot 3\]=2\\cdot 3\\times 10^\{6\}=\\textbf\{6M\}S\.T\.

△3:G​e​n​e​r​a​l​M​a​t​r​i​x​M​u​l​t​i​p​l​i​c​a​t​i​o​n−𝒪⁡\(n3\)\\displaystyle\\triangle\_\{3\}:General\\,Matrix\\,Multiplication\-\\mathcal\{O\}\(n^\{3\}\)△4:\\displaystyle\\triangle\_\{4\}:N​e​g​l​e​c​t​i​n​g​F​P​E​a​n​d​K​e​r​n​e​l​I​n​v​e​r​s​e\\displaystyle Neglecting\\,FPE\\,and\\,Kernel\\,Inverse\\,\(B​o​t​h​c​o​m​p​u​t​e​d​o​n​l​y​o​n​c​e\)\\displaystyle\(Both\\,computed\\,only\\,once\)△5:M​a​t​r​i​x,V​e​c​t​o​r​M​u​l​t​i​p​l​i​c​a​t​i​o​n−𝒪⁡\(n2\)\\displaystyle\\triangle\_\{5\}:Matrix,\\,Vector\\,Multiplication\-\\mathcal\{O\}\(n^\{2\}\)
Explaining△4\\triangle\_\{4\},

- •We neglected the inverse calculation ofF​P​EFPEandKt​r​a​i​nK\_\{train\}\. This is due to the fact that both could be calculated only once at the beginning of the experiment and then be used in all the individual NFs calculations \(The NTK Kernel is shared among the dataset samples\)\. This due to the fact that theF​P​EFPEare fixed to all positional inputs, andKt​r​a​i​nK\_\{train\}is fixed as well for all the correspondingWWs \(In an NF model, theP​EPEinput is fixed for all images \- when they’re at the same size, and the model’s architecture is similar to all Multi\-NFs\. Thus, the NTK Kernel calculation is fixed since its inputs are the PEs data samples and the NF architecture\)\.
- •We neglected the Kernel mapping function time complexity\. This is due to the fact that again, as stated in the previous point, the Kernel matrices \(in our experiment\) can be calculated only once at the beginning and then be used and shared among all the dataset samples\. Another reason which this could help, the kernel mapping matrix differs depending on model architecture, input data, calculation algorithm, etc\.

We can see the radical difference in between the two representational methods in terms of Inference efficiency \(One or even two order of magnitude difference to the favor of the NF\)\. TheWWvector’s representation sizew=k−1×y→\(h⋅w\)×cw=k^\{\-1\}\\times y\\rightarrow\(h\\cdot w\)\\times cis similar to the target image’s size, also\(h⋅w\)×c\(h\\cdot w\)\\times c\. And the results, yet, falls short\.

The NTK method suffers from inefficiency when compared to other methods \(as in the Functa paper, and even the direct raw data input space\) and this inefficiency is shown in three main constellations:

1. 1\.Representational structure efficiency\.
2. 2\.Representational effectiveness\.
3. 3\.Time\-complexity and computational flops efficiency\.

To conclude, NTK is unsuitable for practical representation tasks due to weaknesses in structure, representation, and runtime, posing limitations for applications like model scaling and edge AI \(energy\-efficient systems\)\. We observe three main weaknesses in the NTK method:

1. 1\.Representational structure efficiency:The NTK matrixWWhas a representational size ofℝ\(h⋅w\)×c\\mathbb\{R\}^\{\(h\\cdot w\)\\times c\}, matching the ground truth \(GT\) image size but lacking efficient structure\.
2. 2\.Representational effectiveness:AlthoughWWmatches the image size, it fails to achieve effective representation in terms of performance results\.
3. 3\.Time\-complexity:The NTK method exhibits high time complexity and computational cost, being at least one order of magnitude less efficient in runtime\.

## Appendix ENovel Approaches of Representation Learning via NTK

### E\-ARepresentation Learning via NTK Novel Algorithms

In this paper, we outline the previous limitations, and propose a transformative path forward in applying NTK to representation learning, proposing the integration of our two novel algorithms\. Each algorithm independently addresses a unique, previously unsolved challenge in NTK, and their combined potential could fundamentally overcome existing limitations, paving the way for a significant leap in the field\.

The results of our novel algorithms,

- •Fixing Representational Linearity \- NTK KIP\.
- •NTK Algorithm which suggests a new Efficient meta\-learning approach\.

#### E\-A1NTK\-KIP

In an effort to solve the previous discussed issues, linearity, efficiency, and NTK’s kernel poor representation \(as appears in the hollow experiment\), we came up with the idea of NTK KIP\.

We’ve noticed that thewwvector suffers from the linearity problem \(Section[II\-A](https://arxiv.org/html/2609.03117#S2.SS1),NTK representational linearity\) with respect to the target image, As we recallW=Kt​r​a​i​n​Yt​r​a​i​n\|Yt​r​a​i​n:G​T​O​b​s​e​r​v​e​d​L​a​b​e​l​sW=K\_\{train\}Y\_\{train\}\\mid Y\_\{train\}:GT\\,Observed\\,Labels\.

Hence, if to handle this, we need forKt​r​a​i​nK\_\{train\}\(NTK Kernel\) to beYG​TY\_\{GT\}dependent, which will lead forWWto becomeYYdependent as well\. And the way to assure this is via the input coordinates\. If we can update the coordinates \(or at least some of them\) according toYYthen our PE will becomel​a​b​e​l​slabelsdependent and henceKK\(which uses our NF input’s PE\) will also bel​a​b​e​l​slabelsdependent, leading thatWWwill be non\-linear with respect toYY\. Formally, letNNbe the total number of images in our dataset\. Given imagei∈\[1,…,N\]i\\in\[1,\\,\.\.\.,\\,N\]of size\(h,w,c\)\(h,\\,w,\\,c\), letP=h⋅wP=h\\cdot wbe the total number of pixels inii,p∈\[1,…,P\]p\\in\[1,\\,\.\.\.,\\,P\]be a pixel inii,x∈ℝN×2x\\in\\mathbb\{R\}^\{N\\times 2\}be the PE vector,y∈ℝ\(P×c\)y\\in\\mathbb\{R\}^\{\(P\\times c\)\}be the image labels \(values\)\. We then definexip∈ℝ2x\_\{i\}^\{p\}\\in\\mathbb\{R\}^\{2\}to be the PE coordinate which corresponds to pixelppin imageii, andyipy\_\{i\}^\{p\}to be the value \(label\) of pixelppin imageii\. In the Raw PE \(RPE\) example, we optimizexipx\_\{i\}^\{p\}andyipy\_\{i\}^\{p\}to minimize the reconstruction error for the full image:

arg⁡minxip,yip​‖𝒦⁡\(x,xip\)​𝒦​\(xip,xip\)−1​yip−y‖\\displaystyle\\arg\\min\_\{x\_\{i\}^\{p\},y\_\{i\}^\{p\}\}\\left\\\|\\mathcal\{K\}\(x,x\_\{i\}^\{p\}\)\\;\\mathcal\{K\}\(x\_\{i\}^\{p\},x\_\{i\}^\{p\}\)^\{\-1\}\\;y\_\{i\}^\{p\}\-y\\right\\\|
This representation is therefore no longer a simple linear mapping of the signal, and can potentially lead to extracting more meaningful and useful information from the image\.

This algorithm will also improve NTK’s kernel representation \(as can be seen in theh​o​l​l​o​whollowexperiment\)\. The PEs will be updated via Gradient Descent according to a reconstruction loss objective of the hypothesisY^\\hat\{Y\}model output\. This will lead to not only better feature representation, but also data\(YG​T\)\(Y\_\{GT\}\)distillation\. When the distilled information will be ”held” in the PEs, and they will be updated is such a manner that the NTK Kernel can be built as a better representation concept \(from the leaned objective\), and will be used as an actual mean for learning \(via Kernel gradient backpropagation\) and updating features accordingly\.

Previous research was conducted in similar areas, where NTK’s kernel was used also as a mean for inducing points, yet, in a classification task \(an image as the input, and the output is the class hypothesis\)\[[23](https://arxiv.org/html/2609.03117#bib.bib33)\]\. In our case, the input is the positional coordinates and the output is an RGB value\. In this experiment we aim to distillate our dataset \(positional coordinates\) as much as possible \(At least one order of magnitude, with the goal for two orders\)\.

We tested this method when the trained PEs were once the raw coordinates, and again as the FPEs \(Fourier Positional Encodings\)\. In both experiments we enable labels learning \(yields better results\)\. The results of both experiments can be seen in Fig\.[11](https://arxiv.org/html/2609.03117#S3.F11)\.

In this experiment, for the Raw PEs KIP, one can see that a steep degradation in the performance appears once down scaling the distillation percentage of the observed pixels in the GT image\. We can gain relatively good results at80%∼50%80\\%\\sim 50\\%observed pixels \(P​S​N​R≅36\.74∼23\.72PSNR\\cong 36\.74\\sim 23\.72\)\. Regarding the FPEs KIP experiment, we can notice a much better improvement regarding feature learning, Kernel representation, and distilled Inpainting reconstruction\. Though in the FPE we’re usingkkbands, thus expending the input coordinates features representation, it’s still considering only the mentioned distilled percentage of thel​a​b​e​l​slabels\. The learning and image representation will take place on extra features, yet, the distilled representational coordinates will be maintained\.

Thekkband which was used in the FF transformation isk=20k=20\(as shown in Fig\.[22](https://arxiv.org/html/2609.03117#A2.F22)\), which means, as shown in Appendix[B](https://arxiv.org/html/2609.03117#A2), that each PE 2D coordinate is transformed into ad\+2⋅d⋅k\|d=2,k=20=82d\\,\+\\,2\\cdot d\\cdot k\\,\|\_\{d=2,\\,k=20\}=82Fourier Features\. Hence, the total number of features isO​b​s%⋅Ff​e​a​t​u​r​e​s=x%⋅82Obs\{\\%\}\\cdot F\_\{features\}=x\\%\\cdot 82\. If to take a5%5\\%distilled dataset for instance, the No\. of FF will then be410%410\\%of the whole dataset\. This is still a limitation of this algorithm\. For this is higher than the Raw PEs percentage and about×4\\times 4larger representation than the raw image itself \(image representation number of pixels\)\. Yet, thel​a​b​e​l​slabelspercentage of the target image still holds, and is still the original distilled percentage\. Another thing to be noticed, after applying the Fourier encodings and learning a distilled representation of its transformation, the resulted representation will not be interpreted and might not represent a 2D spatial target position as the RPEs\. These findings lead to an understanding that a significant amount of trained and learned features are required in order to truly grasp and absorb the distilled latent information via the NTK method\. Hence, driving the need for a true learning\-capable NTK algorithm \(See Section[III\-B](https://arxiv.org/html/2609.03117#S3.SS2)\)\.

To conclude the above in terms of Raw PE vs FPE with respect to Distillation percentage, see Table[I](https://arxiv.org/html/2609.03117#S4.T1)\.

In order to be able to have a better perspective regarding the high dimensional model hyperparameters and to project their main harmonic to a hyperplane, we’ve selected the Fourierkb​a​n​d​sk\_\{bands\}and Distillation percentage \(FPE and Target image\) parameters to examine their effect on the Final model’s PSNR\. Results in Fig\.[12](https://arxiv.org/html/2609.03117#S4.F12)\.

We conducted the experiment examining a grid of2525samples in total from each param \(55values per parameter \-kb​a​n​d​sk\_\{bands\}andD​i​s​t​i​l​l​a​t​i​o​nDistillation\)\. In order to fit and approximate in relatively high accuracy the probability hidden function of this distilled space, we used a radial basis function\. And more specific, Quintic function\. We also added the projection of the approximated space function on each 2D plane\.

If we examine the plane projection on theX−ZX\-Zaxes, we can see the same 2D graph shape as in the experiment with the Fourierkb​a​n​d​sk\_\{bands\}\(See Appendix[B](https://arxiv.org/html/2609.03117#A2), Fig\.[22](https://arxiv.org/html/2609.03117#A2.F22)\)\.

In this sweep grid, PSNR is highest at low distillation and decreases as distillation becomes more aggressive, while increasingKbandsK\_\{\\mathrm\{bands\}\}improves PSNR until saturation\. This is as expected\. The distillation is at its lowest \(distillation range:95%∼60%95\\%\\sim 60\\%\),kb​a​n​d​sk\_\{bands\}is at the maximum of the saturated 2D planar\.

As an example for the FPE representation size, let’s consider a transformation consists ofkb​a​n​d​s=30→F​F=122k\_\{bands\}=30\\rightarrow FF=122\(Per positional encoding\), and at50%50\\%distillation\. Then, the representational size,

r​e​p​r​e​s​e​n​t​a​t​i​o​n​\_​s​i​z​e\\displaystyle representation\\\_size=d​i​s​t​i​l​l​a​t​i​o​n⋅i​m​a​g​e​\_​s​i​z​e⋅F​F\\displaystyle=distillation\\cdot image\\\_size\\cdot FF=50%⋅h⋅w⋅122=61⋅h⋅w\\displaystyle=50\\%\\cdot h\\cdot w\\cdot 122=61\\cdot h\\cdot w=61×i​m​a​g​e​\_​s​i​z​e\\displaystyle=61\\times image\\\_size
We were able to achievel​a​b​e​l​slabelspercentage of about22orders of magnitude\(∼95%\)​d​i​s​t​i​l​l​a​t​i​o​n\(\\sim 95\\%\)\\,distillationlower than the GT image\. Yet, the representation size \(Total feature count\) is still relatively large\. Thus, concluding that the NTK method still falls short and relatively struggle with extracting and learning meaningful features\.

To conclude,Neural Field Kernel Inducing Points \(NTK\-KIP\)\.In order to fix the problem of NTK’s𝒲\\mathcal\{W\}Representational Linearity and also Representational Efficiency, we developed a novel algorithm NTK KIP \(Kernel Inducing Points\), which by updating the spacial coordinates input of the NF via NTK’s Kernel derivative while minimizing an objective loss, not only creating a distilled input \(And hence, label\) prior data, and thus a more efficient NTK𝒲\\mathcal\{W\}representation\. But also, a Non\-Linear𝒲\\mathcal\{W\}representational vector,𝒲=𝒦t\(x,x′;θ\)−1𝒴\\mathcal\{W\}=\\mathcal\{K\}\_\{\\mathit\{t\}\}\(\\mathit\{x\},\\mathit\{x\}^\{\{\}^\{\\prime\}\};\\theta\)^\{\\mathit\{\-1\}\}\\mathcal\{Y\}\\\\s\.t\.x=𝒢⁡\(𝒴\)\|𝒢:n​o​n−l​i​n​e​a​r​m​a​p​p​i​n​g\\mathit\{x\}=\\mathcal\{G\}\(\\mathcal\{Y\}\)\\,\|\\,\\mathcal\{G\}:non\-linear\\,mapping →𝒦=ℱ⁡\[g⁡\(𝒴\)\]\|ℱ:k​e​r​n​e​l​f​u​n​c​t​i​o​n\\rightarrow\\,\\mathcal\{K\}=\\mathcal\{F\}\[\\mathit\{g\}\(\\mathcal\{Y\}\)\]\\,\|\\,\\mathcal\{F\}:kernel\\,function →𝒲:n​o​n−l​i​n​e​a​r​m​a​p​p​i​n​g​o​f​𝒴\\rightarrow\\,\\mathcal\{W\}:\\,non\-linear\\,mapping\\,of\\,\\mathcal\{Y\}

With this, we managed to create via the Raw Positional Encoding \(RPE\) an input data distillation, and hence also𝒲\\mathcal\{W\}distillation, of20\\mathit\{20\}% with above36\\mathit\{36\}PSNR, and50\\mathit\{50\}% with above23\\mathit\{23\}PSNR\. And via the Fourier Positional Encoding \(FPE\),80\\mathit\{80\}% with above31\\mathit\{31\}PSNR, and95\\mathit\{95\}% with above20\\mathit\{20\}PSNR\. We discovered that though this algorithm can create a more distilled and a non\-linear representation, it suffers from the lack of learninng and features extraction which NTK’s representational nature suffers from\. Hence, we created the following algorithm \(Section[III\-B](https://arxiv.org/html/2609.03117#S3.SS2)\) to solve the above NTK problem\.

#### E\-A2MetaQuill Algorithm

We’ve discussed the lack and limitations of the NTK method as a representational concept\. Some of which reasons had to do with the Infinite limit, and the lack of feature learning and the actual evolution of NN complex architectures over time\.

In an effort to bridge these gaps, we came up with a novel idea \(which to our knowledge has never been discovered\) which combines Finite\-width NTK, Model Feature learning, and a novel algorithm which achieves the benefits of MAML \(Mode Agnostic meta\-learning\) without its inner loop\. We also supply a theoretical analysis for our novel algorithm\.

For this algorithm development, we consider the Functa \(See Section[II\-C](https://arxiv.org/html/2609.03117#S2.SS3)\) discussed downstream task \(NF NTK’s representation via DM\)\. The main idea is to try and develop a new algorithm which takes into consideration the Feature Learning of an NF model \(Thus, gaining learning capabilities for NTK\), makes better usage of the learned data \(Learning on only a small subset of the data\. Tackling Efficiency\), and builds a whole new concept which has the benefits of Meta\-learning \(Features Learning\. e\.g\. MAML\), yet in a more efficient manner \(Specifically without MAML’s inner loop\)\.

Specifically, in the Functa DM \(Diffusion Model\) downstream task, we explained how, as inspired by the Functa paper, our dataset is being transferred into a dataset of functions \(NFs\), and then we train a DM on those NFs for various tasks and purposes \(Generating new distributions for yet unseen tasks, such that it can be controlled to our own purposes and task dependent\. e\.g\. super\-resolution, matrix completion, etc\.\)\. Each function \(NF\) learns a single image, thus transforming the images dataset into an NFs dataset \(Creating Multi\-NF\)\.

##### Algorithm

In this Algorithm, we’ll be using Finite\-NTK rather than the Infinite theorem\.

Finite\-Width Kernel LearningIt has been observed that the performance of the infinite width NTK can be quite poor compared to finite width networks\[[37](https://arxiv.org/html/2609.03117#bib.bib36)\]\. This is because finite\-width networks can exhibit feature learning, whereas for infinite\-width networks, the underlying feature map is fixed\. In the infinite width, the feature mapϕ⁡\(x\)\\phi\(x\)is given by the Tangent feature mapφ⁡\(x\)=∇θfθ​\(x\)\\varphi\(x\)=\\nabla\_\{\\theta\}f\_\{\\theta\}\(x\)\. As the width ofθ\\thetagoes to infinity\. Becauseθ\\thetais infinite, the feature map is infinite, so we use the kernel trick to perform regression on this feature set\. LetK⁡\(x,x′\)=𝔼θ∼P⁡\(θ\)​\[∇θfθ​\(x\)T⋅∇θfθ​\(x′\)\]K\(x,\\,x^\{\\prime\}\)=\\mathbb\{E\}\_\{\\theta\\sim P\(\\theta\)\}\[\\nabla\_\{\\theta\}f\_\{\\theta\}\(x\)^\{T\}\\cdot\\nabla\_\{\\theta\}f\_\{\\theta\}\(x^\{\\prime\}\)\], asw→∞w\\rightarrow\\infty, for initialization distributionP⁡\(θ\)P\(\\theta\)\. It has been observed that while the infinite width NTK is an inaccurate model of network training, the NTK theory can still be used, but instead using a time and data\-dependent finite NTK\. Specifically, it has been observed that the training process of a NNf⁡\(θ\)\|θ:N​N​p​a​r​a​m​e​t​e​r​sf\(\\theta\)\\,\|\\,\\theta:NN\\,parameters, can be approximated by a first\-order Taylor expansion around a working pointθτ\\theta\_\{\\tau\}, S\.T\.

θτ\|τ∈\{T\},\\displaystyle\\theta\_\{\\tau\}\\,\|\\,\\tau\\in\\\{T\\\},\\,τ:i​n​f​i​n​i​t​e​s​i​m​a​l​N​o\.e​p​o​c​h​s,\\displaystyle\\tau:infinitesimal\\,No\.\\,epochs,\\,T:T​h​e​t​r​a​i​n​i​n​g​p​h​a​s​e​o​f​N​N\\displaystyle T:The\\,training\\,phase\\,of\\,NN
First\-order Taylor expansion,

fθt\+τ​\(x\)≈fθτ​\(x\)\+\(θt\+τ−θτ\)T⋅∇θτfθτ​\(x\)\\displaystyle f\_\{\\theta\_\{t\+\\,\\tau\}\}\(x\)\\approx f\_\{\\theta\_\{\\tau\}\}\(x\)\\,\+\\,\(\\theta\_\{t\\,\+\\tau\}\-\\theta\_\{\\tau\}\)^\{T\}\\cdot\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x\)
Namely, we train forτ\\tauepochs, then perform a first\-order Taylor expansion\. This is like training a linear model with a learned feature map∇θτfθτ​\(x\)\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x\), with a resulting finite NTKKτ​\(x,x′\)=∇θτfθτ​\(x\)T⋅∇θτfθτ​\(x′\)K\_\{\\tau\}\(x,x^\{\\prime\}\)=\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x\)^\{T\}\\cdot\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x^\{\\prime\}\)\. The quantityfθτ​\(x\)\+\(θt\+τ−θτ\)T⋅∇θτfθτ​\(x\)f\_\{\\theta\_\{\\tau\}\}\(x\)\\,\+\\,\(\\theta\_\{t\\,\+\\tau\}\-\\theta\_\{\\tau\}\)^\{T\}\\cdot\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x\)can be computed exactly using forward\-mode auto\-diff\. We could also consider a similar training paradigm, in which we omit the zeroth order contribution, which we called centered training,

fθt\+τ​\(x\)−fθτ​\(x\)≈\(θt\+τ−θτ\)T⋅∇θτfθτ​\(x\)\\displaystyle f\_\{\\theta\_\{t\+\\,\\tau\}\}\(x\)\-f\_\{\\theta\_\{\\tau\}\}\(x\)\\approx\(\\theta\_\{t\\,\+\\tau\}\-\\theta\_\{\\tau\}\)^\{T\}\\cdot\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x\)→fC​e​n​t​e​r​e​d,△​θ,θτ​\(x\)=△​θT⋅∇θτfθτ​\(x\)\\rightarrow f\_\{Centered,\\,\\triangle\\theta,\\,\\theta\_\{\\tau\}\}\(x\)=\\triangle\\theta^\{T\}\\cdot\\nabla\_\{\\theta\_\{\\tau\}\}f\_\{\\theta\_\{\\tau\}\}\(x\)\(12\)
Note that in centered training, we have two free parameters \-△​θ\\triangle\\theta, andθτ\\theta\_\{\\tau\}\.θτ\\theta\_\{\\tau\}defines our feature map, while△​θ\\triangle\\thetais our linear weight \(The Shared\-base weight\)\. As far as we know, no one has considered training these two parameters simultaneously, asθτ\\theta\_\{\\tau\}is typically fixed, but we will be learning both here\.

Why Finite\-Width NTKs matters?As mentioned earlier, finite\-width NTKs can benefit from feature learning\. Feature learning allows networks to quickly learn without much modification of parameters, like meta\-learning\. The Functa paper[II\-C](https://arxiv.org/html/2609.03117#S2.SS3)has a network with shared weightsθw\\theta\_\{w\}, and per\-image biasesθbi\\theta\_\{b\}^\{i\}\. While this may be motivated by the need to reduce the dimensionality of the new data samples representations \(As the training\-set for the diffusion model\), it additionally has the benefit of leverage the shared learned features inθw\\theta\_\{w\}\. Contrasting this with the infinite\-width Functa idea \- there is no feature learning \(As can bee seen from the results in Section[II\-C](https://arxiv.org/html/2609.03117#S2.SS3)\)\. That is, we are modeling theN​FNFtraining process as training infinite width models on each image independently with the frozen infinite\-width NTK feature set, while Functa is modelling training NFs jointly, with weight sharing inθw\\theta\_\{w\}\. However, this may be too inflexible, ideally, we want to adjust all the weights of the network, but only adjust them slightly so that we are approximately in the fine\-tuning regime\. The problem is that there are too many parameters to do this\. This highlights the main difficulty in balancing Functa and the Infinite\-width Functa\. Infinite\-width methods give a tractable way to have more parameters by working in kernel space, but cannot benefit from feature learning\. In contrast, Functa has too few parameters, but gets feature learning, and we cannot increase the number of free parameters in a tractable way\.

MetaQuill Core AlgorithmTo fix this, we propose the MetaQuill Algorithm, which leverages the Feature Learning of neural networks and the parameter flexibility of NTK kernels while achieving meta\-efficiency using only a fraction of the dataset\. The goal of this algorithm is to integrate Finite\-Width NTK, Model Feature Learning, and a novel approach that retains the benefits of Model Agnostic meta\-learning \(MAML\) but with greater efficiency\.

We demonstrate our findings on the Functa problem example\. Yet, the algorithm is general and holds for any other variate of NNs and downstream tasks\.

With regards to Functa, This will become MetaQuill Functa, which uses the Kernel Trick with a learned feature map\. Specifically, from Eq\.[12](https://arxiv.org/html/2609.03117#A5.E12)we have the following model for each NF in the dataset \(indexed by i\),

N​Fi​\(x\)=△​θiT⋅∇θSfθS​\(x\)NF\_\{i\}\(x\)=\\triangle\\theta\_\{i\}^\{T\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x\)\(13\)
That is, we have per\-image weights△​θi\\triangle\\theta\_\{i\}, but a shared feature map given byθS\\theta\_\{S\}\. Now we want to define a diffusion model on the per\-image parameters△​θi\\triangle\\theta\_\{i\}, but that has high dimensionality\. To work around this, we use the kernel trick, and define the weight\-to\-kernel space transform \(and the reverse transform\)\.Transforming from weight space to kernel space \(and vice\-versa\)\. To proceed, we first consider the task of training an NF on a fixed set of input pointsXX\. LetYiY\_\{i\}be the labels for a specific data point\. We assume we are working in the Fourier embedding space, and also that these are shared for every image\. Our image NF loss is,

ℒi=∑xp,yp∈X,Yi12⋅\(N​Fi​\(xp\)−yip\)2\\displaystyle\\mathcal\{L\}\_\{i\}=\\sum\_\{x^\{p\},\\,y^\{p\}\\,\\in\\,X,\\,Y\_\{i\}\}\\frac\{1\}\{2\}\\cdot\(NF\_\{i\}\(x^\{p\}\)\-y\_\{i\}^\{p\}\)^\{2\}→ℒi=∑xp,yp∈X,Yi12⋅\(△​θiT⋅∇θSfθS​\(xp\)−yip\)2\\rightarrow\\mathcal\{L\}\_\{i\}=\\sum\_\{x^\{p\},\\,y^\{p\}\\,\\in\\,X,\\,Y\_\{i\}\}\\frac\{1\}\{2\}\\cdot\(\\triangle\\theta\_\{i\}^\{T\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x^\{p\}\)\-y\_\{i\}^\{p\}\)^\{2\}\(14\)
Whereppindexes the positions in the image \(So,xpx^\{p\}would bept​hp^\{th\}coordinate which corresponds topt​hp^\{th\}pixel in the label\-image, andyipy\_\{i\}^\{p\}would be thept​hp^\{th\}pixel of theit​hi^\{th\}image\)\. Solving for the optimal△​θ\\triangle\\theta, we have,

△​θi=∑wip,xpwip⋅∇θSfθS​\(xp\)\\triangle\\theta\_\{i\}=\\sum\_\{w\_\{i\}^\{p\},\\,x^\{p\}\}w\_\{i\}^\{p\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x^\{p\}\)\(15\)
That is, the optimal△​θ\\triangle\\thetais given by a linear combination of gradients atXX\. Thesewipw\_\{i\}^\{p\}values have a closed form solution,wi=KθS−1⋅yiw\_\{i\}=K\_\{\\theta\_\{S\}\}^\{\-1\}\\cdot y\_\{i\}\.

Wherewiw\_\{i\}is the vector containingwipw\_\{i\}^\{p\}, andKθSK\_\{\\theta\_\{S\}\}is the finite\-width NTK kernel matrix at the index pointsXX, with finite NTK kernelKS​\(x,x′\)=∇θSfθS​\(x\)T⋅∇θSfθS​\(x′\)K\_\{S\}\(x,x^\{\\prime\}\)=\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x\)^\{T\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\(x^\{\\prime\}\)\}\. This has lower dimensionality than△​θ\\triangle\\theta, so we can create a data\-set and train the diffusion model in this space\. This allows for us to transform from△​θ\\triangle\\thetaspace towwspace by multiplying by the sum of input point gradients\. Also, assuming thatXXis the same for every data point \(i\.e\. every image has the same size, and is sampled at the same points\), then we only need to computeKθSK\_\{\\theta\_\{S\}\}and its inverse once\.

MetaQuill Method\. The Algorithm’s Method \(For MetaQuill Functa\) is as such,

1. 1\.Train INRs for the training set with shared/per\-image parameters given by Eq\.[14](https://arxiv.org/html/2609.03117#A5.E14)\. Additionally, addL2L\_\{2\}penalties△​θi\\triangle\\theta\_\{i\}so that the shared feature map does most of the work\. The goal of this is to obtainθS\\theta\_\{S\}\. This can be done with only a subset of the data\.
2. 2\.ComputeKθSK\_\{\\theta\_\{S\}\}using theθS\\theta\_\{S\}recovered from step 1\.
3. 3\.Compute per\-imagewiw\_\{i\}for the rest of the training\-set\.
4. 4\.Train a diffusion model on thosewiw\_\{i\}s \. To generate novel models and hence images or scenes, we transform back to weight\-space for fast forward passes \(so that we could sample a lot of data points without compute more finite\-NTKs which are slow\),
5. 5\.Synthesize a neww^\\hat\{w\}using the diffusion model\.
6. 6\.Compute△​θi=∑w^ip,xpw^ip⋅∇θSfθS​\(xp\)\\triangle\\theta\_\{i\}=\\sum\_\{\\hat\{w\}\_\{i\}^\{p\},\\,x^\{p\}\}\\hat\{w\}\_\{i\}^\{p\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x^\{p\}\)to transform from kernel space to weight space \(Eq\.[15](https://arxiv.org/html/2609.03117#A5.E15)\)\.
7. 7\.Sample new points \(e\.g\. for up\-resolution\) usingN​Fn​e​w​\(x\)=△​θiT^⋅∇θSfθS​\(x\)NF\_\{new\}\(x\)=\\hat\{\\triangle\\theta\_\{i\}^\{T\}\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x\)\(Eq\.[13](https://arxiv.org/html/2609.03117#A5.E13)\)\.

Formally,

Algorithm 1MetaQuill\.TRAIN

I​N​RθSINR\_\{\\theta\_\{S\}\}
θS←Li=∑xp,yp∈X,Yi12⋅\(△​θiT⋅∇θSfθS​\(xp\)−yip\)2\+‖△​θi‖2\\theta\_\{S\}\\leftarrow\{L\}\_\{i\}=\\sum\_\{x^\{p\},\\,y^\{p\}\\,\\in\\,X,\\,Y\_\{i\}\}\\frac\{1\}\{2\}\\cdot\(\\triangle\\theta\_\{i\}^\{T\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x^\{p\}\)\-y\_\{i\}^\{p\}\)^\{2\}\\,\+\\,\\left\\lVert\\triangle\\theta\_\{i\}\\right\\rVert^\{2\}

KθS←N​T​KF​i​n​i​t​e​\(θS\)K\_\{\\theta\_\{S\}\}\\leftarrow NTK\_\{Finite\}\(\\theta\_\{S\}\)

wi=KθS−1⋅yiw\_\{i\}=K\_\{\\theta\_\{S\}\}^\{\-1\}\\cdot y\_\{i\}

Generate new NFs

w^←D​M​I​n​v​e​r​s​e\\hat\{w\}\\leftarrow DM\\,Inverse

Transform from Kernel to weight space

△​θi=∑w^ip,xpw^ip⋅∇θSfθS​\(xp\)\\triangle\\theta\_\{i\}=\\sum\_\{\\hat\{w\}\_\{i\}^\{p\},\\,x^\{p\}\}\\hat\{w\}\_\{i\}^\{p\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x^\{p\}\)

sample a new NF

N​Fn​e​w​\(x\)=△​θiT^⋅∇θSfθS​\(x\)NF\_\{new\}\(x\)=\\hat\{\\triangle\\theta\_\{i\}^\{T\}\}\\cdot\\nabla\_\{\\theta\_\{S\}\}f\_\{\\theta\_\{S\}\}\(x\)

##### Results

When considering the following novel algorithm for our Neural Field’s Inpainting task, the practical experiment as follows,

Training a shared NF for a single image \(See Fig\.[9](https://arxiv.org/html/2609.03117#S3.F9), top row \- digit 5\)\. When training a Multi\-NF on a subset of the training dataset \(100 images in our example\), one shared NF \(N​FSNF\_\{S\}\) learning bothθS\\theta\_\{S\}and△​θi\\triangle\\theta\_\{i\}simultaneously for 100 images \(See Fig\.[9](https://arxiv.org/html/2609.03117#S3.F9), bottom section\)\.

After achieving the shared NF base parameters\(θS\)\(\\theta\_\{S\}\), achieved via Eq\.[14](https://arxiv.org/html/2609.03117#A5.E14)\. We train a new NF from the shared obtainedN​FSNF\_\{S\}model, creatingN​FiNF\_\{i\}\(For random imageii\) to obtain△​θi\\triangle\\theta\_\{i\}\(See Fig\.[10](https://arxiv.org/html/2609.03117#S3.F10), top section \- digit 7\)\.

We can see that so far, the algorithm is working excellently, yielding high accuracy results, all with efficient calculations, run\-time, and resources \(only 100 images of the original samples dataset\)\. Furthermore, as explained in Appendix[E\-A2](https://arxiv.org/html/2609.03117#A5.SS1.SSS2.Px1)\-MetaQuill Core Algorithm, we can notice that the algorithm achieves similar MAML meta\-learning method, yet, without its \(MAML’s\) inner loop\. It makes MAML inner loop process redundant, which is a huge leap\. This leads to a great efficiency in run\-time space\.

For the next phases \(After calculatingKθSK\_\{\\theta\_\{S\}\}\) of calculating thewiw\_\{i\}vector \(considering its correspondingYiY\_\{i\}sample image\), we’re using Finite\-width NTK algorithm\. The results can be seen in Fig\.[10](https://arxiv.org/html/2609.03117#S3.F10), bottom section\.

We can see that in contrary of the infinite width and time NTK’sWiW\_\{i\}vector calculation \(where there’s no feature learning and the parameters of the model aren’t taken under consideration\), when using the Finite\-NTK with Feature Learning, theWiW\_\{i\}vector loses representation power\. When exploring why is that happening, it was discovered that the Kernel matrix \(KθSK\_\{\\theta\_\{S\}\}calculated via Finite\-NTK on NFs\) is ill\-conditioned \(with high condition number\)\. This means, such a matrix is almost singular, and the computation of its inverse, or solution of a linear system of equations is prone to numerical errors\. Thus, leading to blurry reconstructed images\.

From the conducted experiments we found that though our novel suggested algorithm achieved efficient feature\-learning and the desired results\. Due to NTK’s insufficient representational nature, even in the Finite\-Feature Learning NTK approach, when applying the NTK method \(In order to use it as a multi\-model NF replacement\. See Section[II\-C](https://arxiv.org/html/2609.03117#S2.SS3)\), the kernel matrix suffers from high condition number \(Being close to an Ill\-condition matrix\. Thus, approximating it to singularity\)\. Therefore, leading to a blurry and inaccurate image reconstruction\. Hence, we derive that while NTK could be used as a mean for approximately mimicking NN Model’s evolution, at its core it lacks the ability to truly distillate the features which were learned in the training phase of the model and grasping the distribution of the dataset needed for the different desired tasks \(It lacks the ambient spanning base representation of NN mapping functions space \- Here, trained shared Neural Field model space\)\. A major factor affecting this, is NTK’s Representational Linearity \(Linearity of theWWvector\. See Section[II\-A](https://arxiv.org/html/2609.03117#S2.SS1)\-NTK Representational Linearity\), which theN​F​K​I​PNF\\,KIPalgorithm \(See Section[III\-A](https://arxiv.org/html/2609.03117#S3.SS1)\) addresses and solves\. Therefore, we believe that a combined algorithm of the two novel suggested algorithms\(N​F​K​I​P,a​n​d​M​e​t​a​Q​u​i​l​l\)\(NF\\,KIP,\\,and\\,MetaQuill\)can potentially achieve groundbreaking results, gaining NTK Learning capabilities, and Non\-linear NTK representation mapping, all while achieving Efficiency, and Meta\-learning\.

To coclude,NTK Algorithm which suggests a new Efficient meta\-learning approach\.This algorithm tackles a major issue regarding the NTK method, NTK Learning and Feature Extraction\. Thus, giving the NTK method an actual feature learning capabilities\. All while suggesting a new efficient Meta\-Learning approach which is similar to MAML\[[24](https://arxiv.org/html/2609.03117#bib.bib34)\], yet, without the inner loop phase\. We found this very promising as we managed to train a shared NF model with only a subsection of the data \(100 images\), and adapt it to new samples straight from the first gradient update\.

The above algorithm though looks promising, still suffers from the NTK’s Representational Linearity problem, and from an Ill\-condition matrix \(With relatively high condition number\), resulting in blurry reconstruction images\. Thus, we believe a unified algorithm which combines the above two novel approaches, can potentially achieve a new novel NTK method which benefits from Learning capabilities, Non\-linear representation, and a much more Efficient core representation\.

We hope these findings not only advance understanding in NTK\-based representation learning but also inspire the research community to develop more robust theoretical frameworks and innovative methods\. By addressing key limitations and opening new avenues for exploration, we aim to spur future work that further strengthens and expands the theoretical foundations of representation learning\.

#### E\-A3MetaQuill\-KIP Algorithm

MetaQuill\-KIP represents each taskiiby a shared initializationθS\\theta\_\{S\}and a compact task specific offsetΔ​θi\\Delta\\theta\_\{i\}, so thatθi=θS\+Δ​θi\\theta\_\{i\}=\\theta\_\{S\}\+\\Delta\\theta\_\{i\}\. At test time, instead of training a full INR from scratch, we optimize onlyΔ​θi\\Delta\\theta\_\{i\}while keepingθS\\theta\_\{S\}fixed\.

##### Tangent \(NTK linearized\) prediction

Given coordinatesxx\(Fourier positional encodings in our experiments\), the linearized prediction aroundθS\\theta\_\{S\}is

y^tan​\(x\)=JθS​\(x\)​Δ​θi,\\hat\{y\}\_\{\\text\{tan\}\}\(x\)=J\_\{\\theta\_\{S\}\}\(x\)\\,\\Delta\\theta\_\{i\},whereJθS​\(x\)J\_\{\\theta\_\{S\}\}\(x\)is the Jacobian of the INR output with respect to parameters atθS\\theta\_\{S\}\. This is exactly the tangent output used in our implementation\.

##### KIP warm start forΔ​θi\\Delta\\theta\_\{i\}

To obtain an informative initialization forΔ​θi\\Delta\\theta\_\{i\}, we run a KIP style inducing point optimization on a single target image\. We optimize a compact support set of coordinatesXSX\_\{S\}\(and optionally pseudo labelsYSY\_\{S\}\) so that a kernel ridge regression solve on that distilled support predicts the target well\. For inpainting, the KIP objective is restricted to observed pixels only, matching the inpainting protocol\. This stage yields a task specific offsetΔ​θi\(0\)\\Delta\\theta\_\{i\}^\{\(0\)\}that is already structure aligned, but still limited by the linearized NTK regime\.

##### Nonlinear refinement aroundθS\\theta\_\{S\}

Starting fromΔ​θi\(0\)\\Delta\\theta\_\{i\}^\{\(0\)\}, we refine onlyΔ​θi\\Delta\\theta\_\{i\}using the full nonlinear model output

y^nonlin​\(x\)=fθS\+Δ​θi​\(x\),\\hat\{y\}\_\{\\text\{nonlin\}\}\(x\)=f\_\{\\theta\_\{S\}\+\\Delta\\theta\_\{i\}\}\(x\),and optimize a masked reconstruction loss on the observed pixels \(plus a mildℓ2\\ell\_\{2\}regularizer onΔ​θi\\Delta\\theta\_\{i\}\)\. This final stage injects true nonlinearity while preserving fast adaptation, since onlyΔ​θi\\Delta\\theta\_\{i\}is updated\.

##### Why robustness matters

In real sensing pipelines, the observed pixels can be corrupted by sensor noise or impulse artifacts\. We therefore evaluate robustness when only the observed pixels are perturbed, while the target ground truth remains clean\. To isolate observation noise from the intrinsic inpainting difficulty, we keep the evaluation target fixed and corrupt only the observed pixel values supplied to each method\. We focus on two common sensing abstractions: additive Gaussian noise and impulse salt and pepper corruptions, both applied only on the observed set while leaving the hole unobserved\.

MetaQuill\-KIP robustness under observed pixel corruptions\.We evaluate robustness of inpainting when the observed pixels themselves are corrupted\. This setting is practically relevant in sensor noise and imperfect measurements, and it complements the clean mask experiments in the main text\. We use random masks with observed fractionϕ∈\{0\.10,0\.20,0\.40\}\\phi\\in\\\{0\.10,0\.20,0\.40\\\}on Flowers, and we apply corruption only on the observed pixels\. For Gaussian corruption we add i\.i\.d\. noiseϵ∼𝒩⁡\(0,σ2\)\\epsilon\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\)to observed pixels only, whereσ∈\{0\.02,0\.05,0\.10\}\\sigma\\in\\\{0\.02,0\.05,0\.10\\\}is measured in the stored Flowers pixel scale\[−0\.5,0\.5\]\[\-0\.5,0\.5\]and values are clipped back to that range\. For salt and pepper corruption we flip an observed pixel to either−0\.5\-0\.5or0\.50\.5with probabilityp∈\{0\.02,0\.05,0\.10\}p\\in\\\{0\.02,0\.05,0\.10\\\}, again applied only on observed pixels\. Qualitative robustness examples are shown in Figure[25](https://arxiv.org/html/2609.03117#A5.F25), with additional snapshots atσ=0\.02\\sigma=0\.02andp=0\.02p=0\.02forϕ∈\{0\.10,0\.20,0\.40\}\\phi\\in\\\{0\.10,0\.20,0\.40\\\}in Figure[26](https://arxiv.org/html/2609.03117#A5.F26)\.

We report PSNR on the full image as well as PSNR restricted to the hole\. Both metrics follow our implementation convention of normalizing the compared region by its own intensity range before computing mean squared error, which can make the full image PSNR and hole PSNR numerically close in some regimes\. We therefore use hole PSNR mainly as a complementary diagnostic and rely on the qualitative panels to assess structural plausibility under heavy sparsity and corruption\.

Table[VIII](https://arxiv.org/html/2609.03117#A5.T8)summarizes results for Single\-INR \(masked\), NTK\-KIP inpaint, and MetaQuill\-KIP \(Combined\(KIP→\\rightarrownl\)\)\. Across most corruption settings, MetaQuill\-KIP degrades gracefully as noise increases and benefits more from higher observed fraction\. In several low coverage cases, quantitative differences are small, while qualitative results indicate that MetaQuill\-KIP preserves more globally coherent structure, consistent with the observation that PSNR alone can be misleading under extreme sparsity\.

Obs\. fracϕ\\phiCorruption on observed pixelsSingle\-INR\(masked\)KIP\-inpaintCombined\(KIP→\\rightarrownl\)0\.10Gaussianσ=0\.02\\sigma=0\.0211\.78/11\.6313\.90/13\.8712\.24/12\.19Gaussianσ=0\.05\\sigma=0\.0511\.96/11\.8012\.93/12\.9112\.35/12\.30Gaussianσ=0\.10\\sigma=0\.1011\.32/11\.1812\.74/12\.7210\.71/10\.68Salt and pepperp=0\.02p=0\.0212\.20/12\.0412\.96/12\.9311\.54/11\.50Salt and pepperp=0\.05p=0\.0511\.81/11\.6412\.82/12\.8010\.50/10\.46Salt and pepperp=0\.10p=0\.1011\.75/11\.5712\.81/12\.7811\.21/11\.160\.20Gaussianσ=0\.02\\sigma=0\.0213\.06/13\.0113\.41/13\.4013\.56/13\.55Gaussianσ=0\.05\\sigma=0\.0512\.56/12\.5113\.65/13\.6412\.86/12\.84Gaussianσ=0\.10\\sigma=0\.1012\.12/12\.0812\.62/12\.6112\.51/12\.49Salt and pepperp=0\.02p=0\.0212\.15/12\.1012\.48/12\.4813\.19/13\.17Salt and pepperp=0\.05p=0\.0512\.55/12\.5013\.18/13\.1713\.18/13\.16Salt and pepperp=0\.10p=0\.1012\.05/12\.0011\.44/11\.4311\.26/11\.240\.40Gaussianσ=0\.02\\sigma=0\.0213\.16/13\.2414\.34/14\.3417\.38/17\.29Gaussianσ=0\.05\\sigma=0\.0513\.35/13\.4215\.01/15\.0116\.56/16\.47Gaussianσ=0\.10\\sigma=0\.1013\.18/13\.2614\.31/14\.3114\.26/14\.21Salt and pepperp=0\.02p=0\.0212\.97/13\.0512\.98/12\.9916\.43/16\.35Salt and pepperp=0\.05p=0\.0512\.86/12\.9514\.14/14\.1415\.76/15\.69Salt and pepperp=0\.10p=0\.1012\.73/12\.8213\.51/13\.5114\.73/14\.67TABLE VIII:Robustness to observed pixel corruptions on Flowers random mask inpainting\. Each entry reports PSNR on the full image and PSNR restricted to the hole as psnr/psnr\_hole\. Corruptions are applied only on observed pixels, while evaluation uses the clean ground truth\.![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/Robustness/combined_quad.png)Fig\. 25:Robust inpainting qualitative overview on Flowers\. From left to right: ground truth, KIP init \(PSNR11\.0811\.08dB\),θi\\theta\_\{i\}tangent only \(PSNR27\.5727\.57dB\), and Combined\(KIP→\\rightarrownonlin\) \(PSNR38\.7438\.74dB\)\. Hereθi=θS\+Δ​θi\\theta\_\{i\}=\\theta\_\{S\}\+\\Delta\\theta\_\{i\}and onlyΔ​θi\\Delta\\theta\_\{i\}is adapted whileθS\\theta\_\{S\}is fixed\.Gaussianσ=0\.02\\sigma=0\.02Salt and pepperp=0\.02p=0\.02

ϕ=0\.10\\phi=0\.10

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/Robustness/obs_10_corr_gauss_0p02_inpaint_quad.png)![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/Robustness/obs_10_corr_salt_pepper_0p02_inpaint_quad.png)

ϕ=0\.20\\phi=0\.20

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/Robustness/obs_20_corr_gauss_0p02_inpaint_quad.png)![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/Robustness/obs_20_corr_salt_pepper_0p02_inpaint_quad.png)

ϕ=0\.40\\phi=0\.40

![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/Robustness/obs_40_corr_gauss_0p02_inpaint_quad.png)![Refer to caption](https://arxiv.org/html/2609.03117v1/Figures/MQKIP/MetaQuill_KIP/Robustness/obs_40_corr_salt_pepper_0p02_inpaint_quad.png)

Fig\. 26:Robustness to corruption of observed pixels on Flowers random mask inpainting\. The left column uses Gaussian corruption withσ=0\.02\\sigma=0\.02and the right column uses salt and pepper corruption withp=0\.02p=0\.02, both applied only on observed pixels\. Each panel uses the same internal layout: first row from left to right is ground truth, masked input, Single\-INR \(masked\), and KRR inpaint; second row from left to right is Combined\(KRR→\\rightarrownl\), KIP inpaint, and Combined\(KIP→\\rightarrownl\)\.