Scalable Perturbation Learning for Online Self-Supervised Echo State Networks

# Scalable Perturbation Learning for Online Self-Supervised Echo State Networks
Source: [https://arxiv.org/html/2607.06079](https://arxiv.org/html/2607.06079)
###### Abstract

Intelligent systems should not only solve tasks but also adapt under real\-world constraints\. Autonomous adaptation via self\-supervised learning, sequential adaptation via online learning, and memory\-efficient implementation via perturbation\-based learning are important requirements for such systems\. However, these requirements are generally in tension for high\-dimensional systems, because perturbation\-based learning suffers from variance that grows with the dimension of the perturbed variables\.

In this study, we focus on echo state networks \(ESNs\), where this tension naturally arises in large reservoirs\. We propose a perturbation\-based learning rule for online self\-supervised learning in ESNs\. The proposed rule is derived from an orthogonal decomposition of the self\-supervised learning cost, which separates an input\-dependent component from a redundant component determined by the fixed ESN parameters\. By perturbing only the input\-dependent component, the effective perturbation dimension is reduced from the reservoir dimension to the input dimension\.

Thus, the proposed method preserves self\-supervised adaptation, online learning, and scalar\-feedback perturbation learning, while avoiding reservoir\-size\-dependent variance growth\. This suggests a design principle for scalable and hardware\-compatible learning: online learning should be restricted to the dynamically necessary low\-dimensional component of the objective\.

\\affiliation

\[utokyo\] organization=Graduate School of Information Science and Technology, The University of Tokyo, addressline=7\-3\-1 Hongo, city=Bunkyo\-ku, postcode=113\-8656, state=Tokyo, country=Japan

## 1Introduction

Building intelligent systems that can continuously adapt to complex real\-world dynamics is an important goal in machine learning and neuromorphic engineering\[[16](https://arxiv.org/html/2607.06079#bib.bib9),[15](https://arxiv.org/html/2607.06079#bib.bib10)\]\. Such systems must process temporal information under limited computational and memory resources\[[14](https://arxiv.org/html/2607.06079#bib.bib4),[6](https://arxiv.org/html/2607.06079#bib.bib8)\]\. Reservoir computing follows this direction by using a recurrent dynamical system as a temporal feature extractor and reducing the trainable part of the model\[[13](https://arxiv.org/html/2607.06079#bib.bib11),[22](https://arxiv.org/html/2607.06079#bib.bib5)\]\. Echo state networks\[jaeger2001echo\]are a representative model of reservoir computing, in which the reservoir is a fixed recurrent neural network\.

Realizing such continuously adapting systems with echo state networks requires satisfying three conditions\. First, learning should be online and self\-supervised, so that the system can adapt without external target labels in unpredictable environments\[[16](https://arxiv.org/html/2607.06079#bib.bib9)\]\. Second, the reservoir dimension should be scalable, because large reservoirs are often used to represent high\-dimensional dynamics and long temporal dependencies\[[8](https://arxiv.org/html/2607.06079#bib.bib12),[2](https://arxiv.org/html/2607.06079#bib.bib13)\]\. Third, this scaling should not rely on large auxiliary memory or complex error routing\[[6](https://arxiv.org/html/2607.06079#bib.bib8),[10](https://arxiv.org/html/2607.06079#bib.bib15),[11](https://arxiv.org/html/2607.06079#bib.bib14)\]\. This requirement is especially important for hardware implementations that exploit reservoir dynamics directly, rather than relying on general\-purpose computation and full memory access\.

These requirements are in tension\. To explain this point, we denote the input, reservoir, and output dimensions byni​nn\_\{in\},nrn\_\{r\}, andno​u​tn\_\{out\}, respectively\. Although echo state networks can be applied to various temporal tasks by supervised readout learning, only specific tasks admit a self\-supervised formulation\. In this paper, we consider the task studied in our prior work\[[25](https://arxiv.org/html/2607.06079#bib.bib6)\], where the system learns to recover the external input from the reservoir dynamics without external teaching signals\. For this task,no​u​t=ni​nn\_\{out\}=n\_\{in\}\. Our prior work reformulated this problem not as the training of a usualni​n×nrn\_\{in\}\\times n\_\{r\}output map, but as the training of annr×nrn\_\{r\}\\times n\_\{r\}map that reconstructs the reservoir state itself\. As a result of consideration of self\-supervised formulation, the reconstruction target and the reconstruction error both lie in thenrn\_\{r\}\-dimensional reservoir space\.

Batch least squares\[Legendre1805,Gauss1809\]is not suitable for this setting, because it is not online and requires storing data and solving a global inverse problem after data collection\[jaeger2001echo,[7](https://arxiv.org/html/2607.06079#bib.bib16),[13](https://arxiv.org/html/2607.06079#bib.bib11)\]\. Recursive least squares\[[18](https://arxiv.org/html/2607.06079#bib.bib19)\]makes the update online, but it maintains a precision matrix of sizeO​\(nr2\)O\(n\_\{r\}^\{2\}\), which gives a quadratic memory cost\[[21](https://arxiv.org/html/2607.06079#bib.bib17),[13](https://arxiv.org/html/2607.06079#bib.bib11)\]\. Stochastic gradient descent\[[20](https://arxiv.org/html/2607.06079#bib.bib20),[1](https://arxiv.org/html/2607.06079#bib.bib21)\]can be viewed as a sample\-wise online alternative that avoids this quadratic state\. However, it still requires annrn\_\{r\}\-dimensional error signal at each time step in the self\-supervised learning setting\. Thus, even when the memory cost is reduced, learning still requiresO​\(nr\)O\(n\_\{r\}\)error buffers and feedback channels to provide distinct error signals to the reconstruction units\.

Perturbation\-based learning\[[9](https://arxiv.org/html/2607.06079#bib.bib22)\]provides a possible way to avoid such unit\-wise error feedback, but its scalar feedback introduces a variance scaling problem\[[23](https://arxiv.org/html/2607.06079#bib.bib25),[19](https://arxiv.org/html/2607.06079#bib.bib28)\]\. In weight perturbation\[[4](https://arxiv.org/html/2607.06079#bib.bib23)\]and node perturbation\[[3](https://arxiv.org/html/2607.06079#bib.bib24)\], parameter updates are estimated from random perturbations and a scalar change in the objective function\. Thus, thenrn\_\{r\}\-dimensional reconstruction error is compressed into a single global scalar signal\. This removes the need for multi\-channel error routing and replaces it with a scalar feedback signal that can be broadcast globally\[[4](https://arxiv.org/html/2607.06079#bib.bib23),[3](https://arxiv.org/html/2607.06079#bib.bib24)\]\. However, the same compression makes the variance of the gradient estimate increase with network size\[[23](https://arxiv.org/html/2607.06079#bib.bib25),[19](https://arxiv.org/html/2607.06079#bib.bib28)\]\. In the presentnr×nrn\_\{r\}\\times n\_\{r\}learning problem, the variance scales asO​\(nr2\)O\(n\_\{r\}^\{2\}\)for weight perturbation and asO​\(nr\)O\(n\_\{r\}\)for node perturbation\. Thus, asnrn\_\{r\}increases, the signal\-to\-noise ratio of the estimate decreases, making perturbation\-based learning inefficient for high\-dimensional reservoirs\.

In this paper, we address this variance scaling problem by using the mathematical structure of online self\-supervised learning in echo state networks\. We show that the self\-supervised cost function decomposes orthogonally into two terms\. One term is independent of the online input sequence and can be removed by a fixed transformation\. The other term depends on the input\-driven reservoir state sequence and must be learned from online data\. The key observation is that this online term is associated with the external input, whose dimension isni​nn\_\{in\}, rather than with the full reservoir state, whose dimension isnrn\_\{r\}\. We therefore learn only the input\-dependent term online by perturbation\-based learning\. When the learned map is used, we apply the fixed transformation to cancel the offline term\. By applying perturbation\-based learning only to the input\-dependent term, we reduce the variance scaling of the gradient estimate fromO​\(nr\)O\(n\_\{r\}\)toO​\(ni​n\)O\(n\_\{in\}\)\. Therefore, the signal\-to\-noise ratio of the proposed estimate no longer deteriorates with the reservoir dimension\. This property is practically useful becauseni​nn\_\{in\}is usually much smaller thannrn\_\{r\}in large echo state networks\. The proposed learning rule preserves the advantage of scalar global feedback provided by perturbation\-based learning while improving the scalability of online self\-supervised learning\. This makes online self\-supervised learning more compatible with large reservoirs and hardware settings where memory access and feedback routing are limited\.

The remainder of this paper is organized as follows\. Section 2 presents the theoretical formulation of online self\-supervised learning in echo state networks\. Section 2\.1 reviews general online learning rules, namely gradient descent, stochastic gradient descent, weight perturbation, and node perturbation\. Section 2\.2 reviews the self\-supervised formulation of echo state networks introduced in prior work, on which this study is based\. Section 2\.3 presents our main contribution, namely the orthogonal cost decomposition and the resulting perturbation\-based learning rule\. Section 3 verifies the theoretically predicted scaling behavior through numerical experiments and compares the proposed method with weight perturbation and node perturbation\. Section 4 discusses its implications and limitations\. Section 5 concludes the paper\.

## 2Theoretical Formulation

Throughout this section,⟨⋅⟩u\\langle\\cdot\\rangle\_\{u\}denotes the expectation with respect to the distribution of a random variableuu\. For symmetric positive definite matricesGGandHH, we define the induced norms

‖x‖G2\\displaystyle\\\|x\\\|\_\{G\}^\{2\}:=x⊤​G​x,\\displaystyle:=x^\{\\top\}Gx,‖X‖G,H2\\displaystyle\\\|X\\\|\_\{G,H\}^\{2\}:=tr​\(G​X​H​X⊤\),\\displaystyle:=\\mathrm\{tr\}\\\!\\left\(GXHX^\{\\top\}\\right\),and write‖x‖:=‖x‖I\\\|x\\\|:=\\\|x\\\|\_\{I\}and‖X‖:=‖X‖I,I\\\|X\\\|:=\\\|X\\\|\_\{I,I\}for the standard Euclidean and Frobenius norms\.

### 2\.1Learning algorithms

We recall a general formalization of online learning rules\. As a reference, gradient descent\[Cauchy1847\]minimizes the loss using the full\-batch gradient; its online counterparts include stochastic gradient descent\[[20](https://arxiv.org/html/2607.06079#bib.bib20)\], weight perturbation\[[4](https://arxiv.org/html/2607.06079#bib.bib23)\]and node perturbation\[[3](https://arxiv.org/html/2607.06079#bib.bib24)\]\. Consider a system with a parameter matrixWWto be trained and inputzz\. Letℓ​\(W;z\)∈ℝ\\ell\(W;z\)\\in\\mathbb\{R\}denote the instantaneous loss for an inputzz, and define the expected objective by

L​\(W\)=⟨ℓ​\(W;z\)⟩z\.L\(W\)=\\langle\\ell\(W;z\)\\rangle\_\{z\}\.\(1\)We consider the problem of findingWWthat minimizes the expected objectiveL​\(W\)L\(W\)\.

The gradient descent algorithm \(GD\)\[Cauchy1847\]for the objective functionLLis defined as

ΔGD​W=−η​∂L​\(W\)∂W=−η​⟨∂ℓ​\(W;z\)∂W⟩z,\\displaystyle\\Delta^\{\\mathrm\{GD\}\}W=\-\\eta\\frac\{\\partial L\(W\)\}\{\\partial W\}=\-\\eta\\left\\langle\\frac\{\\partial\\ell\(W;z\)\}\{\\partial W\}\\right\\rangle\_\{z\},\(2\)whereη\>0\\eta\>0is a learning rate\.

If the true distribution ofzzis unknown, the expectation⟨⋅⟩z\\langle\\cdot\\rangle\_\{z\}in Eq\. \([2](https://arxiv.org/html/2607.06079#S2.E2)\) is not directly calculable\. The stochastic gradient descent \(SGD\)\[[20](https://arxiv.org/html/2607.06079#bib.bib20)\]approximates GD by replacing the expectation with an empirical average\. In particular, for a given input samplezz, SGD for the objective functionL​\(W\)=⟨ℓ​\(W;z\)⟩zL\(W\)=\\langle\\ell\(W;z\)\\rangle\_\{z\}is defined as

ΔSGD​W=−η​∂ℓ​\(W;z\)∂W\.\\displaystyle\\Delta^\{\\mathrm\{SGD\}\}W=\-\\eta\\frac\{\\partial\\ell\(W;z\)\}\{\\partial W\}\.\(3\)SGD is equivalent to GD on average:

⟨ΔSGD​W⟩z=ΔGD​W\.\\displaystyle\\left\\langle\\Delta^\{\\mathrm\{SGD\}\}W\\right\\rangle\_\{z\}=\\Delta^\{\\mathrm\{GD\}\}W\.\(4\)Thus, SGD provides a valid learning algorithm for minimizing the objective function when the learning rateη\>0\\eta\>0is sufficiently small to suppress the variance of SGD, which is inherited from the randomness ofzz\.

Weight perturbation and node perturbation are zeroth\-order methods that avoid explicit calculation of derivatives by using numerical differentiation\. Specifically, weight perturbation \(WP\)\[[4](https://arxiv.org/html/2607.06079#bib.bib23)\]approximates the derivative∂ℓ​\(W;z\)∂W\\frac\{\\partial\\ell\(W;z\)\}\{\\partial W\}in SGD as follows:

ΔWP​W=−η​ℓ​\(W\+α​Ξ;z\)−ℓ​\(W;z\)α​Ξ,\\displaystyle\\Delta^\{\\mathrm\{WP\}\}W=\-\\eta\\frac\{\\ell\(W\+\\alpha\\Xi;z\)\-\\ell\(W;z\)\}\{\\alpha\}\\Xi,\(5\)whereΞ\\Xiis a perturbation matrix with the same size asWW, whose elements satisfy

⟨Ξi​j​Ξk​l⟩Ξ=δi​k​δj​l,\\left\\langle\\Xi\_\{ij\}\\Xi\_\{kl\}\\right\\rangle\_\{\\Xi\}=\\delta\_\{ik\}\\delta\_\{jl\},\(6\)andα\>0\\alpha\>0is the scale parameter of the perturbation\. For sufficiently smallα\\alpha, WP approximates SGD on average as follows:

⟨ΔWP​W⟩Ξ\\displaystyle\\left\\langle\\Delta^\{\\mathrm\{WP\}\}W\\right\\rangle\_\{\\Xi\}=⟨−η​ℓ​\(W\+α​Ξ;z\)−ℓ​\(W;z\)α​Ξ⟩Ξ\\displaystyle=\\left\\langle\-\\eta\\frac\{\\ell\(W\+\\alpha\\Xi;z\)\-\\ell\(W;z\)\}\{\\alpha\}\\Xi\\right\\rangle\_\{\\Xi\}\(7\)≈−η​⟨\(∑i​j∂ℓ​\(W;z\)∂Wi​j​Ξi​j\)​Ξ⟩Ξ\\displaystyle\\approx\-\\eta\\left\\langle\\left\(\\sum\_\{ij\}\\frac\{\\partial\\ell\(W;z\)\}\{\\partial W\_\{ij\}\}\\Xi\_\{ij\}\\right\)\\Xi\\right\\rangle\_\{\\Xi\}\(8\)=−η​∂ℓ​\(W;z\)∂W\\displaystyle=\-\\eta\\frac\{\\partial\\ell\(W;z\)\}\{\\partial W\}\(9\)=ΔSGD​W\.\\displaystyle=\\Delta^\{\\mathrm\{SGD\}\}W\.\(10\)Thus, WP provides a valid learning algorithm for minimizing the objective function when the learning rateη\>0\\eta\>0is sufficiently small to suppress the variance of WP, which is inherited from the randomnessΞ\\Xi, in addition to that inherited fromzz\.

Node perturbation \(NP\)\[[3](https://arxiv.org/html/2607.06079#bib.bib24)\]follows a similar principle to WP, but uses lower\-dimensional perturbations by exploiting the structure of the derivative to be approximated\. NP assumes that the instantaneous lossℓ​\(W;z\)\\ell\(W;z\)has the form

ℓ​\(W;z\)=ℓ​\(y\),y=W​z\.\\ell\(W;z\)=\\ell\(y\),\\qquad y=Wz\.\(11\)This assumption naturally arises whenWWrepresents network connections and the outputy=W​zy=Wz, rather thanWWitself, is directly evaluated\. Under this assumption, the derivative in SGD has the following structure:

ΔSGD​W=−η​∂ℓ​\(W;z\)∂W=−η​∂ℓ​\(y\)∂y​z⊤\.\\displaystyle\\Delta^\{\\mathrm\{SGD\}\}W=\-\\eta\\frac\{\\partial\\ell\(W;z\)\}\{\\partial W\}=\-\\eta\\frac\{\\partial\\ell\(y\)\}\{\\partial y\}z^\{\\top\}\.\(12\)NP approximates the derivative∂ℓ​\(y\)∂y\\frac\{\\partial\\ell\(y\)\}\{\\partial y\}in SGD as follows:

ΔNP​W=−η​ℓ​\(y\+α​ξ\)−ℓ​\(y\)α​ξ​z⊤,\\displaystyle\\Delta^\{\\mathrm\{NP\}\}W=\-\\eta\\frac\{\\ell\(y\+\\alpha\\xi\)\-\\ell\(y\)\}\{\\alpha\}\\xi z^\{\\top\},\(13\)whereξ\\xiis a perturbation vector with the same size asy=W​zy=Wz, whose elements satisfy

⟨ξi​ξj⟩ξ=δi​j\.\\left\\langle\\xi\_\{i\}\\xi\_\{j\}\\right\\rangle\_\{\\xi\}=\\delta\_\{ij\}\.\(14\)By the same argument as Eq\. \([7](https://arxiv.org/html/2607.06079#S2.E7)\) for WP, NP approximates SGD on average for sufficiently smallα\\alpha\. Thus, NP provides a valid learning algorithm for minimizing the objective function when the learning rateη\>0\\eta\>0is sufficiently small to suppress the variance of NP, which is inherited from the randomness ofξ\\xi, in addition to that inherited fromzz\. In particular, NP can have lower variance than WP when the dimension of the perturbation vectorξ\\xiis smaller than the number of entries of the perturbation matrixΞ\\Xi\. This can be understood as NP achieving a lower\-variance estimate of SGD by exploiting the structural knowledge: the loss evaluates the outputy=W​zy=Wz, rather than the parameter matrixWWitself\.

### 2\.2Self\-Supervised Formulation of Echo State Networks

An echo state network \(ESN\)\[jaeger2001echo\]consists of an inputst∈ℝnins\_\{t\}\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{in\}\}\}, a reservoir statert∈ℝnrr\_\{t\}\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{r\}\}\}, and an outputot∈ℝnouto\_\{t\}\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{out\}\}\}, wherettdenotes discrete time\. The ESN dynamics are given by

rt\+1\\displaystyle r\_\{t\+1\}=σ​\(Win​st\+Wr​rt\),\\displaystyle=\\sigma\(W^\{\\mathrm\{in\}\}s\_\{t\}\+W^\{\\mathrm\{r\}\}r\_\{t\}\),\(15\)ot\\displaystyle o\_\{t\}=Wout​rt,\\displaystyle=W^\{\\mathrm\{out\}\}r\_\{t\},\(16\)whereσ:ℝnr→ℝnr\\sigma:\\mathbb\{R\}^\{n\_\{r\}\}\\to\\mathbb\{R\}^\{n\_\{r\}\}is a nonlinear activation function, andWin∈ℝnr×ninW^\{\\mathrm\{in\}\}\\in\\mathbb\{R\}^\{n\_\{r\}\\times n\_\{\\mathrm\{in\}\}\},Wr∈ℝnr×nrW^\{\\mathrm\{r\}\}\\in\\mathbb\{R\}^\{n\_\{r\}\\times n\_\{r\}\}, andWout∈ℝnout×nrW^\{\\mathrm\{out\}\}\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{out\}\}\\times n\_\{r\}\}are the input, reservoir, and output connection matrices, respectively\.WinW^\{\\mathrm\{in\}\}andWrW^\{\\mathrm\{r\}\}are randomly generated and fixed\. In the standard ESN setting\[jaeger2001echo\], onlyWoutW^\{\\mathrm\{out\}\}is trained so that the ESN outputoto\_\{t\}approximates a desired outputot∗∈ℝnouto\_\{t\}^\{\*\}\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{out\}\}\}, which is usually provided externally\. The optimal readout minimizing the squared loss is given by

W∗out=⟨ot∗​rt⊤⟩ot,rt​⟨rt​rt⊤⟩rt−1\.\\displaystyle W^\{\\mathrm\{out\}\}\_\{\*\}=\\langle o\_\{t\}^\{\*\}r\_\{t\}^\{\\top\}\\rangle\_\{o\_\{t\},r\_\{t\}\}\\langle r\_\{t\}r\_\{t\}^\{\\top\}\\rangle\_\{r\_\{t\}\}^\{\-1\}\.\(17\)Since this solution explicitly depends on the desired outputot∗o\_\{t\}^\{\*\}, this formulation is supervised learning\. Additional treatment is needed if the ESN is required to learn autonomously without explicit external target signals\.

In the special case where the desired output is the input,

ot∗=st,\\displaystyle o\_\{t\}^\{\*\}=s\_\{t\},\(18\)a self\-supervised formulation is possible\[[25](https://arxiv.org/html/2607.06079#bib.bib6)\]\. We introduce an artificial readouto^t∈ℝnr\\hat\{o\}\_\{t\}\\in\\mathbb\{R\}^\{n\_\{r\}\}defined by

o^t=Wdyn​rt,\\displaystyle\\hat\{o\}\_\{t\}=W^\{\\mathrm\{dyn\}\}r\_\{t\},\(19\)whereWdyn∈ℝnr×nrW^\{\\mathrm\{dyn\}\}\\in\\mathbb\{R\}^\{n\_\{r\}\\times n\_\{r\}\}is an artificial readout matrix\. We assume that the activation functionσ\\sigmais invertible, so thatσ−1\\sigma^\{\-1\}exists\. The artificial outputo^t\\hat\{o\}\_\{t\}is trained to approximateσ−1​\(rt\+1\)\\sigma^\{\-1\}\(r\_\{t\+1\}\), which is determined internally by the ESN dynamics\. The optimal solution minimizing the squared loss is

W∗dyn=⟨σ−1​\(rt\+1\)​rt⊤⟩rt,rt\+1​⟨rt​rt⊤⟩rt−1\.\\displaystyle W^\{\\mathrm\{dyn\}\}\_\{\*\}=\\langle\\sigma^\{\-1\}\(r\_\{t\+1\}\)r\_\{t\}^\{\\top\}\\rangle\_\{r\_\{t\},r\_\{t\+1\}\}\\langle r\_\{t\}r\_\{t\}^\{\\top\}\\rangle\_\{r\_\{t\}\}^\{\-1\}\.\(20\)This solution requires only reservoir states and does not explicitly use the inputsts\_\{t\}as the desired output\. Thus, the training ofWdynW^\{\\mathrm\{dyn\}\}can be regarded as self\-supervised learning, withσ−1​\(rt\+1\)\\sigma^\{\-1\}\(r\_\{t\+1\}\)providing an internally available target\.

The self\-supervised learning ofWdynW^\{\\mathrm\{dyn\}\}is related to supervised learning withot∗=sto\_\{t\}^\{\*\}=s\_\{t\}as follows\. Define the transformation𝒫\\mathcal\{P\}by

𝒫​\(Wout\):=Win​Wout\+Wr\.\\displaystyle\\mathcal\{P\}\(W^\{\\mathrm\{out\}\}\):=W^\{\\mathrm\{in\}\}W^\{\\mathrm\{out\}\}\+W^\{\\mathrm\{r\}\}\.\(21\)SinceWinW^\{\\mathrm\{in\}\}andWrW^\{\\mathrm\{r\}\}are randomly generated and fixed, the transformation𝒫\\mathcal\{P\}is static; that is, it can be applied without knowing the dynamic variablessts\_\{t\},rtr\_\{t\}, andot∗o\_\{t\}^\{\*\}\. Using the definitions of the optimal solutions \([17](https://arxiv.org/html/2607.06079#S2.E17)\) and \([20](https://arxiv.org/html/2607.06079#S2.E20)\), together with the ESN dynamics \([15](https://arxiv.org/html/2607.06079#S2.E15)\), we obtain

W∗dyn=𝒫​\(W∗out\)\.\\displaystyle W^\{\\mathrm\{dyn\}\}\_\{\*\}=\\mathcal\{P\}\(W^\{\\mathrm\{out\}\}\_\{\*\}\)\.\(22\)Hence, the optimal self\-supervised readoutW∗dynW^\{\\mathrm\{dyn\}\}\_\{\*\}is obtained from the optimal supervised readoutW∗outW^\{\\mathrm\{out\}\}\_\{\*\}by applying the static transformation𝒫\\mathcal\{P\}\.

Conversely, suppose thatWinW^\{\\mathrm\{in\}\}has full column rank\. Then\(Win\)\+​Win=Inin\(W^\{\\mathrm\{in\}\}\)^\{\+\}W^\{\\mathrm\{in\}\}=I\_\{n\_\{\\mathrm\{in\}\}\}, where\(Win\)\+\(W^\{\\mathrm\{in\}\}\)^\{\+\}denotes the Moore–Penrose inverse ofWinW^\{\\mathrm\{in\}\}\. Thus, the left inverse𝒬\\mathcal\{Q\}of the transformation𝒫\\mathcal\{P\}exists and is defined by

𝒬​\(Wdyn\):=\(Win\)\+​\(Wdyn−Wr\)\.\\displaystyle\\mathcal\{Q\}\(W^\{\\mathrm\{dyn\}\}\):=\(W^\{\\mathrm\{in\}\}\)^\{\+\}\(W^\{\\mathrm\{dyn\}\}\-W^\{\\mathrm\{r\}\}\)\.\(23\)The transformation𝒬\\mathcal\{Q\}is static, as is𝒫\\mathcal\{P\}, and satisfies𝒬∘𝒫=id\\mathcal\{Q\}\\circ\\mathcal\{P\}=\\mathrm\{id\}\. Thus, applying the transformation𝒬\\mathcal\{Q\}to both sides of \([22](https://arxiv.org/html/2607.06079#S2.E22)\) yields

𝒬​\(W∗dyn\)=W∗out\.\\displaystyle\\mathcal\{Q\}\(W^\{\\mathrm\{dyn\}\}\_\{\*\}\)=W^\{\\mathrm\{out\}\}\_\{\*\}\.\(24\)Therefore, supervised learning ofWoutW^\{\\mathrm\{out\}\}in ESNs withsts\_\{t\}as the desired outputot∗o\_\{t\}^\{\*\}can be reformulated as self\-supervised learning ofWdynW^\{\\mathrm\{dyn\}\}, followed by the static transformation𝒬\\mathcal\{Q\}\. In this paper, we focus on the self\-supervised learning ofWdynW^\{\\mathrm\{dyn\}\}instead of the supervised learning ofWoutW^\{\\mathrm\{out\}\}\.

Online learning rules forWdynW^\{\\mathrm\{dyn\}\}are derived from minimization algorithms for the cost functionLsSL​\(Wdyn\)=⟨ℓsSL​\(Wdyn;rt,rt\+1\)⟩rt,rt\+1L^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\}\)=\\langle\\ell^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)\\rangle\_\{r\_\{t\},r\_\{t\+1\}\}, where

ℓsSL​\(Wdyn;rt,rt\+1\)=12​‖Wdyn​rt−σ−1​\(rt\+1\)‖2\.\\displaystyle\\ell^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)=\\frac\{1\}\{2\}\\\|W^\{\\mathrm\{dyn\}\}r\_\{t\}\-\\sigma^\{\-1\}\(r\_\{t\+1\}\)\\\|^\{2\}\.\(25\)Notice that this cost function does not explicitly depend on the supervised signalot∗=sto\_\{t\}^\{\*\}=s\_\{t\}, and neither do the corresponding online learning rules\. As reviewed in Section[2\.1](https://arxiv.org/html/2607.06079#S2.SS1), the online learning rules SGD, WP, and NP are derived as follows:

ΔSGD​Wdyn=−η​∂ℓsSL​\(Wdyn;rt,rt\+1\)∂Wdyn=−η​∂ℓsSL​\(yt;rt\+1\)∂yt​rt⊤,\\displaystyle\\Delta^\{\\mathrm\{SGD\}\}W^\{\\mathrm\{dyn\}\}=\-\\eta\\frac\{\\partial\\ell^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)\}\{\\partial W^\{\\mathrm\{dyn\}\}\}=\-\\eta\\frac\{\\partial\\ell^\{\\mathrm\{sSL\}\}\(y\_\{t\};r\_\{t\+1\}\)\}\{\\partial y\_\{t\}\}r\_\{t\}^\{\\top\},\(26\)ΔWP​Wdyn=−η​ℓsSL​\(Wdyn\+α​Ξ;rt,rt\+1\)−ℓsSL​\(Wdyn;rt,rt\+1\)α​Ξ,\\displaystyle\\Delta^\{\\mathrm\{WP\}\}W^\{\\mathrm\{dyn\}\}=\-\\eta\\frac\{\\ell^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\}\+\\alpha\\Xi;r\_\{t\},r\_\{t\+1\}\)\-\\ell^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)\}\{\\alpha\}\\Xi,\(27\)ΔNP​Wdyn=−η​ℓsSL​\(yt\+α​ξ;rt\+1\)−ℓsSL​\(yt;rt\+1\)α​ξ​rt⊤\.\\displaystyle\\Delta^\{\\mathrm\{NP\}\}W^\{\\mathrm\{dyn\}\}=\-\\eta\\frac\{\\ell^\{\\mathrm\{sSL\}\}\(y\_\{t\}\+\\alpha\\xi;r\_\{t\+1\}\)\-\\ell^\{\\mathrm\{sSL\}\}\(y\_\{t\};r\_\{t\+1\}\)\}\{\\alpha\}\\xi r\_\{t\}^\{\\top\}\.\(28\)Here we abuse notation by writingℓsSL​\(Wdyn;rt,rt\+1\)=ℓsSL​\(yt;rt\+1\)\\ell^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)=\\ell^\{\\mathrm\{sSL\}\}\(y\_\{t\};r\_\{t\+1\}\), whereyt=Wdyn​rty\_\{t\}=W^\{\\mathrm\{dyn\}\}r\_\{t\}\. As we saw in Section[2\.1](https://arxiv.org/html/2607.06079#S2.SS1), SGD, WP, and NP have practical limitations as the reservoir sizenr=dimrtn\_\{r\}=\\dim r\_\{t\}scales, although they are theoretically valid in principle\. Specifically, SGD requires handling the error vector∂ℓsSL​\(yt;rt\+1\)/∂yt\\partial\\ell^\{\\mathrm\{sSL\}\}\(y\_\{t\};r\_\{t\+1\}\)/\\partial y\_\{t\}, whose size isnrn\_\{r\}, and therefore requiresO​\(nr\)O\(n\_\{r\}\)memory and routing costs\. Although perturbation\-based methods WP and NP alleviate these costs by representing the error feedback as a scalar, they instead suffer from a variance problem because they introduce artificial perturbationsΞ\\Xiandξ\\xiof sizesnr2n\_\{r\}^\{2\}andnrn\_\{r\}, respectively\. Consequently, these online learning algorithms are not suitable for implementation on edge devices with large reservoirs\.

Therefore, the remaining question is whether one can retain scalar error feedback while reducing the effective perturbation dimension fromnrn\_\{r\}to a smaller dimension\. In the next section, we show that the orthogonal structure ofLsSLL^\{\\mathrm\{sSL\}\}allows such a reduction\.

### 2\.3Orthogonal Cost Decomposition and Learning Algorithm

We found that the self\-supervised costℓsSL​\(Wdyn;rt,rt\+1\)\\ell^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)admits the following orthogonal decomposition:

ℓsSL​\(Wdyn;rt,rt\+1\)\\displaystyle\\ell^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)=ℓ~sSL​\(Wdyn;rt,rt\+1\)\+12​‖Δ​\(Wdyn\)‖I,rt​rt⊤2,\\displaystyle=\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)\+\\frac\{1\}\{2\}\\\|\\Delta\(W^\{\\mathrm\{dyn\}\}\)\\\|^\{2\}\_\{I,r\_\{t\}r\_\{t\}^\{\\top\}\},\(29\)where

ℓ~sSL​\(Wdyn;rt,rt\+1\)\\displaystyle\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)=12​‖\(Win\)\+​\[Wdyn​rt−σ−1​\(rt\+1\)\]‖\(Win\)⊤​Win2,\\displaystyle=\\frac\{1\}\{2\}\\left\\\|\(W^\{\\mathrm\{in\}\}\)^\{\+\}\\left\[W^\{\\mathrm\{dyn\}\}r\_\{t\}\-\\sigma^\{\-1\}\(r\_\{t\+1\}\)\\right\]\\right\\\|^\{2\}\_\{\(W^\{\\mathrm\{in\}\}\)^\{\\top\}W^\{\\mathrm\{in\}\}\},\(30\)Δ​\(Wdyn\)\\displaystyle\\Delta\(W^\{\\mathrm\{dyn\}\}\)=\(I−Win​\(Win\)\+\)​\(Wdyn−Wr\)\.\\displaystyle=\\left\(I\-W^\{\\mathrm\{in\}\}\(W^\{\\mathrm\{in\}\}\)^\{\+\}\\right\)\(W^\{\\mathrm\{dyn\}\}\-W^\{r\}\)\.\(31\)The decomposition \([29](https://arxiv.org/html/2607.06079#S2.E29)\) leads to two observations\. First, the first term of \([29](https://arxiv.org/html/2607.06079#S2.E29)\) depends onWdynW^\{\\mathrm\{dyn\}\}only through

y~t=\(Win\)\+​Wdyn​rt∈ℝnin\.\\displaystyle\\tilde\{y\}\_\{t\}=\(W^\{\\mathrm\{in\}\}\)^\{\+\}W^\{\\mathrm\{dyn\}\}r\_\{t\}\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{in\}\}\}\.\(32\)Thus, we may write

ℓ~sSL​\(Wdyn;rt,rt\+1\)=ℓ~sSL​\(y~t;rt\+1\)\.\\displaystyle\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)=\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(\\tilde\{y\}\_\{t\};r\_\{t\+1\}\)\.\(33\)This implies the following structure of the derivative ofℓ~sSL\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}with respect toWd​y​nW^\{dyn\}:

∂ℓsSL~∂Wdyn=\(\(Wi​n\)\+\)⊤​∂ℓsSL~∂y~t​rt⊤\.\\displaystyle\\frac\{\\partial\\tilde\{\\ell^\{\\mathrm\{sSL\}\}\}\}\{\\partial W^\{\\mathrm\{dyn\}\}\}=\(\(W^\{in\}\)^\{\+\}\)^\{\\top\}\\frac\{\\partial\\tilde\{\\ell^\{\\mathrm\{sSL\}\}\}\}\{\\partial\\tilde\{y\}\_\{t\}\}r\_\{t\}^\{\\top\}\.\(34\)By applying an argument analogous to node perturbation, we obtain the following perturbation\-based online learning rule:

ΔOurs​Wdyn=−η​ℓ~sSL​\(y~t\+α​ξ~;rt\+1\)−ℓ~sSL​\(y~t;rt\+1\)α​\(\(Win\)\+\)⊤​ξ~​rt⊤,\\displaystyle\\Delta^\{\\mathrm\{Ours\}\}W^\{\\mathrm\{dyn\}\}=\-\\eta\\frac\{\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(\\tilde\{y\}\_\{t\}\+\\alpha\\tilde\{\\xi\};r\_\{t\+1\}\)\-\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(\\tilde\{y\}\_\{t\};r\_\{t\+1\}\)\}\{\\alpha\}\\bigl\(\(W^\{\\mathrm\{in\}\}\)^\{\+\}\\bigr\)^\{\\top\}\\tilde\{\\xi\}r\_\{t\}^\{\\top\},\(35\)whereξ~∈ℝnin\\tilde\{\\xi\}\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{in\}\}\}satisfies

⟨ξ~i​ξ~j⟩ξ~=δi​j\.\\left\\langle\\tilde\{\\xi\}\_\{i\}\\tilde\{\\xi\}\_\{j\}\\right\\rangle\_\{\\tilde\{\\xi\}\}=\\delta\_\{ij\}\.\(36\)As with WP and NP, the proposed rule approximates SGD with respect toℓ~sSL\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}for sufficiently smallα\\alpha:

⟨ΔOurs​Wdyn⟩ξ~\\displaystyle\\left\\langle\\Delta^\{\\mathrm\{Ours\}\}W^\{\\mathrm\{dyn\}\}\\right\\rangle\_\{\\tilde\{\\xi\}\}≈−η​\(\(Win\)\+\)⊤​∂ℓsSL~∂y~t​rt⊤\.\\displaystyle\\approx\-\\eta\(\(W^\{\\mathrm\{in\}\}\)^\{\+\}\)^\{\\top\}\\frac\{\\partial\\tilde\{\\ell^\{\\mathrm\{sSL\}\}\}\}\{\\partial\\tilde\{y\}\_\{t\}\}r\_\{t\}^\{\\top\}\.\(37\)=−η​∂ℓ~sSL​\(Wdyn;rt,rt\+1\)∂Wdyn\.\\displaystyle=\-\\eta\\frac\{\\partial\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\)\}\{\\partial W^\{\\mathrm\{dyn\}\}\}\.\(38\)Unlike WP and NP, whose perturbation dimensions arenr2n\_\{r\}^\{2\}andnrn\_\{r\}respectively, the proposed rule \([35](https://arxiv.org/html/2607.06079#S2.E35)\) uses a perturbation vector of dimensionninn\_\{\\mathrm\{in\}\}\. Thus, the variance of the gradient estimate scales withninn\_\{\\mathrm\{in\}\}, rather than with the reservoir dimensionnrn\_\{r\}\.

Second, the second term of Eq\. \([29](https://arxiv.org/html/2607.06079#S2.E29)\) can be removed without using the datartr\_\{t\}, and without changing the first term\. Define the static transformation

𝒮​\(Wdyn\)=Win​\(Win\)\+​Wdyn\+\(I−Win​\(Win\)\+\)​Wr\.\\displaystyle\\mathcal\{S\}\(W^\{\\mathrm\{dyn\}\}\)=W^\{\\mathrm\{in\}\}\(W^\{\\mathrm\{in\}\}\)^\{\+\}W^\{\\mathrm\{dyn\}\}\+\\left\(I\-W^\{\\mathrm\{in\}\}\(W^\{\\mathrm\{in\}\}\)^\{\+\}\\right\)W^\{\\mathrm\{r\}\}\.\(39\)The static transformation𝒮\\mathcal\{S\}satisfies

ℓ~sSL​\(S​\(Wdyn\);rt,rt\+1\)\\displaystyle\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(S\(W^\{\\mathrm\{dyn\}\}\);r\_\{t\},r\_\{t\+1\}\)=ℓ~sSL​\(Wdyn;rt,rt\+1\),\\displaystyle=\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(W^\{\\mathrm\{dyn\}\};r\_\{t\},r\_\{t\+1\}\),\(40\)Δ∘𝒮​\(Wdyn\)\\displaystyle\\Delta\\circ\\mathcal\{S\}\(W^\{\\mathrm\{dyn\}\}\)=O\.\\displaystyle=O\.\(41\)Therefore, during online learning from sequentially given reservoir states, the second term of \([29](https://arxiv.org/html/2607.06079#S2.E29)\) can be eliminated offline without affecting the online training progress associated with the first term\. In the proposed method, we therefore updateWdynW^\{\\mathrm\{dyn\}\}using the online rule forℓ~sSL\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}, and apply𝒮\\mathcal\{S\}to the currentWdynW^\{\\mathrm\{dyn\}\}at arbitrary times during learning\.

Consequently, the proposed method provides an online self\-supervised learning rule for ESNs whose error\-transmission pathway size and perturbation dimension do not scale with the reservoir dimension\. This property makes the method suitable for high\-dimensional reservoirs and hardware implementations in edge environments\.

## 3Numerical Examples

We demonstrate the features of the proposed perturbation\-based online learning algorithm for self\-supervised learning in ESNs\. Figure[1](https://arxiv.org/html/2607.06079#S3.F1)shows the online learning process of the proposed method \(Ours\), together with SGD, WP, and NP for comparison\. While SGD, WP, and NP optimize the full costℓsSL\\ell^\{\\mathrm\{sSL\}\}, the proposed method optimizes only its projected componentℓ~sSL\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}\(Figure[1](https://arxiv.org/html/2607.06079#S3.F1)A\)\. This is because the remaining orthogonal component inℓsSL\\ell^\{\\mathrm\{sSL\}\}can be removed offline by applying the transformation𝒮\\mathcal\{S\}toWdynW^\{\\mathrm\{dyn\}\}, without using the reservoir statesrtr\_\{t\}provided sequentially\.

The orthogonal decomposition ofℓsSL\\ell^\{\\mathrm\{sSL\}\}also provides a geometric interpretation of the learning algorithm \(Figure[1](https://arxiv.org/html/2607.06079#S3.F1)B\)\. For visualization, we setnr=3n\_\{r\}=3and plot the trajectory of the first column vector of the3×33\\times 3matrixWdynW^\{\\mathrm\{dyn\}\}\. The proposed method uses gradients of the form given in Eq\.[34](https://arxiv.org/html/2607.06079#S2.E34)\. Since the update direction for each column belongs toIm⁡Win\\operatorname\{Im\}W^\{\\mathrm\{in\}\}, the trajectory ofWdynW^\{\\mathrm\{dyn\}\}is confined to the affine subspace

𝒜​\(W0dyn\):=W0dyn\+ℳin,ℳin:=\{Win​Z∣Z∈ℝnin×nr\}\.\\mathcal\{A\}\(W^\{\\mathrm\{dyn\}\}\_\{0\}\):=W^\{\\mathrm\{dyn\}\}\_\{0\}\+\\mathcal\{M\}\_\{\\mathrm\{in\}\},\\qquad\\mathcal\{M\}\_\{\\mathrm\{in\}\}:=\\\{\\,W^\{\\mathrm\{in\}\}Z\\mid Z\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{in\}\}\\times n\_\{r\}\}\\,\\\}\.HereW0dynW^\{\\mathrm\{dyn\}\}\_\{0\}denotes the initial value ofWdynW^\{\\mathrm\{dyn\}\}, andℳin\\mathcal\{M\}\_\{\\mathrm\{in\}\}is the matrix\-valued extension ofIm⁡Win\\operatorname\{Im\}W^\{\\mathrm\{in\}\}\. WhenW0dyn=OW^\{\\mathrm\{dyn\}\}\_\{0\}=O, this affine subspace reduces to the linear subspaceℳin\\mathcal\{M\}\_\{\\mathrm\{in\}\}\.

Thus, the solution obtained by the proposed method corresponds to the projection of the full solution obtained by SGD, WP, or NP onto𝒜​\(W0dyn\)\\mathcal\{A\}\(W^\{\\mathrm\{dyn\}\}\_\{0\}\)\. In general, this projected solution differs from the full solution because the residual component orthogonal toIm⁡Win\\operatorname\{Im\}W^\{\\mathrm\{in\}\}is not updated online\. If the initial value satisfiesW0dyn=𝒮​\(W0dyn\)W^\{\\mathrm\{dyn\}\}\_\{0\}=\\mathcal\{S\}\(W^\{\\mathrm\{dyn\}\}\_\{0\}\), then this residual component has already been removed, and the proposed trajectory lies in the residual\-free affine subspace\. For a general initial value, the corresponding residual\-free solution can be obtained by applying the static transformation𝒮\\mathcal\{S\}to the learnedWdynW^\{\\mathrm\{dyn\}\}\.

![Refer to caption](https://arxiv.org/html/2607.06079v1/x1.png)Figure 1:Features of the proposed perturbation\-based online learning algorithm\. \(A\) Online learning process measured by the decomposed reconstruction errors,RMSE1\+RMSE2\\mathrm\{RMSE\}\_\{1\}\+\\mathrm\{RMSE\}\_\{2\}, for SGD, weight perturbation \(WP\), node perturbation \(NP\), and the proposed method \(Ours\)\. SGD, WP, and NP optimize the full self\-supervised costℓsSL\\ell^\{\\mathrm\{sSL\}\}, whereas Ours optimizes only the projected componentℓ~sSL\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}; the remaining orthogonal component can be removed offline by the static transformation𝒮\\mathcal\{S\}\. \(B\) Geometric interpretation of the learning trajectories\. For visualization,nr=3n\_\{r\}=3is used, and the trajectory of the first column vector ofWdyn∈ℝ3×3W^\{\\mathrm\{dyn\}\}\\in\\mathbb\{R\}^\{3\\times 3\}is shown\. The proposed update is confined to the affine subspaceW0dyn\+ℳinW^\{\\mathrm\{dyn\}\}\_\{0\}\+\\mathcal\{M\}\_\{\\mathrm\{in\}\}, whereℳin\\mathcal\{M\}\_\{\\mathrm\{in\}\}is the matrix\-valued extension ofIm⁡Win\\operatorname\{Im\}W^\{\\mathrm\{in\}\}, and converges to the projected solution\. The corresponding residual\-free solution can be recovered by applying𝒮\\mathcal\{S\}\.Figure[2](https://arxiv.org/html/2607.06079#S3.F2)evaluates the proposed method \(Ours\) in comparison with SGD, WP, and NP in terms of reservoir\-size dependence, learning\-rate sensitivity, and online learning dynamics\.

As reviewed in Section[2\.1](https://arxiv.org/html/2607.06079#S2.SS1), perturbation\-based learning algorithms, including WP, NP, and Ours, approximate SGD on average by estimating the gradient from scalar\-valued error signals using stochastic perturbations\. These stochastic perturbations introduce additional variance into the update rules\. Therefore, the signal\-to\-noise ratio \(SNR\) of the update direction with respect to the reference gradient is inevitably lower for perturbation\-based learning algorithms than for SGD \(Figure[2](https://arxiv.org/html/2607.06079#S3.F2)A\)\. Here, we focus on how this SNR degradation depends on the reservoir size\.

Since the additional variance scales with the perturbation dimension, the SNRs of WP and NP deteriorate as the reservoir sizenrn\_\{r\}increases \(Figure[2](https://arxiv.org/html/2607.06079#S3.F2)A, blue and green lines\)\. In contrast, the perturbation dimension of Ours scales with the input dimensionninn\_\{\\mathrm\{in\}\}, rather than withnrn\_\{r\}\(Figure[2](https://arxiv.org/html/2607.06079#S3.F2)A, red line\)\. Consequently, the SNR of Ours remains approximately independent ofnrn\_\{r\}, indicating that the proposed method is scalable to large reservoirs\.

If the number of iterations is infinite, an infinitesimal learning rate is optimal in principle\. In practice, however, the number of iterations is finite\. Thus, an excessively small learning rate is inefficient, even though it is stable\. On the other hand, since gradients are locally defined and their online estimates contain errors, an excessively large learning rate can cause divergence of the parameters\. Therefore, for a fixed finite number of iterations, there exists an optimal finite learning rate\. A larger SNR implies that each update is better aligned with the reference gradient, allowing efficient learning with a larger learning rate while avoiding divergence\. The numerical results in Figure[2](https://arxiv.org/html/2607.06079#S3.F2)B show that the proposed method attains a lower RMSE over a broader range of learning rates than WP and NP\. This suggests that reducing the effective perturbation dimension improves the practical stability and efficiency of perturbation\-based online learning\.

The learning curves in Figure[2](https://arxiv.org/html/2607.06079#S3.F2)C, obtained using the learning rates selected from Figure[2](https://arxiv.org/html/2607.06079#S3.F2)B, further confirm the effectiveness of the proposed method for large reservoirs\. Ours converges faster and more stably than WP and NP, while retaining the scalar\-valued error feedback characteristic of perturbation\-based learning\.

![Refer to caption](https://arxiv.org/html/2607.06079v1/x2.png)Figure 2:Performance evaluation of the proposed perturbation\-based online learning algorithm\. \(A\) Signal\-to\-noise ratio \(SNR\) of the update direction as a function of the reservoir sizenrn\_\{r\}\. The SNRs of WP and NP deteriorate asnrn\_\{r\}increases because their perturbation dimensions scale with the reservoir size\. In contrast, Ours maintains a constant SNR up to numerical errors because its perturbation dimension is determined by the input dimensionninn\_\{\\mathrm\{in\}\}, not bynrn\_\{r\}\. \(B\) Learning\-rate dependence of the final RMSE for each algorithm\. The proposed method attains lower RMSE over a broader range of learning rates than WP and NP, indicating improved stability of perturbation\-based online learning\. \(C\) Online learning curves using the optimal learning rates selected from panel B\. Ours converges faster and more stably than WP and NP while retaining scalar\-valued error feedback\.
## 4Discussion

In this study, we considered online learning algorithms for ESNs with two requirements: autonomous adaptation through self\-supervised learning and scalability to large reservoir sizes\. Building on the self\-supervised formulation of ESNs proposed in previous work, we compared three existing online learning rules: SGD, weight perturbation \(WP\), and node perturbation \(NP\)\. We argued that SGD requires the transmission and storage of a high\-dimensional error vector, whose memory and routing costs scale with the reservoir size\. Perturbation\-based methods such as WP and NP avoid this difficulty by representing the error feedback as a scalar loss difference\. However, this scalarization comes at the cost of additional variance in the online gradient estimate, and this variance increases with the dimension of the perturbation\. Since the perturbation dimensions of WP and NP scale with the reservoir size, their gradient estimates become increasingly noisy for large reservoirs, limiting their scalability\. To overcome this tradeoff, we proposed a new perturbation\-based online learning rule that inherits the advantages of both approaches\. Because the proposed method is perturbation\-based, it uses scalar\-valued error feedback and therefore avoids memory and routing costs that scale with the reservoir size\. At the same time, by exploiting the specific structure of self\-supervised learning in ESNs, the perturbation dimension is reduced to the input dimension, rather than the reservoir dimension\. Consequently, the proposed method achieves scalar\-valued error feedback without introducing reservoir\-size\-dependent perturbation variance\. Perturbation\-based learning has attracted attention as one possible approach to local learning that estimates gradients without explicit error backpropagation\[[4](https://arxiv.org/html/2607.06079#bib.bib23),[3](https://arxiv.org/html/2607.06079#bib.bib24)\]\. However, it is known to suffer from the variance problem described above, which prevents it from scaling efficiently to large neural networks\[[23](https://arxiv.org/html/2607.06079#bib.bib25)\]\. The present study may provide a hint for how the low\-dimensional structure of a task can be exploited in the design of learning rules\. A limitation of this study, however, is that this idea is demonstrated only in the specific setting of self\-supervised learning in ESNs\. An important direction for future work is therefore to examine whether similar redundancy structures can be identified and exploited in more general models and tasks\. Such investigations may clarify how task\-specific low\-dimensional structures can be incorporated into perturbation\-based learning rules beyond ESNs\.

The key observation behind the proposed method is the orthogonal decomposition of the self\-supervised cost function\. This decomposition separates the cost into an online component, which depends on reservoir states obtained sequentially through the input\-driven dynamics, and an offline component, which can be computed from the fixed reservoir structure\. By optimizing only the online component during sequential learning and eliminating the offline component through a static transformation, the proposed method clarifies which part of the self\-supervised objective should be allocated online computational resources\. This perspective provides a theoretical guideline for designing scalable online learning algorithms in resource\-limited environments such as edge devices\.

The mechanism behind this decomposition can be understood from the relation between supervised learning and self\-supervised learning in ESNs\. In the input reconstruction task considered in this study, the supervised learning objective is redundantly embedded by the fixed input matrixWinW^\{\\mathrm\{in\}\}\. This embedding is justified by the full column rank assumption onWinW^\{\\mathrm\{in\}\}\. Indeed, ifWinW^\{\\mathrm\{in\}\}has full column rank, then it is injective, admits the left inverse\(Win\)\+\(W^\{\\mathrm\{in\}\}\)^\{\+\}, and\(Win\)⊤​Win\(W^\{\\mathrm\{in\}\}\)^\{\\top\}W^\{\\mathrm\{in\}\}is symmetric positive definite\. Therefore,\(Win\)⊤​Win\(W^\{\\mathrm\{in\}\}\)^\{\\top\}W^\{\\mathrm\{in\}\}defines a valid metric on the input space, and the supervised error can be written equivalently as

‖Wout​rt⏟SL​output−st⏟SL​target‖\(Win\)⊤​Win2\\displaystyle\\\|\\underbrace\{W^\{\\mathrm\{out\}\}r\_\{t\}\}\_\{\\mathrm\{SL\\ output\}\}\-\\underbrace\{s\_\{t\}\}\_\{\\mathrm\{SL\\ target\}\}\\\|^\{2\}\_\{\(W^\{\\mathrm\{in\}\}\)^\{\\top\}W^\{\\mathrm\{in\}\}\}=‖Win​Wout​rt−Win​st‖2\\displaystyle=\\\|W^\{\\mathrm\{in\}\}W^\{\\mathrm\{out\}\}r\_\{t\}\-W^\{\\mathrm\{in\}\}s\_\{t\}\\\|^\{2\}\(42\)=‖\(Win​Wout\+Wr\)​rt−Win​st−Wr​rt‖2\.\\displaystyle=\\\|\(W^\{\\mathrm\{in\}\}W^\{\\mathrm\{out\}\}\+W^\{r\}\)r\_\{t\}\-W^\{\\mathrm\{in\}\}s\_\{t\}\-W^\{r\}r\_\{t\}\\\|^\{2\}\.\(43\)Using the ESN dynamics,

σ−1​\(rt\+1\)=Win​st\+Wr​rt,\\sigma^\{\-1\}\(r\_\{t\+1\}\)=W^\{\\mathrm\{in\}\}s\_\{t\}\+W^\{r\}r\_\{t\},\(44\)and the relationWdyn=Win​Wout\+WrW^\{\\mathrm\{dyn\}\}=W^\{\\mathrm\{in\}\}W^\{\\mathrm\{out\}\}\+W^\{r\}, this redundant supervised objective is identified with the self\-supervised objective

‖\(Win​Wout\+Wr\)​rt−Win​st−Wr​rt‖2=‖Wdyn​rt⏟sSL​output−σ−1​\(rt\+1\)⏟sSL​target‖2\.\\displaystyle\\\|\(W^\{\\mathrm\{in\}\}W^\{\\mathrm\{out\}\}\+W^\{r\}\)r\_\{t\}\-W^\{\\mathrm\{in\}\}s\_\{t\}\-W^\{r\}r\_\{t\}\\\|^\{2\}=\\\|\\underbrace\{W^\{\\mathrm\{dyn\}\}r\_\{t\}\}\_\{\\mathrm\{sSL\\ output\}\}\-\\underbrace\{\\sigma^\{\-1\}\(r\_\{t\+1\}\)\}\_\{\\mathrm\{sSL\\ target\}\}\\\|^\{2\}\.\(45\)Thus, the self\-supervised formulation can be regarded as a redundant representation of the supervised input reconstruction task induced by the injective mapWinW^\{\\mathrm\{in\}\}\.

Because this redundancy is introduced throughWinW^\{\\mathrm\{in\}\}, the self\-supervised error can be decomposed according toIm⁡Win⊕\(Im⁡Win\)⟂\\operatorname\{Im\}W^\{\\mathrm\{in\}\}\\oplus\(\\operatorname\{Im\}W^\{\\mathrm\{in\}\}\)^\{\\perp\}\. LetPinP\_\{\\mathrm\{in\}\}andPin⟂P\_\{\\mathrm\{in\}\}^\{\\perp\}denote the orthogonal projections onto these two subspaces\. Then

‖Wdyn​rt−σ−1​\(rt\+1\)‖2\\displaystyle\\\|W^\{\\mathrm\{dyn\}\}r\_\{t\}\-\\sigma^\{\-1\}\(r\_\{t\+1\}\)\\\|^\{2\}\(46\)=‖Wdyn​rt−σ−1​\(rt\+1\)‖Pin2\+‖Wdyn​rt−σ−1​\(rt\+1\)‖Pin⟂2\.\\displaystyle=\\\|W^\{\\mathrm\{dyn\}\}r\_\{t\}\-\\sigma^\{\-1\}\(r\_\{t\+1\}\)\\\|^\{2\}\_\{P\_\{\\mathrm\{in\}\}\}\+\\\|W^\{\\mathrm\{dyn\}\}r\_\{t\}\-\\sigma^\{\-1\}\(r\_\{t\+1\}\)\\\|^\{2\}\_\{P\_\{\\mathrm\{in\}\}^\{\\perp\}\}\.\(47\)The first term is the component inIm⁡Win\\operatorname\{Im\}W^\{\\mathrm\{in\}\}, which contains the contribution of the external inputsts\_\{t\}\. The second term is the orthogonal component\. SinceWin​st∈Im⁡WinW^\{\\mathrm\{in\}\}s\_\{t\}\\in\\operatorname\{Im\}W^\{\\mathrm\{in\}\}, the direct contribution ofsts\_\{t\}vanishes underPin⟂P\_\{\\mathrm\{in\}\}^\{\\perp\}, and hence

‖Wdyn​rt−σ−1​\(rt\+1\)‖Pin⟂2=‖\(Wdyn−Wr\)​rt‖Pin⟂2\.\\displaystyle\\\|W^\{\\mathrm\{dyn\}\}r\_\{t\}\-\\sigma^\{\-1\}\(r\_\{t\+1\}\)\\\|^\{2\}\_\{P\_\{\\mathrm\{in\}\}^\{\\perp\}\}=\\\|\(W^\{\\mathrm\{dyn\}\}\-W^\{r\}\)r\_\{t\}\\\|^\{2\}\_\{P\_\{\\mathrm\{in\}\}^\{\\perp\}\}\.\(48\)Therefore, the self\-supervised cost decomposes as

‖Wdyn​rt−σ−1​\(rt\+1\)‖2=‖Wdyn​rt−σ−1​\(rt\+1\)‖Pin2⏟online component\+‖\(Wdyn−Wr\)​rt‖Pin⟂2⏟offline residual component\.\\displaystyle\\\|W^\{\\mathrm\{dyn\}\}r\_\{t\}\-\\sigma^\{\-1\}\(r\_\{t\+1\}\)\\\|^\{2\}=\\underbrace\{\\\|W^\{\\mathrm\{dyn\}\}r\_\{t\}\-\\sigma^\{\-1\}\(r\_\{t\+1\}\)\\\|^\{2\}\_\{P\_\{\\mathrm\{in\}\}\}\}\_\{\\text\{online component\}\}\+\\underbrace\{\\\|\(W^\{\\mathrm\{dyn\}\}\-W^\{r\}\)r\_\{t\}\\\|^\{2\}\_\{P\_\{\\mathrm\{in\}\}^\{\\perp\}\}\}\_\{\\text\{offline residual component\}\}\.\(49\)The online component corresponds to the redundancy\-reduced part of the self\-supervised objective and is directly related to the original supervised input reconstruction task\. In contrast, the residual component no longer contains the direct input termWin​stW^\{\\mathrm\{in\}\}s\_\{t\}\. It is determined by the mismatch betweenWdynW^\{\\mathrm\{dyn\}\}and the fixed reservoir matrixWrW^\{r\}outsideIm⁡Win\\operatorname\{Im\}W^\{\\mathrm\{in\}\}, and can be removed by the static transformation𝒮\\mathcal\{S\}\.

This observation gives a broader interpretation of the proposed method\. When a supervised task is compatible with the system dynamics, it may be reformulated as a self\-supervised task by introducing a redundant representation through fixed system matrices\. The validity of such a reformulation depends on whether the embedding preserves the original supervised error, asWinW^\{\\mathrm\{in\}\}does here through its full column rank property\. The resulting self\-supervised cost may contain components that are not essential for the original supervised task\. If the introduced redundancy is known, these redundant components can be identified by projection and removed from the online optimization problem\. Thus, self\-supervised reformulation is not merely a way to construct internal targets; it can also reveal how online learning should be restricted to the dynamically necessary component of the objective\. This provides a principle for designing efficient perturbation\-based learning algorithms whose online memory, routing, and perturbation dimensions do not scale with the reservoir size\.

The affine map𝒫\\mathcal\{P\}was originally introduced in\[jaeger2001echo,[12](https://arxiv.org/html/2607.06079#bib.bib3),[17](https://arxiv.org/html/2607.06079#bib.bib2)\]to interpret the learned readoutWoutW^\{\\mathrm\{out\}\}as a learned dynamical system\. Specifically, the supervised learning resultWoutW^\{\\mathrm\{out\}\}defines

f^SL:r↦σ​\[𝒫​\(Wout\)​r\]\.\\displaystyle\\hat\{f\}^\{\\mathrm\{SL\}\}:r\\mapsto\\sigma\\\!\\left\[\\mathcal\{P\}\(W^\{\\mathrm\{out\}\}\)r\\right\]\.\(50\)Previous studies have shown that the learned dynamical systemf^SL\\hat\{f\}^\{\\mathrm\{SL\}\}can mimic the dynamics of the external input,st↦st\+1s\_\{t\}\\mapsto s\_\{t\+1\}\[[12](https://arxiv.org/html/2607.06079#bib.bib3),[17](https://arxiv.org/html/2607.06079#bib.bib2),[5](https://arxiv.org/html/2607.06079#bib.bib1)\]\. In addition, such learned dynamics can be combined with filtering algorithms based on state\-space models for noise filtering\[[24](https://arxiv.org/html/2607.06079#bib.bib27),[25](https://arxiv.org/html/2607.06079#bib.bib6)\]\. These applications are enabled not only by learning the parameterWoutW^\{\\mathrm\{out\}\}, but also by interpreting the learned result through the static transformation𝒫\\mathcal\{P\}\. This suggests that formalizing the static transformation used for interpretation is as important as the parameter learning itself\.

The transformation𝒮\\mathcal\{S\}, which was introduced to eliminate the residual termΔ​\(Wdyn\)\\Delta\(W^\{\\mathrm\{dyn\}\}\)appearing in the orthogonal decomposition of the cost, plays a role parallel to that of𝒫\\mathcal\{P\}forWoutW^\{\\mathrm\{out\}\}\. The parameterWdynW^\{\\mathrm\{dyn\}\}obtained by minimizing the reduced costℓ~sSL\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}preserves the information needed to recover the supervised readout through𝒬\\mathcal\{Q\}\. That is,𝒬​\(Wdyn\)\\mathcal\{Q\}\(W^\{\\mathrm\{dyn\}\}\)gives the corresponding solution of the supervised reconstruction problemWout​rt≈stW^\{\\mathrm\{out\}\}r\_\{t\}\\approx s\_\{t\}\. However, the learned parameterWdynW^\{\\mathrm\{dyn\}\}itself does not generally define the appropriate learned dynamics as

f^sSL:r↦σ​\[Wdyn​r\]\.\\displaystyle\\hat\{f\}^\{\\mathrm\{sSL\}\}:r\\mapsto\\sigma\\\!\\left\[W^\{\\mathrm\{dyn\}\}r\\right\]\.\(51\)This is because the reduced costℓ~sSL\\tilde\{\\ell\}^\{\\mathrm\{sSL\}\}ignores the orthogonal residual componentΔ​\(Wd​y​n\)\\Delta\(W^\{dyn\}\)that is irrelevant to online learning but necessary for interpreting the result as a full reservoir dynamics\. Therefore, the learned parameter should be interpreted through the static transformation𝒮\\mathcal\{S\}\. The corresponding self\-supervised learned dynamics is defined by

f^sSL:r↦σ​\[𝒮​\(Wdyn\)​r\]\.\\displaystyle\\hat\{f\}^\{\\mathrm\{sSL\}\}:r\\mapsto\\sigma\\\!\\left\[\\mathcal\{S\}\(W^\{\\mathrm\{dyn\}\}\)r\\right\]\.\(52\)The transformation𝒮\\mathcal\{S\}can be written as the composition

𝒮=𝒫∘𝒬,\\displaystyle\\mathcal\{S\}=\\mathcal\{P\}\\circ\\mathcal\{Q\},\(53\)where𝒬\\mathcal\{Q\}recovers the corresponding supervised readout fromWdynW^\{\\mathrm\{dyn\}\}, and𝒫\\mathcal\{P\}maps this readout back to the parameter space of reservoir dynamics\. Therefore, the dynamics defined by𝒮​\(Wdyn\)\\mathcal\{S\}\(W^\{\\mathrm\{dyn\}\}\)coincides with the supervised interpretationf^SL\\hat\{f\}^\{\\mathrm\{SL\}\}obtained from the recovered readout𝒬​\(Wdyn\)\\mathcal\{Q\}\(W^\{\\mathrm\{dyn\}\}\)\. Thus, applying𝒮\\mathcal\{S\}to the learned parameter restores the proper dynamical\-system interpretation and allows the self\-supervised learning result to mimic the external input dynamicsst↦st\+1s\_\{t\}\\mapsto s\_\{t\+1\}, in the same sense as the supervised formulation\.

## 5Conclusion

In this study, we demonstrated that online self\-supervised learning in echo state networks admits an orthogonal decomposition of its cost function\. This decomposition separates the objective into two parts: an input\-driven component that must be learned online, and a residual component that can be removed offline by a static transformation determined solely by the fixed reservoir parameters\. Our findings highlight that the effective dimensionality of this online learning problem is governed by the input dimension, not the reservoir dimension\. As a result, scalar\-feedback perturbation learning becomes scalable to large reservoirs without incurring reservoir\-size\-dependent gradient variance\. Accordingly, we propose a design principle for online learning rules: when a task carries a known low\-dimensional structure induced by fixed system parameters, this structure should be actively exploited, restricting online memory, error routing, and perturbation to the dynamically necessary part of the objective\. This perspective reframes self\-supervised reformulation\. It becomes not merely a means of constructing internal targets, but a way of identifying which part of a learning objective genuinely requires online resources\. These results support a broader view of reservoir computing, in which the fixed structure of the reservoir is treated as exploitable prior knowledge rather than a constraint to be circumvented\. More broadly, they point toward local, hardware\-compatible learning rules for resource\-limited neuromorphic systems, and toward an understanding of how biological systems might leverage their own fixed structure to learn efficiently under scalar, globally broadcast feedback\.

## Acknowledgments

This work was partially supported by JSPS KAKENHI Grant Numbers JP20H00596, JP21K12105, JP22K18419, JP24K15161, JP24H02330, JST CREST Grant No\. JPMJCR19K2, JST Moonshot RD Grant No\. JPMJMS2021, Cross\-ministerial Strategic Innovation Promotion Program \(SIP\) on “Integrated Health Care System” Grant No\. JPJ012425\.

## References

- \[1\]L\. Bottou\(2010\)Large\-Scale Machine Learning with Stochastic Gradient Descent\.InProceedings of COMPSTAT’2010,Y\. Lechevallier and G\. Saporta \(Eds\.\),Heidelberg,pp\. 177–186\(en\)\.External Links:ISBN 978\-3\-7908\-2604\-3,[Document](https://dx.doi.org/10.1007/978-3-7908-2604-3%5F16)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p4.3)\.
- \[2\]J\. Dambre, D\. Verstraeten, B\. Schrauwen, and S\. Massar\(2012\-07\)Information Processing Capacity of Dynamical Systems\.Scientific Reports2\(1\),pp\. 514\(en\)\.External Links:ISSN 2045\-2322,[Link](https://www.nature.com/articles/srep00514),[Document](https://dx.doi.org/10.1038/srep00514)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p2.1)\.
- \[3\]I\. R\. Fiete and H\. S\. Seung\(2006\-07\)Gradient learning in spiking neural networks by dynamic perturbation of conductances\.Physical Review Letters97\(4\),pp\. 048104\(eng\)\.External Links:ISSN 0031\-9007,[Document](https://dx.doi.org/10.1103/PhysRevLett.97.048104)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p5.5),[§2\.1](https://arxiv.org/html/2607.06079#S2.SS1.p1.4),[§2\.1](https://arxiv.org/html/2607.06079#S2.SS1.p5.1),[§4](https://arxiv.org/html/2607.06079#S4.p1.1)\.
- \[4\]B\. Flower and M\. Jabri\(1992\)Summed Weight Neuron Perturbation: An O\(N\) Improvement Over Weight Perturbation\.InAdvances in Neural Information Processing Systems,Vol\.5\.External Links:[Link](https://proceedings.neurips.cc/paper/1992/hash/996a7fa078cc36c46d02f9af3bef918b-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p5.5),[§2\.1](https://arxiv.org/html/2607.06079#S2.SS1.p1.4),[§2\.1](https://arxiv.org/html/2607.06079#S2.SS1.p4.1),[§4](https://arxiv.org/html/2607.06079#S4.p1.1)\.
- \[5\]M\. Hara and H\. Kokubu\(2022\-04\)Learning Dynamics by Reservoir Computing \(In Memory of Prof\. Pavol Brunovský\)\.Journal of Dynamics and Differential Equations\(en\)\.External Links:ISSN 1572\-9222,[Link](https://doi.org/10.1007/s10884-022-10159-w),[Document](https://dx.doi.org/10.1007/s10884-022-10159-w)Cited by:[§4](https://arxiv.org/html/2607.06079#S4.p6.7)\.
- \[6\]G\. Indiveri and S\. Liu\(2015\-08\)Memory and Information Processing in Neuromorphic Systems\.Proceedings of the IEEE103\(8\),pp\. 1379–1397\.External Links:ISSN 1558\-2256,[Link](https://ieeexplore.ieee.org/document/7159144),[Document](https://dx.doi.org/10.1109/JPROC.2015.2444094)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p1.1),[§1](https://arxiv.org/html/2607.06079#S1.p2.1)\.
- \[7\]H\. Jaeger and H\. Haas\(2004\-04\)Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication\.Science304\(5667\),pp\. 78–80\.External Links:[Link](https://www.science.org/doi/10.1126/science.1091277),[Document](https://dx.doi.org/10.1126/science.1091277)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p4.3)\.
- \[8\]H\. Jaeger\(2002\)Adaptive Nonlinear System Identification with Echo State Networks\.InAdvances in Neural Information Processing Systems,Vol\.15\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2002/hash/426f990b332ef8193a61cc90516c1245-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p2.1)\.
- \[9\]J\. Kiefer and J\. Wolfowitz\(1952\-09\)Stochastic Estimation of the Maximum of a Regression Function\.The Annals of Mathematical Statistics23\(3\),pp\. 462–466\(en\)\.External Links:ISSN 0003\-4851, 2168\-8990,[Link](https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-23/issue-3/Stochastic-Estimation-of-the-Maximum-of-a-Regression-Function/10.1214/aoms/1177729392.full),[Document](https://dx.doi.org/10.1214/aoms/1177729392)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p5.5)\.
- \[10\]T\. P\. Lillicrap, A\. Santoro, L\. Marris, C\. J\. Akerman, and G\. Hinton\(2020\-06\)Backpropagation and the brain\.Nature Reviews Neuroscience21\(6\),pp\. 335–346\(en\)\.External Links:ISSN 1471\-0048,[Link](https://www.nature.com/articles/s41583-020-0277-3),[Document](https://dx.doi.org/10.1038/s41583-020-0277-3)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p2.1)\.
- \[11\]J\. Lin, L\. Zhu, W\. Chen, W\. Wang, C\. Gan, and S\. Han\(2022\)On\-Device Training Under 256KB Memory\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 22941–22954\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/90c56c77c6df45fc8e556a096b7a2b2e-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p2.1)\.
- \[12\]Z\. Lu, J\. Pathak, B\. Hunt, M\. Girvan, R\. Brockett, and E\. Ott\(2017\-04\)Reservoir observers: Model\-free inference of unmeasured variables in chaotic systems\.Chaos: An Interdisciplinary Journal of Nonlinear Science27\(4\),pp\. 041102\.Note:Number: 4External Links:ISSN 1054\-1500,[Link](https://aip.scitation.org/doi/full/10.1063/1.4979665),[Document](https://dx.doi.org/10.1063/1.4979665)Cited by:[§4](https://arxiv.org/html/2607.06079#S4.p6.3),[§4](https://arxiv.org/html/2607.06079#S4.p6.7)\.
- \[13\]M\. Lukoševičius and H\. Jaeger\(2009\-08\)Reservoir computing approaches to recurrent neural network training\.Computer Science Review3\(3\),pp\. 127–149\.External Links:ISSN 1574\-0137,[Link](https://www.sciencedirect.com/science/article/pii/S1574013709000173),[Document](https://dx.doi.org/10.1016/j.cosrev.2009.03.005)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p1.1),[§1](https://arxiv.org/html/2607.06079#S1.p4.3)\.
- \[14\]W\. Maass, T\. Natschläger, and H\. Markram\(2002\-11\)Real\-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations\.Neural Computation14\(11\),pp\. 2531–2560\(en\)\.External Links:ISSN 0899\-7667, 1530\-888X,[Link](https://direct.mit.edu/neco/article/14/11/2531-2560/6650),[Document](https://dx.doi.org/10.1162/089976602760407955)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p1.1)\.
- \[15\]C\. Mead\(1990\-10\)Neuromorphic electronic systems\.Proceedings of the IEEE78\(10\),pp\. 1629–1636\.External Links:ISSN 1558\-2256,[Link](https://ieeexplore.ieee.org/document/58356),[Document](https://dx.doi.org/10.1109/5.58356)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p1.1)\.
- \[16\]G\. I\. Parisi, R\. Kemker, J\. L\. Part, C\. Kanan, and S\. Wermter\(2019\-05\)Continual lifelong learning with neural networks: A review\.Neural Networks113,pp\. 54–71\.External Links:ISSN 0893\-6080,[Link](https://www.sciencedirect.com/science/article/pii/S0893608019300231),[Document](https://dx.doi.org/10.1016/j.neunet.2019.01.012)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p1.1),[§1](https://arxiv.org/html/2607.06079#S1.p2.1)\.
- \[17\]J\. Pathak, B\. Hunt, M\. Girvan, Z\. Lu, and E\. Ott\(2018\-01\)Model\-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach\.Physical Review Letters120\(2\),pp\. 024102\.Note:Number: 2External Links:[Link](https://link.aps.org/doi/10.1103/PhysRevLett.120.024102),[Document](https://dx.doi.org/10.1103/PhysRevLett.120.024102)Cited by:[§4](https://arxiv.org/html/2607.06079#S4.p6.3),[§4](https://arxiv.org/html/2607.06079#S4.p6.7)\.
- \[18\]R\. L\. Plackett\(1950\)Some Theorems in Least Squares\.Biometrika37\(1/2\),pp\. 149–157\.External Links:ISSN 0006\-3444,[Link](https://www.jstor.org/stable/2332158),[Document](https://dx.doi.org/10.2307/2332158)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p4.3)\.
- \[19\]M\. Ren, S\. Kornblith, R\. Liao, and G\. Hinton\(2022\-09\)Scaling Forward Gradient With Local Losses\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=JxpBP1JM15-)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p5.5)\.
- \[20\]H\. Robbins and S\. Monro\(1951\-09\)A Stochastic Approximation Method\.The Annals of Mathematical Statistics22\(3\),pp\. 400–407\(en\)\.External Links:ISSN 0003\-4851, 2168\-8990,[Link](https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-22/issue-3/A-Stochastic-Approximation-Method/10.1214/aoms/1177729586.full),[Document](https://dx.doi.org/10.1214/aoms/1177729586)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p4.3),[§2\.1](https://arxiv.org/html/2607.06079#S2.SS1.p1.4),[§2\.1](https://arxiv.org/html/2607.06079#S2.SS1.p3.4)\.
- \[21\]D\. Sussillo and L\. F\. Abbott\(2009\-08\)Generating coherent patterns of activity from chaotic neural networks\.Neuron63\(4\),pp\. 544–557\(eng\)\.External Links:ISSN 1097\-4199,[Document](https://dx.doi.org/10.1016/j.neuron.2009.07.018)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p4.3)\.
- \[22\]G\. Tanaka, T\. Yamane, J\. B\. Héroux, R\. Nakane, N\. Kanazawa, S\. Takeda, H\. Numata, D\. Nakano, and A\. Hirose\(2019\-07\)Recent advances in physical reservoir computing: A review\.Neural Networks115,pp\. 100–123\.External Links:ISSN 0893\-6080,[Link](https://www.sciencedirect.com/science/article/pii/S0893608019300784),[Document](https://dx.doi.org/10.1016/j.neunet.2019.03.005)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p1.1)\.
- \[23\]J\. Werfel, X\. Xie, and H\. S\. Seung\(2005\-12\)Learning curves for stochastic gradient descent in linear feedforward networks\.Neural Computation17\(12\),pp\. 2699–2718\(eng\)\.External Links:ISSN 0899\-7667,[Document](https://dx.doi.org/10.1162/089976605774320539)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p5.5),[§4](https://arxiv.org/html/2607.06079#S4.p1.1)\.
- \[24\]T\. Yamada, Y\. Katori, and K\. Fujiwara\(2025\)Numerical Evaluation of a Weakly Supervised Filtering Method Based on Echo State Networks\.Proceedings of the ISCIE International Symposium on Stochastic Systems Theory and its Applications2026,pp\. 109–113\.External Links:[Document](https://dx.doi.org/10.5687/sss.2026.109)Cited by:[§4](https://arxiv.org/html/2607.06079#S4.p6.7)\.
- \[25\]T\. Yamada, Y\. Katori, and K\. Fujiwara\(2026\-01\)Unsupervised Learning in Echo State Networks for Input Reconstruction\.Neural Computation38\(2\),pp\. 198–227\.External Links:ISSN 0899\-7667,[Link](https://doi.org/10.1162/NECO.a.38),[Document](https://dx.doi.org/10.1162/NECO.a.38)Cited by:[§1](https://arxiv.org/html/2607.06079#S1.p3.7),[§2\.2](https://arxiv.org/html/2607.06079#S2.SS2.p2.1),[§4](https://arxiv.org/html/2607.06079#S4.p6.7)\.