backpropagation

ERASE introduces a novel training schedule that detaches subgraphs to overlap backward passes with forward work, improving throughput by up to 9.51% in large-scale recommendation systems while preserving model performance.

The author argues that while tool use allows smaller language models to perform tasks effectively, larger models remain crucial for speed, reliability, and internalized knowledge, emphasizing the ongoing need for scaling in AI.

This paper introduces a biologically motivated neural architecture that respects Dale's constraint—neurons and synapses have fixed sign—while still supporting backpropagation-like learning using non-negative error signals and local Hebbian updates, showing gains on Tiny ImageNet.

A walkthrough of backpropagation by hand through a 3-layer network using matrix multiplication, showing all 11 steps from gradients to weight updates.

本文证明现代深度网络中预测编码（PC）使用的Jacobian转置乘积可以分解为局部可用的项，从而消除了自动求导反向传播的需要。提出的WF-Act-PC方法在CIFAR-10/100和Tiny-ImageNet上匹配甚至超越经过调优的反向传播基线，且性能随深度增加而提升。

This paper traces step-by-step how PyTorch's automatic differentiation engine computes gradients for Physics-Informed Neural Network training, including the two levels of differentiation needed for physics residuals and parameter gradients, using a simple MLP and ODE example.

This tweet lists 7 fundamental math ideas used daily by ML engineers, with brief explanations emphasizing the underlying derivations, such as why gradient descent moves in the steepest direction and why softmax plus cross-entropy yields a clean gradient.

Introduces K-ABENA, a selective gradient computation framework that uses compensated loss-based sample exclusion with unbiased gradient estimation, proving convergence guarantees and showing compute savings of 28-54% without performance degradation across various datasets.

This paper shows that predictive coding networks compute the same gradients as backpropagation in the limit of width much larger than depth, bridging biological learning and standard neural network training.

This paper presents a scalable backpropagation-based algorithm for training deep convolutional networks to run on thermodynamic Ising hardware, achieving 94.9% on CIFAR-10 and 76.0% on CIFAR-100 while analyzing inference cost-accuracy tradeoffs.

A Twitter thread listing 13 core machine learning concepts that interviewers expect candidates to know, covering topics from bias-variance tradeoff to the curse of dimensionality.

Microcrad reimplements Karpathy's micrograd autograd engine in C, providing an educational scalar-valued automatic differentiation library with reference counting and a small neural network, aimed at understanding backpropagation at the scalar level.

This paper proposes a framework for neocortical learning that meets criteria for computational, algorithmic, and implementational plausibility, using error-driven predictive learning via temporal derivatives and corticothalamic circuits. It suggests potential improvements over backpropagation.

This paper diagnoses the loss landscape of gradient-based inversion for the Gray-Scott reaction-diffusion system, showing that direct backpropagation fails due to flat plateaus and sharp cliffs, while PINN components like residual loss smooth the landscape. The findings provide design implications for PINN-type methods.

This paper tracks how different learning rules (backprop, feedback alignment, predictive coding, STDP) affect the alignment of CNN representations with human fMRI across training. It finds that backprop destroys V1 alignment in one epoch, while local rules preserve it, suggesting a trade-off between building higher-level representations and retaining early visual features.

Stanford CS224N course notes provide a clear introduction to the mathematics of backpropagation and gradient computation in neural networks, covering chain rule, computational graphs, and vectorized derivatives.

This paper tracks how supervised training with different learning rules (backpropagation, feedback alignment, predictive coding, STDP) degrades alignment between neural network representations and early visual cortex fMRI data, finding that untrained networks often match or exceed trained ones in V1 alignment.

Sakana AI presents DiffusionBlocks, a method that trains neural networks block-wise by interpreting forward passes as diffusion denoising, significantly reducing memory requirements compared to traditional end-to-end backpropagation.

This thread explains the intuition behind the Jacobian Matrix and its widespread applications in AI and machine learning, including backpropagation, normalizing flows, computer vision, and robotics.

The paper introduces closed-form predictive coding via hierarchical Gaussian filters that restore precision-weighted prediction errors, yielding faster and more efficient training without global error signals, outperforming backpropagation on certain tasks.