Tag
A talk on moving beyond GPUs and backpropagation: the speaker argues that the co-evolution of hardware and optimizers has led to stagnation in FLOPs per joule, claims that within a decade we will abandon backpropagation, and proposes pairing SPSA zeroth-order optimizers with metamer-based hybrid architectures, while betting on alternative paths such as optical computing and neuromorphic computing.
Y Combinator's Paper Club session explores AI compute beyond the transformer-backprop-GPU stack, covering optical computing with light, neuromorphic chip architectures inspired by the brain, and biological neurons trained with reinforcement learning to play Doom.
This paper introduces DVA-Neurons, an adaptive LIF neuron design with 2nd-order synaptic filtering for improved dynamics, and demonstrates scalable multi-neuron spiking networks with physical verification in 14nm CMOS technology.
Spi-Fly is a fruit fly-inspired neural network that uses sparse activity and associative learning to quickly learn odors with minimal memory, demonstrating strong few-shot performance but requiring further testing.
PC-ALM is a local training method that uses layer-local dynamical systems to propagate supervision credit, enabling the training of up to 1000-layer networks without backpropagation while nearly matching its performance.
The author explores an alternative approach to AI perception using wave superposition instead of neural network weights, seeking insights from researchers in related fields.
Researchers have developed a bio-inspired algorithm called Spi-Fly, which mimics fruit fly olfactory systems using sparse coding to enable fast learning and prevent catastrophic forgetting in electronic noses.
This paper proposes a Sparse-Activation-ReLU (SAR) layer for low-latency, energy-efficient virtual sensing, achieving significant improvements in latency-error-energy metrics and reducing errors through synthetic knowledge distillation.
This paper presents a Bayesian control framework that integrates spike-based dynamics with probabilistic inference for adaptive control in nonlinear dynamical systems, using a spiking neural network model demonstrated on a benchmark problem.
The paper presents SAGE, a method that adapts surrogate gradients for Spiking Transformers using attention-derived entropy to improve training accuracy, demonstrated on CIFAR-10/100 datasets.
The article discusses NSRAM, a new artificial neuron on a silicon chip that aims to dramatically improve energy efficiency in AI by mimicking biological neurons, addressing the high power consumption of GPUs in data centers.
Researchers developed an artificial synapse that uses light-color programming to achieve brain-like balanced learning, potentially advancing neuromorphic computing.
Researchers propose using sound waves and phi-bits to create neuromorphic devices that better mimic biological neurons, potentially enabling faster, more energy-efficient computing for pattern recognition and data analysis.
Explores the vast energy efficiency gap between the human brain (15W) and AI hardware (billions of watts needed for real-time simulation), highlighting neuromorphic computing approaches like spin-memristors, phase-change materials, and Super-Turing AI that aim to close this gap.
Researchers engineered a van der Waals crystal that mimics neuronal cells, enabling light-driven learning, a step toward neuromorphic computing.
Northwestern University researchers have printed artificial neurons from MoS2 and graphene ink that produce biologically realistic electrical spikes, which living mouse brain cells recognized as natural signals, a breakthrough with major implications for energy-efficient neuromorphic computing.
The article discusses the 'AI power wall' where compute growth outpaces efficiency gains, proposing four paradigm shifts—neuromorphic, photonic, memory-centric, and approximate computing—to make AI sustainable, and promotes the upcoming 'Watt Matters in AI' conference addressing full-stack energy reduction.
Proposes Selective Alignment Knowledge Distillation (SeAl-KD) for Spiking Neural Networks, which selectively aligns class-level and temporal knowledge by equalizing competing logits at erroneous timesteps and reweighting temporal alignment based on confidence and inter-timestep similarity, achieving consistent improvements over existing distillation methods on static and neuromorphic datasets.