mlp

Prof. Tom Yeh shares an interactive diagram for learning about autoencoders, part of his 'AI by Hand' series focused on multi-layer perceptrons.

The author compressed the Bad Apple animation into a small MLP with sine activations (SIREN), using only 790k parameters (~3MB) to represent 2.7 billion pixels, with improvements like time-stretching and motion-focused sampling reducing validation MSE by ~9x.

This paper evaluates Kolmogorov–Arnold Networks (KANs) versus MLPs as residual branches in hard-constrained recurrent physics-informed networks on an embedded RISC-V platform, finding KANs run slower, consume more energy, and are less dependable under INT8 quantization.

Hazy Research showed that an MLP can be initialized with embedded knowledge without training, and a transformer can then query and use that knowledge, hinting at continual learning capabilities.

This paper presents an audited protocol using orthogonal Procrustes alignment to detect donor-specific functional fingerprints in neural networks after Neural Collapse, demonstrating that distinct functional variations remain distinguishable even after convergence on MNIST with MLP-5 networks.

This paper presents a theoretical account of how MLPs in Transformers store facts at an information-theoretically optimal rate, and provides a closed-form construction that achieves optimal storage capacity with far fewer parameters than prior methods, enabling modular fact editing.

This paper presents a rigorous theoretical framework for adversarial robustness in multilayered perceptrons by reducing the problem to lattice traversal, introducing both sound and complete interval certifications with formal guarantees.

Talos-XII is a CLI simulator for Arknights: Endfield's gacha system, built entirely in Rust with a custom autograd engine and small RL/MLP stack (no external ML frameworks). It uses neural networks for environment modeling and pull-decision policy, and includes sophisticated SIMD dispatch and an open experiment called ACHF for adaptive caching.

This paper shows that the geometric symmetry visible from neural network weights depends on the positional encoding and readout observable, and validates this using MLPs trained on 2D signed distance functions with multiple symmetry groups.

Aurora is a leverage-aware spectral optimizer that addresses neuron death in MLP layers by enforcing row uniformity while preserving the polar factor geometry of Muon updates, achieving state-of-the-art performance on the modded-nanoGPT speedrun benchmark.

Announces an arXiv note on a mathematical symmetry connecting classic MLP to Gated MLP, going beyond empirical performance.

This paper introduces Tapered Language Models (TLMs), an architecture principle that allocates more parameters to earlier layers and fewer to later layers, consistently improving perplexity and downstream performance across multiple architectures without extra cost.

Introduces High-Res Neural Cellular Automata that operates on a coarse lattice and uses a Local Pattern Producing Network to generate high-resolution outputs, enabling efficient procedural generation.

Announces Part 2 of a profiling tutorial covering linear layer tracing, gemm epilogues, MLP tracing, and comparisons of torch compile vs Liger kernels, with a link to the full content.

The article introduces a technique that extracts hidden states from an LLM at the last prompt token to perform classification without text generation, using a small MLP to read the model's internal decision, enabling fast and cheap zero-shot classifiers.

This blog post continues the profiling in PyTorch series, exploring nn.Linear, MLP blocks, and fusion techniques using Triton kernels to optimize performance.

This paper systematically explores hybrid KAN and MLP architectures for IMU-based human activity recognition, achieving a 5.33% average macro F1 improvement over pure MLP baselines.

Nous Research released Contrastive Neuron Attribution (CNA), a method to steer LLM behavior by identifying and ablating sparse circuits in MLP neurons without training sparse autoencoders or degrading general benchmarks, validated on multiple large language models.

NousResearch releases Contrastive Neuron Attribution (CNA), a method to steer LLM behavior by ablating sparse MLP circuits without training autoencoders or degrading benchmarks, validated on refusal circuits across models up to 70B parameters.

This paper identifies a circuit underlying a language-switching backdoor in an 8B-parameter language model, where a three-word Latin trigger redirects English output to French via attention heads and orthogonal latent subspaces, with the final layer MLP converting the latent signal to French logits.