mlp

This paper presents a theoretical account of how MLPs in Transformers store facts at an information-theoretically optimal rate, and provides a closed-form construction that achieves optimal storage capacity with far fewer parameters than prior methods, enabling modular fact editing.

This paper presents a rigorous theoretical framework for adversarial robustness in multilayered perceptrons by reducing the problem to lattice traversal, introducing both sound and complete interval certifications with formal guarantees.

Talos-XII is a CLI simulator for Arknights: Endfield's gacha system, built entirely in Rust with a custom autograd engine and small RL/MLP stack (no external ML frameworks). It uses neural networks for environment modeling and pull-decision policy, and includes sophisticated SIMD dispatch and an open experiment called ACHF for adaptive caching.

This paper shows that the geometric symmetry visible from neural network weights depends on the positional encoding and readout observable, and validates this using MLPs trained on 2D signed distance functions with multiple symmetry groups.

Aurora is a leverage-aware spectral optimizer that addresses neuron death in MLP layers by enforcing row uniformity while preserving the polar factor geometry of Muon updates, achieving state-of-the-art performance on the modded-nanoGPT speedrun benchmark.

Announces an arXiv note on a mathematical symmetry connecting classic MLP to Gated MLP, going beyond empirical performance.

This paper introduces Tapered Language Models (TLMs), an architecture principle that allocates more parameters to earlier layers and fewer to later layers, consistently improving perplexity and downstream performance across multiple architectures without extra cost.

Introduces High-Res Neural Cellular Automata that operates on a coarse lattice and uses a Local Pattern Producing Network to generate high-resolution outputs, enabling efficient procedural generation.

Announces Part 2 of a profiling tutorial covering linear layer tracing, gemm epilogues, MLP tracing, and comparisons of torch compile vs Liger kernels, with a link to the full content.

The article introduces a technique that extracts hidden states from an LLM at the last prompt token to perform classification without text generation, using a small MLP to read the model's internal decision, enabling fast and cheap zero-shot classifiers.

This blog post continues the profiling in PyTorch series, exploring nn.Linear, MLP blocks, and fusion techniques using Triton kernels to optimize performance.

This paper systematically explores hybrid KAN and MLP architectures for IMU-based human activity recognition, achieving a 5.33% average macro F1 improvement over pure MLP baselines.

Nous Research released Contrastive Neuron Attribution (CNA), a method to steer LLM behavior by identifying and ablating sparse circuits in MLP neurons without training sparse autoencoders or degrading general benchmarks, validated on multiple large language models.

NousResearch releases Contrastive Neuron Attribution (CNA), a method to steer LLM behavior by ablating sparse MLP circuits without training autoencoders or degrading benchmarks, validated on refusal circuits across models up to 70B parameters.

This paper identifies a circuit underlying a language-switching backdoor in an 8B-parameter language model, where a three-word Latin trigger redirects English output to French via attention heads and orthogonal latent subspaces, with the final layer MLP converting the latent signal to French logits.

The article presents a discovered spectral ratio between MLP and attention norms that predicts geometric stability in transformer models, with an optimal range of 0.5–2 to prevent rank collapse.

This paper identifies the 'Massive Emergence Layer' where extreme activations in LLMs originate and propagate, proposing a method to mitigate their rigidity and improve model performance on tasks like math reasoning and instruction following.