Tag
TriPLU introduces direct trilinear product feed-forward networks for tiny decoder-only language models, showing improved validation loss over SwiGLU in low-compute regimes.
A reproducibility study reveals asymmetric effects when removing residual connections in Pre-LN transformers: attention-skip removal leads to collapse, while FFN-skip removal allows partial recovery at smaller scales.
The paper introduces DUD (Decoupled Update Dynamics), a framework that separates Feed-Forward Network and Attention contributions via causal interventions to improve uncertainty quantification and calibration in large language models, outperforming state-of-the-art baselines.
This paper investigates using symbolic regression to discover explicit neural network weight-update rules that outperform standard hand-designed optimizers on small symbolic regression benchmarks, achieving an aggregate MSE reduction of 44.47% in 25 out of 30 benchmark/network combinations.
This paper investigates the signed nature of FFN residual writes in long-context retrieval, finding that FFN writes act as suppressors or amplifiers depending on layer and task, and proposes a gradient-based diagnostic to distinguish these roles.
This paper evaluates Kolmogorov-Arnold Networks (KANs) as interpretable components and replacements for transformer feed-forward networks in small language models, finding that while KANs provide a practical audit interface, they show no consistent benchmark advantage over MLP baselines.