Tag
TriPLU introduces direct trilinear product feed-forward networks for tiny decoder-only language models, showing improved validation loss over SwiGLU in low-compute regimes.
A research paper introducing a multi-stage forward-query method to cryptanalytically extract isolated bias-free GLU feed-forward block weights, demonstrating sub-percent recovery accuracy on Qwen, Llama, and Gemma components while noting end-to-end model API attacks remain unsolved.
Prox is a training-free framework for sparse SwiGLU FFNs in LLMs, using approximate intermediate-channel salience to build channel masks without dense computation. It outperforms training-free baselines across ten LLMs, achieving up to 1.99x end-to-end decoding speedup at 70% sparsity.
Proposes Confidence-Aware SwiGLU (κ-SwiGLU) that adjusts expert gate sharpness in Mixture-of-Experts models based on token-level routing confidence, improving performance with minimal computational overhead.