Tag
This paper proposes Threshold Gating (TG) as a unified primitive for neural nonlinearity, showing that standard activation functions can be expressed as instances of TG. The authors validate their approach by converting pretrained models across various architectures without retraining and discuss hardware benefits.
Luthira fully implements MicroGPT on an FPGA array, without GPU, PyTorch, or CPU inference loop, achieving inference speeds of 50,000+ tokens per second, demonstrating the potential of hardware-based Transformers.