I built a new attention mechanism (wave field) — runs 128K context where standard attention OOMs, 80+ tok/s on laptop CPU

Reddit r/LocalLLaMA Papers
attention-mechanism efficient-inference open-source fft wave-field language-model

Summary

A solo researcher introduces Wave Field attention, replacing standard O(N²) dot-product attention with FFT wave convolution, achieving O(N log N) training and O(1) inference per token. Claimed 80+ tok/s on CPU with 128K context and better zero-shot performance than GPT-2 124M.

Hey r/LocalLLaMA — solo researcher here. I built a new attention architecture and want independent testers. Wave Field LLM replaces O(N²) dot-product attention with FFT wave convolution on a field. Training is O(N log N). Inference is O(1) per token — constant speed and memory even as context grows. Important: this is a base completion model, not a chat model. Trained from scratch. No RLHF, no safety tuning, no instruction fine-tune. What I've measured so far: 80+ tok/s on Mac laptop CPU (no GPU for inference) 128K context runs where standard attention OOMs Models from 130M to 1.5B params 130M zero-shot (DCLM CORE) vs GPT-2 124M: Wave Field avg: 46.8% | GPT-2: 26.5% PIQA: 61.7% vs 50.0% ARC Easy: 43.8% vs 25.0% At 32K on H100: 21.8x faster, 5.3x less memory than standard attention. Links: https://github.com/badaramoni/wave-field-llm https://wavefieldlab.com/ Demo: https://www.youtube.com/watch?v=zH7ICaY5iz4 Looking for honest feedback: Benchmark methodology — sound or not? How does quality compare to other ~130M base models? Anything that looks off ? it produces harmful answers Active research, arch v9, patent pending. Happy to answer in comments. If you test it, reply with hardware + tok/s + a sample completion. Bad results welcome.
Original Article

Similar Articles

@tilderesearch: https://x.com/tilderesearch/status/2061771450168889432

X AI KOLs Timeline

Wall Attention generalizes diagonal forget gates to softmax attention, enabling state-of-the-art length extrapolation from 4k to 160k+ context zero-shot and outperforming RoPE and FoX in pretraining. It is released as a drop-in replacement with open-source Triton kernels.

Wall Attention (GitHub Repo)

TLDR AI

Wall Attention is a new attention variant with per-channel, per-timestep multiplicative decay, providing content-dependent forgetting rates and efficient training/decode kernels implemented in Triton.

I released a softmax-free attention model at GPT-2 Medium scale (~354M params, 11.5B tokens): structural sparsity + tile-skipping kernels for long-context VRAM savings. Open weights + custom Triton kernels [R]

Reddit r/MachineLearning

Released RRT-355M, a softmax-free attention model at GPT-2 Medium scale with 354M parameters trained from scratch on 11.5B tokens, using structural sparsity and tile-skipping kernels for long-context efficiency, achieving comparable performance to GPT-2 Medium on a 22-task benchmark.

@VukRosic99: Long-context Transformers hit two walls: quadratic attention compute and a KV cache that reaches hundreds of GB at 1M t…

X AI KOLs Timeline

MiniCPM-SALA is a 9B-parameter hybrid attention model that interleaves sparse and linear attention to overcome the quadratic compute and large KV cache bottlenecks of long-context Transformers. It achieves 3.5x faster inference than Qwen3-8B at 256K tokens and supports up to 1M tokens on consumer GPUs, with a cost-effective continual training approach that reduces training costs by ~75%.