Tag
NVIDIA's Nemotron-3-Super-120B-A12B, a hybrid Mamba and mixture-of-experts model, achieves perfect needle-in-haystack retrieval at 504K tokens using only four RTX 3090 GPUs.
Sub Quadratic claims to have a model with a context of 12 million tokens, but access is limited to partners; it performs well in the "needle in a haystack" test, but lacks evidence of general reasoning ability, raising doubts.