Does subquadratic's 12 million context model claim hold any water?
Summary
The video examines whether a claimed 12 million context model from subquadratic research is credible, analyzing its technical underpinnings and potential limitations.
Similar Articles
Do you guys think subquadratic actually has a 12 million context model
Sub Quadratic claims to have a model with a context of 12 million tokens, but access is limited to partners; it performs well in the "needle in a haystack" test, but lacks evidence of general reasoning ability, raising doubts.
@Hesamation: Remember this? 20 days ago SubQ claimed to have developed a model with 12M context window, 95% cheaper than Opus, and t…
SubQ claimed a breakthrough model with a 12M context window and 95% cost reduction vs Opus, but after promising a paper and model card, they have not delivered, raising strong skepticism of a scam or shady behavior.
Subquadratic AI introduces SubQ-1.1-Small, a new model using Smart Sparse Attention
Subquadratic AI introduces SubQ-1.1-Small, a model leveraging Smart Sparse Attention to achieve near-perfect long-context retrieval up to 12M tokens with up to 1,000x attention compute reduction. It balances long-context optimization with strong general reasoning, outperforming baselines on benchmarks like NIAH and RULER.
More Context, Larger Models, or Moral Knowledge? A Systematic Study of Schwartz Value Detection in Political Texts
A systematic study on detecting Schwartz values in political text, comparing context lengths, model sizes, and retrieval-augmented generation methods. Results show that full-document context improves supervised models but not zero-shot LLMs, while retrieved moral knowledge consistently helps via early fusion.
@no_stp_on_snek: https://subq.mildlyconcerning.com
This article critically analyzes the claims and timeline of the subQ long-context AI technique, highlighting discrepancies and walkbacks from the original announcement.