The article describes extending the Muse Glimmer 30B model's context to 512k tokens using its unique architecture with RoPE in SWA layers and no position encoding in GQA attention layers, with benchmark results showing high performance up to 512k tokens.
My fun weekend project was to try to make the new Muse Glimmer 30B work with a longer context, deciding to go for 512k first. I had expected the usual YaRN shenanigans and maybe a LoRA. I couldn't have been wrong more. Upon closer look, Glimmer turned out to be rather unusual architecturally. The thing that make long-context adaptations painful in other models, full attention layers with token position encoding, it simply not there. Instead, only 2048 tokens-wide SWA layers have RoPE, and full GQA attention layers have no position encoding at all. It appears the model is trained to work with long-distance token relationships inferred from the context and SWA layers. It's a rather bold architecture bet, but it seems Meta managed to pull it off. As a result, the model architecture appears to be uniquely suited for context extension by simple mechanical means. To change model context length from stock 128k to, say, 512k, you need only to change “max_position_embeddings” config setting from 131072 to 524288. What confuses other models, like Qwen3.5 family, Glimmer just takes into its stride. I spent close to 70h of compute on DGX Spark to test stock model with extended context on a variety of benchmarks and found the following on a full-precision model: Needle in a haystack, both 1 and 4 needles: 100% on up to 512k tokens Multi-hop retrieval: 100% on up to 512k tokens Semantic lookups: 100% on up to 512k tokens Counting instances across whole context: degrades from 95% at 32k to around 60% on 385k, and then drops to 22% on 512k. Memory use in a simulated agentic session: 100% up to 385k, then slight drop. NoLiMa: noisy but seems stable on all context up to 512k. LongBench v2: same performance on up to 512k tokens. LongCodeQA: same performance on up to 512k tokens. I've also tried InfBench suite, but the results were noisy and I'm still trying to refine them. Full research dossier, scripts, and reports are here in the open: https://github.com/lobanov/muse-glimmer-long-ctx
User tests Meta's Muse Glimmer 30B on a 2× DGX Spark cluster, extending context from 131K to 1M tokens with YaRN and confirming passing retrieval at 832K tokens. Reports ~3× speedup from DFlash speculative decoding and shares full config.
User reports that Muse Glimmer, a 30B model, fits on a single RTX 3090 with full 256k context using Q4_K_XL quantization and DFlash, achieving 64-124 tok/s and perfect long-context retrieval, unlike comparable models.
Meta's Muse Glimmer is a 30B multimodal Transformer model designed for autonomous agentic tasks on consumer hardware, using a memory hierarchy and quantization to fit within 24-32 GB envelopes.
Meta introduces Muse Glimmer, a new 30B open-weights model under Apache 2.0, optimized for agentic task completion, reliable tool use, and multi-step reasoning. Simon Willison tests it locally with LM Studio and llm-coding-agent.
Meta releases Muse Glimmer, a 30B open-weight multimodal model optimized for local agent workflows, with permissive Apache 2.0 licensing, 4-bit quantization support, speculative decoding, and broad ecosystem integrations.