Tag
Russian mathematicians from startup Mostik have developed a method for AI models to communicate via their weights, enhancing efficiency and performance, as demonstrated with GLM-5.2 and Qwen-3.5 models.
The article explains the architectural differences between Mixture of Experts (MoE) and N-gram techniques in AI models, highlighting how Qwen's new model uses N-gram to offload parameters for improved efficiency by separating reasoning and recalling tasks.
This paper introduces a method to predict middle-layer attention in multimodal LLMs to prune visual tokens efficiently, using question-contrastive teacher selection and cross-modal attention distillation.
The author analyzes model size and performance trends following Deepseek V4 Flash, suggesting that open-source models are shrinking in size while improving, and predicts Opus 4.5-level models could run on consumer laptops within a year.
DeepSeek V4 Flash is about 70% smaller than GLM 5.2 yet outperforms it, implying state-of-the-art-level AI could run on consumer hardware like an RTX 5090 much sooner than expected.
A Reddit user asks how DeepSeek can make a 300B parameter model cheaper than a 9B parameter model, sparking discussion on efficiency and cost.
Presents the insight that CPU decode speed depends on active parameters per token, not total parameters, and proposes a 10B-parameter model using ternary weights and granular MoE to achieve high token rates on mid-level PCs. Reports sandbox measurements showing a speedup from 176 to 848 tok/s on an 8.3M model with minimal quality loss.
The article argues that AI model efficiency is improving so rapidly that the hardware needed for a fixed level of intelligence halves roughly every 3 months, making renting frontier intelligence or owning trailing-edge hardware more economical than buying new hardware.
Explores whether a 35B parameter model could surpass a 1000B model by scaling its search horizon at test time, using structured process feedback rather than brute-force parameter scaling.
This paper introduces byte-exact KV-cache grafting, a technique that makes frozen small language models both more capable and cheaper by depositing verified knowledge as a byte-exact state artifact and restoring it during inference, achieving dramatic token and energy reductions without weight changes.
This study introduces section-aware compression for reasoning traces, training models to drop filler narration while preserving compute and verification spans, matching or exceeding original accuracy while using 2-3 times fewer tokens.
This paper introduces PadCaptioner, a 3B parameter model for omni-modal dense video captioning that uses parallelized autoregressive decoding to achieve high efficiency and quality, outperforming 7B counterparts. A latent planning mechanism enables lossless parallel generation by exploiting weak local dependencies among events.
A new post highlights a drawback of diffusion LLMs: bidirectional attention causes keys and values to drift across steps, breaking KV caching. However, generation quality is robust to slight KV drift, and research has focused on maximizing stale KV reuse without quality degradation.
This paper systematically studies variants of QKV projection sharing in transformers, finding that sharing key and value projections (Q-K=V) achieves 50% KV cache reduction with only 3.1% perplexity degradation, and combining with GQA/MQA can reach up to 96.9% cache reduction—enabling practical on-device inference with minimal quality loss.
Complexity-Balanced Splitting (CBS) partitions the diffusion timeline into segments of equal approximation burden using local complexity measures, improving synthesis quality by ~35% in FID without increasing inference cost.
A new method called Zero-Expert Self-Distillation Adaptation (ZEDA) allows MoE models like Qwen3 and GLM to skip half their expert computations on easy tokens with minimal accuracy loss, achieving ~20% inference speedup by adding dummy experts that output nothing.
Q-ARVD is a novel quantization framework to reduce inference costs of autoregressive video diffusion models by addressing frame-wise sensitivity imbalance and weight outlier patterns.
ZEDA is a low-cost framework that converts post-trained static MoE models into dynamic ones by injecting zero-output experts and using self-distillation, achieving over 50% expert FLOP reduction with marginal accuracy loss on benchmarks.
This paper investigates the parameter-level mechanisms behind the efficiency of On-Policy Distillation (OPD) for large language models, attributing it to early 'foresight' in module allocation and update direction. It proposes EffOPD, a plug-and-play method that accelerates OPD training by 3x without compromising final performance.
SlimSpec introduces a low-rank parameterization for drafter LM-heads to accelerate speculative decoding in LLMs, achieving 4-5x speedup while maintaining full vocabulary support.