Tag
作者调侃称AI竞赛最搞笑的结局是：万亿美元级公司砸数千亿开发超人类智能，几年后却有人能在500美元的笔记本上跑出接近的能力，并表示希望这真的发生。
论文提出 Spectral-MoE，一种在稠密模型转 MoE 时为不同层动态分配专家利用率（utilization budget）的方法，包括静态和动态两种分配器，在 Llama-2-7B、Llama-3-8B、Qwen2.5-7B 上显著降低困惑度并提升下游任务准确率，确立了逐层专家利用率这一新的优化方向。
Intern-Decision introduces a family of compact (0.8B/2B/4B) multimodal decision models using an autoregressive masked language modeling objective to output structured choices with probabilities in a single causal forward pass, achieving 90.02% average accuracy across seven benchmarks with low latency for interactive agents.
STAPLE提出一种将静态稀疏记忆集成到预训练模型的方案，通过残差嵌入模块在有限微调下扩展参数容量，在OLMo-2上显著超越MoE上采样基线，同时保持接近稠密模型的吞吐与更低的内存/能耗。
A user praises Qwen 3.8 27B as an underrated, high-performing model, noting that smaller/local models can match bigger ones and that LLMs still struggle with 'pointing'. They plan to test prompt optimization for further gains.
提出ShellElastic，一种将隐藏维度组织为稠密核心与递增外壳的弹性Transformer架构，使单次全宽前向即可同时包含所有子网络的精确隐藏状态；并引入μE扩展最大化更新参数化（μP），保证子网络在超网络加宽或增加外壳时仍被最大化更新且超参数可迁移，性能优于MatFormer等弹性基线并逼近分别训练的模型。
该论文提出 Stepped MoE，一个将弹性结构与稀疏门控路由统一的框架，使单一 LLM 能同时适应部署约束和任务需求，在 1B-4B 参数规模间灵活切换并优于稠密模型 2-5%。
LittleBit 提出了一种通过低秩潜在矩阵分解与二值化实现超低比特（低至0.1 BPW）LLM 量化的框架，配合多尺度补偿机制，在 Llama2-7B 上以 0.1 BPW 超越了现有 0.7 BPW 方法，实现约 31 倍内存压缩与最高 11.6 倍推理加速。
Samsung Labs 释出 LittleBit 项目（NeurIPS 2025 / ICML 2026），通过潜在空间低秩分解与二值化实现大语言模型的亚 1-bit（低至 0.1 bpw）压缩，并在 LittleBit-2 中引入 Joint-ITQ 潜在几何对齐初始化以提升性能。
提出了SparseDecoding，一种面向解码阶段的训练无关剪枝框架，通过在自回归生成时收集校准数据对齐Hessian分布，并开发优化的N:M稀疏矩阵向量核，在A100上实现最高1.48倍端到端解码加速。
A Reddit post questions how Qwen's 27B model achieves performance comparable to or better than GPT-4o, speculating whether better or higher-quality pretraining data or other new techniques explain its efficiency.
Nokia Applied Research and Florida State University researchers present capability scaling-down laws for LLM compression (pruning, quantization, distillation), showing that sharing density-response relations halves the measurements needed to fit pruning predictors while matching fully fitted regressions within 0.020 nats per token on Pythia and OLMo-2 models.
A tweet arguing that people underestimate the intelligence of GLM 5.3 and GLM 5.3 Flash, noting their impressive capability despite requiring a very small computational footprint — no data center needed.
OpenAI is re-opening its Pro $200 subscription with new usage calculations that reduce effective API spend by half, while ensuring increasing value through model efficiency improvements.
Tibo announces changes to the Pro $200 subscription, effectively halving API spend and promising increased value over time, with new model pricing and upcoming features.
The article advises using Claude Sonnet 5.5 at high effort to save costs, while claiming that Opus 5.5 is the most efficient model available.
A preprint introducing Foil, which flattens MoE experts into fewer layers with more experts per layer and more recurrent passes, plus untied attention per pass, achieving lower pretraining loss at matched parameters and compute.
An Epoch AI report indicates that the cost of achieving a specific level of AI performance has dropped sharply, with a 725-fold reduction in inference cost demonstrated by OpenAI o3 and GPT-5.6 Luna on benchmarks like GPQA Diamond.
The paper proposes CS-MoE, a novel Transformer architecture that shares neural experts across layers to improve parameter utilization, achieving lower perplexity with only 55% of parameters activated.
The author questions why tiny AI models with fewer than 50M parameters or swarms of specialized micro-models are rarely deployed in production, speculating on reasons like tooling biases or the convenience of generalist models.