Tag
Mustafa Suleyman, CEO of Microsoft AI, predicts $100B AI training runs are coming, saying gigawatts of compute are being assembled by a handful of labs, making compute and test-time scaling a seismic competitive advantage.
Nokia Applied Research and Florida State University researchers present capability scaling-down laws for LLM compression (pruning, quantization, distillation), showing that sharing density-response relations halves the measurements needed to fit pruning predictors while matching fully fitted regressions within 0.020 nats per token on Pythia and OLMo-2 models.
斯坦福大学2026年秋季新课CME 295《Transformers与大语言模型》开放选课，第二讲笔记发布，系统梳理模型家族、缩放定律、MoE、架构优化、解码策略与提示范式等LLM核心主题。
A short commentary (linking to an X post) arguing that AI capabilities did not 'go FOOM' — i.e., intelligence did not explode explosively as some predicted — reflecting on the gap between AI doomer/foom scenarios and observed progress.
NYU's Fall 2026 graduate seminar, “Language Models, Reinforcement Learning, Reasoning,” threads frontier research from Transformers to world models across a 13-week course — covering pretraining scaling laws, RLVR/GRPO post-training, alignment and safety, and real-world engineering practices such as Kimi K3. The instructor, Pavel Izmailov, is from Anthropic and previously contributed to OpenAI's o1 and Superalignment teams.
A new paper reveals "mode-hopping" in LLM pre-training, where models repeatedly toggle between shallow pattern-matching and genuine generalization even as training loss stays flat — suggesting more pre-training does not always guarantee better generalization.
A post recommending Lilian Weng's blog Lil'Log, highlighting her seven-step research writing workflow and summarizing her posts on recursive self-improvement, scaling laws, test-time compute, and reward hacking.
The paper revisits scaling laws for reward optimization in AI alignment, showing that performance scales as Θ(√min{log(M), K}) where M is the amount of preference data and K is the KL-divergence budget, and provides an information-theoretic model plus extensive empirical validation with proxy reward models.
该论文提出两种学习得到的潜空间蛋白质语言——PLL（基于冻结ESM-2的4,096状态上下文字母表）和SLL（改进的GCP-VQVAE Lite结构分词器），并分别在其上预训练自回归Transformer（PLLM和SLLM），在蛋白质序列与结构生成上相比氨基酸级自回归模型表现出更好的计算扩展性、采样质量以及与AlphaFold2相比约1000倍的推理速度。
This paper introduces an exact generate–transform decomposition to analyze how small 7–9B LLM teams scale across eight orchestration architectures, finding that scaling returns are sharply task-dependent — Proposer-Critic wins on arithmetic benchmarks but no architecture dominates across all tasks.
This paper finds that smaller frozen models can generate better negative samples (rejects) for preference distillation than self-generated responses, reducing inference cost while training stronger 7B–72B students on code generation and math reasoning. The authors derive a finite-horizon utility bound for DPO in a linearized feature model to explain why task structure and lower-likelihood candidate selection improve reject quality.
This paper studies how unlabeled, AI-generated web text affects language model pretraining by training 800 models and fitting new scaling laws. It finds AI text helps data-starved models initially but quickly turns harmful for human-heavy budgets, proposes a scaling law with separate benefit and harm terms that outperforms existing laws, and releases the WildAI 83B-token corpus and all models.
研究系统评估了合成数据预预训练（PPT）在 500M 到 7B 参数、最高 100B token 预训练预算下的表现，发现 PPT 的 token 效率增益在大规模下依然存在，但这些增益并非来自先前假设的语法先验，而是源于提升长程检索能力的 PPT 任务。
Introduces Loop Scaling Laws, the first scaling law jointly modeling recurrence (looped transformers) and sparsity (MoE) alongside model size and data, showing looped MoEs match models ~2x larger on reasoning benchmarks at matched compute.
This paper measures the in-context binding capacity of language models, finding that recall scales with parameter count according to a power law and analyzing factors like pretraining recipes and interference.
This paper systematically compares scaling laws for encoder-free and encoder-based multimodal models, revealing that encoder-free architectures underperform at small scales but catch up at larger compute budgets, with the visual encoder's advantage diminishing as scale increases.
This paper studies the scaling properties of on-policy distillation (OPD) across weak-to-strong, same-base, and strong-to-weak teacher-student LLM setups, finding a universal 'useful-transfer' regime where held-out accuracy rises linearly with KL divergence, and fitting power laws showing smaller teachers can transfer better at matched gold scores.
The article introduces two attention mechanisms, CoWindow Attention and MassAlloc Attention, which optimize compute allocation in transformers, achieving significant speedups and reduced training FLOPs in benchmarks.
This paper evaluates the capability and efficiency of large language models using Chain-of-Thought reasoning, finding that capability gains diminish with increasing model size, while efficiency shows little improvement.
Michael Yu shares his detailed implementation and learnings from Stanford's CS336 course on building LLMs from scratch, covering tokenizers, transformers, training systems, and alignment techniques.