Tag
Pathway's BDH, a post-transformer architecture, reportedly matches GPT-2 scaling from 10M to 1B parameters while training from scratch on standard GPUs.
Call for papers for the Scaling H2R workshop at CoRL 2026, focusing on scaling laws and diversity in human-to-robot learning. Submissions due Oct 7, 2026.
A discussion question asking about the theoretically optimal quantization bit-width for LLMs under a fixed memory budget, referencing recent 3-bit/2-bit results and scaling-law work from 2025-2026.
A systematic empirical study showing that concept directions extracted from one language model can steer other independently trained models when sufficient scale (≥1.7B parameters) is reached, providing functional evidence for the Platonic Representation Hypothesis and highlighting scale thresholds for cross-model interpretability tools.
This paper investigates whether small foundation models fine-tuned on human behavioral data can serve as cognitive proxies, finding that scale matters little in-distribution but larger models generalize better out-of-distribution.
The paper introduces the Skaling law, a generalized neural scaling law that couples model capacity and data through an interaction exponent, reducing prediction error by 1.5-3x and enabling full-grid extrapolation using roughly 10x less compute.
This paper studies a self-supervised task for generating single-cell gene expression vectors using an autoregressive transformer with a quantized VAE tokenizer. It reports scaling laws and a compute-optimal frontier for single-cell foundation models, with potential fine-tuning for perturbation prediction.
This paper introduces a Capability-Driven Multimodal Scaling Law that predicts VLM performance from LLM textual benchmark capabilities, enabling principled backbone selection across model families without expensive training sweeps.
A research paper presenting scaling laws and design principles for mixture-of-experts diffusion language models, trained as LLaDA MoE v2 (30B-A3B) on 23.5T tokens, approaching Qwen3 on several benchmarks with fewer pretraining tokens.
The tweet highlights the potential of agent harnesses to capture and distill experts' tacit knowledge as new training data, referencing a Berkeley AI Summit talk by Jianfeng Gao on agentic modeling as an emerging AI paradigm.
This paper demonstrates that scaling laws fit on small transformer models can accurately predict the loss of much larger models trained on particle physics jet data, enabling compute budgets to be translated into expected physics performance before large training runs. They release five pretrained models and the full training recipe.
This paper introduces Chimera, a hybrid visual diffusion backbone with a principled scaling recipe, combining Kimi Delta Attention, Multi-head Latent Attention, and sparse Mixture-of-Experts to efficiently handle long-context image and video generation. It also presents HeteroP, a module-wise hyperparameter transfer scheme, and Chinchilla-style scaling laws to train an 11B-parameter model with 2B activated parameters.
An NYT analysis details the unprecedented global build-out of AI data centers and chips, projecting a tenfold increase in AI computing power by 2028, which is expected to drive major breakthroughs in AI capabilities.
Introduces cMoLLM, a convolutionally gated mixture-of-LLMs that routes over end-to-end streams through fully differentiable dynamic convolution, improving perplexity and downstream accuracy under matched compute.
This paper presents a distributed benchmark study on scaling laws for classical machine learning models on tabular data, showing that power-law fits hold for most model families and quantifying replicator-implementation variance across 127 student runs.
The author reflects on past LLM scaling eras (model size, chain-of-thought, agents) and speculates about the next big scaling dimension, suggesting recursive self-improvement as a potential breakthrough.
This paper investigates the scaling properties of native multimodal pre-training, deriving compute-optimal model sizes and token counts under a fixed budget, and revealing distinct scaling behaviors for language and multimodal objectives.
A new paper from NaceAI proposes a hypernetwork-based method for injecting knowledge into large language models without modifying their core parameters, potentially enabling efficient continual learning. The approach uses generated low-rank adapters to encode new facts while keeping the base model frozen.
This paper discovers that larger language models have a hidden auto-regressive risk regime where they commit to low-probability tokens and then snowball errors, causing reliability to degrade faster with scale. It shows that this failure mode is causal, dominant, and invisible to the model's own self-monitoring.
The paper investigates scaling laws for hypernetwork-based knowledge injection into LLMs, finding predictive power law scaling and reliable out-of-distribution generalization, establishing hypernetworks as a scalable alternative to LoRA and full fine-tuning.