Tag
The tweet critiques investors in 2023 who dismissed scaling laws and invested in SaaS/dev tools easily replicable by AI, comparing them to dinosaurs, and includes commentary on missing AI companies like OpenAI and Anthropic.
This paper presents an open-source pretraining recipe that trains 2B-parameter language models on consumer-grade RTX 5090 GPUs for under $7K, achieving performance near larger baselines and deriving cost scaling laws for model training.
This newsletter covers key AI and tech news, including a Pew study indicating that a third of web pages published since ChatGPT's launch show signs of AI authorship, alongside updates on enterprise competition and research on agent safety.
The paper proposes a compute-efficient two-step hyperparameter transfer framework to predict optimal learning rates for large Mixture-of-Experts models, enabling efficient pretraining without costly hyperparameter sweeps.
The article discusses the evolving importance of data versus algorithmic progress in AI development, drawing from recent talks by researchers to highlight a shift in focus.
The article questions how the relatively small Qwen 3.8 27B model achieves high intelligence, raising doubts about current scaling laws and the efficiency of large model parameters.
This paper presents a systematic scaling law study for text-to-image diffusion models, showing they scale predictably but require significantly more data per parameter than language models for optimal training.
Scott Gray, a key early employee at OpenAI, joined in 2016 and participated in the development of Sparse Transformer, Scaling Laws, and GPT-3, primarily responsible for the underlying GPU systems. According to reports, OpenAI has had 12 senior executives leave this year, highlighting the loss of key talent.
A new Meta paper reveals that small models can accurately predict scaling laws but require more extensive hyperparameter tuning. The study finds scaling laws emerge around 4M parameters and become clearer with proper tuning.
Galileo, 388 years ago, discovered the reasons why human giants cannot exist, involving principles of physics and biology.
This paper studies the trade-off in repeating high-quality domain data during LLM pretraining to maintain performance as models scale, finding that optimal repetition counts increase with model size and are negatively correlated with domain validation loss.
A sarcastic tweet mocking frontier AI labs for still pretraining models, claiming the loss can be predicted theoretically without actually training.
This paper introduces MOSAIC, a framework that jointly optimizes sparse Mixture-of-Experts model architecture and hardware systems for large-scale pretraining, showing that compute-optimal sparsity is not necessarily cluster-optimal when MFU, communication costs, and parallel layouts are considered.
Dyna Robotics introduces Dyna-2, a world-action model pre-trained on one million hours of human video, revealing scaling laws and suggesting the embodiment gap is an adaptation problem rather than a knowledge problem.
Dyna-2 is a world-action model pre-trained on over a million hours of human video, showing scaling laws on human data and a human-to-robot transfer scaling law for zero-shot robot performance.
Dyna Robotics introduces Dyna-2, a world-action model pre-trained on one million hours of human video, discovering new scaling laws for robot manipulation.
Meta researchers introduce the Skaling law, a generalized scaling law that couples model capacity and data through a single interaction exponent, reducing prediction error by 1.5–3x and enabling roughly 10x less compute for full-grid extrapolation.
Pathway's BDH, a post-transformer architecture, reportedly matches GPT-2 scaling from 10M to 1B parameters while training from scratch on standard GPUs.
Call for papers for the Scaling H2R workshop at CoRL 2026, focusing on scaling laws and diversity in human-to-robot learning. Submissions due Oct 7, 2026.
A discussion question asking about the theoretically optimal quantization bit-width for LLMs under a fixed memory budget, referencing recent 3-bit/2-bit results and scaling-law work from 2025-2026.