Tag
This paper introduces Power Law Graph Attention (PLGA) and the PLDR-LLM architecture, an exact generalization of scaled dot-product attention using input-generated bilinear operators. It presents theoretical results including an inference-collapse theorem, empirical stability measurements, and machine-checked proofs in Lean 4.
This article explains the Fourier transform of power-law functions, focusing on the interesting case of 1/f noise and the symmetry between time and frequency domains.
This paper presents a scaling law showing that the contextual influence of word order in human language decays approximately as 1/d with distance, as measured by the reduction in perplexity from large language models, across multiple languages and corpora.
This paper presents a distributed benchmark study on scaling laws for classical machine learning models on tabular data, showing that power-law fits hold for most model families and quantifying replicator-implementation variance across 127 student runs.
This paper presents a token-level framework showing that power-law scaling in language model loss arises from the aggregation of sigmoidal learning curves of individual tokens, and demonstrates that reshaping training distributions based on token learning times can accelerate validation loss reduction by 11%.
A comprehensive overview of scaling laws in deep learning, tracing their theoretical roots and empirical findings, and explaining how loss decreases predictably with model size, data, and compute.
This paper investigates the quantitative limits of parametric memory in LLMs using LoRA as a probe, establishing a power law relationship and introducing a threshold-guided optimization method called MemFT for improved memory performance.
This paper investigates growth dynamics in deterministic equational discovery across three toy substrates and two real-world replications, finding substrate-conditional saturating power-law scaling.
Foundational empirical study demonstrating power-law scaling relationships between language model performance and model size, dataset size, and compute budget, with implications for optimal training allocation and sample efficiency.