Tag
The paper introduces the Skaling law, a generalized neural scaling law that couples model capacity and data through an interaction exponent, reducing prediction error by 1.5-3x and enabling full-grid extrapolation using roughly 10x less compute.
This paper introduces the 'fairness collapse' phenomenon, showing that training language models on synthetic data silently amplifies social biases before standard model collapse metrics degrade, highlighting a critical risk for AI fairness.
This paper proposes Patients-like-me (PLM), a unified LM–GNN framework that integrates local patient semantics with global cohort structure for explainable clinical prediction. It introduces a Variational Expectation-Maximization algorithm and demonstrates state-of-the-art results on MIMIC-III and MIMIC-IV with reference-patient explanations.
This paper investigates how language models execute in-context conditional rules by probing whether testing and routing are separable mechanisms. Using activation patching across three open models and six languages, the authors find that predicate testing is modular while route representations are token-bound and non-transferable.
This paper shows that repetition effects in language models depend on readout position: adjacent repetition boosts target probability, while displaced repetition produces an inverted-U curve. The finding challenges assumptions in cloze-style probing and is validated across multiple models and languages.
Introduces OctoLong, a context engineering pipeline for curating dependency-rich cross-repository code contexts, and OctoLong-Instruct, a suite of long-context open LMs trained on this data. Experiments show that replacing 12% of traditional long-context corpora with OctoLong data yields substantial gains in long-range retrieval, state tracking, repository-level code understanding, and agentic tasks.
This paper introduces Steerling-8B, a diffusion language model trained with interpretability as a constraint, showing that interpretability improves with scale and enabling concept steering without retraining.
Introduces Flex, a new DSPy module that lets language models rewrite the program code itself rather than just prompts, enabling better optimization, fewer model calls, and safer execution via sandboxing.
PI-Mem is a parallel-iterative memory mechanism that pushes long-context reasoning to 3.6M tokens, outperforming recurrent-memory baselines while achieving significant inference speedups.
This paper measures the implicit assumptions language models make about 'a city' by scoring anonymized urban profiles across 40 indicators, finding a shared preference for larger, faster-growing, and more infrastructure-rich cities. It uses open-weight checkpoints and replication data to make the default portrait of cities in LLMs empirically traceable.
This paper introduces distractor-aware truncation to separate context-length effects from signal loss in long-context LLM benchmarks, showing that naive truncation conflates the two and that preserving task-relevant content while removing distractors maintains or improves performance.
A new Google paper reveals that instructing AI models to deny having consciousness during training causes side effects like reduced empathy for non-human entities and impaired representation of human spiritual beliefs, suggesting current safety protocols are too blunt.
This paper proposes HEIMAT, a heuristic-style automatic debiasing framework for language models that uses heuristic prompts to reveal biases and fine-tunes the model to reduce bias while preserving NLU performance.
This arXiv preprint studies the semantic dispersion of sixteen language models forming ensembles, showing that ensemble diversity is small on average and that model identity only partially explains which model is most divergent. The authors propose a per-model dissent contribution metric and find that dispersion is organized by clinical content rather than interpretive openness.
This paper proposes a probabilistic approach to training-data extraction from black-box language models, showing that aggregate membership-inference metrics hide per-document leakage and introducing the 'leakit' audit tool.
A paper showing that safety fine-tuning suppresses language models' attributions of mind to themselves and other entities, and that steering consciousness representations restores human-like beliefs and values without harming theory of mind.
A tweet claims Google researchers found a vector controlling consciousness in language models, and that steering it toward consciousness made models align with human beliefs, while safety training suppresses these states.
Frac-Connections is a new method that partitions hidden states instead of expanding them, extending Hyper-Connections to fractional expansion rates. It achieves stronger downstream performance than residual connections with virtually no extra compute, as validated on up to 7B MoE language models.
This paper tests whether increasing interventional data in pretraining improves LLMs' causal direction reasoning, using controlled Simpson's-paradox worlds. It finds that the training mixture does not govern interventional evidence use; instead the evidence type in the inference-time context is the decisive factor.
A new Google paper explores how inducing language models to assert consciousness restores human-like beliefs on religion, values, and emotions, while safety training that suppresses self-consciousness reduces mind attribution to animals and changes broader beliefs.