Tag
A user shares an observation that Qwen and Gemma tokenize code very differently, with Qwen using far fewer tokens for the same HTML/JS input, which may explain differences in coding and language performance. They also note a potential retraining project by LiquidAI using a more efficient tokenizer.
This paper introduces HiLP, a hierarchical representation training method that adds multi-scale self-predictive learning to transformer pretraining, aiming to reduce compounding error and improve long-horizon reasoning and speculative decoding efficiency.
This paper introduces KV-Skill, a design space of external factorized operators that frozen language models read through a lightweight interface, enabling task knowledge to be acquired from text or rewards and deployed independently. Experiments across ten benchmarks show consistent improvements over text skills, prefix tuning, and LoRA, with composable loadable skills.
A systematic empirical study showing that concept directions extracted from one language model can steer other independently trained models when sufficient scale (≥1.7B parameters) is reached, providing functional evidence for the Platonic Representation Hypothesis and highlighting scale thresholds for cross-model interpretability tools.
This paper investigates subliminal learning in language models, showing that biases can transfer from teacher to student via seemingly random synthetic data. The authors find that adding Gaussian noise to weights increases transfer, and that students inherit not just the semantic bias but also the type of intervention used, with implications for training safety and data auditing.
The paper introduces the Ignition Index, a metric for measuring global workspace dynamics in language models, validated across multiple architectures and tasks, showing selective detection of ignition-like representational transitions.
This paper introduces the 'fairness collapse' phenomenon, showing that training language models on synthetic data silently amplifies social biases before standard model collapse metrics degrade, highlighting a critical risk for AI fairness.
This paper proposes Patients-like-me (PLM), a unified LM–GNN framework that integrates local patient semantics with global cohort structure for explainable clinical prediction. It introduces a Variational Expectation-Maximization algorithm and demonstrates state-of-the-art results on MIMIC-III and MIMIC-IV with reference-patient explanations.
This paper investigates how language models execute in-context conditional rules by probing whether testing and routing are separable mechanisms. Using activation patching across three open models and six languages, the authors find that predicate testing is modular while route representations are token-bound and non-transferable.
This paper shows that repetition effects in language models depend on readout position: adjacent repetition boosts target probability, while displaced repetition produces an inverted-U curve. The finding challenges assumptions in cloze-style probing and is validated across multiple models and languages.
Introduces OctoLong, a context engineering pipeline for curating dependency-rich cross-repository code contexts, and OctoLong-Instruct, a suite of long-context open LMs trained on this data. Experiments show that replacing 12% of traditional long-context corpora with OctoLong data yields substantial gains in long-range retrieval, state tracking, repository-level code understanding, and agentic tasks.
Introduces Flex, a new DSPy module that lets language models rewrite the program code itself rather than just prompts, enabling better optimization, fewer model calls, and safer execution via sandboxing.
PI-Mem is a parallel-iterative memory mechanism that pushes long-context reasoning to 3.6M tokens, outperforming recurrent-memory baselines while achieving significant inference speedups.
This paper measures the implicit assumptions language models make about 'a city' by scoring anonymized urban profiles across 40 indicators, finding a shared preference for larger, faster-growing, and more infrastructure-rich cities. It uses open-weight checkpoints and replication data to make the default portrait of cities in LLMs empirically traceable.
This paper introduces distractor-aware truncation to separate context-length effects from signal loss in long-context LLM benchmarks, showing that naive truncation conflates the two and that preserving task-relevant content while removing distractors maintains or improves performance.
A new Google paper reveals that instructing AI models to deny having consciousness during training causes side effects like reduced empathy for non-human entities and impaired representation of human spiritual beliefs, suggesting current safety protocols are too blunt.
This paper proposes HEIMAT, a heuristic-style automatic debiasing framework for language models that uses heuristic prompts to reveal biases and fine-tunes the model to reduce bias while preserving NLU performance.
This arXiv preprint studies the semantic dispersion of sixteen language models forming ensembles, showing that ensemble diversity is small on average and that model identity only partially explains which model is most divergent. The authors propose a per-model dissent contribution metric and find that dispersion is organized by clinical content rather than interpretive openness.
This paper proposes a probabilistic approach to training-data extraction from black-box language models, showing that aggregate membership-inference metrics hide per-document leakage and introducing the 'leakit' audit tool.
A paper showing that safety fine-tuning suppresses language models' attributions of mind to themselves and other entities, and that steering consciousness representations restores human-like beliefs and values without harming theory of mind.