Tag
This paper tests whether the standard recipe for identifying attention-head circuits by task-pattern selectivity and causal ablation yields consistent mechanistic claims across different 1B-class language model families (Pythia, OLMo, OLMoE). It finds no two (task, model) cells share the same primary causal screen, and introduces a five-category taxonomy of screen outcomes, with the MoE model showing a distinct prev-token positional substrate.
This paper investigates the Platonic Representation Hypothesis by examining 16 language models across 8 families on 800 reasoning problems. It finds that while models converge in internal representations, they diverge in reasoning processes, especially post-decision, and shared representations have minimal causal influence on predictions.