layer-routing

MACRO is a framework that learns task-specific execution routes over frozen LLM layers using Markov chain-based routing, improving reasoning accuracy without modifying model weights. It outperforms prior routing approaches while reducing search time significantly.

This paper introduces a question-level audit framework distinguishing 'realized' from 'reachable' answers on LLM benchmarks, showing that aggregate score gains often come from producing already-reachable answers rather than expanding true capability, and that random layer routing matches structured search under matched budgets.

Delta Attention Residuals improve layer-wise routing in transformer models by attending to feature changes (deltas) rather than cumulative hidden states, achieving 1.7–8.2% validation perplexity gains across scales from 220M to 7.6B parameters.