weight-sparse-transformers

This paper introduces an automated LLM pipeline to generate and verify human-readable descriptions of when individual weights in transformers matter, finding that 12-31% of weights in weight-sparse transformers are interpretable globally, outperforming dense transformers.