Tag
The paper introduces the 'direction of ignorance' in LLMs' unembedding geometry, which encodes the training corpus's unigram distribution and acts as a Bayesian prior. It demonstrates how this prior is tempered based on context informativeness, providing insights into model behavior.