language-model-evaluation

Introduces Uncheatable Eval, a dynamic benchmark using compression rates to evaluate language models and mitigate data contamination.

The paper proposes ModelLog, a declarative probabilistic framework for evaluating language models by defining semantic constraints over token predictions, linking evaluation to learning through shared semantics.

This paper introduces WinoQueer-NL, a Dutch dataset for evaluating bias in language models toward LGBTQ+ identities, and finds significant disparities in model outputs, particularly favoring stereotypes for transgender and non-binary identities.