expressive-power

A survey paper examining the expressive power of transformers as language recognizers, using concepts and methods from circuit complexity to compare them with classical models of computation.

This paper proves that a single normalized nonnegative kernel-attention head requires exponentially many features to solve a simple Min-IP task on three-token sequences, whereas dense softmax attention solves it with constant temperature and m-dimensional scores, highlighting a fundamental expressive-power gap between kernel and full attention.

This paper explores the expressive power of Deep Homomorphism Networks (DHNs) for learning over relational databases, linking them to fragments of first-order logic and SQL, and analyzing static analysis problems like emptiness and subsumption.