foundational-research

David Ha and Jürgen Schmidhuber recount how foundational deep learning techniques like transformers, unsupervised pretraining, knowledge distillation, and residual networks were pioneered in Munich in 1991, laying the groundwork for the current AI boom.

A discussion on whether foundational AI research can be done without access to high-performance computing, given that early work like 'Attention is all you need' used consumer GPUs.