The Transformer Pill

Reddit r/ArtificialInteligence News

Summary

A reflection on the broad implications of transformer architectures beyond LLMs, including potential impacts on linguistics, genetics, and causal modeling, comparing their significance to the Haber-Bosch process.

I just watched a YouTube video that vulgarized the maths behind transformers. I feel like I have been living under a rock for the last 10 years. My knowledge of AIs basically stopped at CNNs (Convolutional Neural Networks). The theoretical and practical consequences of transformers are so vast and way beyond the current LLM hype when you understand what it implies: \* In linguistics: it completely shatters many of the dominant ideas in the field like the signifier signified divide and grammar seem to be a system emerging from statistical correlations rather than one we are born with. \* In genetics most genes responsible of monogenic diseases are already well known. What is left are polygenic diseases, like most autoimmune diseases or mental illnesses. Bioinformatics could combine the power of transformers with GWAS data to map the complex relationship between genes and illnesses. \* When transformers are paired with time-series, they cease to be correlation engines and become causality engines. Governments, big fortunes and companies like Palantir are mapping supply chains to predict crises, price hikes and potential wars. When you apply these predictive capabilities to human behavior you get very close to Minority Report. When I tried to find an equivalent in the history of science in term of impact, the only thing I could think of was the Haber-Bosch process which basically defined the whole 20th century (fertilizers, bombs, toxic gases…). What are your insights about the revolution transformers are about to bring that the general public seem to be completely unaware of?
Original Article

Similar Articles

When transformers learn "impossible" languages, what do they learn?

arXiv cs.CL

This paper investigates how transformer language models learn 'impossible' languages with unnatural properties, finding that while grammatical sensitivity degrades gradually, generative production shows pronounced failures, suggesting a linking hypothesis for non-attestation.

How LLMs Actually Work (26 minute read)

TLDR AI

A detailed walkthrough of how transformer-based LLMs work, covering tokenization, embeddings, attention, and next-token prediction without heavy math.