How to Land a Frontier Lab Job

I found a great read from the Gemini Pretraining Area Lead, especially for people that want to work in a frontier “Neolab”.

Demonstrate skills that the lab requires: Vlad suggested working on directions where frontier labs expand their scope.

This includes (but not limited to) two stacks below and above LLM:

• Kernel work: covers the ecosystem of advancing performance work of LLMs. Examples include DSL, FlashAttention, Quantization, and so on.
• Agents work: carefully designed workflows of LLMs for creating useful outputs, e.g., AutoResearch.

Suggested learning resources: Kernel Work:

• Flash Attention Series (1 through 4)
• DSLs (Domain Specific Languages): ThunderKittens and CuTe DSL
• SnapKV: Recommended reading for understanding decoding-side optimizations.

Quantization

• LLM.int8(): A classic foundational paper to learn strong tricks for quantization.
• Chris De Sa’s Group Papers, specifically QuiP, QuiP#, and QTIP (https://arxiv.org/abs/2307.13304)
• AQLM: (https://arxiv.org/abs/2401.06118).

Agentic Research Andrej Karpathy’s “Autoresearch” (https://github.com/karpathy/autoresearch…) AlphaEvolve & FunSearch: (https://arxiv.org/abs/2506.13131)

QuIP: 2-Bit Quantization of Large Language Models With Guarantees

Source: https://arxiv.org/abs/2307.13304 View PDF

Abstract:This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from \\textit\{incoherent\} weight and Hessian matrices, i.e., from the weights being even in magnitude and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale quantization algorithm, and show that our theory also applies to an existing method, OPTQ. Empirically, we find that our incoherence preprocessing improves several existing quantization algorithms and yields the first LLM quantization methods that produce viable results using only two bits per weight. Our code can be found atthis https URL.

Submission history

From: Jerry Chee [view email] **[v1]**Tue, 25 Jul 2023 07:44:06 UTC (87 KB) **[v2]**Mon, 15 Jan 2024 21:54:28 UTC (108 KB)