Machine Learning Systems Notes on GitHub

Over the past few months, the author has deeply studied machine learning systems, mainly focusing on training and inference of large language models. This collection of notes covers distributed computing, parallelism, quantization, and PyTorch internals, most of which come from the author’s experiments.

1. Distributed Techniques - covering distributed training fundamentals: NCCL collectives (gather, all-gather, reduce, all-reduce, scatter, reduce-scatter), mixture-of-experts, parallelism strategies (DP, DDP, ZeRO, tensor/pipeline parallelism), and torch.distributed basics.

2. Quantization - model quantization from first principles: symmetric/asymmetric quantization, LLM.int8(), AWQ, SmoothQuant, GPTQ/OBS/OBQ, and QuIP.

3. PyTorch Internals

4. JAX Scaling Book - some solved exercises from the JAX scaling book

Perfect for those interested in machine learning.

Portal: http://github.com/JINO-ROHIT/ml-systems-notes…

JINO-ROHIT/ml-systems-notes

Source: https://github.com/JINO-ROHIT/ml-systems-notes

ml-systems-notes

a personal collection of notes on ml systems engineering covering distributed computing, parallelism, quantization, and pytorch internals.

everything here is a work in progress. i add notes as i do experiments and projects.

contents

• distributed-techniques - distributed training fundamentals: nccl collectives (gather, all-gather, reduce, all-reduce, scatter, reduce-scatter), mixture-of-experts, parallelism strategies (dp, ddp, zero, tensor/pipeline parallelism), and torch.distributed basics.

• quantization - model quantization from first principles: symmetric/asymmetric quantization, llm.int8(), awq, smoothquant, gptq/obs/obq, and quip.

• torch-notes - pytorch internals

• jax-scaling-book - roofline analysis exercises for matrix multiplication in jax/tpu context.