想深入研究模型，不能只停留在应用层，需要弄懂底层系统是如何训练和优化的。

偶然发现 LLMSys-PaperList，这份精心整理了大模型系统相关的论文合集。

从 2022 年一直更新到 2026 年最新的顶会论文，并按训练、推理、多模态等方向分类。

每篇都标注了出处和发表会议，相当于一份持续更新的文献地图。

GitHub：http://github.com/AmberLJC/LLMSys-PaperList…

除了学术论文，还收录了各大厂的技术报告、开源训练和推理框架、相关课程，以及 DeepSeek、Llama、Qwen 等主流模型的技术文档。

如果我们正在做大模型相关的研究或开发，这份清单值得收藏，省去大量找论文的时间。

AmberLJC/LLMSys-PaperList

Source: https://github.com/AmberLJC/LLMSys-PaperList

Awesome LLM Systems Papers

A curated list of Large Language Model systems related academic papers, articles, tutorials, slides and projects. Star this repository, and then you can keep abreast of the latest developments of this booming research field.

Table of Contents

• LLM Systems
  • Training
    • Pre-training
    • Post Training
    • Fault Tolerance / Straggler Mitigation
  • Serving
    • LLM serving
    • Agent Systems
    • Serving at the edge
    • System Efficiency Optimization - Model Co-design
  • Multi-Modal Training Systems
  • Multi-Modal Serving Systems
• LLM for Systems
• Industrial LLM Technical Report
• ML Conferences
  • NeurIPS 2025
• LLM Frameworks
  • Training
  • Post-Training
  • Serving
• ML Systems
• Survey Paper
• LLM Benchmark / Leaderboard / Traces
• Related ML Readings
• MLSys Courses
• Other Reading

LLM Systems

Training

Pre-training

2026

• Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design | EuroSys’ 26
• Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training | EuroSys’ 26
• RDMA Point-to-Point Communication for LLM Systems: RDMA-based point-to-point communication optimization for distributed LLM systems | MLSys’ 26
• MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs | MLSys’ 26
• Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training
• AXLearn: Modular Large Model Training on Heterogeneous Infrastructure | MLSys’ 26
• MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
• MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production | EuroSys’ 26
• MegaScale-Data: Scaling DataLoader for Multisource Large Foundation Model Training | EuroSys’ 26
• HetAuto: Cross-Cluster Auto-Parallelism for Heterogeneous Distributed Training | EuroSys’ 26
• HARP: Orchestrating Automated Parallel Training on Heterogeneous GPU Clusters | EuroSys’ 26
• Crimson: Collaborative Parameter Updates for Efficient Pipeline Training of Large Language Models | EuroSys’ 26
• Suika: Efficient and High-quality Re-scheduling of 3D-parallelized LLM Training Jobs in Shared Clusters | EuroSys’ 26
• Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering | EuroSys’ 26
• BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models | MLSys’ 26
• MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training | MLSys’ 26
• ProTrain: Efficient LLM Training via Automatic Memory Management | MLSys’ 26
• DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization | MLSys’ 26
• Multipath Collective Communication Beyond Scale-up Networks in GPU Clouds | EuroSys’ 26
• STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training through Spatio-Temporal Allocation Planning | EuroSys’ 26
• Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation | EuroSys’ 26
• Bridging the GPU Utilization Gap: Predictive Multi-Dimensional Resource Scheduling for AI Workloads | EuroSys’ 26
• Reducing the GPU Memory Bottleneck with Lossless Compression for ML | EuroSys’ 26
• Efficient Long-Context LM Training by Core Attention Disaggregation | MLSys’ 26
• Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters | MLSys’ 26
• Unleashing Scalable Context Parallelism via Fully Connected Pipeline | MLSys’ 26
• FlexTrain: Scalable Hybrid-Parallel Training for Long-Context LLMs | MLSys’ 26
• veScale-FSDP: Flexible and High-Performance FSDP at Scale | MLSys’ 26
• HexiScale: LLM Training over Heterogeneous Hardware | MLSys’ 26
• FP8-Flow-MoE: Casting-Free FP8 Recipe for MoE without Double Quantization Error | MLSys’ 26

Systems for Post-training / RLHF

2026

• Laminar: A Scalable Asynchronous RL Post-Training Framework | EuroSys’ 26
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs | EuroSys’ 26
• HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments | MLSys’ 26
• ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems | MLSys’ 26
• Beat the Long Tail: Distribution-Aware Speculative Decoding for Reinforcement Learning | MLSys’ 26
• FLoRIST: Federated Low-Rank Adaptation with Random Subspaces for LLMs | MLSys’ 26

Fault Tolerance / Straggler Mitigation

2026

• GoCkpt: Gradient-Assisted Multi-Step Overlapped Checkpointing for Efficient LLM Training | PPoPP’ 26
• Handling Network Faults in Distributed AI Training: Failover is Now an Option | EuroSys’ 26
• GUARD: Scalable Straggler Detection and Mitigation in LLM Training | MLSys’ 26

Serving

LLM serving

2026

• TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference | Code | MLSys’ 26
• AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving
• SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips | MLSys’ 26
• Scaling Up Efficient Small Language Models Serving: Serving and Deployment for Semantic Job Search | MLSys’ 26
• OptiKIT: Meeting SLOs, Slashing Hours - Automated Enterprise LLM Optimization | MLSys’ 26
• BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching | ASPLOS’ 26
• SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding with Disaggregated Pipeline and Fused Kernels | ASPLOS’ 26
• MuxWise: Towards High-Goodput LLM Serving with Prefill-decode Multiplexing | ASPLOS’ 26
• MoEless: Efficient MoE LLM Serving via Serverless Computing
• BiScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS
• Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference
• MineDraft: A Framework for Batch Parallel Speculative Decoding — overlaps drafting and verification across two batches, hiding draft latency. Up to +75% throughput, -39% latency. Integrated into vLLM. | NUS & MIT
• Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start
• AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding | EuroSys’ 26
• FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters | EuroSys’ 26
• Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading | EuroSys’ 26
• KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving | EuroSys’ 26
• AdaGen: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Inference Serving | EuroSys’ 26
• SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference | EuroSys’ 26
• High Throughput and Low Latency LLM Serving via Adaptive KV Caching | EuroSys’ 26
• PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Dropping | EuroSys’ 26
• PiLLM: Resource-Efficient LLM Inference Using Workload Prediction | EuroSys’ 26
• Automated End-to-End Model Serving with Cooperative Compilation and Scheduling | EuroSys’ 26
• MFS: An Efficient Model Family Serving System for LLMs | EuroSys’ 26
• CRAFT: Cost-aware Expert Replica Allocation with Fine-Grained Layerwise Estimations for Efficient MoE Serving | MLSys’ 26
• MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing | MLSys’ 26
• FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management | MLSys’ 26
• Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost | MLSys’ 26
• SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models | MLSys’ 26
• BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization | MLSys’ 26
• From Tokens to Layers: Redefining Stall-Free Scheduling for LLM Serving with Layered Prefill | MLSys’ 26
• HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving | MLSys’ 26
• BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching | MLSys’ 26
• GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving | MLSys’ 26
• PRISM: Parametrically Refactoring Inference for Speculative Decoding Draft Models | MLSys’ 26
• FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models | MLSys’ 26
• Efficient Data Passing for Serverless Inference Workflows: A GPU-Centric Approach | EuroSys’ 26
• TrustWeave: Integrity Measurement and Attestation for Multi-Cloud LLMs | EuroSys’ 26
• Stream2LLM: Overlapping Context Streaming and Prefill for Low-Latency LLM Serving | MLSys’ 26
• Locality-Aware Beam Scheduling for Efficient Test-Time Compute | MLSys’ 26
• Optimizing Deployment Configurations for LLM Inference | MLSys’ 26
• ContextPilot: Fast Long-Context Inference via Context Reuse | MLSys’ 26
• Speculative Decoding: Performance or Illusion? | MLSys’ 26
• SHIP: SRAM-Based Huge Inference Pipelines for Fast LLM Serving | MLSys’ 26
• BEAM: Joint Resource-Power Optimization for LLM Inference | MLSys’ 26
• Beyond the Buzz: A Pragmatic Take on Inference Disaggregation | MLSys’ 26
• PLA-Serve: Prefill-Length-Aware LLM Serving System | MLSys’ 26
• Accelerating Reasoning Model Inference with Sparse Self-Speculative Decoding | MLSys’ 26
• FaaScale: Unlocking Fast LLM Scaling for Serverless Inference | MLSys’ 26
• Breaking the Ice: Analyzing Cold Start Latency in vLLM | MLSys’ 26
• Demystifying the Mixture of Experts Serving Tax | MLSys’ 26
• RaidServe: High-Performance Resilient LLM Serving | MLSys’ 26
• Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problem | MLSys’ 26
• ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

Agent Systems

2026

• DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference | DeepSeek
• AIMS: Cost-Efficient LLM-Based Agent Deployment in Hybrid Cloud-Edge Environments | EuroSys’ 26
• From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents | EuroSys’ 26
• Hippocampus: An Efficient and Scalable Memory Module for Agentic AI | MLSys’ 26
• PROMPTS: Performance Optimization via Multi-Agent Planning for Test-time Compute Scaling | MLSys’ 26
• TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval | MLSys’ 26
• OpenHands Software Agent SDK | MLSys’ 26
• FlashAgents: Accelerating Multi-Agent LLM Systems via Streaming Prefill Overlap | MLSys’ 26
• AgenticCache: Cache-Driven Asynchronous Planning for Agentic LLM Systems | MLSys’ 26
• Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation | MLSys’ 26
• Ontology-Guided Long-Term Agent Memory for Conversational RAG | MLSys’ 26
• OSWorld-Human: Benchmarking Efficiency of Computer-Use Agents | MLSys’ 26

Serving at the edge

2026

• TZ-LLM: Protecting On-Device Large Language Models with Arm TrustZone | EuroSys’ 26
• TailorLLM: Collaborative End-Cloud Inference of Large and Small Language Models Based on Low-Rank Adaptation | EuroSys’ 26
• Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices | EuroSys’ 26
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones | EuroSys’ 26
• On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwarding | EuroSys’ 26
• SwiftFL: Enabling Speculative Training for On-Device Federated Deep Learning | EuroSys’ 26
• viNPU: Optimizing Vision Transformer Inference on Mobile NPUs | EuroSys’ 26
• Efficient, VRAM-Constrained Cross-Lingual Model Inference on Client Devices | MLSys’ 26
• Rethinking DVFS for Mobile LLMs: CORE for Energy-Efficient On-Device Inference | MLSys’ 26
• IntAttention: Fully Integer Attention Pipeline for Edge LLM Inference | MLSys’ 26

System Efficiency Optimization - Model Co-design

2026

• Reducing GPU Memory Fragmentation via Spatio-Temporal Allocation Planning | EuroSys’ 26
• SAS: Sparse Attention Synthesizer for Efficient Language Model Inference | EuroSys’ 26
• LLMFolder: Revisiting Constant Folding in Large Language Models | EuroSys’ 26
• FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling (Blackwell) | MLSys’ 26
• BLASST: Dynamic Blocked Attention Sparsity for Scalable Transformer Inference | MLSys’ 26
• Attribution-based Sparse Activation in Large Language Models | MLSys’ 26
• MixLLM: LLM Quantization with Global Mixed-Precision between Output and Embeddings | MLSys’ 26
• MAC-Attention: Match-Amend-Complete Attention for Efficient Long-Context Inference | MLSys’ 26
• Flashlight: PyTorch Compiler Extensions for Attention Variants | MLSys’ 26
• CAGE: Curvature-Aware Gradient Estimation for Quantization-Aware Training | MLSys’ 26
• OPKV: Recallable Sparsity in Paged KV Cache for Efficient LLM Inference | MLSys’ 26
• Using Span Queries to Optimize Cache and Attention Locality | MLSys’ 26

Multi-Modal Training Systems

• DISTMM: Accelerating distributed multimodal model training | NSDI’ 24
• Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
• Addressing Model and Data Heterogeneity in Multimodal Large Language Model Training | PKU
• Cornstarch: Distributed Multimodal Training Must Be Multimodality-Aware | UMich
• PipeWeaver: Addressing Data Dynamicity in Large Multimodal Model Training with Dynamic Interleaved Pipeline | SJTU
• MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production | EuroSys’ 26

Multi-Modal Serving Systems

• xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism
• MOSEL: Inference Serving Using Dynamic Modality Selection
• Approximate Caching for Efficiently Serving Diffusion Models | Adobe Research
• Generative AI Beyond LLMs: System Implications of Multi-Modal Generation | Meta
• Characterizing and Efficiently Accelerating Multimodal Generation Model Inference | Meta
• DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models | MIT
• LongVILA: Scaling Long-Context Visual Language Models for Long Videos | NVIDIA
• FlexCache: Flexible Approximate Cache System for Video Diffusion | University of Waterloo
• DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving
• PATCHEDSERVE: A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving
• ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
• TetriServe: Efficient DiT Serving for Heterogeneous Image Generation
• dInfer: An Efficient Inference Framework for Diffusion Language Models
• Fast-dLLM v2: Efficient Block-Diffusion LLM
• Argus: Quality-Aware High-Throughput Text-to-Image Inference Serving System
• Cornserve: Efficiently Serving Any-to-Any Multimodal Models
• HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving
• Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing
• VoxServe: Streaming-Centric Serving System for Speech Language Models
• dLLM-Serve: Taming the Memory Footprint Crisis for Efficient Diffusion LLM Serving
• HADIS: Hybrid Adaptive Diffusion Model Serving for Efficient Text-to-Image Generation
• Efficient Multimodal Serving via Module Multiplexing | EuroSys’ 26
• FlashPS: Efficient Generative Image Editing with Mask-aware Caching and Scheduling | EuroSys’ 26
• StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation | MLSys’ 26
• SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding | MLSys’ 26
• Million-Scale Text-to-Video Retrieval with Hyperdimensional Computing | EuroSys’ 26
• TriInfer: Hybrid Encode-Prefill-Decode Disaggregation for Multimodal LLM Inference | MLSys’ 26
• CDLM: Consistency Diffusion Language Models for Faster Text Generation Sampling | MLSys’ 26
• db-SP: Accelerating Sparse Attention for Visual Generative Models | MLSys’ 26
• TiDAR: Think in Diffusion, Talk in Autoregression for Multimodal Generation | MLSys’ 26

LLM for Systems

• Large Language Models for Compiler Optimization
• The Hitchhiker’s Guide to Program Analysis: A Journey with Large Language Models
• LLM-Assisted Code Cleaning For Training Accurate Code Generators | UCB
• Efficient Multi-Task Large Model Training via Data Heterogeneity-aware Model Management
• If At First You Don’t Succeed, Try, Try, Again…? | SOSP’ 24
• Aceso: Efficient Parallel DNN Training through Iterative Bottleneck Alleviation | EuroSys ’24
• GMorph: Accelerating Multi-DNN Inference via Model Fusion | EuroSys ’24
• Automatic Root Cause Analysis via Large Language Models for Cloud Incidents | EuroSys ’24
• KNighter: Transforming Static Analysis with LLM-Synthesized Checkers | SOSP’ 25
• Barbarians at the Gate: How AI is Upending Systems Research
• Let the Barbarians In: How AI Can Accelerate Systems Performance Research
• AI Research Engineering Skills Library: A collection of AI research engineering skills and best practices
• K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model
• AI-Driven Research for Databases: Automated database optimization via co-evolving evaluators and AI-generated solutions
• No More Translation at Runtime: LLM-Empowered Static Binary Translation | EuroSys’ 26
• Unified LLM Model for PPA Prediction from Hardware Code | MLSys’ 26
• Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems | MLSys’ 26
• AccelOpt: Self-Improving LLM Agentic System for Kernel Optimization | MLSys’ 26
• VeriMoA: Mixture-of-Agents for Spec-to-HDL Verification and Generation | MLSys’ 26

Industrial LLM Technical Report

ML Conferences

NeurIPS 2025

A curated collection of NeurIPS 2025 papers focused on efficient systems for generative AI models. The collection includes papers on:

• Architecture & Efficient Mechanisms - Efficient attention, KV-cache systems, speculative decoding
• Model Compression & Quantization - Quantization, pruning, KV cache compression
• Inference & Serving - LLM serving, scheduling, distributed inference
• Multi-Modal & Diffusion - VLM efficiency, diffusion optimization
• Reinforcement Learning - RL training infrastructure, policy optimization
• Training Systems - Distributed training, memory efficiency

See the full NeurIPS 2025 collection for detailed categorization and paper summaries.

LLM Frameworks

Training

• DeepSpeed: a deep learning optimization library that makes distributed training and inference easy, efficient, and effective | Microsoft

• Accelerate | Hugging Face

• LLaVA

• Megatron | Nvidia

• NeMo | Nvidia

• torchtitan | PyTorch

• torchtune: PyTorch-native fine-tuning library for LLMs with minimal dependencies | PyTorch

• veScale | ByteDance

• DeepSeek Open Infra

• VeOmni: Scaling any Modality Model Training

• Cornstarch: Distributed Multimodal Training Must Be Multimodality-Aware | UMich

• GPT-NeoX: Model-parallel autoregressive LLM training combining Megatron and DeepSpeed | EleutherAI

• nanotron: Minimalistic 3D-parallel (tensor/pipeline/data) LLM training framework | Hugging Face

• litgpt: 20+ LLM implementations with pre-training and fine-tuning recipes | Lightning AI

• LLaMA-Factory: Unified efficient fine-tuning of 100+ LLMs and VLMs via LoRA, full fine-tuning, and RL methods | ACL’ 24

• Unsloth: 2-5x faster LLM fine-tuning with ~80% less memory via custom Triton/CUDA kernels

• Post-Training

  • PEFT: Parameter-efficient fine-tuning library (LoRA, QLoRA, Prompt Tuning, IA3, etc.) | Hugging Face
  • TRL: Transformers Reinforcement Learning
  • OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework based on Ray
  • VeRL: Volcano Engine Reinforcement Learning for LLMs
  • rLLM: Reinforcement Learning for Language Agents
  • SkyRL: A Modular Full-stack RL Library for LLMs
  • AReal: Distributed RL System for LLM Reasoning
  • ROLL: Reinforcement Learning Optimization for Large-Scale Learning
  • slime: a LLM post-training framework aiming for RL Scaling
  • RAGEN: Training Agents by Reinforcing Reasoning
  • Agent Lightning: Train ANY AI Agents with Reinforcement Learning
  • LMFlow: Extensible toolkit for fine-tuning and inference of large foundation models
  • NeMo-Aligner: Scalable alignment toolkit for SFT, PPO, DPO, and SteerLM on NeMo | Nvidia

Serving

• llama.cpp: LLM inference in C/C++ with GGUF quantization; supports CPU, Metal, CUDA, and wide hardware
• Ollama: Local LLM serving with model management and OpenAI-compatible API
• TensorRT-LLM | Nvidia
• Triton Inference Server: Production multi-framework model serving platform with dynamic batching | Nvidia
• Ray-LLM | Ray
• TGI | Hugging Face
• vLLM | UCB
• SGLang | UCB
• LMDeploy: LLM compression, deployment, and serving toolkit with TurboMind persistent batching engine | InternLM
• LightLLM: Lightweight Python LLM serving with tri-process architecture decoupling prefill and decode
• DeepSpeed-MII: Low-latency, high-throughput LLM inference powered by DeepSpeed | Microsoft
• CTranslate2: Fast C++/Python inference engine for Transformer models with int8/int16 quantization | OpenNMT
• Petals: Distributed LLM inference and fine-tuning across volunteer GPUs in a BitTorrent-like fashion | ACL’ 23
• KV Transformers
• Dynamo: A Datacenter Scale Distributed Inference Serving Framework | Nvidia
• LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
• aibrix: Cost-efficient pluggable infrastructure for GenAI inference (KV cache routing, autoscaling, disaggregated prefill) | vLLM Project

ML Systems

Survey Paper

• Efficient Large Language Models: A Survey
• Challenges and Applications of Large Language Models
• Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models
• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems

LLM Benchmark / Leaderboard ? Traces

• LLM Energy Leaderboard | Umich
• LLM-Perf Leaderboard | HuggingFace
• Aviary Explorer | Anyscale
• Open LLM Leaderboard | HuggingFace
• HELM | Stanford
• LMSYS | UCB
• Towards Efficient and Reliable LLM Serving: A Real-World Workload Study
• FlashInfer-Bench / LLMInfer-Bench: Benchmarking LLM Inference Kernels and Systems | MLSys’ 26
• DriftBench: Measuring and Predicting Infrastructure Drift in LLM Serving Systems | MLSys’ 26
• Charon: A Unified Simulator for LLM Training and Inference | MLSys’ 26
• ProfInfer: eBPF-based Fine-Grained LLM Inference Profiler | MLSys’ 26

Related ML Readings

• Large Transformer Model Inference Optimization
• Transformer Inference Arithmetic
• The Transformer Family Version 2.0
• Full Stack Optimization of Transformer Inference: a Survey | UCB
• The Smol Training Playbook: The Secrets to Building World-Class LLMs | Hugging Face
• The Ultra-Scale Playbook: Training LLMs on GPU Clusters | Hugging Face

MLSys Courses

• Systems for Machine Learning | (Stanford)[https://cs229s.stanford.edu/fall2023/]
• Systems for Generative AI | (Umich)[https://github.com/mosharaf/eecs598/tree/w24-genai]
• Systems for AI - LLMs | (GT)[https://cs8803-sp24.anand-iyer.com/]

Other Reading

• A curated list of Large Language Model
• AI systems paper list
• A baseline repository of Auto-Parallelism in Training Neural Networks
• Numbers every LLM Developer should know
• 100,000 H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability, Failures, Checkpointing
• OpenAI Keynote on Building Scalable AI Infrastructure
• Awesome ML SYS Tutorial