Megathread for listing latest open source projects, research papers that are helping optimizations, efficiencies and accessibility to Open Source LLM and related hardware, software ?
This megathread compiles a list of the latest open-source projects, research papers, and hardware innovations focused on optimizing inference, efficiency, and accessibility for open-source LLMs and related technologies.
I start with some informations gathered thorough endless posts reading on this sub and online: Inference and hardware optimization projects https://dwarfstar.sh/ - inference engine optimization proposal https://github.com/JustVugg/colibri - Treats VRAM + RAM + storage as one managed inference memory hierarchy, with expert streaming, caches and a strong focus on MoE. https://openfreedom.it/ - agentic harness proposal https://github.com/exo-explore/exo - Core source for heterogeneous/topology-aware Mac clustering https://github.com/ml-explore/mlx/discussions/3481 - JACCL real-world TB5 transfer experiment https://github.com/georgiedekker/mlx_distributed_ring_inference - distributed inference through TCP/Ring over TB3/TB4, without requiring RDMA https://github.com/sqliteai/warp - WARP — Weight-Aware Runtime and Paging https://github.com/kqb/mlx-od-moe - on-demand experts on Apple Silicon: memory-mapped expert storage, shadow predictor, prefetcher and LRU https://www.houmo.cn/1/35/NewsDetails.html - Houmo is developing DRAM-PIM with compute embedded in DRAM arrays, targeting >1 TB/s internal bandwidth and ~3× energy-efficiency improvement over its current generation. https://www.d-matrix.ai/wp-content/uploads/2024/11/d-Matrix-WhitePaper-Technical-FINAL.pdf and d-Matrix 3DIMC announcement - 3D DRAM + digital in-memory compute, highly aligned with the “move compute toward weights rather than weights toward compute” idea https://www.lucebox.com/ - workstation optimized for local AI use for approx. 7000$ Inference Research papers MDI-LLM - Model-Distributed Inference for LLMs at the Edge Model partitioning across low-power nodes and recurrent pipeline parallelism to reduce device idle time. MDI-LLM paper WDMoE - Wireless Distributed Mixture of Experts Distributes experts across edge/mobile devices and jointly optimizes expert selection and communication latency. Includes a physical NVIDIA Jetson testbed. WDMoE paper OD-MoE - On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference Very relevant to our expert-prediction idea. Uses a predictor to forecast experts several layers ahead and loads them just in time across distributed nodes. Reports 99.94% expert-prediction accuracy and about 75% of fully cached decoding speed while using one-third the GPU memory in its tested setup. OD-MoE paper MoE-SpeQ - speculative decoding + proactive expert prefetching Almost directly relevant to the question we uncovered around streamed MoEs. A draft model predicts future experts so their transfer can overlap computation. Reports up to 2.34× over its offloading baseline. MoE-SpeQ paper SP-MoE - speculative decoding and prefetching for MoEs Speculation-aware expert offloading, speculative expert prefetch, asynchronous batched I/O and compute/I/O pipelining. SP-MoE paper MoE-Spec - Expert Budgeting for Efficient Speculative Decoding Important counterargument to “speculation automatically fixes MoE.” Shows that verifying deeper speculative trees can activate too many unique experts, increasing memory pressure; proposes explicit expert budgeting. MoE-Spec paper
An open, in-progress handbook explaining LLM inference internals including GPU memory hierarchy, KV cache, batching, and popular inference engines like vLLM and TensorRT-LLM.
A tweet thread shares a curated list of resources for learning LLM inference, covering tokenization, GPU hardware, optimization techniques like quantization and speculative decoding, and inference engines such as vLLM and SGLang, followed by an Nvidia architect's talk summary.
A comprehensive guide to optimizing local LLM inference on consumer hardware, covering tools like llama.cpp, vLLM, and LM Studio, with practical advice on memory hierarchy, layer placement, and common failure modes.
LLMSys-PaperList is a curated reading list on GitHub that organizes LLM systems research papers and resources into practical categories such as training systems, serving systems, and multi-modal coverage, helping AI/ML engineers and researchers stay updated.