@berryxia: Clarifying Large Model Formats Once and For All! Let's Dive In! Many friends have been discussing the myriad formats of large models and wondering what the differences are. Thus, I decided to write a piece to clarify local large model formats like GGUF and MLX. Simply put, GGUF is a single-file format developed by the llama.cpp team and is now the most mainstream choice for local inference....

X AI KOLs Timeline Tools
llm-formats gguf mlx local-inference quantization llama-cpp hardware-optimization

Summary

This article provides a detailed comparison of the features and application scenarios of mainstream local large model file formats such as GGUF, MLX, and Safetensors, helping developers choose the optimal format based on their hardware environment.

Clarifying Large Model Formats Once and For All! Let's Dive In! Many friends have been discussing the myriad formats of large models and wondering what the differences actually are. Thus, I decided to write a piece to clarify local large model formats like GGUF and MLX. Simply put, GGUF is a single-file format developed by the llama.cpp team and is now the most mainstream choice for local inference. The older GGML format has been largely phased out; now, a single .gguf file contains the header, metadata, and all tensors, resulting in very fast loading speeds. It also supports various K-quants quantization methods, allowing flexible mixing from 2-bit to 8-bit. Tools like llama.cpp, Ollama, and LM Studio essentially rely on it, and there are abundant ready-to-use GGUF versions on Hugging Face. MLX, on the other hand, is a framework optimized specifically by Apple for M-series chips, with mlx-lm being its LLM implementation. The model format is not a single file but a directory containing config, tokenizer, and weights (commonly in .npz or safetensors formats). It runs exceptionally smoothly on Macs due to the unified memory architecture, ensuring stability in loading and context processing. It also supports direct 4-bit quantization or mixed precision during conversion. There are also other less mainstream core formats: * **PyTorch / Safetensors**: The standard for training and sharing. * **ONNX**: Suitable for cross-framework deployment. * **TensorRT**: Dedicated to NVIDIA's ultimate performance needs. The most interesting aspect is: GGUF offers a single file, the most flexible quantization, and strong cross-platform compatibility, making it particularly suitable for CPUs and low-spec devices. MLX offers the best speed and fine-tuning experience on Macs. Safetensors is large in size but convenient for sharing; ONNX is more oriented toward production deployment. In fact, these formats are not directly used during the pre-training phase. After large models are trained, they exist as PyTorch/Safetensors checkpoints. Quantization (PTQ) is performed during subsequent conversion steps, where GGUF and MLX excel in handling quantization efficiently. In summary, Mac users should prioritize MLX, while cross-platform or low-spec devices should choose GGUF. Safetensors remains the most reliable option for training and sharing. For those interested in tinkering with local LLMs, feel free to check out the comparison chart.
Original Article

Similar Articles

@NFTCPS: Running large models locally – looking at the hundreds of GB of weights and VRAM requirements, most people are discouraged immediately, and I was too. Colibri changes the game by treating VRAM, RAM, and disk as a unified hierarchy, streaming weights from disk on demand. Written in pure C with zero dependencies, it already has over 25,000 stars. Here are a few points: …

X AI KOLs Timeline

Introducing colibri, an open-source inference engine written in pure C that unifies VRAM, RAM, and disk as a hierarchical structure to stream large model weights, supporting various cutting-edge MoE models to run locally on consumer hardware, lowering the barrier to using large models.

@wsl8297: Sharing an easy-to-read open-source book 'Foundations of Large Models'. From an introduction to large language models to architectural evolution, then to key technologies such as Prompt engineering, parameter-efficient fine-tuning, model editing, retrieval-augmented generation (RAG), all in one book. GitHub: https://github.com/ZJU-LLMs/…

X AI KOLs Timeline

The Zhejiang University team open-sourced an easy-to-understand textbook on large models 'Foundations of Large Models', covering from architectural evolution to key technologies like RAG, accompanied by the Agent-Kernel multi-agent framework.

@cevenif: For those running local LLMs on Macs, here's a tool worth watching — Rapid-MLX. It delivers 2-4x faster inference on M-series chips than Ollama, thanks to being built directly on Apple's MLX framework for more thorough utilization of the chip architecture. Key highlights: KV cache pruning plus…

X AI KOLs Timeline

Rapid-MLX is a local LLM inference tool optimized for Apple M-series chips. Built on the MLX framework, it achieves 2 to 4 times faster inference than Ollama, supports multiple models, tool calling, and an OpenAI API-compatible interface.

@NFTCPS: 4GB VRAM running 70B large model? It actually works! AirLLM did a clever trick — layered inference, not loading the whole model into VRAM at once, but layer by layer, compute and discard, squeezing the giant into a small GPU. The best part: 100% open source, freebie warning https://github.com/0xSo…

X AI KOLs Timeline

AirLLM is a fully open-source tool that uses layered inference (loading and releasing VRAM layer by layer) to enable 70B large language models to run on GPUs with only 4GB VRAM, without quantization, distillation, or pruning. It already supports running Llama3.1 405B on 8GB VRAM.