MLX 16/8/4/2-bit quants of nvidia/llama-embed-nemotron-8b

Reddit r/LocalLLaMA Models
mlx quantization embedding llama nemotron apple-silicon open-source

Summary

The user converted Nvidia's Llama-Embed-Nemotron-8B model to MLX format with fp16, 8-bit, 4-bit, and 2-bit quantizations, enabling in-process embedding loading on Apple Silicon via mlx-embeddings.

I converted nvidia/llama-embed-nemotron-8b to MLX fp16, 8-bit, 4-bit, and 2-bit (for my OCD) and put it on HuggingFace: [ncorder/llama-embed-nemotron-8b-mlx-fp16](https://huggingface.co/ncorder/llama-embed-nemotron-8b-mlx-fp16) [ncorder/llama-embed-nemotron-8b-mlx-8bit](https://huggingface.co/ncorder/llama-embed-nemotron-8b-mlx-8bit) [ncorder/llama-embed-nemotron-8b-mlx-4bit](https://huggingface.co/ncorder/llama-embed-nemotron-8b-mlx-4bit) [ncorder/llama-embed-nemotron-8b-mlx-2bit](https://huggingface.co/ncorder/llama-embed-nemotron-8b-mlx-2bit) -- I was running this model using GGUFs + llama-server for local semantic search over an Obsidian vault and some other projects. It worked fine but I got tired of managing a whole HTTP server just for embeddings and also wanted Apple Silicon optimizations. The MLX version loads in-process via mlx-embeddings, no server. from mlx_embeddings import load_model, encode model, tokenizer = load_model("ncorder/llama-embed-nemotron-8b-mlx-4bit") embeddings = encode(model, tokenizer, ["your text here"]) Enjoy!
Original Article

Similar Articles

Qwen3.8-27B Hybrid IQ4_XS quantization for 16GB gang

Reddit r/LocalLLaMA

This is a quantized version of the Qwen3.8-27B AI model using IQ4_XS quantization, optimized for 16GB RAM systems, with instructions for local deployment using various tools like llama.cpp and Ollama.

Qwen3.6-35B-A3B-Abliterated-Heretic-MLX-4bit

Reddit r/LocalLLaMA

The user reviews a quantized and fine-tuned version of the Qwen3.6-35B model optimized for Apple Silicon via MLX, praising its speed, intelligence, and lack of safety disclaimers.