Tag
Comparison of four large language models (≤120B parameters) on deep context performance using Strix Halo hardware. Nemotron Super excels in prompt processing speed at deep context depths compared to GPT-OSS and Qwen models.
A user shares their experience running local LLMs on Mac, noting that prompt processing is slow for AI agents compared to Nvidia GPUs, and recommends cloud models like Deepseek unless privacy is a concern.
A pull request for llama.cpp fixes the constant prompt processing issue that occurs when using OpenCode or Pi with the library.
A user benchmarks the Nvidia 5090 RTX GPU for LLM inference using llama.cpp, measuring prompt processing and token generation at various power levels, finding that prompt processing is more sensitive to power limits than token generation, and noting differences from the 4090 RTX.
The article shares a performance optimization trick for llama.cpp, showing that increasing the micro-batch size (`-ub`) combined with partial CPU offloading (`--n-cpu-moe`) can drastically improve prompt processing speed for large models like gpt-oss-120b on consumer GPUs.