The user describes struggles with context, compaction, and memory management for local AI models using pi.dev plugins and seeks suggestions for solutions that handle varying model context windows and VRAM limitations.
I have been battling with my Qwen3.8:27b setup on my rtx 5080 16gb. I am using llama.cpp to run a nvfp4 version of qwen3.8:27b llama-b10699-bin-win-cuda-13.3-x64\llama-server.exe -hf williamliao/Qwen3.8-27B-NVFP4-GGUF:NVFP4 --jinja --chat-template-file d:\qwen\chat_template.jinja --reasoning-format deepseek --host 127.0.0.1 --offline -np 1 --threads 16 --threads-batch 16 --log-verbosity 4 --no-log-prefix --no-log-timestamps --no-mmproj-offload --mmproj D:\OllamaModels\blobs\sha256-ac3714bfdddeca31351f2752bf1a63f266f4df87c0b68c895e44945ca704448e --load-mode none --cache-type-k q4_0 --cache-type-v iq4_nl --flash-attn on -b 512 -ub 512 -c 49152 -a qwen3.8:27b So that is giving me a 48k context, and a speed of around 12t/s My problem is that pi.dev fails to compact at the right times, and then the model stops early. I have tried to use pi install npm:max-context //this doesnt work properly at all pi install npm:pi-observational-memory //this one works kind of, I had it set to compact on threshold. It managed to auto compact 4 times, with a 0.75 ratio set, but it still missed the mark and failed. pi install npm:pi-blackhole //seemed promising, but failed to compact at all, I am not sure what I set wrong. if I look at the https://pi.dev/packages page, there are lots of plugins for either context management, compaction, memory management, or both. there are way too many for me to just randomly keep trying. other AI has been very unhelpful. Google, OpenAI, Copilot, all full of nonsense. A lot of these plugins seem to be aimed at frontier model management. I am also disappointed that compaction settings are best suited to the specific model being used and its context window, but the built in tools in pi.dev architecturally keep the compaction settings completely separate from model settings, which seems a bit of an issue if you change models often. I may want to use qwen3.8 like above, but not necessarily for everything. I also want to use gemma, other qwen models, etc depending on my use case. I dont want to have to manually resetup context settings in pi.dev for every different model. What do you guys suggest, especially for models that have smaller context windows due to my vram shortage ? Thanks in Advance.
A developer discusses challenges and seeks advice on enhancing reasoning and memory handling in a local AI companion, focusing on context selection and dealing with small models.
The user asks for recommendations on the best local AI models for different VRAM classes (8-16GB, 24-32GB, 128GB), mentioning Gemma4, Qwen, and DeepSeek variants, as they work on improving local model support in Hermes.
A discussion about the performance of running large language models locally on CPU, especially with large context sizes, and the challenges of VRAM constraints.
A discussion thread sharing configurations and tips for running AI models like Qwen3.8-27B on 16 GB VRAM Windows systems, focusing on memory optimization techniques.
The user discusses their experience with Qwen 3.6 27B for local coding tasks and asks for recommendations for larger models (100B+) suitable for systems with 224GB of VRAM.