How to run Prism Bonsai 27B

According to their GitHub, the file needed is the -Q2_0.gguf version, which requires compiling their version of llama.cpp. Easiest way to download is to use huggingface-cli if you have it available: hf download prism-ml/Ternary-Bonsai-27B-gguf --include "*-Q2_0.gguf*" If not, just download the file from https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf/tree/main and save it somewhere. Now, we have to compile llama.cpp: ``` git clone https://github.com/PrismML-Eng/llama.cpp prism-llama.cpp cd prism-llama.cpp For Mac: cmake -B build -DGGML_METAL=ON cmake --build build --config Release -j$(sysctl -n hw.logicalcpu) ``` I'm using -DGGML_METAL=ON since this is a mac. If you are on another platform and need CUDA you can use something like this: ``` cmake -B build -DGGML_CUDA=ON cmake --build build --config Release -j$(nproc) ``` Note also the varying format of -j that's just the number of cores you want to use for compilation. Now, you can run llama-server via: build/bin/llama-server -hf prism-ml/Ternary-Bonsai-27B-gguf:Ternary-Bonsai-27B-Q2_0 --port 8080 --host 0.0.0.0 --temp 0.7 --top-p 0.95 --top-k 20 If you downloaded the file by hand, pass -m with the path to the gguf instead of the -hf switch. Add any extra parameters as needed. --temp 0.7 --top-p 0.95 --top-k 20 are the recommended defaults according to their page. Finally, you can visit localhost:8080 in your browser to interact with it. Note that --host 0.0.0.0 will bind with all interfaces, so it will listen internally and externally. If you don't need that, remove that switch.