The article presents a benchmark tool for evaluating local LLM configurations, focusing on VRAM usage, performance metrics, and hardware optimization to assist developers in optimizing setups.
New/Old benchmark that provides a lot of answers for local LLM. I present to you a new test that I developed somewhat by accident: https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF/tree/main/ctx-cliff Its original goal was to test whether a model fits into VRAM under a specific llama-server configuration. Theoretically a simple matter, but when you want to squeeze the absolute maximum out of your hardware and configure the server manually, things get quite complicated—especially when using MTP, ngram, dflash, etc. For example, if you own an NVIDIA card and want to maximize VRAM usage, it turns out that setting the environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 is critical. When defined, CUDA allocations pass through cudaMallocManaged (ggml-cuda.cu:181), handing over memory management to the NVIDIA card's hardware MMU: Elimination of fragmentation (4 KB / 2 MB granularity vs. VMM Pool): The standard VMM allocator (cuMemCreate) in llama.cpp forces large, rigid block allocations and triggers a hard OOM if the card lacks contiguous space. cudaMallocManaged operates on very fine-grained physical pages (4 KB / 2 MB), allowing the NVIDIA driver to stitch together small free fragments of VRAM without throwing an allocation error. Without this parameter, you effectively lose a lot of VRAM capacity. However, enabling it also activates the VRAM-to-RAM offload mechanism, so you need to determine how much context space you actually have under your specific conditions. That is why this test was originally created—you can clearly see the "cliff" when VRAM runs out. Generally, NVIDIA's MMU behavior is quite complex, and this cliff can sometimes be surprising. Besides prefill and decode speed, the test also measures wall time (total request handling time). If the model and the llama-server configuration are flawed, this time can drastically increase with a growing context because the model starts re-reading the entire context from the beginning—completely breaking agentic workflows. Additionally, the script detects empty responses and anomalies (>1000 t/s). If such anomalies occur consistently, the quantization is broken. So, by observing the occurrence of anomalies and the wall time, you can determine with a very good approximation whether a given model and llama-server configuration are suitable for actual work. Here is an example output of the script for the reference model cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF with the llama-server settings below: llama-server \ -m "$MODEL_PATH" \ -a Qwen3.6-27B \ --ctx-size 110000 \ --n-gpu-layers 99 \ --cache-type-k q4_0 \ --cache-type-v q4_0 \ --batch-size 512 \ --ubatch-size 128 \ --flash-attn on \ --host 0.0.0.0 \ --port 8081 \ --reasoning on \ --reasoning-format none \ --reasoning-budget 32000 \ -t 8 \ -tb 8 \ --parallel 1 \ --metrics \ --merge-qkv \ -khad \ -vhad \ --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \ --defrag-thold 0.1 \ --jinja \ --cont-batching \ --temp 1.0 \ --top-k 20 \ --min-p 0.00 \ --top-p 0.95 \ --presence-penalty 0.0 \ --repeat-last-n 512 \ --repeat-penalty 1.00 1. Reference Model Results python3 ctx-cliff.py --file tests/code_4M.py --start 2000 --end 109000 --step 2000 --n-predict 512 ctx | prefill| decode| MTP| wall| status ------------------------------------------------- 1999 | 1021.1 | 46.72| 0/0| 11.4s| OK 3925 | 1320.9 | 46.06| 0/0| 12.6s| OK 6017 | 1261.5 | 45.12| 0/0| 13.0s| OK 8065 | 1293.4 | 44.20| 0/0| 13.2s| OK 10218 | 1191.9 | 43.36| 0/0| 13.6s| OK 12489 | 1184.3 | 42.39| 0/0| 14.0s| OK 14525 | 1228.9 | 41.66| 0/0| 15.9s| OK 16108 | 1258.8 | 41.16| 0/0| 17.6s| OK 18976 | 1237.8 | 40.13| 0/0| 20.8s| OK 20476 | 1058.4 | 39.67| 0/0| 15.0s| OK 22574 | 1091.4 | 39.22| 0/0| 15.8s|STOP@463 24950 | 1082.3 | 38.15| 0/0| 16.3s| OK 26551 | 1060.3 | 37.79| 0/0| 18.0s| OK 29197 | 1058.8 | 37.16| 0/0| 20.7s| OK 30559 | 1059.6 | 36.81| 0/0| 22.1s| OK 32691 | 1048.1 | 36.17| 0/0| 24.5s| OK 34235 | 1046.3 | 35.75| 0/0| 26.2s| OK 36569 | 1037.3 | 35.22| 0/0| 28.7s| OK 38356 | 1027.6 | 34.94| 0/0| 30.7s| OK 40912 | 1014.6 | 34.14| 0/0| 33.8s| OK 42569 | 1010.3 | 34.00| 0/0| 35.6s| OK 44532 | 1002.9 | 33.27| 0/0| 32.1s|STOP@316 47017 | 994.5 | 32.53| 0/0| 44.2s| OK 48257 | 997.5 | 32.83| 0/0| 47.7s| OK 51210 | 996.4 | 32.06| 0/0| 53.1s| OK 52481 | 826.8 | 31.97| 0/0| 18.4s| OK 54608 | 803.1 | 31.43| 0/0| 18.9s| OK 56263 | 775.9 | 31.16| 0/0| 18.6s| OK 58871 | 847.3 | 30.54| 0/0| 24.2s| OK 60014 | 795.5 | 30.38| 0/0| 21.1s| OK 62496 | 825.7 | 29.95| 0/0| 26.7s| OK 64364 | 780.5 | 29.67| 0/0| 23.2s| OK 65843 | 775.2 | 29.08| 0/0| 25.5s| OK 67130 | 746.5 | 28.85| 0/0| 22.2s| OK 68719 | 757.5 | 28.73| 0/0| 24.3s| OK 70803 | 753.8 | 28.45| 0/0| 27.3s| OK 72826 | 712.6 | 28.20| 0/0| 22.2s| OK 74889 | 723.3 | 27.72| 0/0| 25.3s| OK 76819 | 725.8 | 27.43| 0/0| 28.1s| OK 78975 | 723.9 | 27.09| 0/0| 31.4s| OK 81045 | 678.9 | 26.81| 0/0| 23.4s| OK 83184 | 755.7 | 26.48| 0/0| 36.9s| OK 85162 | 712.9 | 26.14| 0/0| 41.0s| OK 87191 | 684.5 | 25.90| 0/0| 31.5s| OK 89098 | 728.9 | 25.66| 0/0| 44.9s| OK 90969 | 706.3 | 25.30| 0/0| 50.0s| OK 93074 | 696.8 | 25.12| 0/0| 53.6s| OK 95132 | 654.1 | 24.84| 0/0| 34.1s| OK 97250 | 614.2 | 24.56| 0/0| 25.3s| OK 99301 | 680.1 | 24.27| 0/0| 40.2s| OK 101183 | 629.1 | 24.14| 0/0| 31.9s| OK 103237 | 668.2 | 23.83| 0/0| 46.8s| OK 105209 | 624.7 | 23.64| 0/0| 38.8s| OK 107265 | 655.6 | 23.37| 0/0| 53.9s| OK A model with a similar PPL but smaller, generated using https://github.com/Thireus/GGUF-Tool-Suite. The model parameters are identical. You can see one anomaly, which means the model completely failed. Additionally, there are a lot of STOPs. The script commands the model to continue generating the code up to 512 tokens; if it finishes much earlier, it means it gave up—which is not a good sign. 2. Thireus Model (Same Parameters) python3 ctx-cliff.py --file tests/code_4M.py --start 2000 --end 109000 --step 2000 --n-predict 512 ctx | prefill| decode| MTP| wall| status ------------------------------------------------- 1998 | 1066.5 | 46.88| 0/0| 11.4s| OK 3925 | 1235.6 | 46.17| 0/0| 12.7s| OK 6017 | 1210.7 | 45.29| 0/0| 13.1s| OK 8065 | 1240.4 | 44.65| 0/0| 13.1s| OK 10213 | 1192.5 | 44.06| 0/0| 13.4s| OK 12491 | 1185.0 | 43.29| 0/0| 13.8s| OK 14525 | 1175.7 | 42.50| 0/0| 13.8s| OK 16107 | 1113.7 | 41.89| 0/0| 14.1s| OK 18975 | 1093.7 | 40.82| 0/0| 15.2s| OK 20478 | 1051.8 | 40.30| 0/0| 14.9s| OK 22569 | 1050.0 | 39.66| 0/0| 15.7s| OK 24955 | 1017.4 | 38.79| 0/0| 16.4s| OK 26546 | 982.8 | 38.33| 0/0| 15.7s| OK 29207 | 988.5 | 37.55| 0/0| 17.1s| OK 30540 | 939.4 | 38.14| 0/0| 3.5s| STOP@46 32705 | 933.1 | 36.57| 0/0| 16.4s| OK 34235 | 919.3 | 36.11| 0/0| 16.6s| OK 36570 | 972.2 | 35.51| 0/0| 20.7s| OK 38357 | 906.9 | 35.32| 0/0| 11.1s|STOP@245 40905 | 891.6 | 34.40| 0/0| 17.8s| OK 42573 | 849.8 | 33.93| 0/0| 17.1s| OK 44530 | 863.8 | 33.54| 0/0| 19.5s| OK 47019 | 861.3 | 32.96| 0/0| 20.3s| OK 48258 | 889.6 | 32.69| 0/0| 24.0s| OK 51204 | 834.0 | 31.96| 0/0| 22.3s| OK 52488 | 802.6 | 31.76| 0/0| 19.1s| OK 54607 | 791.4 | 31.32| 0/0| 18.8s| OK 56259 | 794.5 | 31.00| 0/0| 20.4s| OK 58873 | 789.4 | 31.19| 0/0| 8.7s| STOP@46 60016 | 727.6 |ANOMALY| 0/0| 1.6s| STOP@1 62496 | 776.2 | 31.24| 0/0| 9.0s| STOP@29 64364 | 803.6 | 29.57| 0/0| 31.3s| OK 65843 | 755.4 | 29.31| 0/0| 24.1s| OK 67129 | 715.9 | 29.06| 0/0| 20.7s| OK 68719 | 727.0 | 28.77| 0/0| 23.0s| OK 70804 | 731.2 | 28.57| 0/0| 26.0s| OK 72828 | 704.1 | 28.08| 0/0| 22.9s| OK 74885 | 752.9 | 27.64| 0/0| 31.8s| OK 76819 | 708.7 | 28.45| 0/0| 11.5s| STOP@36 78976 | 737.6 | 27.30| 0/0| 37.8s| OK 81044 | 678.9 | 27.01| 0/0| 25.4s| OK 83184 | 679.5 | 26.64| 0/0| 28.8s| OK 85162 | 677.7 | 26.39| 0/0| 31.9s| OK 87190 | 649.3 | 26.10| 0/0| 25.5s| OK 89100 | 651.2 | 25.86| 0/0| 28.6s| OK 90967 | 652.3 | 25.53| 0/0| 31.7s| OK 93075 | 651.7 | 25.26| 0/0| 35.2s| OK 95132 | 626.9 | 25.02| 0/0| 27.8s| OK 97248 | 623.9 | 24.77| 0/0| 31.5s| OK 99303 | 623.2 | 24.46| 0/0| 35.0s| OK 101182 | 603.1 | 24.25| 0/0| 26.9s| OK 103236 | 604.7 | 24.03| 0/0| 30.5s| OK 105211 | 601.1 | 23.81| 0/0| 34.0s| OK 107265 | 599.9 | 23.51| 0/0| 37.8s| OK I improved the KV cache to 5_0/4_1 and unfortunately, it doesn't help (but at least there is no anomaly). 3. Thireus Model (KV cache 5_0/4_1) python3 ctx-cliff.py --file tests/code_4M.py --start 2000 --end 109000 --step 2000 --n-predict 512 ctx | prefill| decode| MTP| wall| status ------------------------------------------------- 2000 | 1084.7 | 46.89| 0/0| 11.3s| OK 3923 | 1239.4 | 45.72| 0/0| 12.8s| OK 6016 | 1208.8 | 45.13| 0/0| 13.1s| OK 8066 | 1183.9 | 44.50| 0/0| 13.2s| OK 10212 | 1187.1 | 43.90| 0/0| 13.5s| OK 12492 | 1182.5 | 43.10| 0/0| 13.8s| OK 14525 | 1174.1 | 42.20| 0/0| 13.9s| OK 16108 | 1113.4 | 43.54| 0/0| 2.5s| STOP@24 18973 | 1089.8 | 40.52| 0/0| 15.3s| OK 20482 | 1050.8 | 39.93| 0/0| 15.1s| OK 22566 | 1043.5 | 39.41| 0/0| 15.8s| OK 24953 | 1042.4 | 38.50| 0/0| 18.4s| OK 26552 | 991.8 | 38.03| 0/0| 15.8s| OK 29194 | 1028.3 | 37.28| 0/0| 19.5s| OK 30562 | 1026.9 | 36.86| 0/0| 22.0s| OK 32692 | 1022.7 | 36.30| 0/0| 25.4s| OK 34234 | 1020.3 | 35.85| 0/0| 28.1s| OK 36569 | 1016.1 | 35.24| 0/0| 31.7s| OK 38356 | 1010.2 | 34.74| 0/0| 34.8s| OK 40910 | 887.9 | 34.09| 0/0| 18.8s| OK 42570 | 875.7 | 33.66| 0/0| 18.6s| OK 44532 | 852.3 | 33.26| 0/0| 18.8s| OK 47017 | 845.9 | 32.68| 0/0| 19.6s| OK 48257 | 850.6 | 32.43| 0/0| 21.1s| OK 51211 | 847.9 | 31.73| 0/0| 25.0s| OK 52480 | 842.7 | 31.47| 0/0| 26.7s| OK 54609 | 839.9 | 31.03| 0/0| 29.5s| OK 56262 | 837.3 | 30.67| 0/0| 31.7s| OK 58872 | 828.8 | 30.35| 0/0| 24.0s|STOP@174 60015 | 826.2 | 29.93| 0/0| 36.9s| OK 62494 | 820.2 | 29.49| 0/0| 40.3s| OK 64364 | 813.8 | 29.18| 0/0| 42.9s| OK 65844 | 809.5 | 28.80| 0/0| 45.1s| OK 67130 | 807.6 | 29.44| 0/0| 30.4s| STOP@42 68719 | 802.3 | 28.43| 0/0| 49.2s| OK 70804 | 796.7 | 29.31| 0/0| 35.0s| STOP@27 72827 | 791.2 | 27.80| 0/0| 55.2s| OK 74887 | 786.5 | 27.32| 0/0| 58.4s| OK 76819 | 780.3 | 27.12| 0/0| 61.3s| OK 78975 | 774.6 | 26.79| 0/0| 64.7s| OK 81045 | 769.1 | 27.56| 0/0| 49.8s| STOP@32 83185 | 762.2 | 26.31| 0/0| 71.3s| OK 85162 | 759.4 | 25.98| 0/0| 74.3s| OK 87189 | 753.7 | 25.76| 0/0| 77.6s| OK 89099 | 748.6 | 25.32| 0/0| 83.3s| OK 90966 | 744.0 | 24.88| 0/0| 90.3s| OK 93077 | 626.4 | 24.79| 0/0| 29.3s| OK 95133 | 631.5 | 24.52| 0/0| 32.7s| OK 97247 | 630.8 | 24.29| 0/0| 36.3s| OK 99302 | 597.6 | 23.96| 0/0| 27.1s| OK 101182 | 651.8 | 23.86| 0/0| 42.2s| OK 103238 | 601.5 | 23.63| 0/0| 33.9s| OK 105210 | 637.7 | 23.36| 0/0| 49.4s| OK 107265 | 560.5 | 23.14| 0/0| 27.8s| OK Now, an even smaller model with MTP Qwen3.8-27B.i1-thireus-37087.gguf (also from the https://github.com/Thireus/GGUF-Tool-Suite project): 4. Smaller MTP Model (Qwen3.8-27B.i1-thireus-37087.gguf) llama-server \ -m "$MODEL_PATH" \ -a Qwen3.6-27B \ --ctx-size 110000 \ --n-gpu-layers 99 \ --cache-type-k q4_0 \ --cache-type-v q4_0 \ --spec-type mtp:n_max=3 \ --batch-size 512 \ --ubatch-size 128 \ --flash-attn on \ --host 0.0.0.0 \ --port 8081 \ --reasoning on \ --reasoning-format none \ --reasoning-budget 32000 \ -t 8 \ -tb 8 \ --parallel 1 \ --metrics \ --merge-qkv \ -khad \ -vhad \ --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \ --defrag-thold 0.1 \ --jinja \ --cont-batching \ --temp 1.0 \ --top-k 20 \ --min-p 0.00 \ --top-p 0.95 \ --presence-penalty 0.0 \ --repeat-last-n 512 \ --repeat-penalty 1.00 python3 ctx-cliff.py --file tests/code_4M.py --start 2000 --end 109000 --step 2000 --n-predict 512 ctx | prefill| decode| MTP| wall| status ------------------------------------------------- 1998 | 896.4 | 64.97| 163/283| 6.2s|STOP@366 3925 | 1003.6 | 73.53| 283/429| 8.9s| OK 6017 | 979.6 | 56.21| 2/6| 2.3s| STOP@9 8065 | 1003.7 | 69.92| 177/252| 6.9s|STOP@341 10213 | 968.4 | 91.63| 346/397| 7.8s| OK 12491 | 962.3 | 88.80| 345/393| 8.2s| OK 14525 | 951.1 | 97.63| 369/381| 7.4s| OK 16107 | 889.1 | 95.37| 372/391| 7.2s| OK 18975 | 896.0 | 93.95| 376/385| 8.7s| OK 20478 | 854.9 | 60.77| 275/399| 10.2s| OK 22569 | 836.4 | 86.47| 369/397| 8.5s| OK 24955 | 834.6 | 45.57| 17/39| 4.0s| STOP@48 26546 | 792.2 | 81.67| 367/397| 8.3s| OK 29207 | 804.0 | 83.34| 374/385| 9.5s| OK 30540 | 764.8 |ANOMALY| 0/0| 1.8s| STOP@1 32705 | 784.0 | 82.90| 379/384| 9.0s| OK 34235 | 771.5 | 79.05| 374/389| 8.5s| OK 36570 | 756.3 | 37.68| 174/443| 16.7s| OK 38357 | 739.5 | 72.35| 365/389| 9.6s| OK 40905 | 741.6 | 64.33| 346/397| 11.4s| OK 42573 | 702.4 | 58.60| 330/392| 11.2s| OK 44530 | 706.4 | 65.28| 356/399| 10.7s| OK 47019 | 728.9 | 63.39| 354/399| 16.5s| OK 48258 | 671.2 | 40.75| 189/325| 11.7s|STOP@399 51204 | 684.8 | 70.32| 380/386| 11.7s| OK 52488 | 645.8 | 69.75| 380/385| 9.4s| OK 54607 | 655.6 | 66.82| 376/384| 11.0s| OK 56259 | 644.3 | 66.37| 377/383| 10.3s| OK 58873 | 645.4 | 36.43| 5/8| 4.5s| STOP@14 60016 | 616.8 | 41.44| 284/423| 14.3s| OK 62496 | 650.2 |ANOMALY| 0/0| 8.1s| STOP@1 64364 | 655.7 | 29.32| 174/409| 28.3s| OK 65843 | 650.3 | 38.03| 270/399| 26.7s| OK 67129 | 647.0 | 40.49| 294/432| 27.9s| OK 68719 | 643.6 | 49.23| 340/401| 28.3s| OK 70804 | 639.5 | 49.77| 345/394| 31.5s| OK 72828 | 637.3 | 35.01| 260/402| 39.1s| OK 74885 | 632.3 | 46.68| 339/403| 38.9s| OK 76819 | 628.4 | 34.19| 261/399| 46.2s| OK 78976 | 626.2 | 33.06| 255/414| 50.2s| OK 81044 | 622.8 | 46.84| 349/408| 49.2s| OK 83184 | 618.8 | 41.50| 326/417| 54.3s| OK 85162 | 614.6 | 41.77| 328/398| 57.7s| OK 87190 | 611.1 | 33.41| 276/398| 64.4s| OK 89100 | 607.9 | 41.96| 337/403| 64.7s| OK 90967 | 605.4 | 43.65| 59/69| 57.8s| STOP@89 93075 | 602.0 | 35.67| 308/428| 73.9s| OK 95132 | 598.6 | 34.31| 302/432| 78.3s| OK 97248 | 594.5 | 35.33| 310/409| 81.8s| OK 99303 | 590.5 | 48.65| 376/383| 81.8s| OK 101182 | 588.6 | 35.24| 313/399| 89.3s| OK 103236 | 584.3 | 28.77| 263/400| 96.6s| OK 105211 | 580.8 | 28.18| 263/421|100.8s| OK 107265 | 577.3 | 32.55| 306/424|102.5s| OK I improved the KV cache to 5_0/4_1 and unfortunately, it doesn't help. Additionally, you can see the cliff (running out of VRAM) at around 107k ctx: 5. Smaller MTP Model (KV cache 5_0/4_1) python3 ctx-cliff.py --file tests/code_4M.py --start 2000 --end 109000 --step 2000 --n-predict 512 ctx | prefill| decode| MTP| wall| status ------------------------------------------------- 1999 | 891.2 | 71.04| 261/394| 7.8s| OK 3924 | 1012.0 | 83.35| 313/390| 8.1s| OK 6011 | 986.5 | 71.66| 280/420| 9.3s| OK 8068 | 971.0 | 58.74| 211/409| 10.9s| OK 10211 | 974.4 | 92.03| 351/387| 7.8s| OK 12493 | 969.9 | 50.84| 174/415| 12.5s| OK 14526 | 957.7 | 62.36| 267/419| 10.4s| OK 16104 | 950.2 | 96.22| 376/388| 9.4s| OK 18976 | 948.0 | 90.77| 372/387| 13.3s| OK 20476 | 943.3 | 74.76| 334/390| 16.7s| OK 22573 | 941.9 | 83.97| 365/389| 19.3s| OK 24951 | 834.7 | 82.14| 367/403| 9.1s| OK 26549 | 797.2 | 82.96| 371/390| 8.2s| OK 29200 | 806.2 | 84.26| 378/388| 9.4s| OK 30551 | 770.6 | 75.50| 362/404| 8.6s| OK 32701 | 772.6 | 73.78| 360/394| 9.8s| OK 34234 | 770.6 | 75.96| 369/396| 8.8s| OK 36570 | 758.1 | 43.86| 31/55| 4.8s| STOP@73 38356 | 742.0 | 69.43| 360/402| 9.8s| OK 40903 | 744.5 | 71.90| 369/389| 10.6s| OK 42576 | 700.6 | 63.96| 350/400| 10.4s| OK 44528 | 711.8 | 70.52| 372/394| 10.1s| OK 47020 | 710.2 | 63.48| 357/404| 11.6s| OK 48258 | 671.8 | 71.63| 380/385| 9.1s| OK 51203 | 688.9 | 63.51| 363/393| 12.4s| OK 52490 | 647.7 | 68.03| 377/380| 9.6s| OK 54608 | 656.8 | 63.12| 368/397| 11.4s| OK 56256 | 646.3 | 60.42| 362/401| 11.1s| OK 58874 | 648.5 | 63.90| 375/386| 12.1s| OK 60017 | 665.3 | 60.19| 367/396| 17.6s| OK 62495 | 632.9 | 45.47| 71/88| 6.6s|STOP@119 64364 | 615.8 | 29.87| 375/387| 20.3s| OK 65843 | 503.4 | 37.08| 273/415| 17.0s| OK 67129 | 561.9 | 45.21| 324/414| 13.8s| OK 68720 | 573.6 | 49.34| 343/401| 13.3s| OK 70803 | 580.2 | 28.51| 17/40| 5.4s| STOP@48 72829 | 585.3 | 44.45| 327/388| 15.1s| OK 74885 | 558.0 | 36.83| 286/428| 17.7s| OK 76820 | 556.5 | 21.75| 8/34| 5.5s| STOP@42 78975 | 569.7 | 35.10| 277/393| 22.8s| OK 81045 | 564.2 | 32.13| 258/419| 27.9s| OK 83184 | 570.7 | 42.34| 333/401| 27.7s| OK 85162 | 563.8 | 30.75| 253/419| 35.9s| OK 87191 | 566.8 | 31.06| 263/420| 39.3s| OK 89097 | 564.1 | 23.56| 8/17| 27.3s| STOP@25 90967 | 560.6 | 32.94| 288/422| 45.3s| OK 93074 | 558.3 | 26.24| 220/427| 53.2s| OK 95133 | 557.2 | 30.89| 275/405| 54.0s| OK 97248 | 555.1 | 42.17| 349/401| 53.5s| OK 99303 | 556.5 |ANOMALY| 0/0| 45.0s| STOP@1 101182 | 556.8 | 28.24| 263/403| 66.5s| OK 103236 | 522.2 | 32.99| 345/410| 71.1s| OK 105209 | 514.6 | 28.73| 347/388| 78.0s| OK 107267 | 539.1 | 9.49| 75/142| 81.3s|STOP@190
Homebench is a zero-config terminal tool that benchmarks locally-run LLMs for speed, memory, and quality, presenting a live leaderboard. It supports Ollama, LM Studio, llama.cpp, vLLM, and OpenAI-compatible servers.
A detailed benchmark of 20 small LLMs quantized for a 6GB GPU, measuring speed and VRAM usage at various context lengths, with qualitative probing for tool-use and instruction following. The report aims to help users with modest hardware choose models for local, private automation tasks.
whichllm is an open-source Python tool that auto-detects your GPU/CPU/RAM and ranks the best local LLMs from HuggingFace that fit your system, using real benchmarks rather than size heuristics.
A web tool that calculates whether a given local LLM can run on specified hardware by factoring in quantization, KV cache, and VRAM overhead, providing estimated speed and memory usage.
A comprehensive guide to optimizing local LLM inference on consumer hardware, covering tools like llama.cpp, vLLM, and LM Studio, with practical advice on memory hierarchy, layer placement, and common failure modes.