Saluki 27B: "96% of Qwen 3.8’s performance at ~1/7 the size"

# Underdog Saluki 27B Source: [https://underdog.ai/saluki](https://underdog.ai/saluki) Our models Qwen3\.8\-27B is an open source model that beats Claude Opus 4\.6 Max, the frontier model from six months ago, according to[Artificial Analysis](https://artificialanalysis.ai/models/comparisons?compare=qwen3-8-27b,claude-opus-4-6-adaptive)\. We squeezed it into a 7\.89 GB file that runs on standard llama\.cpp\. We call it**Saluki**\. It’s the best 2\-bit build of this model we know of at picking the right tool, and it needs no special software to run\. File size7\.89 GBFromabout 54 GBUnderdog Bench88 / 120Runs on16 GB laptop## The short version We built a few 2\-bit versions of Qwen3\.8\-27B along the way\. Then we tested them against the other small versions of the same model people are likely to compare us with\. - Our builds score highest on**Underdog Bench**, our tool\-calling test, and are clearly the best at picking the right tool\. ## Best for - **Agents that use tools:**picking the right function, filling in arguments, chaining calls\. - **Local assistants**that follow instructions closely, on hardware you already own\. - **Private and offline work,**with no data leaving the machine\. ## Good to know - **Built for agents, not math olympiads\.**Saluki keeps 82 to 85% of the full model’s score on competition math \(AIME\)\. For everyday reasoning and tool use, it holds up far better\. - **It likes to think things through\.**With thinking on, Saluki reasons carefully before it answers, and the default effort gives the best results\. For snappy tool calls, turn thinking off\. - **Use a forgiving tool\-call parser for parallel calls\.**Some multi\-call replies come out in a slightly different format\. A lenient parser recovers most of them \(55 of 100 passed, against 42 with strict parsing\)\. - **Wordplay at the letter level**\(palindromes, vowel patterns\) is its least favorite kind of instruction\. - **Text in, text out\.**This release doesn’t take images\. ## The headline: Underdog Bench Tool calling is what turns a chat model into an agent\. The model has to pick the right function, fill in the right arguments and know when no tool fits\. Underdog Bench is 120 tasks from the[Berkeley Function Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html), 24 in each of five kinds of tool use\. We picked and froze them on 27 September, before any of these models were tested\. Every model ran the same harness and settings in its own runtime: ours in stock llama\.cpp, the others in the runtimes their files are built for\. Tasks passed, out of 120 - Main7\.9 GB91 - Saluki7\.89 GB88 - Mia 2\-bit9\.1 GB85 - Full\-size Qwen3\.8\-27Babout 54 GB84 - ISTA 2\-bit8\.4 GB76 - Ternary Bonsai 25\.95 GB70 SalukiMain, our other buildOther 2\-bit buildsFull\-size modelModelSizePassedof 120Still solvesof the full model’s 84Mainours7\.9 GB9177 \(92%\)Salukiours, launching7\.89 GB8876 \(90%\)Mia 2\-bitEXL3 2\.0bpw9\.1 GB8563 \(75%\)ISTA 2\-bitIQ2\_XS8\.4 GB7663 \(75%\)Ternary Bonsai 2PTQ1\_05\.95 GB7068 \(81%\)Full\-size Qwen3\.8\-27Breferenceabout 54 GB8484 Our Main build tops Underdog Bench at 91\. Saluki scores 88\. Both beat every other 2\-bit build we tested\. We also keep far more of what the full\-size model gets right: 76 or 77 of its 84 solved tasks, against 68 at most for the other 2\-bit builds\. Both of our builds passed the full\-size model’s score\. We don’t claim they’re smarter than the original\. 120 tasks is a modest test, and part of that lead is likely run\-to\-run variation\. ### Where each model is strong Kind of task24 eachMainSalukiMiaTernary Bonsai 2Call the one right function23241923Pick the right function from several22231221Know when no tool fits19181918Same function, several calls1612212Several different functions1111146 Saluki is clearly the best at choosing the right tool:**47 of 48**on the first two rows, against Mia’s 31\. That’s the everyday skill for an assistant juggling lots of tools\. ## Several tool calls at once Real assistants often need to do several things in one go: “check the weather in Paris, London and Tokyo”, or “book the table and add it to my calendar”\. Getting that right in a single reply is what parallel tool calling measures\. We took 100 such tasks from the Berkeley Function Calling Leaderboard\. The tasks and the scoring were written down and frozen before any model ran\. Every model ran with thinking off and the same settings, each in its own runtime\. Tasks passedof 100Mia9\.1 GBSaluki7\.89 GBFull\-sizeabout 54 GBTernary Bonsai 25\.95 GBStrict scoring69423511Forgiving small formatting slips70553511Same function called several times43 of 6222 of 6224 of 623 of 62Several different functions26 of 3820 of 3811 of 388 of 38 Mia is the clear leader here, and we say so\. It’s also a bigger file: 9\.1 GB, against Saluki’s 7\.89 GB\. Saluki beats the full\-size model it came from, which, with thinking off as here, often makes one call and stops when the task needs several\. We’re not perfect here either\. On 23 tasks Saluki’s reply didn’t contain a call the checker could read, mostly formatting slips, which the forgiving score partly recovers\. That’s the next thing we’re tightening\. ## Coding agent and web browsing These two tests are the closest to real agent work\. They take hours to run, so we ran them on Saluki, plus browsing for Main\. In the coding test the model gets 50 real GitHub issues from SWE\-bench Verified\. It has to explore the code, find the bug and fix it on its own, and the fix is checked by the project’s own tests\. In the browsing test \(WebWalkerQA\) the model starts on a real website’s homepage and has to click its way to the answer to a question, 150 questions in total\. TestSalukiTernary Bonsai 2Full\-sizeCoding agent: issues fixedof 50302933Coding agent: still fixedof the 33 the full\-size model fixed272533Web browsing: answered correctlyof 1506040\.0%5536\.7%not run Main answered 48 of the 150 browsing questions \(32\.0%\)\.On the coding test Saluki kept most of what the full\-size model could do: 27 of its 33 fixes\. Every model’s browsing answers were graded by the same judge with the same rules\. Main browses less well, which is one reason we’re launching Saluki\. ## IFEval: following simple, checkable instructions IFEval is the standard instruction\-following test: 541 prompts with rules a program can check, like “answer in under 100 words”, “no commas” or “wrap the whole reply in double quotes”\. We ran Saluki with Qwen’s recommended thinking settings \(temperature 0\.6, default thinking effort\)\. ModelIFEvalSourceSaluki93\.5 / 90\.9loose / strictOur runTernary Bonsai 291\.31Ternary Bonsai 2’s model cardFull\-size Qwen3\.8\-27B91\.50Ternary Bonsai 2’s model card On the loose measure Saluki scores above the full\-size model’s public number\. Two caveats: the public number doesn’t say whether it’s loose or strict, and it comes from a different harness\. Our numbers count the 6 prompts we didn’t finish as failures, so they’re the cautious version\. ## Runs anywhere llama\.cpp runs Saluki uses standard llama\.cpp formats, so it loads in stock[llama\.cpp](https://github.com/ggml-org/llama.cpp), and in apps built on it, with no custom build\. Mia’s files are in[ExLlamaV3](https://github.com/turboderp-org/exllamav3)’s format, so they need ExLlamaV3 and won’t load in llama\.cpp at all\. To be fair, we ran every model in its own runtime, which is its best case\. Thanks to[ISTA\-DASLab](https://huggingface.co/ISTA-DASLab)for the 2\-bit release Saluki is built on, and to the Qwen team for Qwen3\.8\-27B\.