d1 (2 minute read)

# Thread by @liquidai on Thread Reader App Source: [https://threadreaderapp.com/thread/2105003472332693869.html](https://threadreaderapp.com/thread/2105003472332693869.html) ## More from @liquidai [](https://threadreaderapp.com/user/liquidai) Aug 24 Today we release Pipette, a model evaluation suite for on\-device intelligence, in partnership with @ArtificialAnlys Most benchmarking platforms are optimized to measure core capabilities and speed profile of foundation models served in the cloud\. Pipette gives the field a common, reproducible way to measure the quality, speed, latency, and memory use of AI models on devices such as phones, laptops, PCs, AI boxes, and embedded hardware\. \> Pipette is open source \> In Pipette, models get compated as model \+ quantization \+ runtime \+ device from one interface\. \> It comes with a warehouse of verified benchmark results, currently with 10k\+ results across 35 model classes, 7 quants, llama\.cpp runtimes, and 4 devices\. \> Pipette is a dynamic platform, allowing new contributions from day one, adding new devices, runtimes, model families, and quantization levels\. 🧵[](https://pbs.twimg.com/media/HQfxwNma4AETD1r.jpg) What is being released today: 1\.An interactive dashboard for exploring and comparing on\-device model performance:[pipette\.liquid\.ai](https://pipette.liquid.ai/) 2\.A public dataset containing more than 10k results, including 1000\+ tested performance configurations across models, quantization levels, runtimes, and devices\. Inspect the current breakdown:[pipette\.liquid\.ai/coverage](https://pipette.liquid.ai/coverage) 3\.Benchmark clients for macOS, Windows, iOS, and Android that run versioned benchmark definitions on target devices:[github\.com/Liquid4All/pip…](https://github.com/Liquid4All/pipette-clients) 4\.A companion Artificial Analysis dashboard presenting Pipette results and the aggregate benchmark scores:[artificialanalysis\.ai/hardware\-infer…](http://artificialanalysis.ai/hardware-inference-stack/mobile-phones) 5\.Apache 2\.0\-licensed open\-source infrastructure for benchmark management, execution, submissions, and evaluation scoring: [github\.com/Liquid4All/pip…](https://github.com/Liquid4All/pipette-mgmt) [github\.com/Liquid4All/pip…](https://github.com/Liquid4All/pipette-clients) [github\.com/Liquid4All/pip…](https://github.com/Liquid4All/pipette-scores) 6\.Native iOS and Android apps for running benchmarks directly on\-device\. iOS App:[apps\.apple\.com/us/app/pipette…](https://apps.apple.com/us/app/pipette-by-liquid/id6772314671) Android app:[play\.google\.com/store/apps/det…](https://play.google.com/store/apps/details?id=ai.liquid.pipette) 2/  Get started: The initial public release covers about 35 model classes from several providers, with 7 llama\.cpp quantization levels\. Laptop and desktop runs cover all four levels, while current phone runs cover all quant levels\.\. The release includes context lengths from 256 to 8,192 tokens, where device memory allows\. Current devices include a MacBook Pro with M5 Max, iPhone 17 Pro, and Samsung Galaxy S26 Ultra, AMD Ryzen AI Max\+ 395 with Radeon 8060S \(coming live soon\)\. More models, runtimes, and devices are on the way\. \> If you are choosing a model, inspect the runtime, quantization, and device you plan to ship\. \> If that configuration is missing, run the Pipette clients on your hardware and help test the third\-party submission workflow\. \> Every accepted submission expands the configurations represented in the dataset\. \> We encourage model providers, runtime developers, hardware teams, and application developers to test the configurations they care about\. \> If there is something you want measured, open an issue and tell us\. 3/  Read 4 tweets [](https://threadreaderapp.com/user/liquidai) Aug 20 Today, we release DSpark draft models for LFM2\.5\-1\.2B\-Instruct, LFM2\.5\-2\.6B, and LFM2\.5\-8B\-A1B\. These add a speculative decoding path that trades a minimal memory increase for a large decoding speedup without changing output quality\. A lightweight draft model proposes a block of candidate tokens and the target model verifies them in a single forward pass\. Across MATH500, GSM8K, HumanEval, MBPP, and MT\-Bench at batch size 1: \> Up to 3\.18x throughput on an H100: LFM2\.5\-8B\-A1B on MATH500, 428 → 1362 tok/s \> Up to 2\.87x on an M4 Max MacBook Pro: LFM2\.5\-1\.2B\-Instruct on HumanEval, 136 → 389 tok/s \> LFM2\.5\-2\.6B means: 2\.67x on the H100 \(323 → 864 tok/s\), 2\.27x on device \(61 → 139 tok/s\) \> Under greedy decoding, the emitted sequence is identical to baseline by construction, so benchmark accuracy is unchanged\. Each draft model is around 300M parameters, with embedding and LM head tied to its target model\. The gain shows most in agentic workloads, where the model reasons before every tool call and the user waits through it all: on BFCL multi\-tool scenarios, DSpark cuts LFM2\.5\-2\.6B latency by nearly 50% on average\. 🧵 Measured with llama\.cpp and Metal on an M4 Max MacBook Pro, FP16 GGUF weights, batch size 1, temperature 0: \> LFM2\.5\-1\.2B\-Instruct: 2\.54x mean, 138 → 350 tok/s \> LFM2\.5\-2\.6B: 2\.27x mean, 61 → 139 tok/s \> Day\-one GGUF checkpoints, with the LFM\-compatible DSpark integration open\-sourced upstream in llama\.cpp In agentic workloads the model reasons before every tool call, and the user waits through it all\. That is where speculation pays most: on BFCL multi\-tool scenarios, DSpark cuts LFM2\.5\-2\.6B latency by nearly 50% on average\. 2/[](https://pbs.twimg.com/media/HQLeuANbgAMRlVR.png) A parallel draft backbone proposes a block of tokens in one forward pass, a Markov\-chain head adds dependency between neighboring draft tokens, and the target model verifies the whole block at once\. We follow the DSpark recipe with an extended data mixture of SFT, chat, code, and function\-calling data\. Every draft model and every ablation is trained on @AMD MI325x GPUs\. Ablations set the architecture: \> 5 layers, block of 9, attention\-only for the first version \> 15 epochs per draft model, tracking validation loss and acceptance rate at every epoch \> Mean acceptance of 10 drafted tokens: 6\.95 for LFM2\.5\-8B\-A1B, 5\.02 for LFM2\.5\-1\.2B\-Instruct, 4\.81 for LFM2\.5\-2\.6B 3/ Read 4 tweets [](https://threadreaderapp.com/user/liquidai) Aug 19 Today, we release updated 4\-bit checkpoints for LFM2\.5\-230M, LFM2\.5\-350M, LFM2\.5\-1\.2B\-Instruct, and LFM2\.5\-2\.6B trained with Quantization\-Aware Distillation \(QAD\)\. These checkpoints recover accuracy lost to 4\-bit quantization while retaining the low memory footprint and high decode throughput of the Q4\_0 format\. All four checkpoints reach roughly 97% of their BF16 averages\. 🧵[](https://pbs.twimg.com/media/HQFyrq4aAAAeYaK.jpg) Post\-training quantization \(PTQ\) compresses an already\-trained model without retraining it to adapt to quantization error, which can degrade model quality at lower precisions\. QAD addresses this by simulating quantization during training and teaching the resulting quantized student to match a full\-precision teacher\. The final checkpoint remains a standard Q4\_0 GGUF, so it requires no specialized inference runtime\. \(2/4\) Quality recovery is 70\.6%, 73\.4%, 65\.5%, and 48\.4% of the BF16\-to\-Q4\_0 quality gap at 230M, 350M, 1\.2B, and 2\.6B respectively\. The 230M and 350M QAD Q4\_0 checkpoints match Q5\_K\_M quality within evaluation variance at a 4\-33% higher decode throughput when benchmarked across a MacBook Pro, NucBox EVO\-X2, Samsung Galaxy S26 Ultra, and Raspberry Pi 5\. The 1\.2B and 2\.6B QAD Q4\_0 checkpoints match Q4\_K\_M quality at a 3\-14% higher throughput on those same devices\. \(3/4\) Read 4 tweets [](https://threadreaderapp.com/user/liquidai) Jul 7 Today we release Antidoom, an open\-source method that removes a common failure mode in reasoning models: the doom loop\. Doom\-loop rates before and after, with eval scores up across the board: \> Early LFM2\.5\-2\.6B checkpoint: 10\.2% → 1\.4% \> Qwen3\.5\-4B: 22\.9% → 1% \(greedy sampling\) 🧵 A doom loop happens when the model emits a span, then repeats it over and over until the context window runs out\. Small reasoning models hit it most on hard math and coding\. The usual fixes are stopgaps\. Applying repetition\_penalty can degrade quality\. RL needs calibrated rewards and costly rollouts\. \(2/6\)[](https://pbs.twimg.com/media/HMoUmBGXUAAwNHE.jpg) Our approach is surgical\. The loop almost always starts on one overtrained token, often an interruptive the model overproduces \("Wait," "So," "Alternatively"\)\. We retrain that single token to prefer coherent alternatives and leave the rest of the distribution largely intact\. \(3/6\)[](https://pbs.twimg.com/media/HMoUnEZW0AIAU6u.jpg) Read 6 tweets [](https://threadreaderapp.com/user/liquidai) Mar 31 Today, we release LFM2\.5\-350M\. Agentic loops at 350M parameters\. A 350M model trained for reliable data extraction and tool use, where models at this scale typically struggle\. <500MB when quantized, built for environments where compute, memory, and latency are constrained\. 🧵[](https://pbs.twimg.com/media/HEwWZTlaoAAjOVx.jpg) Trained on 28T tokens with scaled RL, LFM2\.5\-350M is a step change from LFM2\-350M: \> instruction following: 18\.20 → 40\.69 \> data extraction: 11\.67 → 32\.45 \> tool use: 22\.95 → 44\.11 These are the capabilities that matter in production\. What matters isn’t parameter count, it’s what you can actually run: \> runs across CPUs, GPUs, and mobile \> fast, efficient, low\-latency \> consistent structured outputs \> reliable function calling and agent workflows This enables a different deployment model: move high\-frequency, structured workloads closer to the data: on\-device, on\-prem, or at the edge instead of routing everything through a large cloud model\. Read 6 tweets [](https://threadreaderapp.com/user/liquidai) Mar 5 \> 385ms average tool selection\. \> 67 tools across 13 MCP servers\. \> 14\.5GB memory footprint\. \> Zero network calls\. LocalCowork is an AI agent that runs on a MacBook\. Open source\. 🧵[](https://pbs.twimg.com/media/HCqDtQAbwAACYjP.jpg) Building a local AI agent sounds great until you try to use one all day\. The hard part isn’t getting a model to understand you\. It’s getting it to choose the right tool and do it fast enough that the experience feels interactive\. So we put LFM2\-24B\-A2B to the test on laptops, building an open\-source desktop agent called LocalCowork\. Everything runs locally: \> the model \> the tools \> the data No cloud\. No API keys\. Nothing leaves the machine\. Read 6 tweets