No Dedicated GPU, 7.9B Model Hits 47 token/s on This Mac

Ling-3.0-tiny was running, so I first asked a trivial question: “Calculate 17×23, output only the final integer.” The screen returned 391.

There’s nothing to brag about with these three digits; they simply show that Ling-3.0-tiny’s official INT4 weights are fully loaded locally, Apple GPU is participating in inference, and the service can indeed be called via local APIs. At this point, deployment is considered complete.

The machine here is a MacBook Pro with Apple M5 chip, 10-core CPU, 10-core GPU, and 32GB unified memory.

No NVIDIA GPU, no DGX Spark.

Ordinary Mac users encountering a new model first need to know if their machine can fit it, how speed and memory perform, and finally what they can actually do with it. I ran through all three questions.

7.9B Parameters, Why It’s Still Called “Tiny”

The official model page shows Ling-3.0-tiny has 7.9B total parameters, but only about 1.3B are activated per token. It’s a sparse MoE architecture with 128 routing experts—each token activates 8 of them, plus a shared expert.

Think of it as a sizable small team. When work arrives, you don’t call everyone into the meeting room—just pick the most suitable few to handle it. The model retains the 7.9B total parameter scale, but each token only activates about 1.3B, reducing actual computation.

The attention structure is a 3:1 mix of KDA and MLA—a detail worthy of its own deep dive. For local users, this design targets something very specific: it aims to compress long-context processing and inference costs into what a personal computer can handle.

The official versions are BF16, FP8, and INT4:

BF16 weights are about 14.7GiB, FP8 about 7.8GiB, and INT4 comprises 32 weight files totaling about 5.4GiB.

This machine runs INT4.

32GB unified memory isn’t scarce, but macOS, daily apps, model weights, and context cache all draw from it. INT4 leaves the most headroom—ideal for keeping the model running in the background long-term, or working on the model while doing other tasks.

I didn’t do a full side-by-side comparison of BF16 and FP8 on this machine, so I won’t invent speed rankings. INT4 is the version that’s been tested, verified, and left running as a persistent service on this M5.

Getting It Running on Mac Involved a Bit of Hassle

Ling-3.0-tiny isn’t yet in Ollama’s stable release roster; on Mac, you need to use an unmerged experimental branch, then leverage MLX and Metal to access Apple GPU.

This time, I pinned the specific source commit of the experimental branch, added CMake and Metal Toolchain, built the MLX Metal v4 runtime, and downloaded the official INT4 weights—32 safetensors files totaling 5,805,705,224 bytes. After import, the local model is named ling-3-tiny:int4.

The service is managed by macOS’s launchd. It starts automatically on login, restarts after crashes, and its API only listens on 127.0.0.1—other devices on the same LAN can’t access it by default.

After 30 minutes idle, the model unloads from memory, but the API service stays; on the next request, it reloads.

Both Ollama’s native API and OpenAI-compatible endpoints work. Chat tools and scripts that support custom OpenAI API addresses can connect, though some clients may need adjustments to authentication or endpoint configuration.

How Fast Does This M5 Actually Run?

For typical Chinese, JSON, and code tasks, sustained generation speed falls roughly between 47 and 49 tokens per second.

A 741-token continuous Chinese generation ran at 47.63 token/s. In a chat window, this feels quite smooth—text streams out continuously, often faster than reading speed.

After completely unloading the model, cold start takes 1.7 to 2.9 seconds. When the model stays in memory, the first token in a streaming request appears in 145 milliseconds—a warm-start figure, distinct from a cold boot.

Peak runtime memory for short prompts is about 6.04GiB.

I also fed it a 4,677-token long document, asking it to find the hidden code LING-0160. It found it correctly in one attempt, with input processing speed around 444 token/s and peak runtime memory rising to 9.11GiB.

A 32GB machine still has noticeable headroom. But as context lengthens, the KV cache grows too—you can’t extrapolate 128K or 256K usage from an 8K footprint.

The official introduction mentions higher performance numbers, including over 160 token/s on specific platforms and 100–105 token/s on DGX Spark. Those figures come from their own hardware and test conditions. These results are from this M5, 32GB, INT4 setup. Mixing performance from different machines makes articles more exciting but sets readers up for disappointment after setup.

It Works, But Makes Specific Mistakes

Benchmarks only tell part of the story. I assigned it several tasks closer to daily work.

Structured information extraction was clean. Given an order description, asked to return only JSON, it correctly extracted name, city, amount, and refund status, with correct field types.

In a coding test, it wrote a Python function to merge intervals. Syntax check passed, and all five functional test cases passed.

The prompt clearly said “No Markdown,” yet it eagerly wrapped the output in a code fence. The code works, but format instructions weren’t fully followed.

Tool calling also succeeded. Given a weather query function, it accurately selected lookup_weather with “Hangzhou” as the parameter. This means the model can integrate with search, calendar, databases, or local scripts—its utility doesn’t stop at the chat box.

Chinese summarization had another small issue. Content was summarized correctly, and it was asked for three bullet points—but it only wrote two. The meaning was understood, but the count wasn’t adhered to.

Thinking mode issues were more obvious. I asked it a classic three-box label problem, allowing 2,048 thinking tokens. The model kept circling in deductions, hitting the length limit without producing a final answer. After disabling Thinking, it could deduce the correct “one time,” but the process was again too long and hit the length limit.

These shortcomings are worth noting. A local model being runnable doesn’t mean every task is suited for direct handoff.

JSON, text processing, code drafts, and tool routing already have decent usability. Strict formatting, complex reasoning, and single-answer-required workflows still need validation.

Who Is It Best For?

Ling-3.0-tiny INT4 on a 32GB Mac feels like a local assistant you can summon anytime.

Individual users can use it to handle private documents, summarize long materials, draft code, or integrate it into Obsidian, editors, or personal automation scripts. Model weights and inference stay local—daily processed text never leaves for the cloud.

Developers can also use the OpenAI-compatible API to shift parts of their cloud-model workflows to local testing.

Its current concurrency is set to 1. If two requests come in simultaneously, they queue. Single-user use and live demos work fine, but multi-user service would require reworking concurrency, memory, and long-term stability tests.

Mac support currently comes from an experimental branch. As of August 16, 2026, stable Ollama hasn’t merged this code yet. Current deployment is pinned to a verified commit—future runtime upgrades will require retesting imports, inference, tool calls, and memory.

You can chase new versions, but ideally not five minutes before a client demo.

After Running It, Redefining “Tiny”

“Small models” used to imply “if it runs at all, don’t expect much quality.”

This run shows Ling-3.0-tiny is ready for daily workflows. With nearly 8B total capacity, activating only a portion per token, it fits comfortably on a 32GB Mac, and speeds are now at a level you’d actually want to use daily.

It also has clear boundaries. Thinking loops, occasional format noncompliance, and current single-user concurrency. Keeping these issues visible in the article makes its value clearer.

The model currently runs on this M5 Mac at http://127.0.0.1:11434, named ling-3-tiny:int4.

Open an editor or script, send a request, and it starts generating.

For a personal computer without a dedicated GPU, this is quite useful—what do you think?

References

• Ling-3.0-tiny Official Model Page
• Ling-3.0-tiny INT4 Official Model Page
• Ling-3.0-tiny Official Introduction
• Ollama’s Ling and MLX Support Progress