Nanbeige4.2-3B on Apple Silicon: Fixing Deployment Bugs and Decreasing Looped Transformer Memory Overhead

# Nanbeige4.2-3B on Apple Silicon: Fixing Deployment Bugs and Decreasing Looped Transformer Memory Overhead
Source: [https://arxiv.org/html/2608.13987](https://arxiv.org/html/2608.13987)
\\setCJKmainfont

FandolHei\-Regular\.otf

August 14, 2026

###### Abstract

[Nanbeige4\.2\-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B)\([Nanbeige Team 2026](https://arxiv.org/html/2608.13987#bib.bib3)\)is a 3B\-parameter agentic model built around a Looped Transformer \(LT\)\([Bae et al\. 2026](https://arxiv.org/html/2608.13987#bib.bib1)\)that reuses one stack of layers for a second forward pass, adding effective depth without additional parameters\. Evaluated on Apple Silicon \(MPS\), we identify five independent bugs which prevent the released checkpoint from running via Hugging Facetransformersout of the box \(including a silently\-zeroed RoPE buffer and calls to removedtransformerscache APIs\)\. Furthermore, we show that fixing these bugs is still not sufficient for agentic tasks, due to the LT’s layer\-reuse strategy \(which effectively doubles peak attention memory\) used to achieve parameter efficiency\. We thus introduce a chunked\-prefill strategy which alleviates the incurred memory\-capacity penalty, extending allowable context width by2\.7×2\.7\\timeson 32 GiB shared memory\. However, even with the reduced memory overhead, we show that patches are required to renderNanbeige4\.2\-3Busable; resolving both system prompt and MPS\-native memory bugs finally allows reliable evaluation on standard MCP and tool\-calling benchmarks\. On a subset of MCPMark, the debugged model completes up to30%of real agentic tasks \(up from the original’s 0%\), while, on BFCL, it is near\-perfect at single tool calls \(yet fails the majority of multi\-tool tests\)\. We release the patched checkpoint, system prompt optimizer, and evaluation harnesses at[github\.com/johnhalloran/Nanbeige4\.2\-3B\-mps\-fix](https://github.com/johnhalloran321/Nanbeige4.2-3B-mps-fix)\.

## 1Introduction

Nanbeige4\.2\-3Bhas recently been released as a capable small language model \(SLM\) specifically designed for improved capability at agentic tasks\. The model utilizes a Loop Transformer \(LT\) architecture to trade off decoding compute without requiring scaling parameter count\. The released model card reports competitive or better results than larger models \(Qwen3\.5\-4B, Qwen3\.5\-9B\) on agentic and office\-workflow benchmarks\([Nanbeige Team 2026](https://arxiv.org/html/2608.13987#bib.bib3)\), credited to the added effective depth of its Looped Transformer architecture\. However, running the released checkpoint in a ReAct\-style agentic harness viatransformerson MPS \(Apple Silicon\) surfaces both stability and correctness problems rendering the model unusable out of the box\.

Herein, we identify five initial bugs—e\.g\., a silently\-zeroed RoPE buffer, calls to removedtransformerscache APIs, etc\.—and the necessary fixes to render the released checkpoint usable on MPS\. However, the resulting model remains unscalable for agentic tasks due to the increased memory needs of the LT architecture, despite the 3B parameters inbf16on a 32 GiB shared memory system; inherently, the LT enables parameter efficiency at the cost of double the peak attention memory \(due to the recursive looping over layers in the forward pass\), which prohibits the long reasoning traces required for agentic tasks\. Thus, we introduce a chunked\-prefill strategy which relieves LT memory overhead, more than doubling allowable context\-width evaluation\. However, the resulting model reveals further deficiencies: \(1\)Nanbeige4\.2\-3B’s trained\-in tool\-use system prompt is silently replaced, not merged, the moment a caller supplies any system message, and \(2\) an MPS out\-of\-memory \(OOM\) error can permanently degrade the serving process’s usable memory budget, surfacing only while evaluating the debugged model on MCPMark\. The gamut of bug and model fixes—five initial bug fixes, an alternative prefilling algorithm, system prompt correction, and MPS\-specific OOM fix—allow the true reproducible evaluation of the model on standard agentic and tool\-use benchmarks\.

## 2Five Initial Deployment Bugs

Loading on Apple Silicon via

model=AutoModelForCausalLM\.from\_pretrained\(”Nanbeige/Nanbeige4\.2\-3B”,

trust\_remote\_code=True,

device=”mps”,

\)

fails or silently misbehaves for five independent reasons\. We confirm each by direct reproduction against the unmodified checkpoint\.

1. 1\.RoPE buffer persistence \(dominant bug\)\.The model’sinv\_freqrotary\-embedding buffer is silently zeroed on load and never repopulated before the first forward pass\. RoPE therefore contributes zero positional information to attention:the model runs without knowing token order\. This manifests as fluent\-looking but positionally\-incoherent generation rather than a crash, which makes it easy to miss without directly inspecting buffer values after load\.
2. 2\.RoPE\-config dispatchKeyError\.A bug in the custom modeling code’s RoPE\-type dispatch raisesKeyErrorfor a subset of otherwise valid config values \(Table[1](https://arxiv.org/html/2608.13987#S2.T1)\)\. This fires during model construction, before device placement or a single forward pass, and is not MPS\-specific \(it blocks loading on any device\)\.
3. 3\.Cache API sentinel mismatch\.Calling the model’sforward\(\)directly with the defaultpast\_key\_values=None, instead of throughgenerate\(\), calls an API removed in currenttransformersreleases \(DynamicCache\.from\_legacy\_cache\(\.\.\.\)\)\.
4. 4\.Position\-IDs re\-trim\.A bug during position\-tracking in the custom attention code produces a hard crash on device MPS specifically \(does not reproduce on CPU\)\.
5. 5\.Tied\-weights key format\.An incompatible tied\-weights key naming convention breakssave\_pretrained\(\)\. Even a successfully\-patched, running model cannot be re\-serialized without an additional fix\.

We fix all five via sibling\-file monkeypatching \(never modifying cachedtransformerspackage files\) in the[johnhalloran/Nanbeige4\.2\-3B\-mps\-fix](https://huggingface.co/johnhalloran/Nanbeige4.2-3B-mps-fix)checkpoint\. None of the five is related to the memory or system\-prompt issues in Sections[3](https://arxiv.org/html/2608.13987#S3)–[4](https://arxiv.org/html/2608.13987#S4), which persist after all five are fixed\.

Table[1](https://arxiv.org/html/2608.13987#S2.T1)gives the exact trigger, line number in the unmodifiedmodeling\_nanbeige\.py, and error string or symptom for each bug, againsttransformers==5\.8\.1\(the version used throughout this paper\); full diffs and reproduction scripts are in the artifact repository’spatch/directory \(Section[6](https://arxiv.org/html/2608.13987#S6)\)\.

Table 1:Exact trigger, source line, and error/symptom for each bug, againsttransformers==5\.8\.1\. Line numbers refer to the unmodified checkpoint\.
## 3The Looped\-Transformer Memory Tradeoff

Nanbeige4\.2\-3B’s LT\([Bae et al\. 2026](https://arxiv.org/html/2608.13987#bib.bib1)\)feeds the hidden state through the full stack ofLLphysical transformer layers, then re\-feeds the output of that pass through the sameLLlayers a second time before producing logits: two effective passes \(2​L2Leffective layer\-executions\) fromLLlayers’ worth of parameters\. This effectively improves model quality without scaling parameters—e\.g\., LTs outperforming larger non\-looped models at a fixed parameter budget\([Bae et al\. 2026](https://arxiv.org/html/2608.13987#bib.bib1)\)\. However, parameter counts are kept low at the expense of additional compute and memory requirements\.

Naively, self\-attention’s peak activation memory during prefilling is dominated by materializing an attention\-score tensor proportional to\(prompt\_len\)2\(\\text\{prompt\\\_len\}\)^\{2\}per layer pass\. Compared to their non\-looped counterparts, LTs double required prefilling memory, as theO⁡\(prompt\_len2\)O\(\\text\{prompt\\\_len\}^\{2\}\)attention computation over the same prompt is repeated once per loop\. Thus, despite the total\-parameter count savings enabled by looped weight\-sharing during pretraining, naive prefilling results in twice the full quadratic attention cost during inference\.

For small language models \(SLMs\) on large dedicated hardware \(e\.g\., H200s\), this doubling is usually not an issue\. However, on Apple Silicon’s unified memory—shared between the OS, competing processes, and the model with no page\-out path comparable to CUDA’s memory management—it can be catastrophic for agentic tasks\.

### 3\.1Balancing Looped Memory Use via Chunked Prefilling

As opposed to*naive prefilling*—wherein the full\(prompt\_len×prompt\_len\)\(\\text\{prompt\\\_len\}\\times\\text\{prompt\\\_len\}\)attention\-score tensor is computed once per loop iteration in a single forward pass—we show thatNanbeige4\.2\-3B’s memory use may be significantly decreased via*chunked prefilling*\. Chunked prefilling processes the prompt in fixed\-size chunks, growing the Key\-Value cache incrementally between chunks the same way ordinary autoregressive decoding already does\.

We replace the single\-shotmodel\(input\_ids=full\_prompt, \.\.\.\)prefill call with a loop that processes the prompt in fixed\-size chunks \(256 tokens by default\), incrementally growing aDynamicCachebetween chunks, before handing the remainder off togenerate\(\):

total\_len=input\_ids\.shape\[1\]

iftotal\_len<=chunk\_size:

returnmodel\.generate\(input\_ids=input\_ids,max\_new\_tokens=max\_new\_tokens,\*\*gen\_kwargs\)

cache=DynamicCache\(\)

n\_full\_chunks=\(total\_len\-1\)//chunk\_size

withtorch\.no\_grad\(\):

foriinrange\(n\_full\_chunks\):

start,end=i\*chunk\_size,i\*chunk\_size\+chunk\_size

outputs=model\(

input\_ids=input\_ids\[:,start:end\],past\_key\_values=cache,

use\_cache=True,cache\_position=torch\.arange\(start,end\),

\)

cache=outputs\.past\_key\_values

returnmodel\.generate\(input\_ids=input\_ids,past\_key\_values=cache,

max\_new\_tokens=max\_new\_tokens,\*\*gen\_kwargs\)

This bounds the peak per\-step attention\-score tensor to\(chunk\_size×running\-total\)\(\\text\{chunk\\\_size\}\\times\\text\{running\-total\}\), independent of how long the prompt is, at the cost of splitting one forward pass into several sequential sub\-calls instead of one\. Bit\-identical outputs were verified against naive prefill\.

### 3\.2LongBench\-Pro results

We demonstrate the memory benefits of chunked prefilling \(CP\) over naive prefilling \(NP\) using 50 long samples fromLongBench\-Pro\([Chen et al\. 2026](https://arxiv.org/html/2608.13987#bib.bib2)\)and single\-turn queries of eight lengths varying from 1024 to12,24412,244tokens\. Each sample is tokenized using theNanbeige4\.2\-3Btokenizer and truncated to the target length\. To measure maximum memory throughput, we calculate the max batch size per length and prefilling strategy by doubling the batch size until failure, repeating this process 3 times\. All evaluations were performed on aApple M2 Maxwith 32 GiB of shared memory\. Results are in Table[2](https://arxiv.org/html/2608.13987#S3.T2)\.

Table 2:NP vs CP evaluated over 50 LongBench\-Pro samples, averaged over 3 repeated experiments\. “–” denotes the method could not complete even batch=1\.The maximum length possible under CP \(11231\) is significantly larger than NP \(4096\)\. However, CP trades memory requirements for time; atprompt\_len=1024\\text\{prompt\\\_len\}\{\}=1024, CP allows twice the amount of batch\-parallelism, while only being22\.8%22\.8\\%slower than NP\. This tradeoff is most noticeable atprompt\_len=2048\\text\{prompt\\\_len\}\{\}=2048, where CP allows 4 times the batch\-parallelism while being40\.9%40\.9\\%slower\. We note that, for CP, per\-chunk subcalls incur a fixed cost, paidchunk\_sizetimes regardless of batch size\. Thus, this overhead becomes amortized when the batch size is large, but becomes a larger portion of total runtime when only smaller batch sizes are present, e\.g\.,prompt\_len=4096\\text\{prompt\\\_len\}\{\}=4096, which achieves lower throughput than batch size=1=1evaluations over longer prompts\. We note that folding the sequential per\-chunk subcalls into fewer large GPU operations via kernel fusion would reduce this per\-call overhead directly\.

## 4System\-Prompt Regression

Independent of the previously discussed memory issues,Nanbeige4\.2\-3B’s chat template \(chat\_template\.jinja, in the\{% if tools %\}branch\) performs the following if/else onmessages\[0\]:

- •If the caller suppliesanysystem message, it is used verbatim, with a trailing"\\n\\n"the template appends\.
- •Otherwise, the template injects a hardcoded default \(Nanbeige’s own trained\-in tool\-use system prompt, beginning ‘‘你是一位工具函数调用专家\.\.\.’’111Translation: “You are a tool\-function\-calling expert\. You will be given a question and a set of possible tool functions\. Based on the question, you need to make one or more function/tool calls to accomplish the goal — please do your best to explore solving the problem through tools\. If no function is usable, reply directly to the user in natural language\. If the given question is missing parameters required by a function, use natural language to ask the user for the necessary information\. If the call results are already sufficient to answer the user’s question, summarize the results and reply to the user in natural language\.”\) withnotrailing separator before the\# Toolssection that follows\.

Any caller\-supplied system message thus silently discards the model’s own trained default instead of extending it\. Any tools\-plus\-user\-message request produces a clean, correctly\-formatted single tool call, but with no system message; if a system message is added—even one such as “You are an assistant with MCP tools”—multi\-tool\-call outputs break into malformed text\.

Re\-supplying the original text as an explicit system message does not fix this behavior\. Byte\-identical content through the explicit\-system\-message branch still breaks, because that branch’s own auto\-appended"\\n\\n"differs from the zero\-extra\-whitespace auto\-insert branch’s output by exactly two characters\. The released checkpoint’s tool\-calling reliability is calibrated to the exact byte sequence its own default rendering path produces, consistent with its tool\-use SFT/RL data having only ever been rendered through the auto\-insert branch and never with a caller\-supplied system message\.

Fix:remove the system message from the chat template, such that the template takes its own zero\-extra\-whitespace auto\-insert path\. Then insert the caller’s system content in the rendered string after the auto\-insert default \(never through the default template’s aforementioned branch\)\. This generates single\-tool\-call outputs while including the caller’s system content\.

We note that a closely related \(but a mechanically distinct bug\) has been independently reported against this model:[llama\.cpp PR \#26324](https://github.com/ggml-org/llama.cpp/pull/26324)documents Nanbeige4\.2\-3B emitting<tool\_call\>with a trailing space instead of<tool\_call\>\\nfor roughly 25% of calls, breaking tag\-matching in that inference engine, and notes the same template structure is shared by \(though not observed to trigger the same failure in\) Qwen3\-Coder and Qwen3\.5\-4B\. Both bugs sit in the same chat\-template/generation pipeline and are independent evidence that this checkpoint’s tool\-calling reliability is template and whitespace sensitive\.

## 5Evaluation

We evaluate the combined fixes on a 10 task subset ofMCPMark\([Wu et al\. 2025](https://arxiv.org/html/2608.13987#bib.bib4)\)—which tests MCP\-tool use capabilities on multi\-turn tasks while grading tool\-generated responses—and a 150 subset of theBerkeley Function\-Calling Leaderboard\(BFCL\)\([Yan et al\. 2024](https://arxiv.org/html/2608.13987#bib.bib5)\)–which tests tool selection ability without considering tool execution outputs\.

MPS memory bug\.Running the test suite end\-to\-end surfaced a bug unrelated to the model: a single caughtRuntimeError: MPS backend out of memorypermanently degrades the harness process’s usable MPS memory budget for the rest of its life\. Neithertorch\.mps\.empty\_cache\(\)norgc\.collect\(\)reclaim it in a controlled tests; only a process restart does\. Uncorrected, one task’s OOM—expected forMCPMark, where multi\-turn tasks can grow past the max\-token ceiling \(12,24412,244, per Table[2](https://arxiv.org/html/2608.13987#S3.T2)\)—cascades into spurious OOMs on every later, unrelated task in the same long\-lived server\. Restarting the harness fresh before each task \(mirroring the subprocess\-per\-trial isolation already used for Table[2](https://arxiv.org/html/2608.13987#S3.T2)\) fixes this\.

### 5\.1MCPMark\(Filesystem subset, easy tier\)

Table 3:MCPMarkFilesystem \(easy tier\), patched checkpoint, per\-task server isolation, usingMCPMark’s default 1 hour timeout\.Evaluations were run overMCPMark’s Filesystem suite of 10 easy tasks \(which do not require API credentials\), using the benchmark’s default timeout of 1 hour\. The original \(unpatched\) checkpoint cannot be evaluated \(on any device\) due to bug 2 of Section[2](https://arxiv.org/html/2608.13987#S2)\. The patched checkpoint—with CP \(Section[3\.1](https://arxiv.org/html/2608.13987#S3.SS1)\) and all described bug fixes \(Sections[2](https://arxiv.org/html/2608.13987#S2)and[4](https://arxiv.org/html/2608.13987#S4)\)—scores3/10 \(30%\)\. Details for each per\-task run are in Table[3](https://arxiv.org/html/2608.13987#S5.T3)\.

For the failed pattern\_matching task, the model correctly calls the MCP toolread\_multiple\_filesonce, but repeats a long absolute path 21 times, leading to a long context\-width that eventually times out\. The remaining failures accumulate context over multiple turns and eventually exceed memory capacity before solving the underlying task\.

### 5\.2Tool\-calling correctness in isolation from decode throughput \(BFCL\)

To stress test the tool calling ability of the model \(and not necessarily end\-to\-end task effectiveness\), we evaluate the debugged/scalable model on a 150 task subset ofBFCL’s non\-live, single\-turn categories\([Yan et al\. 2024](https://arxiv.org/html/2608.13987#bib.bib5)\):simple\_python\(one correct call\),multiple\(pick 1 ofNNcandidate functions\),parallel\(emit 2\+ calls to the same function\),parallel\_multiple\(2\+ calls to different functions\), andirrelevance\(correctly emit no call at all\)\. These are AST/exact\-match graded, require no external LLM judge, and complete in one generation each \(10–20s\), so there is no wall\-clock or multi\-turn\-accumulation confound \(as inMCPMark\)\.

Table 4:BFCL, patched checkpoint, 30 tasks per category\.The model reliably recognizes when*not*to call a tool \(100% onirrelevance\) and is moderately reliable on a single, well\-specified call \(63\.3%\)\. It is specifically weak at emitting*multiple*tool calls in one turn: on both parallel categories, the dominant failure is producing the wrong number of function calls, almost always one call where two were required\. This is a distinct, format\-level limitation from anything in Section[5\.1](https://arxiv.org/html/2608.13987#S5.SS1)—it would appear even on hardware with sufficient memory capacity to avoid OOMs and speed to avoid timeouts\.

## 6Conclusions and Artifacts

The releasedNanbeige4\.2\-3Bcheckpoint cannot run reliably via Hugging Facetransformerson Apple Silicon, blocked first by five independent deployment bugs and then by the memory overhead of its Looped Transformer architecture\. We fix all five bugs, introduce chunked prefilling to more than double the usable context width under Apple Silicon’s shared\-memory limits, and resolve a system\-prompt regression and an MPS memory bug that otherwise block reliable agentic evaluation\. The resulting patched checkpoint completes up to 30% of a real MCPMark agentic subset \(up from 0%\) and is near\-perfect at single BFCL tool calls, though it still fails the majority of multi\-tool\-call tests\.

## References

- Bae et al\. \(2026\)Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se\-Young Yun\.Mixture\-of\-recursions: Learning dynamic recursive depths for adaptive token\-level computation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026\.URL[https://arxiv\.org/abs/2507\.10524](https://arxiv.org/abs/2507.10524)\.
- Chen et al\. \(2026\)Ziyang Chen, Xing Wu, Junlong Jia, Chaochen Gao, Qi Fu, Debing Zhang, and Songlin Hu\.Longbench pro: A more realistic and comprehensive bilingual long\-context evaluation benchmark\.*arXiv preprint arXiv:2601\.02872*, 2026\.URL[https://arxiv\.org/abs/2601\.02872](https://arxiv.org/abs/2601.02872)\.
- Nanbeige Team \(2026\)Nanbeige Team\.Nanbeige4\.2\-3b: Unlocking agentic capabilities in a compact model\.*arXiv preprint arXiv:2607\.22083*, 2026\.URL[https://arxiv\.org/abs/2607\.22083](https://arxiv.org/abs/2607.22083)\.
- Wu et al\. \(2025\)Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fanshi Zhang, Yaoqi Ye, Jiawei Wang, Zirui Wang, Jinjie Ni, Yufan Yang, Arvin Xu, and Michael Qizhe Shieh\.Mcpmark: A benchmark for stress\-testing realistic and comprehensive mcp use\.*arXiv preprint arXiv:2509\.24002*, 2025\.URL[https://arxiv\.org/abs/2509\.24002](https://arxiv.org/abs/2509.24002)\.
- Yan et al\. \(2024\)Fanjia Yan, Huanzhi Mao, Charlie Cheng\-Jie Ji, Tianjun Zhang, Shishir G\. Patil, Ion Stoica, and Joseph E\. Gonzalez\.Berkeley function calling leaderboard \(bfcl\)\.UC Berkeley Gorilla Project, 2024\.URL[https://gorilla\.cs\.berkeley\.edu/leaderboard\.html](https://gorilla.cs.berkeley.edu/leaderboard.html)\.