SkillMem: Compact External Skill Memory for Latent Multi-Agent Reasoning

arXiv cs.AI Papers
multi-agent latent-reasoning memory skill-learning efficiency distillation qwen3

Summary

SkillMem introduces a compact external latent memory for reusable skills in latent multi-agent reasoning, achieving 38× storage reduction and up to 4.23× inference speedup across benchmarks with Qwen3 models from 4B to 14B.

arXiv:2610.37935v1 Announce Type: new Abstract: Latent multi-agent systems reduce inference cost by keeping intermediate reasoning and inter-agent communication in continuous representations.\ Reusable skills, however, are commonly supplied as text or stored as layer-wise Key-Value states or skill-specific model parameters, introducing repeated processing or substantial per-skill storage.\ We introduce \textbf{SkillMem}, which represents each reusable skill as a small external latent memory and makes it directly available during latent multi-agent reasoning.\ A role-conditioned cross-attention reader allows the same memory to guide different agents according to their roles and current hidden states.\ We train the skill memories and shared reader through behavioral distillation, matching the change in token predictions induced by a textual skill while keeping the backbone frozen.\ For our experiments, we obtain textual skills from paired successful and failed multi-agent trajectories, although existing skills can also be compiled directly.\ Across six benchmarks and Qwen3 models ranging from 4B to 14B parameters, SkillMem provides strong gains for smaller backbones while preserving efficient latent multi-agent inference across model scales.\ We use the same $8\times256$ memory configuration across all evaluated model scales, requiring 9.6\,KB of skill-specific storage per skill, including its retrieval embedding.\ For a library of 100 skills, SkillMem reduces total storage by more than $38\times$ compared with alternative skill representations and achieves up to a $4.23\times$ end-to-end inference speedup.\ These results show that compact procedural memories can be reused without sacrificing the efficiency of latent multi-agent reasoning.
Original Article
View Cached Full Text

Cached at: 10/31/26, 09:13 AM 

# SkillMem: Compact External Skill Memory for Latent Multi-Agent Reasoning
Source: [https://arxiv.org/abs/2610.37935](https://arxiv.org/abs/2610.37935)
[View PDF](https://arxiv.org/pdf/2610.37935)[HTML \(experimental\)](https://arxiv.org/html/2610.37935v1)

> Abstract:Latent multi\-agent systems reduce inference cost by keeping intermediate reasoning and inter\-agent communication in continuous representations\.\\ Reusable skills, however, are commonly supplied as text or stored as layer\-wise Key\-Value states or skill\-specific model parameters, introducing repeated processing or substantial per\-skill storage\.\\ We introduce \\textbf\{SkillMem\}, which represents each reusable skill as a small external latent memory and makes it directly available during latent multi\-agent reasoning\.\\ A role\-conditioned cross\-attention reader allows the same memory to guide different agents according to their roles and current hidden states\.\\ We train the skill memories and shared reader through behavioral distillation, matching the change in token predictions induced by a textual skill while keeping the backbone frozen\.\\ For our experiments, we obtain textual skills from paired successful and failed multi\-agent trajectories, although existing skills can also be compiled directly\.\\ Across six benchmarks and Qwen3 models ranging from 4B to 14B parameters, SkillMem provides strong gains for smaller backbones while preserving efficient latent multi\-agent inference across model scales\.\\ We use the same $8\\times256$ memory configuration across all evaluated model scales, requiring 9\.6\\,KB of skill\-specific storage per skill, including its retrieval embedding\.\\ For a library of 100 skills, SkillMem reduces total storage by more than $38\\times$ compared with alternative skill representations and achieves up to a $4\.23\\times$ end\-to\-end inference speedup\.\\ These results show that compact procedural memories can be reused without sacrificing the efficiency of latent multi\-agent reasoning\.

## Submission history

From: Yuxuan Zhu \[[view email](https://arxiv.org/show-email/9cb118cf/2610.37935)\] **\[v1\]**Tue, 27 Oct 2026 20:25:47 UTC \(1,646 KB\)

Similar Articles

SimpleMem: Efficient Lifelong Memory for LLM Agents

Papers with Code Trending

Introduces SimpleMem, an efficient memory framework for LLM agents that uses semantic lossless compression to improve accuracy and reduce token consumption, achieving 26.4% F1 improvement and up to 30x reduction in inference-time token usage.

SkillEffect: Checked Lowering for Memory-Bounded Agent Tools

arXiv cs.AI

SkillEffect introduces a checked-lowering runtime for AI agent tools that enforces memory bounds through audited implementations, significantly reducing peak memory usage and improving completion rates under memory constraints.