@Zen_with_AI: Kubernetes is designed for always-on, long-lived Pods. AI agents are the opposite… They only work for seconds, and are idle 90–95% of the time — waiting on LLMs, waiting on tools, waiting for human approval. The real-world result: clusters are almost always paying for idle CP…

X AI KOLs Following Tools

Summary

Agent Substrate (donated by Google to CNCF, now released as 1.0) is a zero-trust sandbox runtime for AI agents that achieves higher-density agent scheduling and second-level recovery on Kubernetes by snapshotting idle actors and mapping them onto a small pool of warmed workers.

Kubernetes is designed for always-on, long-lived Pods. AI agents are the opposite… They only work for seconds, and are idle 90–95% of the time — waiting on LLMs, waiting on tools, waiting for human approval. The real-world result: clusters are almost always paying for idle CPU and memory… But at KubeCon North America 2026, Agent Substrate (donated by Google to CNCF, now at 1.0) became a hot topic… because it separates state from compute. It treats each session as an Actor (carrying its memory, files, and identity) and places it on top of a smaller pool of warmed Workers (pre-warmed Pods)… https://github.com/agent-substrate/substrate…
Original Article
View Cached Full Text

Cached at: 11/16/26, 09:00 AM

Kubernetes was designed for always-on Pods (long-lived workloads). AI agents are the exact opposite… they only work for a few seconds, and spend 90–95% of their time idle — waiting on the LLM, waiting on tools, waiting for human approval. The practical result: the cluster is almost always paying for idle CPU and memory…

But at KubeCon North America 2026, Agent Substrate (donated to the CNCF by Google, and now at 1.0) became the headline topic… because it separates state from compute. It treats every session as an Actor (with memory, files, and identity) layered on top of a much smaller pool of Workers (pre-warmed Pods)…

https://github.com/agent-substrate/substrate


agent-substrate/substrate

Source: https://github.com/agent-substrate/substrate

Agent Substrate

Zero-trust sandboxing for millions of agents on Kubernetes.

License (https://opensource.org/licenses/Apache-2.0) | Release (https://github.com/agent-substrate/substrate/releases) | CI (https://github.com/agent-substrate/substrate/actions/workflows/pr-workflow.yaml?query=branch%3Amain) | govulncheck (https://github.com/agent-substrate/substrate/actions/workflows/govulncheck.yaml) | GitHub stars (https://github.com/agent-substrate/substrate/stargazers)

Quickstart · Security · Architecture · Contributing

What is Agent Substrate?

Agent Substrate is a security-hardened, zero-trust runtime for sandboxed agents, engineered to safely run millions of agents and tools with 10x higher density than standard containers. Purpose-built for the era of autonomous agents, Substrate delivers fast resume operations in less than 500ms, with more than 500 activations per second. Agents get isolated compute, identity, networking, and storage. Sensitive credentials are kept out of reach from untrusted code. It supports multiple sandbox technologies, including microVMs and gVisor, while enabling a consistent lifecycle and management of all sandboxes.

At its core, Agent Substrate maps a larger set of “actors” (stateful workloads such as agents) onto a smaller pool of ready “workers”. Because agents spend most of their lives waiting on models, tools, or humans, Substrate snapshots idle actors out of memory and restores them when a request arrives.

Substrate is a low-opinion runtime, not an agent SDK. The workloads it runs do not have to be AI agents, though autonomous, untrusted, mostly-idle agents are the target for its isolation and scheduling model. It runs on standard Kubernetes: Pods, autoscaling, NetworkPolicies. It adds agent-specific scheduling, snapshotting, identity, and network mediation. This allows agentic, inference, training, and reinforcement learning workloads to share one infrastructure plane.

Security Model

Substrate treats every actor as untrusted and mutually hostile, so tenants can share physical nodes without sharing a security boundary. Every control operates within the cold-start and resume budget. See the Security Guide for the full list of security principles and threat model.

  • Always sandboxed, VM-grade boundaries: Every actor runs inside a dedicated gVisor (https://gvisor.dev/) (runsc) or Cloud Hypervisor (https://www.cloudhypervisor.org/) microVM sandbox with segregated compute, network, and storage. Plain Linux containers are not supported.
  • Per-agent cryptographic identity: Every active actor receives a short-lived SPIFFE (https://spiffe.io/) X.509 certificate by default, and the egress gateway can mint per-actor JWTs on demand when an egress policy calls for one. Both carry the actor identity and atespace (the Substrate namespace grouping), and are designed to federate with external identity systems.
  • Zero credentials in the sandbox: Actors never hold actor identity JWTs/certs, API keys, or cloud tokens. The built-in egress gateway (atenet-egress) injects credentials into approved outbound requests outside the sandbox, so a prompt-injected or compromised agent has nothing to exfiltrate.
  • Default-deny networking: Actors start with a private network stack, no inbound path except through the ingress router (atenet-router), no lateral actor-to-actor or control-plane access, and outbound traffic restricted by the egress gateway.
  • Pluggable policy at the edge: The egress gateway acts as an external Policy Enforcement Point (PEP) that is fully replaceable, allowing you to attach your own authorization, DLP, prompt-injection scanning, or cloud IAM policy engine without placing policy code inside the untrusted sandbox.

Performance & Multiplexing

Agent Substrate Demo (https://www.youtube.com/watch?v=ZEzkCFJkzjY)

Watch the Agent Substrate cluster multiplex ~250 stateful actors across just 8 physical pods.

This demo highlights the core developer experience and “Agentic Infrastructure” capabilities of Substrate:

  1. Actor Teleport: Suspend and resume actors onto any available worker in the pool, when needed.
  2. Full-State Persistence: Working memory (RAM) and filesystem state are captured in snapshots and restored intact across hibernation cycles, without losing in-flight sessions or tool state.
  3. 10x–30x Oversubscription: Multiplex a large registry of stateful actors onto a small pool of warm worker pods, paying for compute only while an agent is actively thinking or executing a tool.

To reproduce this demo in your own cluster, see the Counter Demo. For more walkthroughs, visit the agent-substrate YouTube channel (https://www.youtube.com/channel/UCN9PPqlTtVxlcpbQ-NWpfZQ).

Star History

Framework Agnostic & Compatibility

Agent Substrate is framework and agent harness agnostic. Because it runs standard OCI container images inside gVisor or microVM sandboxes and enforces security outside the workload, it hosts agents built on any stack without SDK changes:

  • Agent Development Kit (ADK): Preserves session state across invocations as actor state.
  • LangChain: Drop-in execution environment for LangChain agents and tool calls.
  • Claude Code, Codex, and Antigravity: High-density, stateful coding environments that keep system and filesystem state across sessions while isolating untrusted code execution.
  • Model Context Protocol (MCP): Deploy sandboxed MCP servers as Substrate Actors with per-actor identity and egress controls.

Ecosystem & Examples

  • Agent Executor (https://github.com/google/ax): A distributed agent runtime that demonstrates building a secure, hyper-scalable agent harness on Agent Substrate (see the announcement blog (https://cloud.google.com/blog/products/ai-machine-learning/agent-executor-googles-distributed-agent-runtime) and integration guide (https://github.com/google/ax#2-deploy-the-control-plane)).
  • kagent (https://github.com/kagent-dev/kagent): A CNCF Sandbox project and Kubernetes-native framework for building, deploying, and managing AI agents that uses Agent Substrate to run sandboxed, stateful agent workloads (see the announcement blog (https://kagent.dev/blog/the-future-of-kagent)).

Status and compatibility

Agent Substrate has reached 1.0. We are committed to compatibility and stability of the public API, CLI, and configuration formats. We will continue to add new features, but we will not break existing APIs or configuration formats without a major version bump. See “Compatibility and schema evolution” in CONTRIBUTING.md for more details.

Supported Kubernetes Releases

Currently we aim to support the latest stable release (https://kubernetes.io/releases/) of Kubernetes, and the previous minor release.

Community

For announcements, technical discussions, and community support, please join the ate-dev (https://groups.google.com/g/ate-dev) Google Group. We host a weekly community meeting every Thursday from 10:00am - 11:00am PST.

  • Video call link: https://meet.google.com/uhq-cxvn-dhy
  • Or dial: (US) +1 253-289-6971 PIN: 787 664 574 59#
  • More phone numbers: https://tel.meet/uhq-cxvn-dhy?pin=9044088223662
  • Meeting notes (https://docs.google.com/document/d/1obSIvfcafLNniLYTQCcT2eCgxHqa2AQ3Ga7YTsju49s) for the weekly sync meeting
  • Recordings and transcripts (https://drive.google.com/corp/drive/u/0/folders/1rX1S6vPxPrR8dA1lEBuBEXkGKjHtG-mL) of all community meetings

We also have channels in the CNCF slack; request an invite here (https://slack.cncf.io/) if you don’t have access.

  • #substrate-users (https://cloud-native.slack.com/archives/C0B6RCAJULW) to discuss using substrate.
  • #substrate-dev (https://cloud-native.slack.com/archives/C0B6M3E2J3D) to discuss developing substrate.

Developing

Please see CONTRIBUTING.md for guidelines on contributing to the project. We welcome contributions of all kinds, but the project is VERY young. Our immediate focus is on building out the core system and demos, so we may not be able to review or merge contributions that don’t align with those goals in the near term.

Quickstart (Development)

To quickly set up the complete environment:

  1. Make sure you have Go (https://go.dev/doc/install), kubectl (https://kubernetes.io/docs/tasks/tools/), and docker (https://www.docker.com/) installed and configured on your dev machine. We will automatically manage other dependencies via Go, including kind (https://kind.sigs.k8s.io/). A Linux host that will run more than about a thousand sandboxes also needs a larger kernel neighbor table (net.ipv4.neigh.default.gc_thresh1/2/3 = 4096/16384/32768); hack/create-kind-cluster.sh warns when the host is at the default of 1024.

  2. Run the following steps:

# create cluster and local registry
hack/create-kind-cluster.sh

# make ate-setup target the kind cluster; set this in every shell that runs
# ate-setup, including for the demo walkthroughs
export ATE_INSTALL_KIND=true

# install Substrate
go run ./cmd/ate-setup deploy ate-system --credential-provider='{"name":"k8s.io"}' \
  --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client

# install counter demo
go run ./cmd/ate-setup deploy demo counter

# install kubectl-ate
go install ./cmd/kubectl-ate

# create a counter actor in the demo's atespace (--template names the
# actor template, resolved in the actor's atespace)
kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter

# serve a local proxy on port 8000 that sends requests to the actor through the
# network router
kubectl ate proxy actor my-counter-1 -a ate-demo-counter

{"name":"k8s.io"} deploys the bundled credential provider with an empty, default-deny namespace policy. Before an actor can have a credential injected, grant its atespace the namespaces whose Secrets it may read; see Enable it.

  1. In a separate terminal, send an HTTP request to increment the counter:
curl -X POST -i http://localhost:8000/

Custom Setup and Deployment

You can deploy or cleanup specific Agent Substrate components using the installation script. See go run ./cmd/ate-setup for all options.

# Re-deploy only ate-apiserver of the ATE system
go run ./cmd/ate-setup deploy apiserver

# Delete everything (core system and all demos)
go run ./cmd/ate-setup delete all

Tearing down local kind resources

If you need to delete the local kind cluster and its registry (if it was created by hack/create-kind-cluster.sh):

./hack/delete-kind-cluster.sh

GKE Quickstart

To deploy on Google Kubernetes Engine, follow the official GKE Installation Guide (https://docs.cloud.google.com/kubernetes-engine/ai-ml/install-overview-substrate). It lists the prerequisites and gives the one-line installer command. The installer’s source and detailed options are in substrate-gke (https://github.com/ai-on-gke/substrate-gke).

Demos

We provide several sample applications demonstrating Agent Substrate’s capabilities:

  1. Counter Demo: A stateful Go HTTP server demonstrating state preservation across suspends/resumes, and on-demand actor resumption and routing via the Substrate router.
  2. Sandbox Demo (Antigravity): A secure, sandboxed execution environment (running Alpine Linux) that allows arbitrary shell execution while preserving filesystem state across sessions.
  3. Claude Code Multiplex: Demonstrates oversubscribing physical hardware by multiplexing multiple Claude Code agents onto a limited pool of workers.
  4. Multi-Template: Two ActorTemplates running different binaries share one WorkerPool, even though the templates live in different atespaces.
  5. Request Parking: An oversubscribed pool where the router holds inbound requests until a worker frees up, instead of returning 503.

Documentation & Guides

  • Architecture: How the control plane, node supervisor, and networking stack fit together.
  • API Configuration Guide: Detailed reference for configuring WorkerPools, ActorTemplates, Secrets, and Volumes.
  • Full CLI Documentation: Installation and usage for kubectl-ate.
  • Glossary: Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateworker) and how they relate.
  • Integration Repositories: Where integrations live, how their repositories are named, and how fixes flow back to core.
  • Observability Guide: Guide to actor logging, metrics, and distributed tracing.
  • Extending Substrate: How to augment or replace the built-in ingress and egress gateways.
  • Authentication Guide: Which credential people, pods and CI jobs should use with the Substrate API, and how to trust more JWT issuers.
  • Actor JWTs: What an actor JWT contains, how relying parties verify it, and how to rotate the signing key.
  • Federating actor JWTs: Serve the actor JWT issuer publicly and set up a relying party outside the cluster to trust it.
  • Rotating the actor JWT signing key: The rotation steps, how long to wait between them, and replacing a compromised key.
  • Egress Traffic: Which protocols an Actor may reach the outside world with and which are blocked.
  • Enabling man-in-the-middle (MITM) interception for Actor Egress policy: Egress policies such as header injection depend on MITM interception of Actor traffic. This guide explains how an Actor should be configured to enable interception.
  • Request Parking: How the router parks requests through transient worker-pool saturation.
  • Sandbox Asset Cache: How a node caches the sandbox binaries actors run on, decides which runtimes it needs, and evicts to a budget.
  • Upgrade Runbook: Upgrade a running install to a newer supported release, and roll back if needed.
  • Node Storage: How to give snapshot I/O its own disk.
  • Security Guide: Security overview, security principles, and threat modeling.
  • Benchmarking Guide: Locust-based load tests, monitoring stack, and the orchestrated benchmark harness.

Tour

Commands

  • cmd/ateapi: The core control plane API server exposing gRPC endpoints to manage actor and worker lifecycles.
  • cmd/atelet: A node-level DaemonSet that supervises physical worker pods, coordinates snapshotting, and manages state transfers.
  • cmd/atecontroller: A Kubernetes controller that reconciles WorkerPool custom resources.
  • cmd/ate-idp-server: Serves the OpenID discovery document and key set that relying parties use to verify actor JWTs.
  • cmd/atenet: A combined network

Similar Articles

AI agents might need their own Kubernetes moment!

Reddit r/AI_Agents

Discusses the operational challenges of deploying AI agents at scale, drawing a parallel to how Kubernetes solved container orchestration. Suggests the agent ecosystem needs a similar infrastructure breakthrough.

What if an agent runtime never tells the model what to do next?

Reddit r/AI_Agents

作者介绍了一种名为 LCRK(LIA Cognitive Runtime Kernel)的被动式智能体运行时架构:基础设施只检测事件、维护持久内部状态并提供工具访问,不负责编排下一步动作,由模型自主决定是否行动、做什么或保持静默。该系统不依赖 LangChain、CrewAI 或 AutoGen,源码暂未公开。

@Saboo_Shubham_: GitHub Repo:

X AI KOLs Following

Agent Substrate is a Google open-source runtime that enables high-density lifecycle management for large-scale AI agent deployments, multiplexing many stateful agents onto fewer physical workers via Kubernetes and microVM/gVisor sandboxes with sub-second suspend/resume.