AICR v1.0: Open, stable, and verifiable GPU cluster configuration (6 minute read)

# AICR v1.0: Open, stable, and verifiable GPU cluster configuration Source: [https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration](https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration) GPU\-accelerated Kubernetes clusters depend on compatible versions across dozens of components, each on its own release cycle: host kernels, GPU drivers, container runtimes, networking, storage, operators, and workload frameworks\. A configuration that works for one service, GPU generation, and Kubernetes release may silently fail for another, and tracing version conflicts after deployment is slow and error\-prone\. [NVIDIA AI Cluster Runtime](https://github.com/NVIDIA/aicr)\(AICR\) addresses this with version\-locked, validated recipes for GPU cluster configuration\. Each recipe pins the component combinations that work together, renders deployment artifacts for Helm, Argo CD, Flux, or Helmfile, and carries signed validation evidence from the hardware it was tested on\. The[v1\.0 release](https://github.com/NVIDIA/aicr/releases/tag/v1.0.0)of AICR establishes a stable compatibility contract across its CLI, REST API, Go SDK, bundle layout, and artifact schemas so operators, integrators, and contributors can build on AICR’s public interfaces with confidence\. *Figure 1: AICR coverage across Kubernetes services, multiple generations of NVIDIA GPU accelerators, supported Linux operating systems, deployment frameworks, and workload platforms* The[validation dashboard](https://validation.aicr.run/)lets operators find recipes by service, GPU, operating system, workload intent, and optional platform, then inspect each recipe’s status and any published evidence for the hardware configuration tested\. Integrators can build against AICR’s public interfaces under the v1\.x compatibility policy\. Contributors can propose recipes for environments the maintainers cannot test, validate them on their own clusters, and submit signed evidence for maintainer review\. That recipe model is also finding uses across the ecosystem\.[Pulumi Labs](https://www.pulumi.com/registry/packages/nvidia-aicr/)exposes AICR through an infrastructure\-as\-code provider, while[Mirantis’s k0rdent integration](https://github.com/k0rdent/catalog/blob/main/apps/aicr/data.yaml)packages it for multi\-cluster management\. Together, they demonstrate the value of defining GPU\-accelerated Kubernetes configuration once and consuming it through different tools\. Today, AICR has over 100 distinct contributors, with almost half from outside of NVIDIA\! ## Why GPU cluster configuration needs a reproducible contract[https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration#why_gpu_cluster_configuration_needs_a_reproducible_contract](https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration#why_gpu_cluster_configuration_needs_a_reproducible_contract) GPU\-accelerated Kubernetes clusters depend on compatible versions and settings across host kernels, GPU drivers, container runtimes, Kubernetes, networking, storage, device plugins, operators, schedulers, and workload frameworks\. These components follow different release cycles; upgrading one can break a previously working combination\. A configuration validated for one service, GPU generation, fabric type, machine shape, and Kubernetes release may fail for another, and small version differences can be difficult to trace after deployment\. Even if every component installs successfully, the cluster may not meet the recipe’s intended configuration\. Installation doesn’t confirm that components are healthy, that required capabilities like gang scheduling or accelerator discovery work, or that measured results meet a recipe’s performance thresholds\. Knowledge of which combinations work and how they were validated has lived in separate validation systems, deployment scripts, and operational runbooks\. That makes it hard for teams to discover, reproduce, and update working configurations\. Over the past six months, AICR has grown from a handful of recipes to a library spanning major Kubernetes services and the current NVIDIA accelerator portfolio, rendered as deployer\-neutral bundles\. We added live\-cluster validation, signed evidence, public evidence aggregation, and supply\-chain verification\. For v1\.0, we also added committed compatibility baselines and merge\-blocking checks around the public integration surfaces\. ## From observed state to a verifiable result[https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration#from_observed_state_to_a_verifiable_result](https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration#from_observed_state_to_a_verifiable_result) AICR provides four core capabilities: - **Snapshot**records observed cluster state, including Kubernetes, operating system, kernel, GPU, and topology information\. - **Recipe**describes the desired, version\-locked component configuration and the constraints and validation phases that apply to it\. - **Bundle**renders the recipe into artifacts for the operator’s preferred deployment tooling\. - **Validation**compares the recipe with observed state and, where declared, runs deployment, conformance, and performance checks against the cluster\. These four capabilities are deliberately independent\. A snapshot records the observed state; it is not a desired configuration\. A recipe describes the desired configuration; it does not reconcile a cluster\. Common open source CD tools like Helm, Argo CD, Flux, or Helmfile apply or reconcile the bundle into the cluster\. AICR can then validate the running cluster against the recipe, and record signed evidence of the result\. These capabilities can be combined in multiple sequences\. Snapshot data or explicit target criteria can produce a recipe\. A recipe can produce a bundle for an existing deployer\. The recipe and observed cluster state feed validation\. An operator explicitly verifies bundles and evidence when they invoke the corresponding command\. *Figure 2: The NVIDIA AI Cluster Runtime lifecycle* For example, an operator can select EKS, GB300, Ubuntu, training, and Kubeflow; resolve those criteria to a pinned recipe; render the recipe for Argo CD; deploy it through the existing GitOps workflow; and validate the running cluster against the same recipe\. The intended configuration does not change if the operator instead renders it for Helm, Flux, or Helmfile\. ## What’s new in AICR v1\.0[https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration#what%E2%80%99s_new_in_aicr_v10](https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration#what%E2%80%99s_new_in_aicr_v10) AICR v1\.0 defines compatibility rules for its public CLI, REST API, Go SDK, bundle layout, and artifact schemas\. It also lets operators inspect the validation evidence published for each Supported recipe: what was tested, which checks passed, and who signed the results\. AICR v1\.0 sets compatibility rules for: - `aicr`CLI’s public commands, flags, exit semantics, and structured output - `aicrd`REST API and OpenAPI contract - exported API of the[`github\.com/NVIDIA/aicr/pkg/client/v1`](https://github.com/NVIDIA/aicr/tree/main/pkg/client/v1)package - generated bundle layout and AICR artifact schemas Each public interface has a committed baseline checked before changes merge\. The release policy also defines semantic breaking changes\. After v1\.0, removing or incompatibly changing a stable public interface requires a new major release\. For Go integrators,`pkg/client/v1`exposes the supported workflow without requiring imports from AICR’s internal packages\. The CLI and REST server use the same facade, reducing the risk that one public entry point behaves differently from another\. ## Try AICR and contribute[https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration#try_aicr_and_contribute](https://developer.nvidia.com/blog/aicr-v1-0-open-stable-and-verifiable-gpu-cluster-configuration#try_aicr_and_contribute) Try a recipe for your environment, inspect its status and any published evidence, and run the dashboard’s`aicr evidence verify`command where evidence is available\. Contributions are especially useful for hardware and cluster combinations outside current project coverage\. You can: - Contribute features, integrations, or documentation - Propose a recipe for hardware, OS, or a service not yet represented in AICR - Validate a recipe in your own cluster and[submit signed evidence](https://github.com/NVIDIA/aicr/blob/main/docs/contributor/evidence-publishing.md) - Report bugs, share feedback, or request features through GitHub Issues Start with the[project repository](https://github.com/NVIDIA/aicr),[contributing guide](https://github.com/NVIDIA/aicr/blob/main/CONTRIBUTING.md), and[issue tracker](https://github.com/NVIDIA/aicr/issues)\.