Scaling frontier LLM inference across GPU nodes opens unauthenticated control-plane ports: our benchmark on Ray + vLLM on Kubernetes

As frontier model inference scales beyond single 8xH100/H200 boxes into multi-node pipeline and tensor parallelism, models rely heavily on distributed frameworks like Ray, KubeRay and vLLM. We benchmarked a multi-node distributed LLM cluster on AWS EKS to inspect the runtime network surface and see what happens between nodes under vendor defaults: What we found: - 15 of 17 active listening sockets never appeared in declared containerPort manifests. - A neighbour container in an unrelated namespace was able to connect to Ray GCS (6379) and raylet worker RPCs (10002 to 10006) over unauthenticated cleartext gRPC. - Four default security scanners (Trivy, Checkov, Kubescape, kube-linter) missed this runtime exposure completely because they do not inspect the RayCluster custom resource. - A single-GPU vLLM deployment exposed 26 API routes with no authentication configured. Mitigations and performance cost: - Ingress default-deny NetworkPolicy blocked all neighbour-pod access to the Ray control plane in steady state without adding measurable latency. - Encrypting inter-node GPU traffic via WireGuard (Cilium chained with AWS VPC CNI) successfully carried the workload, but introduced a 3.4% to 6.5% throughput drop and a 3.2% to 8.4% latency increase under load. As AI models become more distributed and autonomous agents run inside cluster infrastructure, securing inter-node control planes becomes just as critical as protecting the weights. Full report: https://sorami.com.au/research/distributed-llm-inference-hidden-network/ Hardening guide: https://sorami.com.au/guides/securing-ray-vllm-on-kubernetes/ Data and logs: https://github.com/Sorami-Consulting-AU/distributed-llm-inference-hidden-network