How Kubernetes Probes Work

# How Kubernetes probes work | ngrok blog Source: [https://ngrok.com/blog/probes](https://ngrok.com/blog/probes) I’m going to show you,*really show you*, how probes work in Kubernetes\. How they can make your application more resilient, and how they can help you prevent avoidable mistakes\. Like restart loops that take hours to recover from, and dropping requests during rollouts\. Every interactive demo in this post uses[webernetes](https://github.com/ngrok/webernetes), my partial port of Kubernetes to TypeScript\. It contains more than 100,000 lines of ported Kubernetes Go code to run a simulated cluster*right here in your browser*\. I verified the behaviour of these demos against k3s and managed to find a bug in Kubernetes\! More on that later\. ### What you will learn ## [https://ngrok.com/blog/probes#a-pod-without-probes](https://ngrok.com/blog/probes#a-pod-without-probes)A pod without probes I want to run a pod with a single container\. Here’s its manifest,`pod\-a\.yaml`: ### pod\-a\.yaml ``` 1apiVersion: "v1"2kind: "Pod"3metadata:4 name: "pod-a"5spec:6 containers:7 - name: "app"8 image: "my-app:latest" ``` This image,`my\-app:latest`, spends a few seconds initialising before listening on port 8080\. You will see this below when you clickrestartto send thecontainera signal, causing it to crash and get started back up by Kubernetes\. You canpauseorresetany demo at any time\. - 0/2Restart containerNot yet complete\. After the first crash, thecontainerrestarts straight away\. After the second, Kubernetes imposes aCrashLoopBackOffon it before starting it again\. By default this delay is 10 seconds, doubling with each crash up to a maximum wait of 5 minutes\. I shortened it to 3 seconds for this demo\. In both cases, Kubernetes considers the containerReadyas soon as it starts, even though we know it’s not\. It’s still doingstartupwork and not listening on port 8080\. Next I’ll addpod\-b, which sends arequesttopod\-aevery 2 seconds\. Throughout the post, you can think ofpod\-bas any source of client traffic: an ingress controller, a load balancer, inter\-service requests, etc\. If yourestartpod\-ain the demo below while arequestis on its way, that request willfail\. - Cause a request to failNot yet complete\. From the moment you restart thecontaineruntil itsstartupwork finishes, requests willfail, even though the container is consideredReady\! This is not what I want\. I need Kubernetes to know whenpod\-ais ready to receive traffic\. For this, Kubernetes gives us**probes**\. Probes are periodic checks sent to containers to determine their health\. They come in three flavours: - Startup probesdetermine whether my application inside the container has started\. - Readiness probesdetermine whether my application is ready to receive traffic\. - Liveness probesdetermine whether my application needs to be restarted\. It sounds likestartup probesare best suited to the problem I showed you in the demos above, so let’s start there\. ## [https://ngrok.com/blog/probes#startup-probes](https://ngrok.com/blog/probes#startup-probes)Startup probes Below, I’ve added astartup probeto`pod\-a\.yaml`: ### pod\-a\.yaml ``` 1apiVersion: "v1"2kind: "Pod"3metadata:4 name: "pod-a"5spec:6 containers:7 - name: "app"8 image: "my-app:latest"9 startupProbe:10 httpGet:11 path: "/startup"12 port: 808013 periodSeconds: 114 failureThreshold: 5 ``` It’s an`httpGet`probe that sends a`GET /startup`request to the pod on port 8080\. Status codes 200\-399 count as a success\. This happens every`periodSeconds`seconds, and is allowed to fail`failureThreshold`consecutive times before Kubernetes kills the container\. This gives my container ~5 seconds to complete itsstartupwork\. Kubernetes also supports`tcpSocket`,`exec`, and`grpc`probes\. These establish a TCP connection, run a command inside the container, or call the gRPC health\-checking protocol to establish container health\. You can[read about them in the Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/probes/)\. I’ll be using`httpGet`throughout this post\. Probes are sent by a process called thekubelet\. Each node in the cluster has its own kubelet, and it’s the kubelet’s job to make sure the right pods are running and being probed for each node\. When yourestartpod\-abelow, it now shows asNotReady\. Kubernetes is now*aware*thatpod\-ahasn’t initialised yet\. It only becomesReadyafter the firststartup probesucceeds\. - 0/2Restart containerNot yet complete\. kubelet NotReadyis the default for pods with containers that have astartup probe\. However, even when not ready,pod\-bstill sends requests topod\-aand those requests stillfailduring the container’sstartupperiod\. This is because I’ve configuredpod\-bto send requests directly topod\-a’s IP address, which bypasses the readiness mechanism\. ### I'm lying a bit about NotReady *Technically*Kubernetes doesn’t have a`NotReady`condition, it has a`Ready`condition that can be`True`,`False`, or`Unknown`\. I’m referring to it as`NotReady`because it was shorter than having`Ready=True`or`Ready=False`in the demos\. To fix thesefailedrequests I need to graduate to a more production\-grade setup: multiple copies ofpod\-awith requests load\-balanced between them\. I’m going to create a ReplicaSet configured to run 2 replicas ofpod\-aand a Service to load balance between them\. ### replica\-set\-a\.yaml ``` 1apiVersion: "apps/v1"2kind: "ReplicaSet"3metadata:4 name: "replica-set-a"5spec:6 # Run 2 copies of the pod defined under `template`.7 replicas: 28 selector:9 matchLabels:10 # Consider pods with this label to be part of this replica set.11 app: "pod-a"12 template:13 metadata:14 labels:15 app: "pod-a"16 spec:17 # The same pod spec from before.18 containers:19 - name: "app"20 image: "my-app:latest"21 startupProbe:22 httpGet:23 path: "/startup"24 port: 808025 periodSeconds: 126 failureThreshold: 5 ``` ### service\-a\.yaml ``` 1apiVersion: "v1"2kind: "Service"3metadata:4 name: "service-a"5spec:6 selector:7 # Load-balance between pods that have this label.8 app: "pod-a"9 ports:10 # Send requests to this port on the pods.11 - port: 8012 targetPort: 8080 ``` pod\-bwill from now on send requests to the DNS name Kubernetes creates for the Service, in this case`service\-a\.default\.svc\.cluster\.local`, instead of directly to an individual pod\. Kubernetes uses a pod’s`Ready`condition to include or exclude it from Service load balancing\. Below you can click therestartbutton to crash only the topcontainer\. Notice that when the top container isstarting up, requests are always sent to the bottom container\. When a container isNotReady, it marks the whole pod not ready and it won’t get traffic from any Services it is part of\. - 0/2Restart top containerNot yet complete\. kubelet Despite this, requests*can*stillfailif they’re in\-flight when you restart the top container\. This happens because therestartbutton crashes the container abruptly\. It doesn’t get a chance to finish in\-flight requests\. The better thing to do here is*delete*the pod and rely on the ReplicaSet to bring up a new one\. This is better for 2 reasons: 1. Kubernetes gives pods a 30\-second termination grace period by default, which I’ve configured to 2 seconds in this post so you don’t have to wait\. When deleted, pods are consideredterminatingand Kubernetes removes them from any Services they’re part of\. They won’t receive any new requests\. 2. ReplicaSets don’t countterminatingpods as active replicas, so they create replacements as soon as the deleted pod is terminating\. Together, graceful termination and thestartup probekeep requests away from containers that are starting or stopping\. In this next demo, clickingdeletewon’t cause any requests frompod\-btofail\. - 0/2Wait for containers to be readyNot yet complete\. kubelet There’s always a pod ready to service a newrequest, making it safe to delete pods without interrupting user traffic\. ### How does this grace period actually work? ### [https://ngrok.com/blog/probes#how-to-misconfigure-a-startup-probe](https://ngrok.com/blog/probes#how-to-misconfigure-a-startup-probe)How to misconfigure a startup probe Earlier I mentioned that I’m giving my pod ~5 seconds to complete itsstartupwork by setting`failureThreshold`to 5 with a`periodSeconds`of 1\. Choose these values on your own containers carefully\. Too little time can cause a container to crash\-loop\. Setting the`failureThreshold`below will restart thecontainerwith the new value\. Set it to 1 or 2 and see what happens\. - Make**pod\-a**crash loopNot yet complete\. kubelet After a few restarts,pod\-ais put inCrashLoopBackOff\. Thestartup probenever gives thecontainerenough time to start, so this demo crash\-loops until you set`failureThreshold`back to 3 or above\. When configuring this for your own containers, choose values that allow for your worst\-case startup time\. ## [https://ngrok.com/blog/probes#readiness-probes](https://ngrok.com/blog/probes#readiness-probes)Readiness probes After anystartup probesucceeds,readiness probesmonitor the container for the rest of its life\. Failing a readiness probe marks the containerNotReadyand removes it from receiving requests for any Service it is part of\. I’ve modified`pod\-a\.yaml`to have just a readiness probe for now: ### pod\-a\.yaml ``` 1apiVersion: "v1"2kind: "Pod"3metadata:4 name: "pod-a"5spec:6 containers:7 - name: "app"8 image: "my-app:latest"9 readinessProbe:10 httpGet:11 path: "/ready"12 port: 808013 periodSeconds: 314 failureThreshold: 115 successThreshold: 1 ``` I’m sending it to the`/ready`endpoint every 3 seconds\. After a single failure, the container gets theNotReadycondition\. Switch`/ready`in the demo below from200to503and watch the container become not ready\. - Wait for**pod\-a**to become readyNot yet complete\. kubelet ### Out\-of\-band probing The demo above sets`failureThreshold`and`successThreshold`to 1, but I don’t want a single transient failure to remove my pods from their Services\. Below I’ve set the thresholds to 2\. Set`/ready`to503again and notice it now takes 2 failures before thecontainerbecomesNotReady\. - Wait for**pod\-a**to become readyNot yet complete\. kubelet You may notice here that when flipping from ready to not ready, an out\-of\-band probe can be fired\. This is for the same reasons as before\. The pod isNotReadyand its status just got updated\. By default`successThreshold`is 1 and`failureThreshold`is 3\. Generally good defaults that I don’t recommend changing unless you have a great reason\. ### [https://ngrok.com/blog/probes#why-do-we-need-startup-probes-if-we-have-readiness-probes](https://ngrok.com/blog/probes#why-do-we-need-startup-probes-if-we-have-readiness-probes)Why do we need startup probes if we have readiness probes? The demos above only use areadiness probe\. Probing starts straight away and doesn’t succeed until my container has finished itsstartupwork\. This is exactly the job mystartup probewas doing, so why do we need both probe types? A few good reasons: 1. Startup probes delay readiness and liveness starting until initialisation is complete\. 2. They allow startup to have a separate`periodSeconds`and`failureThreshold`, so slow initialisation can be probed more frequently than steady\-state readiness and liveness\. 3. Repeated startup failures kill the container and apply its restart policy\. Readiness failures don’t\. A restart*could*help a stuck container become ready\. You can use multiple probes at the same time\. For example, I might send astartup probeevery second to detect initialisation quickly, then slow down to every 5 seconds for myreadiness probeto reduce steady\-state probe load on thecontainerandkubelet\. ### pod\-a\.yaml ``` 1apiVersion: "v1"2kind: "Pod"3metadata:4 name: "pod-a"5spec:6 containers:7 - name: "app"8 image: "my-app:latest"9 startupProbe:10 httpGet:11 path: "/startup"12 port: 808013 periodSeconds: 114 failureThreshold: 515 readinessProbe:16 httpGet:17 path: "/ready"18 port: 808019 periodSeconds: 5 ``` Readiness probesdon’t start until thestartup probesucceeds\. I’ve started the demo below paused so you can see it from the start\. Hit theplaybutton when you’re ready, and pressresetif you want to start again from the beginning\. This guarantee, that readiness probes don’t start until startup succeeds, allows me to check startup\-specific things in the`/startup`endpoint\. I could make sure initial configs have been loaded, caches have been pre\-warmed and so on\. In practice, startup probes are less commonly used than readiness probes\. It’s nice to know they’re there as an option if I need them, though\. If you do find yourself wishingreadiness probescould restartcontainers, though, I have just the thing for you\. ## [https://ngrok.com/blog/probes#liveness-probes](https://ngrok.com/blog/probes#liveness-probes)Liveness probes The final probe type is theliveness probe\. This probe works just like thereadiness probe, but instead of marking a containerNotReadywhen it reaches its`failureThreshold`, the liveness probe kills the container\. Kubernetes then applies the Pod’s`restartPolicy`, which defaults to`"Always"`and means a killed container will be restarted\. ### pod\-a\.yaml ``` 1apiVersion: "v1"2kind: "Pod"3metadata:4 name: "pod-a"5spec:6 containers:7 - name: "app"8 image: "my-app:latest"9 startupProbe:10 httpGet:11 path: "/startup"12 port: 808013 periodSeconds: 114 failureThreshold: 515 livenessProbe:16 httpGet:17 path: "/live"18 port: 808019 periodSeconds: 220 failureThreshold: 1 ``` This helps when the container can’t recover on its own, such as when its main thread has deadlocked or a critical background thread has died\. If I can reliably detect these conditions, I can fail the liveness probe and rely on Kubernetes to restart the container\. The demo below showspod\-agetting sentstartup probesuntil it finishes itsstartup, after which theliveness probesbegin\. Set the`/live`endpoint to return503to see thecontainerget restarted\. - Cause a liveness restartNot yet complete\. kubelet ### There's something wrong with the demo above\.\.\. ### [https://ngrok.com/blog/probes#how-to-misconfigure-a-liveness-probe](https://ngrok.com/blog/probes#how-to-misconfigure-a-liveness-probe)How to misconfigure a liveness probe It would be a bad idea for myliveness probeto check if my database is healthy\. A blip in the database could cause all of mycontainersto crash\-loop if it lasts long enough\. When you take the database down in the demo below, thepod\-aliveness probeswill fail\. After a few failures, each container will go intoCrashLoopBackOff\. To stress how bad this can be, I’ve made the backoff delay scale like it does in real Kubernetes: 10 seconds at first, doubling for each crash\. Go and cause some havoc\! - Take the database downNot yet complete\. database kubelet This problem gets worse if clients retry\. It hasn’t come up in any other demos so far, but mypod\-acontainers can only handle 3requestsper second\. If they get more than that, they get overloaded and crash\! I’ve configuredpod\-bin the demo below to retry failed requests in a loop, and set the maximumCrashLoopBackOffdelay to 5 seconds again\. Cause another outage, and see if you can recover from it\. - Take the database downNot yet complete\. database kubelet The retries create what’s called a**thundering herd**, which causes a**cascading failure**\. It doesn’t matter that the database is up, anycontainerthat dares to recover gets a laser beam of traffic that kills it again\. Probes, sadly, can’t help me get out of this\. I would need to create some way to only let a small percentage of traffic through, allowing the containers time to recover, then ramp back up to full traffic over time\. Or if I have control over the clients, for example they’re a mobile app I’ve also created, I could add a backoff delay to the retries\. This would slow the traffic growth, making it easier to recover\. The best thing I can do, though, is**avoid this mistake in the first place**\. Fail aliveness probeonly when the failure is local to onecontainerand a restart is likely to restore it\. Don’t fail on conditions that will be true for all of your containers at the same time\. ## [https://ngrok.com/blog/probes#probes-and-deployments](https://ngrok.com/blog/probes#probes-and-deployments)Probes and Deployments The last thing I want to touch on is how probes affect Deployments\. In Kubernetes, most of a Pod’s`spec`is immutable\. The way to update an immutable field is to create a new Pod and delete the old one\. Deployments manage this replacement as a “rollout\.” Let’s take`deployment\-a\.yaml`here as an example: ### deployment\-a\.yaml ``` 1apiVersion: "apps/v1"2kind: "Deployment"3metadata:4 name: "deployment-a"5spec:6 replicas: 37 strategy:8 type: "RollingUpdate"9 rollingUpdate:10 maxUnavailable: "25%"11 maxSurge: "25%"12 selector:13 matchLabels:14 app: "pod-a"15 template:16 metadata:17 labels:18 app: "pod-a"19 spec:20 containers:21 - name: "app"22 image: "my-app:latest"23 ports:24 - name: "http"25 containerPort: 808026 startupProbe:27 httpGet:28 path: "/startup"29 port: "http"30 periodSeconds: 131 successThreshold: 132 failureThreshold: 5 ``` I’ve highlighted the`strategy`because it’s the part that controls how new Pods get rolled out\. Deployments start off by creating a ReplicaSet to bring up the`replicas`I’ve configured\. Changing a Deployment’s`template`after it has been created makes a new, second ReplicaSet configured with this new`template`\. The Deployment then scales up the new ReplicaSet while scaling down the old one, based on the`strategy`parameters\. Here’s what each`strategy`parameter means: 1. `type: "RollingUpdate"`updates the Pods gradually rather than all at once\. If you did want all at once, you would use`type: "Recreate"`\. This first scales the old ReplicaSet to 0, then the new one to the configured`replicas`\. This causes downtime, so it’s not the default\. 2. `maxUnavailable: "25%"`allows`floor\(3 \* 0\.25\) = 0`unavailable replicas, so all 3 must remain available during the rollout\. 3. `maxSurge: "25%"`allows`ceil\(3 \* 0\.25\) = 1`extra pod above`replicas`during the rollout, so in our case 4 replicas are allowed to exist\. It’s a lot, so clickingdeploybelow may help you better understand\. Remember that the rollout has to keep 3 podsReadyat all times, and is allowed to go up to 4 replicas thanks to`maxSurge`\. Pods that areterminatingdon’t count as available, so you will see more than 4 replicas at times\. - 0/3Wait for deployment readyNot yet complete\. kubelet 0\.0s The rollout can only create 1 extra pod, and has to wait for that pod to becomeReadybefore it can kill an old pod\. This means that probes play a direct role in how fast a rollout can go\. You should see that with the above configuration, it takes about 11 seconds to finish\. Also notice that norequestsfrompod\-bfail\. Below, I’ve changed`periodSeconds`from 1 to 5\. See how long it takes todeploywith this longer period\. - 0/3Wait for deployment readyNot yet complete\. kubelet 0\.0s It now takes about 19\-20 seconds for this rollout to complete\. Longerstartup probeperiods delay rollouts because each replacement pod has to wait until it passes the probe\. Keep this in mind when tuning your own probes\. Lastly, what happens if I update a Deployment and have*no probes at all*? In the demo below, you will notice that a rollout will cause a small number of requests tofailbecause the newcontainershaven’t finished theirstartup\. - 0/3Wait for deployment readyNot yet complete\. kubelet 0\.0s A rollout without probes happens very quickly because each container is considered ready as soon as it starts\. This causes a small number of requests tofailbecause thecontainershaven’t finishedstartupyet\. ## [https://ngrok.com/blog/probes#tips-for-designing-good-probe-endpoints](https://ngrok.com/blog/probes#tips-for-designing-good-probe-endpoints)Tips for designing good probe endpoints ### [https://ngrok.com/blog/probes#startup](https://ngrok.com/blog/probes#startup)Startup 1. Use them when startup is slow or variable, or you have initialisation work that can get stuck and needs to be restarted\. 2. Probe frequently to detect initialisation quickly\. If you do lower`periodSeconds`, make sure to increase`failureThreshold`to maintain the total time you wait for startup\. Target your worst\-case startup time, plus a little headroom\. 3. Take advantage of a separate`/startup`endpoint if there are checks you can do to be certain initialisation has finished\. If not, using the same endpoint as your liveness check is reasonable\. ### [https://ngrok.com/blog/probes#readiness](https://ngrok.com/blog/probes#readiness)Readiness 1. Keep this probe cheap and conservative\. Fail it only when removing a pod from serving is likely to improve overall service health\. 2. **Prefer not**to fail readiness based on the status of shared dependencies like database servers and third\-party APIs\. Include a dependency only when a container truly can’t serve useful traffic without it\. 3. **Prefer not**to fail readiness in response to high CPU or memory\. If your service is near total capacity, removing a replica may cause a cascading failure\. ### [https://ngrok.com/blog/probes#liveness](https://ngrok.com/blog/probes#liveness)Liveness 1. Fail this probe**only**when it’s*very*likely a container is stuck and a restart will help\. If you aren’t sure, return success\. 2. **Don’t**fail liveness based on the status of shared dependencies like database servers and third\-party APIs\. 3. **Don’t**fail liveness in response to high CPU or memory\. ### [https://ngrok.com/blog/probes#general-advice](https://ngrok.com/blog/probes#general-advice)General advice 1. Keep probes bounded and cheap\.Startup probeshave a bit more wiggle room than the other two, but they still consume cluster resources that could be spent serving user traffic\. 2. The default`failureThreshold`is 3\. Lower it only when immediate intervention is worth the risk of reacting to a transient failure\. ## [https://ngrok.com/blog/probes#probe-playground](https://ngrok.com/blog/probes#probe-playground)Probe playground Below is a demo that lets you set whatever probe parameters you want\. Changes won’t be applied until you pressdeploy\. It’s surprisingly easy to get yourself into unrecoverable situations, so don’t feel bad about using theresetbutton\. 0\.0s periodSeconds1successThreshold1failureThreshold5initialDelaySeconds0 periodSeconds5successThreshold1failureThreshold1initialDelaySeconds0 periodSeconds5successThreshold1failureThreshold3initialDelaySeconds0 requests per second2max retries0 ## [https://ngrok.com/blog/probes#conclusion](https://ngrok.com/blog/probes#conclusion)Conclusion Probes are tricky to get right\. By*showing*you how they work, and letting you cause some chaos, you’re now better equipped to make informed decisions about your own probes\. If you have feedback about this post, or you’re curious about[webernetes](https://github.com/ngrok/webernetes), I would love to talk to you\! Email me at[s\.rose@ngrok\.com](mailto:[email protected])\. ### The shameless plug ngrok have a[first\-party Kubernetes Operator](https://github.com/ngrok/ngrok-operator)\! It supports both the Ingress and Gateway APIs, as well as letting you declaratively create agent endpoints in your cluster\. You can learn more at our[docs](https://ngrok.com/docs/k8s)\.