Liveness and Readiness Probes

Introduction: When "Running" Isn't Enough

Imagine you've deployed your web application to production. You check kubectl get pods and see STATUS: Running for all your containers. Everything looks perfect, so you go home for the evening. But at 2 AM, you get paged because users are reporting error messages and timeouts. You check again — the pods still show as Running . What's going on? A container can be technically running but completely nonfunctional. Maybe the application deadlocked while processing a request. Maybe it ran out of database connections and can't serve traffic anymore. The pod status only tells Kubernetes that the process is alive, not whether it's actually healthy. In this lesson, you'll learn how health probes enable Kubernetes to automatically detect these problems and take corrective action, whether that means restarting the container or temporarily stopping traffic until it recovers.

Understanding Health Probe Types

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

How Probes Work: Types and Configuration

Kubernetes supports three different mechanisms for performing health checks, each suited to different types of applications. The httpGet probe makes an HTTP GET request to a specified path and port on the container. If the response status code is between 200 and 399, the probe is considered successful. This is the most common probe type for web applications and REST APIs because these applications already expose HTTP endpoints. For example, you might configure a probe to hit /health or /ready endpoints that your application provides specifically for health checking.

The exec probe runs a command inside the container. If the command exits with status code 0, the probe succeeds. This is useful for applications that don't expose HTTP endpoints, such as databases or message queues. For instance, you might run pg_isready to check if PostgreSQL is accepting connections, or a custom script that verifies specific application conditions. The tcpSocket probe attempts to open a TCP connection to a specified port. If the connection succeeds, the probe passes. This is the simplest probe type and works well for any network service where you just need to verify that the port is accepting connections, without caring about the application-level protocol.

For this lesson, we'll focus on httpGet probes because they're the most widely applicable and because that's what our nginx example uses. When you configure an httpGet probe, you specify the path (like / or /health), the port (like 80 or 8080), and optionally headers or scheme (HTTP vs. HTTPS). Kubernetes will make HTTP GET requests to http://<pod-ip>:<port><path> and evaluate the response. If you're running a simple web server like nginx, checking the root path / works fine. For more complex applications, you'd typically create dedicated health check endpoints that verify database connections, cache availability, and other dependencies.

Several timing parameters control how probes behave. The initialDelaySeconds parameter tells Kubernetes how long to wait after the container starts before performing the first probe. This is critical because most applications need a few seconds to initialize. If you start probing immediately, you'll get false failures while the application is still starting up. The periodSeconds parameter specifies how often to run the probe after the initial delay. For example, periodSeconds: 10 means Kubernetes checks every 10 seconds. You want frequent enough checks to detect problems quickly, but not so frequent that you overwhelm the application with health check requests.

The timeoutSeconds parameter (defaults to 1) sets how long to wait for a probe response before considering it failed. If your application is slow to respond, increase this value. The failureThreshold parameter (defaults to 3) determines how many consecutive probe failures are required before taking action. This prevents transient network issues or momentary spikes from triggering unnecessary restarts or traffic removal. The successThreshold parameter (defaults to 1 for liveness and readiness) specifies how many consecutive successes are needed to consider the probe successful again after it was failing. For most use cases, the default values work well, but understanding these parameters helps you tune probe behavior for your specific application.

Adding Probes to Your Deployment

Let's build a deployment that includes both liveness and readiness probes for an nginx web server. We'll start with the basic structure you learned in previous lessons, then add the probe configurations step by step. Start by creating a file called deployment-probes.yaml with the following content that defines the deployment metadata and replica settings:

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-probes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app

This creates a deployment named web-app-probes that will manage two pod replicas. The selector tells the deployment to manage any pods with the label app: web-app. Now, let's add the pod template that defines what each replica should look like:

YAML

  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: nginx
          image: nginx:1.25

So far, this is familiar — we're defining a pod template with a single container running nginx version 1.25. The pod gets labeled with app: web-app so the deployment can track it. Now comes the important part: adding the liveness probe configuration:

YAML

          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 10

The livenessProbe section goes directly under the container definition because each container can have its own probe. We've configured an httpGet probe that makes requests to path: / (the root URL) on port: 80 (nginx's default HTTP port). This means Kubernetes will perform an HTTP GET request to http://<pod-ip>:80/ to check if the container is alive. nginx is configured by default to serve a welcome page at /, so a healthy nginx container will respond with an HTTP 200 status code, making the probe succeed.

The initialDelaySeconds: 10 setting tells Kubernetes to wait 10 seconds after the container starts before performing the first liveness probe. This gives nginx time to initialize and start listening on port 80. If we didn't include this delay, the first few probes might fail simply because nginx hasn't finished starting up yet, potentially causing Kubernetes to restart a perfectly healthy container. The periodSeconds: 10 setting means that after the initial delay, Kubernetes will check the liveness probe every 10 seconds. If three consecutive checks fail (the default failureThreshold), Kubernetes will restart the container.

Now, let's add the readiness probe configuration right after the liveness probe:

YAML

          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 5

The readinessProbe configuration looks similar to the liveness probe but with different timing parameters. We're using the same httpGet mechanism to check the same path and port — this is common for simple applications where "alive" and "ready" mean essentially the same thing. However, notice that we've set initialDelaySeconds: 5 instead of 10. The readiness probe can start checking sooner because we're not worried about restarting the container if it's not ready yet — we just want to know when we can start sending it traffic. We also check more frequently with periodSeconds: 5 because we want to detect when the pod becomes ready as quickly as possible so we can add it to the Service endpoints and start routing traffic to it.

In a more complex application, you might use different endpoints for liveness and readiness. For example, the liveness probe might hit /health, which just checks if the application process is responding, while the readiness probe might hit /ready, which additionally verifies that database connections are available and caches are loaded. For our nginx example, checking the root path works well for both probes because nginx is either serving the page (healthy and ready) or not responding at all (unhealthy and not ready).

Here's the complete deployment with all sections combined:

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-probes
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 5

Save this complete configuration in deployment-probes.yaml. You now have a deployment that not only runs nginx containers but also continuously monitors their health and readiness, enabling Kubernetes to automatically handle failures and manage traffic routing intelligently.

Deploying and Inspecting Probe Configuration

Let's deploy this health-monitored application to your cluster and verify that the probes are configured correctly. Apply the deployment using kubectl:

Shell

kubectl apply -f deployment-probes.yaml

You should see confirmation that the deployment was created:

text

deployment.apps/web-app-probes created

The Kubernetes API server has received your deployment configuration, and the deployment controller is now working to create your two replica pods. Let's check if the pods were created and are running:

Shell

kubectl get pods -l app=web-app

The -l app=web-app flag filters to show only pods with the app=web-app label. You should see output like this:

text

NAME                              READY   STATUS    RESTARTS   AGE
web-app-probes-6d8f9b7c5d-m8x4j   1/1     Running   0          30s
web-app-probes-6d8f9b7c5d-t5k9p   1/1     Running   0          30s

Both pods show STATUS: Running and READY: 1/1, which means they started successfully and their readiness probes are passing. The RESTARTS column shows 0, indicating that the liveness probes have been succeeding and Kubernetes hasn't needed to restart any containers. If you run this command very quickly after applying the deployment (within 5-10 seconds), you might briefly see READY: 0/1 because the readiness probe hasn't passed yet. Once the probe succeeds, the status changes to 1/1.

Now, let's inspect one of these pods in detail to see exactly how the probes are configured. Copy one of the pod names from your output (your actual pod names will be different than the example shown above) and run:

Shell

kubectl describe pod web-app-probes-6d8f9b7c5d-m8x4j

Replace web-app-probes-6d8f9b7c5d-m8x4j with your actual pod name. The describe command outputs a lot of information about the pod. Scroll down to find the container section, which will look something like this:

text

Containers:
  nginx:
    Container ID:   containerd://abc123def456...
    Image:          nginx:1.25
    Port:           <none>
    State:          Running
      Started:      Mon, 15 Jan 2024 10:30:45 -0800
    Ready:          True
    Liveness:       http-get http://:80/ delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:80/ delay=5s timeout=1s period=5s #success=1 #failure=3

This output confirms that both probes are configured exactly as we specified. The Liveness line shows http-get http://:80/ indicating an HTTP GET probe to port 80 at path /. The timing parameters are displayed: delay=10s (initialDelaySeconds), period=10s (how often it checks), timeout=1s (how long to wait for a response), #success=1 (successThreshold), and #failure=3 (failureThreshold). The Readiness line shows similar information with the different timing we configured.

If you don't want to scroll through all the describe output, you can use grep to extract just the probe information. Run this command:

Shell

kubectl describe pod web-app-probes-6d8f9b7c5d-m8x4j | grep -A3 "Liveness\|Readiness"

The -A3 flag tells grep to show 3 lines After each match, which is usually enough to see the probe configuration. You'll get output focusing just on the probes:

text

    Liveness:       http-get http://:80/ delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:80/ delay=5s timeout=1s period=5s #success=1 #failure=3

This confirms that Kubernetes applied your probe configurations correctly. The kubelet on each node is now actively running these health checks according to the schedule you defined. Every 10 seconds, it checks liveness, and every 5 seconds, it checks readiness. As long as the probes keep succeeding, the pod continues running and receiving traffic.

Understanding Probe Behavior and Events

Understanding what happens when probes fail is crucial for debugging and troubleshooting. When a liveness probe fails repeatedly (three times by default), Kubernetes considers the container to be in an unrecoverable state and takes corrective action. The kubelet kills the container and starts a new one in its place. You'll see the RESTARTS column in kubectl get pods increment each time this happens. The pod itself isn't deleted — it's the same pod object, but the container inside has been restarted. This automatic recovery is one of Kubernetes' most powerful features: it can detect and fix broken containers without any human intervention.

When a readiness probe fails, Kubernetes takes a gentler approach. The container keeps running (it's not restarted), but the pod is removed from any Service endpoints that route traffic to it. This means if you have a Service pointing to pods with label app: web-app, and one pod's readiness probe starts failing, that pod will stop receiving traffic from the Service. The other healthy pods continue handling requests. Kubernetes keeps checking the readiness probe, and as soon as it starts succeeding again, the pod is added back to the Service endpoints and begins receiving traffic. This behavior is perfect for scenarios like temporary database connection issues or cache rebuilding — you want the pod to recover on its own without a full restart.

Let's look at how to monitor probe status and events. The Events section at the bottom of kubectl describe pod output shows a timeline of what's happened to the pod, including probe failures. Run the describe command again and look for the Events section:

Shell

kubectl describe pod web-app-probes-6d8f9b7c5d-m8x4j | grep -A10 "Events"

The -A10 flag shows 10 lines after the "Events" header. For a healthy pod, you'll see events like:

text

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  2m    default-scheduler  Successfully assigned default/web-app-probes-6d8f9b7c5d-m8x4j to minikube
  Normal  Pulling    2m    kubelet            Pulling image "nginx:1.25"
  Normal  Pulled     2m    kubelet            Successfully pulled image "nginx:1.25"
  Normal  Created    2m    kubelet            Created container nginx
  Normal  Started    2m    kubelet            Started container nginx

These events show the normal pod lifecycle: scheduled to a node, image pulled, container created, and started. If a probe had failed, you'd see additional events. For example, if the liveness probe failed three times, you'd see:

text

Warning  Unhealthy  30s   kubelet  Liveness probe failed: Get "http://10.244.0.5:80/": dial tcp 10.244.0.5:80: connect: connection refused
Normal   Killing    30s   kubelet  Container nginx failed liveness probe, will be restarted

The Unhealthy warning shows what happened when the probe failed — in this case, it couldn't connect to the container's port 80. The Killing event shows that Kubernetes decided to restart the container. These events are invaluable for understanding why a container restarted or why it's not receiving traffic.

You can watch pods in real time to see how they transition through different states. Run this command:

Shell

kubectl get pods -l app=web-app --watch

The --watch flag keeps the command running and updates the output whenever the pod status changes. You'll see the current state first, and then any changes will appear as new lines. This is useful for observing how pods recover from probe failures. Press Ctrl+C to stop watching when you're done. In a healthy system, you shouldn't see much activity — the pods stay in Running status with READY: 1/1. But during deployments, scaling operations, or when probes fail, you'll see the status change in real time.

Introduction: When "Running" Isn't Enough

Understanding Health Probe Types

Join the 1M+ learners on CodeSignal

When to Use Liveness and Readiness Probes

How Probes Work: Types and Configuration

Adding Probes to Your Deployment

Deploying and Inspecting Probe Configuration

Understanding Probe Behavior and Events

Summary: Building Self-Healing Applications