Systematic Troubleshooting in Kubernetes

You've spent the last four lessons building a solid foundation for reliable Kubernetes applications — organizing environments with namespaces, managing resources with requests and limits, implementing health checks with probes, and enabling automatic scaling with HPA. But here's the reality: even with all these reliability features in place, things will still break. A deployment might not start, pods might crash unexpectedly, or services might fail to communicate with each other.

When these issues happen in production at 2 a.m., you need a systematic way to diagnose problems quickly and confidently. In this lesson, you'll learn a repeatable troubleshooting workflow that guides you from "something's broken" to "I know exactly what's wrong" using eight diagnostic steps that work for most operational problems in Kubernetes.

To practice realistic troubleshooting, you need an application that mirrors what you'll encounter in production. Most real-world applications aren't single deployments — they're multi-tier systems with frontend components that talk to backend services, databases, and external APIs. We're going to deploy a simplified version of this architecture: a frontend service running nginx that theoretically communicates with a backend service, also running nginx for simplicity. While both tiers use the same image in this example, they represent distinct layers of your application stack, and each has its own deployment and service resource.

Let's build this setup step by step, starting with the frontend deployment. Create a file called frontend-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 2

This creates a deployment named frontend with two replicas for high availability. Now add the selector and pod template:

  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          ports:
            - containerPort: 80

The selector tells the deployment to manage pods labeled with app: frontend, and the template defines those pods with an nginx container exposing port 80. Save this file — it's a standard deployment you've seen in previous lessons. Next, create frontend-service.yaml to expose this deployment:

apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  selector:
    app: frontend
  ports:
    - port: 80
      targetPort: 80

This Service resource creates a stable endpoint called frontend that routes traffic to any pod with the app: frontend label on port 80. The service abstraction is critical because it gives us a predictable DNS name (frontend) even as individual pods come and go. Now, let's create the backend tier. Create backend-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          ports:
            - containerPort: 80

The backend deployment is similar to the frontend but only runs one replica and uses the app: backend label to distinguish it. Finally, create backend-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  selector:
    app: backend
  ports:
    - port: 80
      targetPort: 80

Now you have four manifest files that define a complete multi-tier application. Apply all of them to your cluster in one command:

kubectl apply -f frontend-deployment.yaml -f frontend-service.yaml -f backend-deployment.yaml -f backend-service.yaml

Kubernetes will create all four resources and confirm their creation:

deployment.apps/frontend created
service/frontend created
deployment.apps/backend created
service/backend created

This multi-tier setup gives you a realistic troubleshooting environment. When you encounter issues, you'll need to determine which tier is broken, whether it's a deployment problem or a service configuration issue, and how the components interact. This mirrors real production scenarios where problems can occur at multiple layers of your application stack. On CodeSignal, this environment comes pre-configured, but understanding how to build it yourself is important for working on your own clusters.