Implementing Health Checks | CodeSignal Learn

Implementing Health Checks | CodeSignal Learn

Introduction

Welcome back to Advanced NGINX Configuration and Monitoring! You've done excellent work learning URL rewriting and custom error pages. Now, we're moving into a crucial aspect of building resilient production systems: ensuring that your application remains available even when individual backend servers experience problems. In this lesson, we'll focus on implementing health checks and automatic failover . We'll configure NGINX to monitor backend server health, automatically route traffic away from failing servers, and expose health check endpoints that can be used by monitoring tools. By the end of this lesson, you'll understand how to set failure thresholds, handle various types of errors gracefully, and create dedicated health check routes with appropriate timeout configurations.

Understanding Health Checks and Failover

When running applications at scale, backend servers can fail for many reasons: network issues, resource exhaustion, application bugs, or planned maintenance. Without proper health checking, NGINX might continue sending requests to failed servers, resulting in poor user experience and cascading failures. Health checks solve this problem by continuously monitoring backend server status and adjusting traffic routing accordingly. NGINX supports two approaches: Passive health checks : Monitor the results of actual client requests. If a server fails to respond correctly during regular traffic, it's temporarily removed from rotation. Active health checks : Proactively send test requests at regular intervals, available only in NGINX Plus . In this lesson, we'll implement passive health checks, which work well for most applications and are available in the open-source version of NGINX . The key is configuring thresholds that balance quick failure detection against false positives from temporary network hiccups.

Configuring the Upstream Block

Let's start building our configuration by defining multiple backend servers in an upstream block: nginxhttp { upstream flask_backend { server 127.0.0.1:5000; server 127.0.0.1:5001; server 127.0.0.1:5002; } } The upstream directive creates a named group of backend servers that NGINX will load balance across. In this case, we have three Flask application instances running on ports 5000, 5001, and 5002. By default, NGINX distributes requests using a round-robin algorithm, cycling through each server in order.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Setting Failure Thresholds

Now we'll add passive health check parameters to each server definition: upstream flask_backend { server 127.0.0.1:5000 max_fails=1 fail_timeout=5s; server 127.0.0.1:5001 max_fails=1 fail_timeout=5s; server 127.0.0.1:5002 max_fails=1 fail_timeout=5s; } These parameters control when NGINX considers a server unavailable: max_fails=1: The number of unsuccessful connection attempts or error responses before marking the server as down. Setting this to 1 means a single failure triggers removal from rotation. fail_timeout=5s: Serves two purposes: it defines the time window for counting failures and how long the server remains unavailable before NGINX retries it. With these settings, if any server fails just once, NGINX stops sending requests to it for 5 seconds. This aggressive approach works well when you want fast failure detection and have enough healthy servers to handle the load.

Proxying API Requests

We need a location block that proxies client requests to our backend servers: server { listen 3000; location /api/ { proxy_pass http://flask_backend/; } } The trailing slash in both /api/ and http://flask_backend/ is significant: it strips the /api prefix when forwarding requests. A request to /api/users becomes /users at the backend, which is often the desired behavior when backends don't include a common prefix in their routes.

Handling Request Failures

To make our failover mechanism more robust, we'll add the proxy_next_upstream directive: location /api/ { proxy_next_upstream error timeout http_502 http_503 http_504; proxy_pass http://flask_backend/; } The proxy_next_upstream directive specifies which failure conditions should trigger NGINX to try the next server in the upstream group: error: Connection establishment failed or an error occurred while communicating with the server. timeout: The backend took too long to respond. http_502, http_503, http_504: Specific HTTP error codes indicating server problems. This configuration ensures that if one backend returns a 502 Bad Gateway or times out, NGINX automatically retries the request with another server in the pool. The combination of passive health checks and smart retry logic creates a resilient system that handles transient failures gracefully.

Creating a Health Check Endpoint

Let's add a dedicated endpoint for monitoring our backend health: location /health { proxy_pass http://flask_backend/status; } This location creates a /health endpoint on our NGINX server that proxies to /status on the backends. External monitoring tools can periodically query this endpoint to verify that at least one backend server is responding correctly. Notice that we're not using a trailing slash here because we want to preserve the exact path structure.