Welcome back to Advanced NGINX Configuration and Monitoring! You've done excellent work learning URL rewriting and custom error pages. Now, we're moving into a crucial aspect of building resilient production systems: ensuring that your application remains available even when individual backend servers experience problems.

In this lesson, we'll focus on implementing health checks and automatic failover. We'll configure NGINX to monitor backend server health, automatically route traffic away from failing servers, and expose health check endpoints that can be used by monitoring tools. By the end of this lesson, you'll understand how to set failure thresholds, handle various types of errors gracefully, and create dedicated health check routes with appropriate timeout configurations.

When running applications at scale, backend servers can fail for many reasons: network issues, resource exhaustion, application bugs, or planned maintenance. Without proper health checking, NGINX might continue sending requests to failed servers, resulting in poor user experience and cascading failures.

Health checks solve this problem by continuously monitoring backend server status and adjusting traffic routing accordingly. NGINX supports two approaches:

• Passive health checks: Monitor the results of actual client requests. If a server fails to respond correctly during regular traffic, it's temporarily removed from rotation.
• Active health checks: Proactively send test requests at regular intervals, available only in NGINX Plus.

In this lesson, we'll implement passive health checks, which work well for most applications and are available in the open-source version of NGINX. The key is configuring thresholds that balance quick failure detection against false positives from temporary network hiccups.

Let's start building our configuration by defining multiple backend servers in an upstream block:

The upstream directive creates a named group of backend servers that NGINX will load balance across. In this case, we have three Flask application instances running on ports 5000, 5001, and 5002. By default, NGINX distributes requests using a round-robin algorithm, cycling through each server in order.

Now we'll add passive health check parameters to each server definition:

These parameters control when NGINX considers a server unavailable:

• max_fails=1: The number of unsuccessful connection attempts or error responses before marking the server as down. Setting this to 1 means a single failure triggers removal from rotation.
• fail_timeout=5s: Serves two purposes: it defines the time window for counting failures and how long the server remains unavailable before NGINX retries it.

With these settings, if any server fails just once, NGINX stops sending requests to it for 5 seconds. This aggressive approach works well when you want fast failure detection and have enough healthy servers to handle the load.

We need a location block that proxies client requests to our backend servers:

The trailing slash in both /api/ and http://flask_backend/ is significant: it strips the /api prefix when forwarding requests. A request to /api/users becomes /users at the backend, which is often the desired behavior when backends don't include a common prefix in their routes.

To make our failover mechanism more robust, we'll add the proxy_next_upstream directive:

The proxy_next_upstream directive specifies which failure conditions should trigger NGINX to try the next server in the upstream group:

• error: Connection establishment failed or an error occurred while communicating with the server.
• timeout: The backend took too long to respond.
• http_502, http_503, http_504: Specific HTTP error codes indicating server problems.

This configuration ensures that if one backend returns a 502 Bad Gateway or times out, NGINX automatically retries the request with another server in the pool. The combination of passive health checks and smart retry logic creates a resilient system that handles transient failures gracefully.

Let's add a dedicated endpoint for monitoring our backend health:

This location creates a /health endpoint on our NGINX server that proxies to /status on the backends. External monitoring tools can periodically query this endpoint to verify that at least one backend server is responding correctly. Notice that we're not using a trailing slash here because we want to preserve the exact path structure.

In this lesson, we've built a complete health checking and failover system for backend services. We learned how to configure upstream blocks with multiple servers, set failure thresholds using max_fails and fail_timeout, implement automatic retry logic with proxy_next_upstream, and create dedicated health check endpoints with aggressive timeout settings.

These patterns form the foundation of highly available web architectures. When combined with the error handling techniques from our previous lesson, they ensure that your users experience minimal disruption even when backend services fail. The passive health checks we've implemented work continuously in the background, automatically removing unhealthy servers and restoring them once they recover.

Now it's time to put these concepts into practice; the upcoming exercises will challenge you to configure your own health checking systems and see firsthand how NGINX maintains service availability through automatic failover!