Error Rate Alerts with Multi-Threshold Routing

Imagine receiving an alert: "10 errors detected in the payments service." You need to investigate. But when you open your dashboard, you see the service is handling 10,000 requests per minute—those 10 errors represent a 0.1% error rate. Meanwhile, your auth service also has 10 errors, but it's only handling 50 requests per minute—a 20% error rate where one in five login attempts is failing.

The same error count has completely different meanings depending on traffic volume. This is where error rates come in. By calculating the percentage of failed requests, you add context that helps you understand problem severity. A service experiencing 100 errors might be failing catastrophically if it only receives 200 total requests, or it might be performing within acceptable bounds if it's processing 50,000 requests.

This lesson teaches you to calculate error rate percentages from success and failure counters, create alerts with different severity thresholds, and understand how multi-threshold alerting enables intelligent notification routing. You'll build two alert rules monitoring the same metric: one that fires at 5% errors, and another that fires at 15% errors. This foundation prepares you for advanced routing patterns where different severity levels trigger different notification channels.

Your PostgreSQL database contains a table called requests_slo that tracks service reliability by recording successes and failures in pre-aggregated time windows:

CREATE TABLE public.requests_slo (
  ts timestamptz NOT NULL,
  service text NOT NULL,
  ok_count int NOT NULL,
  err_count int NOT NULL
);

Each row represents a time window during which your monitoring system counted successful requests and failed requests separately. This pre-aggregation pattern is common in observability systems, particularly for high-traffic services where storing every individual request would create enormous data volumes.

Your monitoring system collects these counts every minute, so each service generates one row per minute with that minute's success and failure totals. Sample data:

During the minute ending at 14:46, the auth service handled 498 successful requests and 2 failures. Payments had one failure among 408 requests, while search had zero failures among 407 requests. During the previous minute at 14:45, search experienced 3 failures among 164 requests, while auth had 2 failures among 157 requests. These raw counts tell part of the story, but calculating the percentage gives you the context needed for threshold-based alerting.