Alert Query Multi-Condition Filters

Introduction: Events vs Numbers

In the previous lesson, you monitored CPU usage —a number that goes up and down. You asked: "Is CPU above 80%?" This works great for metrics. But what about application logs? Your services generate thousands of log entries every minute: "User logged in successfully," "Payment processed," "Database connection failed," "API request completed." You don't want to measure these—you want to filter them. You need alerts that say "notify me when ERROR logs appear" rather than "notify me when a number is too high." This lesson shows you how Grafana handles event-based alerts using multi-condition filters.

Alert Philosophy: Symptoms, Not Causes

Before building any alert, understand this rule: alert on symptoms, not root causes . A symptom is what users experience. Users can't log in. That's a symptom. The root cause might be database connection pool exhaustion, but that's a technical detail one layer removed from user impact. Why does this matter? If you alert on every technical anomaly—CPU spikes, memory increases, cache misses—you'll get hundreds of alerts daily. Engineers start ignoring them. Critical alerts get lost in noise. But if you alert when users can't log in, that's always worth investigating. Grafana excels at symptom-based monitoring because you can combine conditions. Instead of alerting on a single database error (which might be a transient blip), you can alert when ERROR logs appear in your auth service AND those errors persist for more than 2 minutes AND the error rate exceeds 5% of total requests. This multi-condition approach filters out noise.

Your Log Data Structure

Your PostgreSQL database has a log_events table that captures logs from your microservices: SQL CREATE TABLE public.log_events (ts timestamptz NOT NULL, level text NOT NULL, -- 'INFO', 'WARN', 'ERROR' service text NOT NULL, -- 'auth', 'payments', 'search', 'api' message text NOT NULL); If you query recent data, you might see: ts level service message 2024-01-15 14:23:00+00 INFO auth User login successful 2024-01-15 14:23:30+00 ERROR search Database connection timeout 2024-01-15 14:24:15+00 ERROR auth Failed to verify JWT token Notice how INFO logs vastly outnumber ERROR logs—maybe a 1000:1 ratio in production. Most operations succeed, so you need filtering to focus only on the errors that matter.

ts	level	service	message
2024-01-15 14:23:00+00	INFO	auth	User login successful
2024-01-15 14:23:30+00	ERROR	search	Database connection timeout
2024-01-15 14:24:15+00	ERROR	auth	Failed to verify JWT token

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Building the Filter Query

Your alert query needs to return a numeric value that Grafana can evaluate. While you can view individual log rows in Explore or table panels, alert rules need aggregated data they can compare against thresholds.

For log-based alerts, the most common approach is to count matching rows. Start by counting all ERROR logs:

SQL

SELECT
  COUNT(*) as value
FROM public.log_events
WHERE level = 'ERROR'

SELECT
  COUNT(*) as value
FROM public.log_events
WHERE level = 'ERROR'

This counts how many ERROR logs exist, but it searches all historical data. You need to focus on recent logs using Grafana's time filter macro:

SQL

WHERE level = 'ERROR'
  AND $__timeFilter(ts)

WHERE level = 'ERROR'
  AND $__timeFilter(ts)

The $__timeFilter(ts) macro limits results to your alert's evaluation window, like "last 5 minutes." Both conditions must be true—that's what the AND operator means. You're building a multi-condition filter.

Here's your complete basic query:

SQL

SELECT
  COUNT(*) as value
FROM public.log_events
WHERE level = 'ERROR'
  AND $__timeFilter(ts)

SELECT
  COUNT(*) as value
FROM public.log_events
WHERE level = 'ERROR'
  AND $__timeFilter(ts)

When this runs, you might get a result like value = 3, meaning three ERROR logs occurred in your evaluation window. Grafana can evaluate this number against your alert threshold. If you set a threshold of 0 (fire when any errors exist), the alert triggers when the count is greater than 0. If you set a threshold of 5 (fire only on persistent errors), the alert triggers when the count exceeds 5.

Want to focus on specific services? Add more conditions. For payment service errors only:

SQL

SELECT
  COUNT(*) as value
FROM public.log_events
WHERE level = 'ERROR'
  AND service = 'payments'
  AND $__timeFilter(ts)

SELECT
  COUNT(*) as value
FROM public.log_events
WHERE level = 'ERROR'
  AND service = 'payments'
  AND $__timeFilter(ts)

Or monitor multiple critical services:

SQL

SELECT
  COUNT(*) as value
FROM public.log_events
WHERE level = 'ERROR'
  AND service IN ('payments', 'auth')
  AND $__timeFilter(ts)

SELECT
  COUNT(*) as value
FROM public.log_events
WHERE level = 'ERROR'
  AND service IN ('payments', 'auth')
  AND $__timeFilter(ts)

Each additional condition refines your focus, helping ensure the alert only fires for genuinely important scenarios. The query returns a single number—the count of matching errors—which the alert engine uses to determine whether your threshold is exceeded.

Alert States and State Transitions

Before understanding how Grafana evaluates your queries, you need to understand the three states an alert can be in: Normal, Pending, and Alerting.

Normal means everything is okay. Your query returns a value that doesn't exceed the threshold. For example, if your query counts ERROR logs and returns 0, and your threshold is "greater than 0," the alert stays in Normal state. No notifications are sent. This is where your alert should spend most of its time.

Pending means the threshold has been exceeded, but not long enough to confirm a real problem. If your query suddenly returns 5 errors and your threshold is 0, the alert moves to Pending. But it won't send notifications yet—it's waiting to see if this is a brief spike or a sustained issue. You configure the pending duration in your alert rule, typically 1-5 minutes. If the condition remains true for the entire pending period, the alert transitions to Alerting. If the condition resolves before the pending period expires, the alert returns to Normal without ever sending a notification.

Alerting means the threshold has been exceeded for longer than the pending period. Notifications are sent to your configured contact points. The alert stays in Alerting state as long as the condition remains true. When your query returns a value below the threshold again (like when errors stop), the alert immediately returns to Normal state and can optionally send a "resolved" notification.

Here's how state transitions work with a concrete example:

Normal → Pending: At 14:23:00, your query starts returning count = 3 (errors exist). Your threshold is 0 and your pending period is 2 minutes. The alert enters Pending state.
Pending → Alerting: At 14:25:00 (2 minutes later), if errors still exist (count still > 0), the alert transitions to Alerting. Notifications are sent.
Alerting → Normal: At 14:27:00, your query returns count = 0 (errors stopped). The alert immediately returns to Normal. If you've configured "send resolved notifications," Grafana sends an "all clear" message.

The pending period prevents alert noise. Without it, a single transient error (lasting 5 seconds) would trigger a notification. With a 2-minute pending period, only errors that persist meaningfully long enough to indicate a real problem cause notifications.

Grafana evaluates your alert at regular intervals—the evaluation interval you configure when creating the rule, typically 1 minute. At each evaluation, Grafana runs your query and checks if the threshold is exceeded. The pending period is measured in evaluation cycles, not wall-clock time. If your evaluation interval is 1 minute and your pending period is 2 minutes, the alert must exceed the threshold for 2 consecutive evaluations before firing.

Understanding these states helps you design better alerts. If you see alerts rapidly transitioning between states (Normal → Pending → Normal → Pending), your conditions might be too sensitive or your threshold too low. If alerts stay in Pending but never reach Alerting, increase your evaluation interval or decrease your pending period. The goal is steady state transitions: Normal most of the time, brief Pending periods for investigation, and Alerting only when action is required.

Introduction: Events vs Numbers

Alert Philosophy: Symptoms, Not Causes

Your Log Data Structure

Join the 1M+ learners on CodeSignal

Building the Filter Query

Alert States and State Transitions

How Grafana Evaluates Filtered Queries

Testing Before Deployment

Routing Alerts with Labels

Summary