Spike Detection and Alert Annotations

Introduction

Your first CPU alert used the Last reduce function—it checked the most recent measurement at evaluation time. If CPU was at 85% when the alert checked, the condition triggered. If it had dropped to 75% by the next check, the condition cleared. This works well for relatively stable metrics that change gradually. But here's the problem. Consider your edge server's latency over a five-minute period: 45ms \to 48ms \to 380ms \to 52ms \to 47ms If your alert checks at the end of that window, it sees 47ms and concludes everything is fine. The 380ms spike that frustrated users never registers because it happened between evaluations. Your alert remained Normal while customers experienced slow page loads. This lesson teaches you to choose the right reduce function for different problem patterns. You'll learn when Max catches critical spikes that Last misses, how Mean identifies trending problems while filtering noise, when Min helps detect unexpected drops, and how alert annotations transform generic notifications into actionable messages with context, runbook links, and precise measurements.

Understanding Reduce Functions Through Real Scenarios

Every Grafana alert reduces time-series data into a single value for threshold comparison. The reduce function you choose determines which aspect of your data gets evaluated. Think of it as asking different questions about the same measurements. Last - "What's happening right now?" When you use Last, you're checking current state. Is the disk 90% full at this exact moment? The catch: It misses historical context within your evaluation window. A brief CPU spike to 98% that resolved before the next check goes undetected because Last only sees the current 45%. Max - "What's the worst it got during this period?" For latency monitoring, this is often what you actually care about. Even a single request taking 2 seconds represents a terrible user experience for whoever made that request. Using Max ensures your alert catches every spike, no matter how brief. If any measurement in your evaluation window exceeded 300ms, Max surfaces that value for threshold comparison. Mean - "What's the typical level?" This smooths out single-point anomalies while catching sustained elevation. Consider latency measurements of 180ms, 185ms, 190ms, 195ms, 188ms —nothing individually crosses your 200ms threshold, but the mean of 188ms indicates trending upward toward a problem. Mean helps you catch gradual degradation before it becomes critical. Min - "What's the best it achieved?" Less commonly used, but valuable for throughput metrics where you expect a minimum level of activity. If your payment processing system normally handles 500-1000 transactions per minute, but Min shows only 50 transactions even at peak times, you've discovered a capacity problem. The system isn't entirely down, but it's severely constrained.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Building Your Annotation Set

Scroll down to the Add annotations section. You'll create five annotations that transform your generic alert into a self-contained incident report.

Start by clicking Add annotation. Set the key to summary and the value to Edge server latency spike detected. This provides a human-readable one-line description that appears prominently in notifications. Recipients immediately understand what failed without needing to decode technical jargon.

Add a second annotation with key description and value Latency exceeded 300ms threshold. Check network conditions and server load. This gives the on-call engineer immediate context about what the alert means and suggests initial troubleshooting directions.

Create a third annotation with key runbook_url and value https://wiki.company.com/runbooks/latency-troubleshooting. Replace this example URL with your actual internal documentation link. Engineers can click directly to detailed investigation procedures rather than searching wikis. This single link saves critical minutes during incident response.

Now you'll use annotation templates to include dynamic data. Add an annotation with key affected_host and value {{ $labels.metric }}. The double curly braces indicate a template variable that Grafana replaces with actual data when the alert fires. The $labels.metric variable contains the value from your query's metric column—the host name. If edge-eu-west triggered the alert, the notification shows affected_host: edge-eu-west. One alert rule monitors all hosts, but each notification identifies exactly which host has the problem.

Finally, add an annotation with key max_latency_ms and value {{ $values.B }}. The $values.B template variable contains the actual value from expression B—your Max reduce function's output. If latency spiked to 380ms, the notification shows max_latency_ms: 380. This tells engineers the severity at a glance. A 310ms spike might warrant checking monitoring dashboards. A 950ms spike demands immediate action.

Now click Save rule and exit.

Use Case	Reduce Function	Why
Current state monitoring (disk space, memory usage)	`Last`	Metric changes gradually over time
User-impacting spikes (latency, errors)	`Max`	Even one bad experience matters
Trending problems (CPU usage, request rates)	`Mean`	Identify patterns while filtering noise
Minimum throughput (transaction rates, messages processed)	`Min`	Detect capacity constraints

Introduction

Understanding Reduce Functions Through Real Scenarios

Join the 1M+ learners on CodeSignal

Creating a Spike Detection Alert with Max

What This Configuration Means

Enriching Alerts with Annotations

Annotations: Context That Travels With Alerts

Building Your Annotation Set

How Annotations Transform Notifications

Choosing the Right Reduce Function

Decision Guide: Which Reduce Function?

Summary