What an Alert Storm Actually Is

An alert storm is what happens when one real incident produces dozens or hundreds of individual alerts in a short window. You get paged once — then again thirty seconds later, and again, and again, until Slack is a wall of red and your phone looks like it has been compromised. This is not a minor annoyance. It is the single most common reason on-call engineers stop trusting their own alerting system, and untrusted alerts are the same as no alerts at all.

Alert storms almost never come from a single badly configured rule. They come from three systemic problems layered on top of each other:

Flap sensitivity — the monitor reports UP/DOWN/UP/DOWN on each check because the service is genuinely unstable or because one probe has network issues.
No cooldown — the same rule fires on every failing check during a sustained outage instead of once per incident.
No consensus — the alert fires on the first failing probe without waiting to see if the other locations agree, so a single regional issue wakes up the whole team.

Fix these three and the storm dies. This post walks through the three fixes in the order you should apply them, with concrete settings and what each one is buying you.

Fix #1: Consecutive Failures Kill Flap Alerts

The first setting every alert rule needs is a consecutive failures threshold. Instead of firing on the first failing check, the rule waits for N back-to-back failures before it considers the monitor actually down. A value of 3 is a good default — it smooths out single-check flaps without delaying real alerts significantly.

How it plays out in practice, with checks at 30-second intervals:

Without consecutive threshold: one transient DNS hiccup → one page. One TLS handshake timeout → another page. Even if the service is fine 10 seconds later, you already got woken up.
With threshold of 3: the same hiccup is absorbed silently. The rule only fires if the failure persists for 90 seconds — which almost always means a real problem.

The tradeoff is a 60–90 second detection delay on real outages. For 99% of services this is a bargain — the alternative is a team that has muted your monitoring system entirely. If you are running something where 60 seconds of delay is genuinely unacceptable (financial trading, life safety), use 2 instead of 3, but do not use 1. One-check alerts are a guarantee of storm behavior.

Fix #2: Cooldowns Stop Repeat Pages During Sustained Outages

Once an alert rule has fired, it should not fire again for a defined window, regardless of how many more checks fail. That window is the cooldown. A 15-minute default is a good starting point.

The math is brutal without a cooldown. Suppose your API goes down at 14:00 and stays down for two hours. With 30-second checks and no cooldown, a rule with "any failing check triggers" produces 240 separate notifications. Multiply that by three alert contacts and you have 720 messages to deal with in the middle of an actual outage — exactly when your attention should be on fixing the problem.

With a 15-minute cooldown, the same two-hour outage produces eight alerts. Still useful — you know it is ongoing — without drowning anyone.

Some teams go further and set the cooldown to equal the incident's expected duration (30 minutes for a small team, an hour for a larger one), on the theory that if the issue is not fixed by then, a fresh alert serves as a nudge. This works well in practice. What does not work is no cooldown — any nonzero value is vastly better than zero.

A well-designed alerting system separates "incident created" from "monitor still failing." The first should page. The second should appear on a dashboard, not in Slack. If your tool cannot tell the difference, that is a tool problem, not a tuning problem.

Fix #3: Multi-Location Consensus Kills Regional Noise

Every serious monitoring tool runs checks from multiple probe locations in parallel. What matters is how the alerting logic handles disagreement between those locations. There are three useful strategies:

Any — alert fires if even one probe reports the monitor down. Fastest detection, most noise. Use this only for rare, high-value monitors where false positives are acceptable.
All — alert fires only if every probe agrees. Quietest strategy, but real regional failures (say, a bad peering arrangement) can cause a true outage for half your users with no alert at all. Use this cautiously.
Majority — alert fires if more than half the probes report failure. This is the default you should use for almost everything. It eliminates single-region false positives while still catching real outages within a few seconds.

A concrete example: your monitor runs from 4 locations — US East, EU Central, Singapore, Cape Town. A transient BGP issue affects only Cape Town for three minutes. With "any," you get an alert. With "majority," the check fails in Cape Town but the other three report success, so no alert fires — correctly, because the service is actually fine for 75% of your users.

If your alerting tool does not expose a consensus strategy, check what it does by default. Most cheap tools use "any" because it is the simplest to implement, and most of your alerting noise comes from that choice.

Layer All Three: A Real-World Rule Configuration

Here is the setup we recommend for a standard production API monitor, combining all three fixes:

Check interval: 30 seconds
Probe locations: 4 (one per continent where you have users)
Consensus strategy: majority
Consecutive failures: 3
Cooldown: 15 minutes

What this buys you: real outages are detected within 90 seconds of failing in 3+ of 4 locations. You get one alert when the incident opens, and then — for that 15-minute window — no further pages, even if every subsequent check fails. The recovery notification fires automatically when checks return to normal.

For non-production environments, you can relax the consecutive failures to 5 or 6 and raise the cooldown to 30 minutes. For genuinely critical monitors (payment processing, login), tighten to consecutive failures of 2 but keep majority consensus — it is the consensus setting, not the consecutive setting, that really controls false positives.

Signs Your Current Alerting Is Storming

If you are not sure whether you have an alert-storm problem, here are the symptoms:

Your team has a muted Slack channel just for monitoring alerts.
Engineers routinely say "ignore that one, it's flaky" about specific monitors.
Your post-incident reviews mention "we didn't see the page because too many came in."
Your on-call rotation has informal rules about which alerts to investigate and which to skip.
New engineers are told "you'll get used to the noise" during onboarding.

All five are variants of the same problem: alerts are not trusted, so they are not acted on. Fixing storms is not about getting more signal, it is about restoring trust in the existing signal.

How GoPinger Ships These Defaults

Every alert rule in GoPinger exposes the three settings above by default:

Consecutive failures with a configurable threshold (default 3)
Cooldown in minutes (default 15)
Location strategy — any, all, or majority (default: majority)

There are six condition types that all share the same settings: monitor down, response time above threshold, SSL expiry, SSL invalid, HTTP status mismatch, and keyword mismatch. You tune them once per rule, mix and match per monitor, and the alerting engine handles the rest. Recovery notifications are sent automatically when checks return to normal.

You can also send test alerts to verify the entire delivery path — email, Slack webhook, custom webhook — without waiting for a real incident.

Good alerting has a lot in common with good spam filtering. The goal is not to catch everything; it is to catch the things that matter with few enough false positives that humans still read the notifications.

See the full feature details on the alert rules feature page, pair it with alert contacts for delivery, or read our guide on fixing alert fatigue in DevOps for the wider picture. Consecutive-failure thresholds, cooldowns, and majority consensus are built into GoPinger's alert rules — see pricing to find the plan that fits your monitoring needs.