Writing alert rules that do not cry wolf
How threshold-plus-duration rules, incident grouping, maintenance windows and quiet hours combine to keep alerting useful instead of noisy.
Last updated 2026-08-26
Most alert fatigue comes from one design mistake: firing on an instantaneous threshold. CPU touches 95% for a single sample during a nightly backup and someone's phone goes off. Rules in Servers Sentinel take a duration alongside the threshold, and that one addition removes the majority of false alarms.
The shape of a rule
A rule names a metric, a comparator, a threshold and a period the condition has to hold for before it fires:
| Field | Example | Notes |
|---|---|---|
| Metric | cpu.usage | Any collected metric, including custom ones pushed through the API |
| Comparator | > | >, >=, <, <=, == |
| Threshold | 90 | Percent, bytes, or the metric's own unit |
| For | 10m | The condition must hold continuously for this long |
| Severity | warning / critical | Drives channel routing and escalation |
| Scope | Global or a server group | Rules can target the whole fleet or one group |
Incident grouping
When a rack loses power, forty hosts stop reporting within the same few seconds. Forty alerts tell you nothing that one alert would not. Related firings are folded into a single incident with the affected hosts listed on it, so the page you get says *a datacenter is down*, not *server-17 is down* forty times.
An incident carries a timeline of everything that happened while it was open, and can be acknowledged and resolved, with a postmortem attached.
Suppression
- Maintenance windows - schedule one before a planned reboot and rules stay silent for the hosts in scope. The metrics are still collected and stored; only notification is suppressed.
- Quiet hours - route non-critical severities away from phones outside working hours while critical ones still get through.
- Flap damping - a condition that crosses back and forth around a threshold does not re-notify on every crossing.
Channels
A firing rule can deliver to e-mail, Telegram, Slack, or an arbitrary webhook. Routing is by severity, so a warning can land in a team channel while a critical also pages. Webhook payloads carry the incident ID, which makes them straightforward to key against in an on-call tool.
The health score
Every host also carries a single recomputed health score. It is the number to put on a wall display: it folds resource pressure, failed services, security events and probe results into one figure, so a fleet of two hundred hosts can be read at a glance rather than scanned.
The score is a summary, not a trigger. Alert on the underlying conditions; use the score to decide which host to look at first.