What a server health score should actually measure

Every monitoring product eventually ships a health score, and most of them are useless for the same reason: they average things that should never have been averaged. A host at 99% disk and 5% CPU is not "52% healthy". It is about to fall over.

A score is a triage tool, not an alarm

The mistake is treating the score as a trigger. If the score is what pages you, it has to be precise, and no single number can be precise about a hundred different failure modes. So we gave it a narrower job: decide which host to look at first. Alerts still fire on specific conditions with specific thresholds. The score only ranks.

That reframing makes the design tractable. A ranking function does not need to be right about severity; it needs to be monotonic - worse should always sort higher.

Worst-of, not average-of

The core rule is that resource pressure contributes by its worst dimension, not its mean. A host that is fine on three axes and critical on the fourth is a host in trouble, and the score has to say so. Averaging is what lets a full disk hide behind an idle CPU.

On top of that base, we subtract for things that are categorically bad rather than continuous:

• failed systemd units or stopped Windows services;
• a reboot that is required and has not happened;
• SMART pre-failure indicators, or a degraded RAID array;
• failing uptime probes and TLS certificates near expiry;
• an active brute-force attempt against SSH or RDP.

Stability matters more than resolution

A score that moves every tick is noise on a wall display, and people stop reading it. Ours is deliberately coarse and hysteretic: it moves in steps, and it takes a sustained change to move it. If you are watching the number rather than the graph, you want it to mean something when it changes.

Read how the alerting side works in writing alert rules that do not cry wolf.