Monitoring that only wakes you when it should

Monitoring usually gets deployed after an outage nobody knew about for three hours. The result is a system that starts notifying about everything — and within two weeks a mailbox rule moves those notifications into a separate folder.

At that point things are worse than before the rollout, because there is a belief that something is watching, and nothing is.

Two things to keep apart

Collecting data and notifying are not the same thing, even though one tool does both.

Collect plenty: load, disk space, service availability, response times, temperatures, array status. That is cheap, gets in nobody’s way and pays off when diagnosing.

Notify rarely. That is the whole craft.

The typical mistake is treating it as one decision: since we collect, let us notify. Hence a hundred alerts a day.

A test before every alert

Before a notification rule is created, answer three questions:

1. Is somebody supposed to do something when it arrives? If not, this is not an alert — it is a chart.
2. Does it need a reaction now, or can it wait until morning? Separating those two categories matters more than the alert’s wording.
3. How often will this happen in normal operation? If more than once a week, the threshold is set wrong.

An alert that fails those three questions becomes a chart. Charts are looked at while diagnosing; alerts wake people up.

Thresholds that do not shout without reason

Do not alert on an instantaneous value. CPU load jumps to a hundred per cent during every backup. An alert makes sense only on a sustained value — fifteen minutes above the threshold, for instance.

Disk space: alert ahead of time, not after the fact. A notification at 95% arrives when hours are left. Far more useful is an alert on the rate of growth: “at the current rate this disk fills in three days”. Then there is time to react calmly.

Service availability: check what the user does. A running process does not mean the service responds. Checking that a process exists is cheap and misleading; checking the response to a real request is correct.

Backups: alert on the absence of success, not on success. A notification about a successful backup gets read for a week. A notification that a backup did not run, or that its size deviates from previous ones, arrives rarely — and will therefore be read.

Three levels instead of one

A single channel for everything causes most of the misunderstandings. A sensible split:

Wakes you at night. The company cannot work: the file server, mail, the link or the production system is down. There are a handful of these, not dozens.

To look at in the morning. A disk filling up, a service that restarted overnight, a backup that took three times longer than usual.

To review at the next check. Trends, resource use, things worth noticing but not worth interrupting work for.

Write this down together with the list of what belongs in which group — otherwise in six months everything is back in one bucket.

Something worth saying plainly to a client

Round-the-clock monitoring does not mean round-the-clock support. The system can watch continuously; the person reacting to its notifications works agreed hours, unless the contract says otherwise.

Mixing those two is the most common source of disappointment: the company pays for monitoring and assumes somebody picks up at three in the morning. Settle it explicitly when the contract is signed, not at the first night-time outage.

The measure that tells the truth

Well-tuned monitoring is recognised not by how much data it collects but by the fact that a notification prompts interest rather than irritation.

If the first reaction to an alert is “not again” — the rule is wrong. If the first reaction is “let me check” — it works as it should.