Who monitors the monitor?

The problem is structural, not a configuration mistake

Monitoring works by putting an observer outside the thing being observed. Your monitoring server sits outside your web app, so when the app dies the server notices and tells you. Nothing sits outside the monitoring server. It is the last node in the chain, and the chain has no one to report its own break.

People try to close this with self-checks: a monitor in Uptime Kuma pointed at Uptime Kuma, a health endpoint the server scrapes on itself, a cron job that restarts the container if it stops responding. All of them share one defect. They run on the machine that is failing. When the process is dead, the self-check is dead; when the host is powered off, the self-check is off with it. A watcher inside the failure domain is not a watcher.

Four failures your own monitor cannot report

• The monitoring process died. Container OOM-killed, database corrupted after an unclean shutdown, an upgrade that failed halfway. The dashboard is a connection error, but nothing sends you a connection error.
• The host died. Power supply, disk, kernel panic, a UPS holding a battery that turned out to be dead. Everything on that box went quiet at once, including the thing whose job was to tell you.
• The network went down. Your ISP, your router, your upstream switch. The monitor may be running perfectly and even recording failures locally, but no alert can leave the building.
• The notification path broke. An expired SMTP password, a rotated Telegram bot token, an ntfy topic you renamed, a Slack webhook whose app got uninstalled. Worse than the others, because everything looks green while you are silently unreachable.

The fourth is the one that bites hardest in practice. Hardware fails rarely; credentials rot constantly. Most people discover their alerting has been broken for weeks only when a real outage goes unannounced.

The classic DIY answers

Two designs come up over and over in self-hosting circles. Both are legitimate. Both have running costs that are easy to underestimate.

The first is a second instance on a VPS. Run another Uptime Kuma (or Gatus, or Zabbix) on a cheap cloud box, have each instance watch the other, and let whichever survives raise the alarm. This works and it is fully under your control. It is also a second server to patch, a second set of certificates to renew, a second notification integration to keep credentials fresh in, and a monthly bill. You closed the blind spot by acquiring a machine that now has its own.

The second is a heartbeat service plus a push notification channel. The monitoring box periodically pings an external endpoint; when the pings stop, the endpoint alerts you, often via ntfy to your phone. Much lighter, and the right shape. The usual friction is assembling it from three separate free-tier accounts with nowhere to see the state of all of it, so the pieces drift.

The backstop pattern

A backstop is a small external service whose only job is to watch the watcher. It does not replace your monitoring stack, and it does not need to know about your two hundred containers, your Prometheus rules, or your Grafana dashboards. It needs to know one thing: whether your monitoring is alive and able to speak. Four properties make it work where a self-check does not.

• It lives outside your failure domain: different building, power, ISP, provider. Your outage is not its outage.
• It has an independent notification path. If your alerts go out over your own SMTP server, the backstop must not.
• It is small enough not to need monitoring of its own. If you find yourself patching your backstop, it has grown too large.
• It works in both directions: active checks against whatever is publicly reachable, and heartbeat ingest for whatever is not.

Honest scoping: one outside observer is enough

The most common overcorrection is to conclude that self-hosting was the mistake and migrate everything to a SaaS. That is usually the wrong trade. Your local stack is fast, private, unlimited, and tuned to your environment, and it can see inside your network in ways no external service will.

What it cannot do is see itself. So add one outside observer, not a second monitoring platform. In practice that is two or three monitors: a heartbeat your monitoring box pings on a schedule, an HTTP or keyword check against your public status page or reverse proxy, and perhaps an SSL check on the certificate everything else depends on. Undownable's free plan covers ten monitors at five-minute checks with every alert channel included, several times more than a backstop needs. The whole configuration on your side can be one crontab line:

# Tell an outside service the monitoring box is still alive, every 5 minutes.
*/5 * * * * /usr/bin/curl -fsS -m 10 --retry 3 https://undownable.com/ping/YOUR-MONITOR-ID > /dev/null

When those pings stop, for any reason at all, something outside your house tells you. That is the difference between finding out in ninety seconds and finding out on Thursday when you happen to open the dashboard.

Where to start

• Pick the machine whose silence would be most expensive: usually the monitoring host itself.
• Add a heartbeat monitor and put the curl line in its crontab, with the expected interval a little longer than the cron schedule so a slow minute does not page you.
• Add an active check against anything you already expose publicly, to catch the case where the box is up but the path from the outside world is broken.
• Send the backstop's alerts somewhere different from your normal alerts. Two channels that fail together are one channel.
• Break it on purpose once. An untested backstop is a belief, not a control.