A dead man's switch for cron jobs

What a dead man's switch is

The mechanism is inverted: instead of your job reporting failure, it reports success, and an external service alerts when the success reports stop. You create a heartbeat monitor, which gives you a private ingest URL like https://undownable.com/ping/{id}. Your job calls it at the end of every successful run. If a ping does not arrive inside the expected window, the monitor goes down and your alert channels fire.

The name comes from the rail industry: a lever the driver must keep held down, so that an incapacitated driver stops the train. The property that makes it useful is the same here. Doing nothing is the failure signal, so no specific failure has to be anticipated in advance.

Why this beats checking logs

Every log-based approach shares a flaw: it can only detect failures that produced output. Consider what actually goes wrong with scheduled work.

• The crontab was wiped by a config-management run, an image rebuild, or a migration to a new host. There is no job, so there is no log.
• The cron daemon is not running, or was never enabled in a container.
• The job runs but hangs forever on a network mount or a lock, producing nothing.
• The machine is off. This one is obvious in hindsight and invisible in practice.
• The job failed and did email you, but the mail bounced because the box has no working MTA, which is the normal state of a modern server.

A dead man's switch catches all five, because all five look identical from the outside: the ping did not arrive. And unlike log parsing, the detection logic lives somewhere the failure cannot reach.

The cron recipe

Chain the ping onto the job with a double ampersand, so it runs only when the job exits zero. A failed run sends nothing, and the silence becomes the alert:

# Nightly backup at 02:30. Ping only on a clean exit.
30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1 && /usr/bin/curl -fsS -m 10 --retry 3 https://undownable.com/ping/YOUR-MONITOR-ID > /dev/null

The curl flags are doing real work. -f makes curl exit non-zero on an HTTP error instead of cheerfully saving an error page, -s hides the progress meter, -S keeps genuine error messages visible despite -s, -m 10 caps the whole operation so a hung ping cannot wedge the job, and --retry 3 rides out a transient blip.

Two cron traps to avoid. Cron runs with a minimal PATH, usually just /usr/bin and /bin, so use the absolute path to curl rather than trusting your interactive environment. And percent signs are special in a crontab: cron converts the first unescaped % into a newline and feeds everything after it to the command as standard input. A date format has to be escaped:

# Wrong: cron truncates the command at the first %
0 3 * * * /usr/local/bin/report.sh --for $(date +%Y-%m-%d)

# Right
0 3 * * * /usr/local/bin/report.sh --for $(date +\%Y-\%m-\%d)

Inside a script

For anything with more than one step, put the ping at the end of the script under set -e, so any failing step aborts before the ping is reached:

#!/usr/bin/env bash
set -euo pipefail

PING_URL="https://undownable.com/ping/YOUR-MONITOR-ID"

restic backup /srv /home --tag nightly
restic forget --keep-daily 7 --keep-weekly 4 --prune
restic check --read-data-subset=5%

# Reached only if every command above succeeded.
curl -fsS -m 10 --retry 3 "$PING_URL" > /dev/null

This is stronger than pinging from cron, because it asserts on the whole pipeline rather than the exit code of a wrapper. A backup that runs, writes nothing, and exits zero is still a failed backup; put the verification step before the ping and the switch covers it.

systemd timers

With a timer-driven oneshot unit, ExecStartPost is the natural place for the ping: systemd runs it only after ExecStart has completed successfully, and skips it entirely if the unit fails.

# /etc/systemd/system/nightly-backup.service
[Unit]
Description=Nightly restic backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 https://undownable.com/ping/YOUR-MONITOR-ID

# /etc/systemd/system/nightly-backup.timer
[Unit]
Description=Run the nightly backup at 02:30

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

Enable it with systemctl enable --now nightly-backup.timer. Persistent=true makes systemd run a missed occurrence once the machine comes back, which is usually what you want for backups.

The silence handles the missing case. For an immediate, loud signal on an actual failure, add an OnFailure drop-in. %n expands to the failing unit's full name and arrives in the templated unit as %i:

# /etc/systemd/system/nightly-backup.service.d/onfailure.conf
[Unit]
OnFailure=notify-failure@%n.service

# /etc/systemd/system/notify-failure@.service
[Unit]
Description=Report a failed unit (%i)

[Service]
Type=oneshot
ExecStart=/usr/bin/curl -fsS -m 10 -H "Title: %i failed" -H "Priority: high" -H "Tags: rotating_light" -d "Unit %i entered the failed state on %H" https://ntfy.sh/your-topic

Drop that one template on a box and every unit can reference it. The two mechanisms are complementary: OnFailure tells you a run broke, the heartbeat tells you a run never happened.

GitHub Actions and other CI

A final step with no if condition inherits the implicit success gate, so it runs only when everything before it passed:

      - name: Signal the heartbeat
        run: curl -fsS -m 10 --retry 3 "$HEARTBEAT_URL"
        env:
          HEARTBEAT_URL: ${{ secrets.HEARTBEAT_URL }}

One trap worth knowing: the moment you write any custom if condition, the implicit success gate disappears and your condition becomes the entire test. A step guarded only by a branch check will happily run after upstream failures, so keep the gate explicit:

        if: success() && github.ref == 'refs/heads/main'

The same shape works for scheduled workflows, which is worth doing: GitHub automatically disables scheduled workflows in a public repository after 60 days with no repository activity, and a dead man's switch is the only thing that will tell you it happened.

Intervals and grace periods

A heartbeat monitor has two timing settings, and the alert window is their sum. The expected ping interval is how often you promise to check in; the grace period is extra slack on top, defaulting to sixty seconds. A job pinging every five minutes with sixty seconds of grace is overdue at six minutes, and the evaluation that notices runs on the same interval cadence, so in practice the alert lands somewhere in the following few minutes rather than on the exact second.

For jobs that run less often than the longest interval option, carry the rest in the grace period. A nightly backup pairs a fifteen-minute interval with a grace period of 86400 seconds, giving a window of roughly twenty-four hours and a quarter, which tolerates the job starting late without tolerating it never starting.

One more setting worth changing: heartbeats are already confirmed by the grace period, so the usual "confirm across two failed checks" default just doubles your time to alert. Set failures before down to 1 on heartbeat monitors.

What happens when the pings stop

The monitor transitions to down and opens an incident, which notifies every channel you have attached: email, ntfy (including self-hosted servers with access tokens), Telegram, Slack, Discord, or a webhook of your own. Delivery is exactly-once, and flap suppression keeps a job that recovers and re-fails from becoming a notification storm. A recovery notice goes out on the next successful ping. If the job is expected to be offline, a maintenance window suppresses alerts without denting the uptime record.