Watch a nightly backup job

You’ll get: an incident the moment a nightly backup fails, with the job’s own output attached, and automatic resolution the next time it succeeds.

Before you start

Create a heartbeat monitor for this job (see Heartbeat monitors), with its expected interval matching your schedule (daily, for a nightly job) and a grace period generous enough to cover a slow run. Copy the monitor’s key, iik_hb_….

1. Wrap the backup command

A heartbeat’s URL takes the job’s exit status as its last path segment — 0 for success, 1–255 for a failure, or the word fail if you’d rather not deal with numbers. Since a shell exit code is already 0–255, you can pass it straight through:

backup-with-heartbeat.sh

#!/usr/bin/env bash
set -uo pipefail
HEARTBEAT_URL="https://infrainbox.example.com/v1/heartbeats/iik_hb_XXXXXXXXXXXX"

output="$(/usr/local/bin/run-backup.sh 2>&1)"
code=$?

curl -fsS -m 10 --data-binary "$output" "$HEARTBEAT_URL/$code" >/dev/null

exit "$code"

A non-zero code opens or continues the monitor’s incident, with $output (up to 16 KiB) attached as the incident’s detail — exactly what you’d want to read at 7 a.m. A 0 is a plain success, same as pinging the bare HEARTBEAT_URL with no code at all.

2. Point your scheduler at the wrapper

0 3 * * * /usr/local/bin/backup-with-heartbeat.sh

If you already use restic, borgmatic or rsync

• restic: wrap it exactly as above — a plain restic backup … already exits non-zero on failure.
• borgmatic: it already knows how to ping a Healthchecks-style URL from its own hooks (check your version’s healthchecks:/ping_url setting) — point that straight at your monitor’s URL. Borgmatic also pings /start and /log by default; InfraInbox accepts both and currently just ignores them, so there’s nothing extra to configure.
• rsync: rsync’s own exit code covers most failures (a 2>&1 | tee capture works the same way as the wrapper above); a full sync that partially fails but still exits 0 won’t be caught this way, so check its output for rsync error: lines if that matters to you.

Check it works

• After a normal night, the monitor reads Up.
• Force a failure (point the job at a bad path, or just exit 1 in a test copy of the script) and confirm an incident opens with the captured output visible on it.
• Fix it and run the wrapper again: the incident resolves on its own.