Skip to content

Heartbeats and ping checks

Every other source tells InfraInbox when something is wrong. A check tells InfraInbox when something has gone quiet — a host that stopped pinging, a job that stopped running. It’s the one signal a dead host can still send, because it works by absence: you ping it on a schedule, and InfraInbox alerts when a ping doesn’t show up.

Purpose Proves Default interval Default grace Default severity
liveness A host or site is up 60 seconds 180 seconds CRITICAL
job A job ran and succeeded 1 day 1 hour ERROR

Only a heartbeat can move its source between Online and Offline; a missed or failed ping check degrades the source without marking it offline — a backup that didn’t run isn’t the same as the host being down.

On a source’s page, open the Checks tab. It’s one coverage list — a Proxmox source’s Nodes and Backup jobs groups above the Ping checks group of everything else — with a one-line summary at the top counting each state apart (up, waiting for its first ping, paused, late, down): “All N checks up” only when every one of them is. + Add check offers Ping check (for a job or script) and Host heartbeat (for a host that pings, not offered on a source with a watcher, which pings its own). Either opens a panel on the right, with Cancel and Add at the foot:

  • Name — whatever you’ll recognize it by.
  • Expect a ping — Every … (a number of minutes, hours or days) or On a schedule (every day, or every week on a given day, at a time). A schedule shows the crontab line to run the job by, for example cron: 0 2 * * 0; Every … needs none. InfraInbox itself keeps only how often: it counts from the last ping, not from the clock.
  • Then wait — extra time before a late ping counts as down.

Save it and InfraInbox issues a key shaped iik_hb_…, with the ping URL and the snippets to paste. It’s separate from the source’s own iik_src_… key — a heartbeat key only works on heartbeat routes.

Each row says what it is without opening it (Ping check · Job · Daily · +1 h grace · last ping …) and opens the same panel. There you can change it with the save bar (a save or Discard closes the panel), Show ping URL again with a copy button, and, under More, Pause, make a New ping URL (the old one stops at once) or Delete it.

A check marked created by the watcher holds a key only the watcher uses, so its panel has no ping URL and no New ping URL: the watcher pings it by itself. Its watcher’s row, in the Checks tab’s Watchers section, shows its state. Name a check yourself the same as one a watcher would create and the watcher’s own enrolment is refused rather than handed your check’s key: rename one of them.

Proxmox backup jobs, watched by their reports

Section titled “Proxmox backup jobs, watched by their reports”

Proxmox reports a backup that fails, never one that doesn’t run at all: a disabled job, a node that was down at backup time, a job switched back to email. On a Proxmox source, the Checks tab lists each scheduled backup job that has reported (Backup jobs), an unwatched one with its last report and a Watch this job button; Watch this job turns it into a check with no ping to set up: the job’s own backup reports count as its runs. A node or a job that’s already watched opens the same panel a ping check does — pause it, get its ping URL again, or delete it — right from its row.

  • It runs daily alerts after 27 hours without a report (a day, plus 3 hours of grace).
  • It runs weekly alerts after 7 days and 3 hours.

A failed run still counts as a run here: the failure raises its own incident. Only scheduled runs appear; a manual Backup now doesn’t.

A check’s state is new until its first ping, then up, late (past its interval but still inside grace — this alerts nobody), down (past the full deadline, or a failure arrived), or paused.

There are two forms. Pick whichever suits the client.

Key in the URL (works from a plain crontab with nothing else configured):

Terminal window
curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…

Key in a header (keeps the key out of shell history and process listings):

Terminal window
curl -fsS -m 10 --retry 3 -X POST -H "Authorization: Bearer iik_hb_…" https://infrainbox.example.com/v1/heartbeats

A plain ping (GET, HEAD or POST, either form) is a success: the check’s deadline resets to interval + grace, and if it was down, that incident resolves.

Append a signal to report more than “alive”: /v1/heartbeats/{token}/{signal} (URL-key form) or /v1/heartbeats/{signal} (header form).

Signal Meaning
0 Success — same as a plain ping.
fail, or an exit status 1–255 The check goes DOWN immediately (or the incident takes one more occurrence if already down).
start Accepted and recorded, but doesn’t change state — some tools (borgmatic) send this by default before a run.
log Same: accepted, no state change.

Anything else — an exit status above 255 included — is rejected. Send the job’s log as the request body (--data-binary @job.log, up to 16 KiB) and, on a failure, it becomes the incident’s detail.

A heartbeat ping every minute:

* * * * * curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…

On a host where you’d rather not touch a user’s crontab, install it as a system cron file instead:

Terminal window
echo '* * * * * root curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…' \
> /etc/cron.d/infrainbox-heartbeat

A job that reports its own exit status:

Terminal window
your-job > /tmp/job.log 2>&1
curl -fsS -m 10 --retry 3 --data-binary @/tmp/job.log \
"https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?"

A restic backup, reporting both ends of the run:

Terminal window
curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…/start
restic backup /data > /tmp/restic.log 2>&1
curl -fsS -m 10 --retry 3 --data-binary @/tmp/restic.log \
"https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?"

borgmatic has a built-in Healthchecks-compatible hook, so it needs no wrapper at all:

# borgmatic config.yaml
healthchecks:
ping_url: https://infrainbox.example.com/v1/heartbeats/iik_hb_…

For a job already run by a .timer unit, wrap the ping in ExecStartPost/ExecStopPost rather than a separate script:

/etc/systemd/system/nightly-backup.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/nightly-backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…

If the job itself can fail non-zero, prefer the exit-status form so a failure opens an incident instead of silently not pinging:

ExecStart=/bin/sh -c '/usr/local/bin/nightly-backup.sh; \
curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?'
/etc/systemd/system/nightly-backup.timer
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
[Install]
WantedBy=timers.target

Wait past a check’s interval plus grace without pinging it (or send /fail once) and confirm it turns down and an incident opens; ping it again and confirm the incident resolves.