Heartbeats and ping checks
Every other source tells InfraInbox when something is wrong. A check tells InfraInbox when something has gone quiet — a host that stopped pinging, a job that stopped running. It’s the one signal a dead host can still send, because it works by absence: you ping it on a schedule, and InfraInbox alerts when a ping doesn’t show up.
Heartbeats vs. ping checks
Section titled “Heartbeats vs. ping checks”| Purpose | Proves | Default interval | Default grace | Default severity |
|---|---|---|---|---|
liveness |
A host or site is up | 60 seconds | 180 seconds | CRITICAL |
job |
A job ran and succeeded | 1 day | 1 hour | ERROR |
Only a heartbeat can move its source between Online and Offline; a missed or failed ping check degrades the source without marking it offline — a backup that didn’t run isn’t the same as the host being down.
Create one
Section titled “Create one”On a source’s page, open the Checks tab. It’s one coverage list — a Proxmox source’s Nodes and Backup jobs groups above the Ping checks group of everything else — with a one-line summary at the top counting each state apart (up, waiting for its first ping, paused, late, down): “All N checks up” only when every one of them is. + Add check offers Ping check (for a job or script) and Host heartbeat (for a host that pings, not offered on a source with a watcher, which pings its own). Either opens a panel on the right, with Cancel and Add at the foot:
- Name — whatever you’ll recognize it by.
- Expect a ping — Every … (a number of minutes, hours or days) or On a schedule (every day, or every week on a given day, at a time). A schedule shows the crontab line to run the job by, for example
cron: 0 2 * * 0; Every … needs none. InfraInbox itself keeps only how often: it counts from the last ping, not from the clock. - Then wait — extra time before a late ping counts as down.
Save it and InfraInbox issues a key shaped iik_hb_…, with the ping URL and the snippets to paste. It’s separate from the source’s own iik_src_… key — a heartbeat key only works on heartbeat routes.
Each row says what it is without opening it (Ping check · Job · Daily · +1 h grace · last ping …) and opens the same panel. There you can change it with the save bar (a save or Discard closes the panel), Show ping URL again with a copy button, and, under More, Pause, make a New ping URL (the old one stops at once) or Delete it.
A check marked created by the watcher holds a key only the watcher uses, so its panel has no ping URL and no New ping URL: the watcher pings it by itself. Its watcher’s row, in the Checks tab’s Watchers section, shows its state. Name a check yourself the same as one a watcher would create and the watcher’s own enrolment is refused rather than handed your check’s key: rename one of them.
Proxmox backup jobs, watched by their reports
Section titled “Proxmox backup jobs, watched by their reports”Proxmox reports a backup that fails, never one that doesn’t run at all: a disabled job, a node that was down at backup time, a job switched back to email. On a Proxmox source, the Checks tab lists each scheduled backup job that has reported (Backup jobs), an unwatched one with its last report and a Watch this job button; Watch this job turns it into a check with no ping to set up: the job’s own backup reports count as its runs. A node or a job that’s already watched opens the same panel a ping check does — pause it, get its ping URL again, or delete it — right from its row.
- It runs daily alerts after 27 hours without a report (a day, plus 3 hours of grace).
- It runs weekly alerts after 7 days and 3 hours.
A failed run still counts as a run here: the failure raises its own incident. Only scheduled runs appear; a manual Backup now doesn’t.
A check’s state is new until its first ping, then up, late (past its interval but still inside grace — this alerts nobody), down (past the full deadline, or a failure arrived), or paused.
Sending a ping
Section titled “Sending a ping”There are two forms. Pick whichever suits the client.
Key in the URL (works from a plain crontab with nothing else configured):
curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…Key in a header (keeps the key out of shell history and process listings):
curl -fsS -m 10 --retry 3 -X POST -H "Authorization: Bearer iik_hb_…" https://infrainbox.example.com/v1/heartbeatsA plain ping (GET, HEAD or POST, either form) is a success: the check’s deadline resets to interval + grace, and if it was down, that incident resolves.
Signal endpoints
Section titled “Signal endpoints”Append a signal to report more than “alive”: /v1/heartbeats/{token}/{signal} (URL-key form) or /v1/heartbeats/{signal} (header form).
| Signal | Meaning |
|---|---|
0 |
Success — same as a plain ping. |
fail, or an exit status 1–255 |
The check goes DOWN immediately (or the incident takes one more occurrence if already down). |
start |
Accepted and recorded, but doesn’t change state — some tools (borgmatic) send this by default before a run. |
log |
Same: accepted, no state change. |
Anything else — an exit status above 255 included — is rejected. Send the job’s log as the request body (--data-binary @job.log, up to 16 KiB) and, on a failure, it becomes the incident’s detail.
A heartbeat ping every minute:
* * * * * curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…On a host where you’d rather not touch a user’s crontab, install it as a system cron file instead:
echo '* * * * * root curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…' \ > /etc/cron.d/infrainbox-heartbeatA job that reports its own exit status:
your-job > /tmp/job.log 2>&1curl -fsS -m 10 --retry 3 --data-binary @/tmp/job.log \ "https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?"A restic backup, reporting both ends of the run:
curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…/startrestic backup /data > /tmp/restic.log 2>&1curl -fsS -m 10 --retry 3 --data-binary @/tmp/restic.log \ "https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?"borgmatic has a built-in Healthchecks-compatible hook, so it needs no wrapper at all:
# borgmatic config.yamlhealthchecks: ping_url: https://infrainbox.example.com/v1/heartbeats/iik_hb_…systemd timer
Section titled “systemd timer”For a job already run by a .timer unit, wrap the ping in ExecStartPost/ExecStopPost rather than a separate script:
[Service]Type=oneshotExecStart=/usr/local/bin/nightly-backup.shExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…If the job itself can fail non-zero, prefer the exit-status form so a failure opens an incident instead of silently not pinging:
ExecStart=/bin/sh -c '/usr/local/bin/nightly-backup.sh; \ curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?'[Timer]OnCalendar=*-*-* 03:00:00Persistent=true
[Install]WantedBy=timers.targetCheck it works
Section titled “Check it works”Wait past a check’s interval plus grace without pinging it (or send /fail once) and confirm it turns down and an incident opens; ping it again and confirm the incident resolves.
- Proxmox: what Proxmox can’t tell you — node and backup-job heartbeats built into the wizard
- Docker and host watcher — creates and pings its own host heartbeat automatically