Heartbeat monitors

Every other source tells InfraInbox when something is wrong. A heartbeat monitor tells InfraInbox when something has gone quiet — a host that stopped pinging, a job that stopped running. It’s the one signal a dead host can still send, because it works by absence: you ping it on a schedule, and InfraInbox alerts when a ping doesn’t show up.

Liveness vs. job monitors

Purpose	Proves	Default interval	Default grace	Default severity
liveness	A host or site is up	60 seconds	180 seconds	CRITICAL
job	A job ran and succeeded	1 day	1 hour	ERROR

Only a liveness monitor can move its source between Online and Offline; a missed or failed job monitor degrades the source without marking it offline — a backup that didn’t run isn’t the same as the host being down.

Create one

On a source’s detail page, open the Heartbeats tab → New monitor, and set:

• Name — whatever you’ll recognize it by.
• Expected every (minutes) — how often it should ping.
• Grace before alerting (minutes) — extra time before a late ping counts as down.

Save it and InfraInbox issues a key shaped iik_hb_…, shown once. It’s separate from the source’s own iik_src_… key — a heartbeat key only works on heartbeat routes.

A monitor’s state is new until its first ping, then up, late (past its interval but still inside grace — this alerts nobody), down (past the full deadline, or a failure arrived), or paused.

Sending a ping

There are two forms. Pick whichever suits the client.

Key in the URL (works from a plain crontab with nothing else configured):

curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…

Key in a header (keeps the key out of shell history and process listings):

curl -fsS -m 10 --retry 3 -X POST -H "Authorization: Bearer iik_hb_…" https://infrainbox.example.com/v1/heartbeats

A plain ping (GET, HEAD or POST, either form) is a success: the monitor’s deadline resets to interval + grace, and if it was down, that incident resolves.

Signal endpoints

Append a signal to report more than “alive”: /v1/heartbeats/{token}/{signal} (URL-key form) or /v1/heartbeats/{signal} (header form).

Signal	Meaning
0	Success — same as a plain ping.
fail, or an exit status 1–255	The monitor goes DOWN immediately (or the incident takes one more occurrence if already down).
start	Accepted and recorded, but doesn’t change state — some tools (borgmatic) send this by default before a run.
log	Same: accepted, no state change.

Anything else — an exit status above 255 included — is rejected. Send the job’s log as the request body (--data-binary @job.log, up to 16 KiB) and, on a failure, it becomes the incident’s detail.

Note

Unknown or revoked heartbeat keys always answer 404, never 401 — so a request never reveals whether a monitor exists.

Tip

A ping URL you paste into a chat gets fetched by that chat’s own link preview. InfraInbox recognizes common preview and crawler requests (by their User-Agent, or a browser’s Sec-Purpose: prefetch) and answers them without counting the ping — a pasted link can’t accidentally keep a monitor alive.

Cron

A liveness ping every minute:

* * * * * curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…

On a host where you’d rather not touch a user’s crontab, install it as a system cron file instead:

echo '* * * * * root curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…' \
  > /etc/cron.d/infrainbox-heartbeat

A job that reports its own exit status:

your-job > /tmp/job.log 2>&1
curl -fsS -m 10 --retry 3 --data-binary @/tmp/job.log \
  "https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?"

A restic backup, reporting both ends of the run:

curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…/start
restic backup /data > /tmp/restic.log 2>&1
curl -fsS -m 10 --retry 3 --data-binary @/tmp/restic.log \
  "https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?"

borgmatic has a built-in Healthchecks-compatible hook, so it needs no wrapper at all:

# borgmatic config.yaml
healthchecks:
    ping_url: https://infrainbox.example.com/v1/heartbeats/iik_hb_…

systemd timer

For a job already run by a .timer unit, wrap the ping in ExecStartPost/ExecStopPost rather than a separate script:

/etc/systemd/system/nightly-backup.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nightly-backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…

If the job itself can fail non-zero, prefer the exit-status form so a failure opens an incident instead of silently not pinging:

ExecStart=/bin/sh -c '/usr/local/bin/nightly-backup.sh; \
  curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?'

/etc/systemd/system/nightly-backup.timer

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

Check it works

Wait past a monitor’s interval plus grace without pinging it (or send /fail once) and confirm it turns down and an incident opens; ping it again and confirm the incident resolves.

Next

• Proxmox: what Proxmox can’t tell you — node and backup-job heartbeats built into the wizard
• Docker and host watcher — creates and pings its own host heartbeat automatically