# Heartbeat monitors

> Catch silence itself — a host, or a job, that should have reported and didn't.

Web page: https://infrainbox.app/docs/sources/heartbeats/

Every other source tells InfraInbox when something is wrong. A heartbeat monitor tells InfraInbox when something has gone quiet — a host that stopped pinging, a job that stopped running. It's the one signal a dead host can still send, because it works by *absence*: you ping it on a schedule, and InfraInbox alerts when a ping doesn't show up.

## Liveness vs. job monitors

| Purpose | Proves | Default interval | Default grace | Default severity |
|---|---|---|---|---|
| `liveness` | A host or site is up | 60 seconds | 180 seconds | CRITICAL |
| `job` | A job ran and succeeded | 1 day | 1 hour | ERROR |

Only a liveness monitor can move its source between **Online** and **Offline**; a missed or failed job monitor degrades the source without marking it offline — a backup that didn't run isn't the same as the host being down.

## Create one

On a source's detail page, open the **Heartbeats** tab → **New monitor**, and set:

- **Name** — whatever you'll recognize it by.
- **Expected every** (minutes) — how often it should ping.
- **Grace before alerting** (minutes) — extra time before a late ping counts as down.

Save it and InfraInbox issues a key shaped `iik_hb_…`, shown once. It's separate from the source's own `iik_src_…` key — a heartbeat key only works on heartbeat routes.

A monitor marked **created by the watcher** holds a key only the [watcher](https://infrainbox.app/docs/sources/watcher.md) uses, so its menu has no **Show ping URL** or **Rotate key**: the watcher pings it by itself.

A monitor's state is `new` until its first ping, then `up`, `late` (past its interval but still inside grace — this alerts nobody), `down` (past the full deadline, or a failure arrived), or `paused`.

## Sending a ping

There are two forms. Pick whichever suits the client.

**Key in the URL** (works from a plain crontab with nothing else configured):

```bash
curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…
```

**Key in a header** (keeps the key out of shell history and process listings):

```bash
curl -fsS -m 10 --retry 3 -X POST -H "Authorization: Bearer iik_hb_…" https://infrainbox.example.com/v1/heartbeats
```

A plain ping (GET, HEAD or POST, either form) is a success: the monitor's deadline resets to interval + grace, and if it was down, that incident resolves.

## Signal endpoints

Append a signal to report more than "alive": `/v1/heartbeats/{token}/{signal}` (URL-key form) or `/v1/heartbeats/{signal}` (header form).

| Signal | Meaning |
|---|---|
| `0` | Success — same as a plain ping. |
| `fail`, or an exit status `1`–`255` | The monitor goes DOWN immediately (or the incident takes one more occurrence if already down). |
| `start` | Accepted and recorded, but doesn't change state — some tools (borgmatic) send this by default before a run. |
| `log` | Same: accepted, no state change. |

Anything else — an exit status above 255 included — is rejected. Send the job's log as the request body (`--data-binary @job.log`, up to 16 KiB) and, on a failure, it becomes the incident's detail.

> **Note:** Unknown or revoked heartbeat keys always answer 404, never 401 — so a request never reveals whether a monitor exists.

> **Tip:** A ping URL you paste into a chat gets fetched by that chat's own link preview. InfraInbox recognizes common preview and crawler requests (by their User-Agent, or a browser's `Sec-Purpose: prefetch`) and answers them without counting the ping — a pasted link can't accidentally keep a monitor alive.

## Cron

A liveness ping every minute:

```cron
* * * * * curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…
```

On a host where you'd rather not touch a user's crontab, install it as a system cron file instead:

```bash
echo '* * * * * root curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…' \
  > /etc/cron.d/infrainbox-heartbeat
```

A job that reports its own exit status:

```bash
your-job > /tmp/job.log 2>&1
curl -fsS -m 10 --retry 3 --data-binary @/tmp/job.log \
  "https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?"
```

A restic backup, reporting both ends of the run:

```bash
curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…/start
restic backup /data > /tmp/restic.log 2>&1
curl -fsS -m 10 --retry 3 --data-binary @/tmp/restic.log \
  "https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?"
```

borgmatic has a built-in Healthchecks-compatible hook, so it needs no wrapper at all:

```yaml
# borgmatic config.yaml
healthchecks:
    ping_url: https://infrainbox.example.com/v1/heartbeats/iik_hb_…
```

## systemd timer

For a job already run by a `.timer` unit, wrap the ping in `ExecStartPost`/`ExecStopPost` rather than a separate script:

```ini
# /etc/systemd/system/nightly-backup.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/nightly-backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…
```

If the job itself can fail non-zero, prefer the exit-status form so a failure opens an incident instead of silently not pinging:

```ini
ExecStart=/bin/sh -c '/usr/local/bin/nightly-backup.sh; \
  curl -fsS -m 10 --retry 3 -o /dev/null https://infrainbox.example.com/v1/heartbeats/iik_hb_…/$?'
```

```ini
# /etc/systemd/system/nightly-backup.timer
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target
```

## Check it works

Wait past a monitor's interval plus grace without pinging it (or send `/fail` once) and confirm it turns **down** and an incident opens; ping it again and confirm the incident resolves.

## Next

- [Proxmox: what Proxmox can't tell you](https://infrainbox.app/docs/sources/proxmox.md#what-proxmox-cant-tell-you) — node and backup-job heartbeats built into the wizard
- [Docker and host watcher](https://infrainbox.app/docs/sources/watcher.md) — creates and pings its own host heartbeat automatically