Alert on failed systemd units

You’ll get: an incident naming the unit, the moment systemd marks it as failed — no polling, nothing extra to install.

Before you start

An ingest key from a generic InfraInbox source, shaped like iik_src_… (see Events API: curl and scripts).

1. Write the reporting script

#!/usr/bin/env bash
set -euo pipefail
unit="$1"
curl -fsS -m 10 \
  -H "Authorization: Bearer ${INFRAINBOX_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"title\":\"${unit} failed\",\"severity\":\"ERROR\",\"type\":\"systemd.unit_failed\",\"resource\":\"unit:${unit}\",\"dedupKey\":\"systemd:${unit}\"}" \
  "${INFRAINBOX_URL}"

sudo chmod +x /usr/local/bin/infrainbox-unit-failed.sh

dedupKey is the unit’s own name, so repeated failures of the same unit fold into one incident instead of opening a new one each time.

2. Create the reporting unit

# /etc/systemd/system/infrainbox-alert@.service
[Unit]
Description=Report %i's failure to InfraInbox

[Service]
Type=oneshot
Environment=INFRAINBOX_URL=https://infrainbox.example.com/v1/events
Environment=INFRAINBOX_KEY=iik_src_XXXXXXXXXXXX
ExecStart=/usr/local/bin/infrainbox-unit-failed.sh %i

3. Point a unit at it

For any unit you want watched, add a drop-in (sudo systemctl edit <unit>) or edit its file directly:

[Unit]
OnFailure=infrainbox-alert@%n.service

Reload and test:

sudo systemctl daemon-reload
sudo systemd-run --unit=infrainbox-test-fail /bin/false

Check it works

The incident appears in InfraInbox naming infrainbox-test-fail.service.
systemctl status infrainbox-alert@infrainbox-test-fail.service shows the reporting run itself exited cleanly.