# Alert on failed systemd units

> An OnFailure= template unit that posts to InfraInbox the moment systemd marks any unit as failed.

Web page: https://infrainbox.app/docs/cookbook/systemd-failures/

**You'll get:** an incident naming the unit, the moment `systemd` marks it as failed — no polling, nothing extra to install.

## Before you start

An ingest key from a generic InfraInbox source, shaped like `iik_src_…` (see [Events API: curl and scripts](https://infrainbox.app/docs/sources/events-api.md)).

## 1. Write the reporting script

```bash
#!/usr/bin/env bash
# /usr/local/bin/infrainbox-unit-failed.sh
set -euo pipefail
unit="$1"
curl -fsS -m 10 \
  -H "Authorization: Bearer ${INFRAINBOX_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"title\":\"${unit} failed\",\"severity\":\"ERROR\",\"type\":\"systemd.unit_failed\",\"resource\":\"unit:${unit}\",\"dedupKey\":\"systemd:${unit}\"}" \
  "${INFRAINBOX_URL}"
```

```bash
sudo chmod +x /usr/local/bin/infrainbox-unit-failed.sh
```

`dedupKey` is the unit's own name, so repeated failures of the same unit fold into one incident instead of opening a new one each time.

## 2. Create the reporting unit

```ini
# /etc/systemd/system/infrainbox-alert@.service
[Unit]
Description=Report %i's failure to InfraInbox

[Service]
Type=oneshot
Environment=INFRAINBOX_URL=https://infrainbox.example.com/v1/events
Environment=INFRAINBOX_KEY=iik_src_XXXXXXXXXXXX
ExecStart=/usr/local/bin/infrainbox-unit-failed.sh %i
```

## 3. Point a unit at it

For any unit you want watched, add a drop-in (`sudo systemctl edit <unit>`) or edit its file directly:

```ini
[Unit]
OnFailure=infrainbox-alert@%n.service
```

Reload and test:

```bash
sudo systemctl daemon-reload
sudo systemd-run --unit=infrainbox-test-fail /bin/false
```

## Check it works

- The incident appears in InfraInbox naming `infrainbox-test-fail.service`.
- `systemctl status infrainbox-alert@infrainbox-test-fail.service` shows the reporting run itself exited cleanly.

> **Note:** systemd has no built-in counterpart to `OnFailure=` for "this unit used to fail and now doesn't", so nothing here resolves the incident automatically. Resolve it by hand once you've fixed the unit, or have your fix's own deploy step POST an event with `"action":"resolve"` and the same `dedupKey`.