# ZFS and disk health alerts

> A ZED zedlet and a smartd exec script that post pool and disk health straight to InfraInbox.

Web page: https://infrainbox.app/docs/cookbook/zfs-and-disks/

**You'll get:** one incident per pool when ZFS reports a vdev problem, and one per disk when `smartd` flags a SMART failure — deduplicated so a noisy pool doesn't spam you with a new incident per event.

## Before you start

An ingest key from a generic InfraInbox source, shaped like `iik_src_…` (see [Events API: curl and scripts](https://infrainbox.app/docs/sources/events-api.md)).

## ZFS: a ZED zedlet

ZED (the ZFS Event Daemon) runs every executable script under `/etc/zfs/zed.d/` whose name starts with `all-` for every event it sees. Create one:

```bash
#!/usr/bin/env bash
# /etc/zfs/zed.d/all-infrainbox.sh
set -uo pipefail
URL="https://infrainbox.example.com/v1/events"
KEY="iik_src_XXXXXXXXXXXX"
pool="${ZEVENT_POOL:-unknown}"
class="${ZEVENT_SUBCLASS:-${ZEVENT_CLASS:-}}"
dedup="zfs:${pool}"

post() {
  curl -fsS -m 10 -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d "$1" "$URL"
}

case "$class" in
  statechange)
    if [ "${ZEVENT_VDEV_STATE_STR:-}" = "ONLINE" ]; then
      post "{\"title\":\"$pool is healthy again\",\"action\":\"resolve\",\"dedupKey\":\"$dedup\"}"
    else
      post "{\"title\":\"$pool: ${ZEVENT_VDEV_PATH:-a device} is ${ZEVENT_VDEV_STATE_STR:-degraded}\",\"severity\":\"CRITICAL\",\"type\":\"zfs.statechange\",\"resource\":\"zpool:$pool\",\"dedupKey\":\"$dedup\"}"
    fi
    ;;
  checksum|io|probe_failure)
    post "{\"title\":\"$pool: $class on ${ZEVENT_VDEV_PATH:-a device}\",\"severity\":\"WARNING\",\"type\":\"zfs.$class\",\"resource\":\"zpool:$pool\",\"dedupKey\":\"$dedup\"}"
    ;;
esac
```

```bash
sudo chmod +x /etc/zfs/zed.d/all-infrainbox.sh
sudo systemctl restart zfs-zed   # or `zed`, depending on your distribution
```

Test it without waiting for a real fault, by invoking it with the same environment variables ZED would set:

```bash
sudo ZEVENT_POOL=tank ZEVENT_SUBCLASS=statechange ZEVENT_VDEV_STATE_STR=FAULTED \
  ZEVENT_VDEV_PATH=/dev/sdb1 /etc/zfs/zed.d/all-infrainbox.sh
```

## Disks: a smartd exec script

```bash
#!/usr/bin/env bash
# /usr/local/bin/smart-to-infrainbox.sh
set -uo pipefail
URL="https://infrainbox.example.com/v1/events"
KEY="iik_src_XXXXXXXXXXXX"
dev="${SMARTD_DEVICE:-unknown}"
msg="${SMARTD_MESSAGE:-SMART failure}"

curl -fsS -m 10 \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d "{\"title\":\"${SMARTD_DEVICESTRING:-$dev}: $msg\",\"severity\":\"CRITICAL\",\"type\":\"smart.failure\",\"resource\":\"disk:$dev\",\"dedupKey\":\"smart:$dev\"}" \
  "$URL"
```

```bash
sudo chmod +x /usr/local/bin/smart-to-infrainbox.sh
```

Add it to `/etc/smartd.conf`, either on a specific drive's line or the catch-all:

```
DEVICESCAN -a -M exec /usr/local/bin/smart-to-infrainbox.sh
```

```bash
sudo systemctl restart smartd
```

To test it without waiting for a real failure, add `-M test` temporarily — it runs the exec script once, at smartd's next start, as a test:

```
DEVICESCAN -a -M exec /usr/local/bin/smart-to-infrainbox.sh -M test
sudo systemctl restart smartd
# then remove -M test once you've confirmed the incident arrived
```

> **Warning:** `$SMARTD_MESSAGE` can contain characters that break naive JSON like the examples above if you extend them (a stray `"`). Keep the message short, or build the request body with a proper JSON tool (`jq -n --arg msg "$SMARTD_MESSAGE" '...'`) if you do.

## Check it works

- The forced ZED test above opens a CRITICAL incident for pool `tank`; re-running it with `ZEVENT_VDEV_STATE_STR=ONLINE` resolves it.
- The `smartd -M test` run produces one incident per configured disk; disable `-M test` afterwards so it doesn't re-alert on every smartd restart.

`smartd` itself has no clean "back to healthy" signal the way ZED's `statechange` does, so a disk's incident stays open until you resolve it by hand once you've dealt with the drive.