ZFS and disk health alerts

You’ll get: one incident per pool when ZFS reports a vdev problem, and one per disk when smartd flags a SMART failure — deduplicated so a noisy pool doesn’t spam you with a new incident per event.

Before you start

An ingest key from a generic InfraInbox source, shaped like iik_src_… (see Events API: curl and scripts).

ZFS: a ZED zedlet

ZED (the ZFS Event Daemon) runs every executable script under /etc/zfs/zed.d/ whose name starts with all- for every event it sees. Create one:

/etc/zfs/zed.d/all-infrainbox.sh

#!/usr/bin/env bash
set -uo pipefail
URL="https://infrainbox.example.com/v1/events"
KEY="iik_src_XXXXXXXXXXXX"
pool="${ZEVENT_POOL:-unknown}"
class="${ZEVENT_SUBCLASS:-${ZEVENT_CLASS:-}}"
dedup="zfs:${pool}"

post() {
  curl -fsS -m 10 -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d "$1" "$URL"
}

case "$class" in
  statechange)
    if [ "${ZEVENT_VDEV_STATE_STR:-}" = "ONLINE" ]; then
      post "{\"title\":\"$pool is healthy again\",\"action\":\"resolve\",\"dedupKey\":\"$dedup\"}"
    else
      post "{\"title\":\"$pool: ${ZEVENT_VDEV_PATH:-a device} is ${ZEVENT_VDEV_STATE_STR:-degraded}\",\"severity\":\"CRITICAL\",\"type\":\"zfs.statechange\",\"resource\":\"zpool:$pool\",\"dedupKey\":\"$dedup\"}"
    fi
    ;;
  checksum|io|probe_failure)
    post "{\"title\":\"$pool: $class on ${ZEVENT_VDEV_PATH:-a device}\",\"severity\":\"WARNING\",\"type\":\"zfs.$class\",\"resource\":\"zpool:$pool\",\"dedupKey\":\"$dedup\"}"
    ;;
esac

sudo chmod +x /etc/zfs/zed.d/all-infrainbox.sh
sudo systemctl restart zfs-zed   # or `zed`, depending on your distribution

Test it without waiting for a real fault, by invoking it with the same environment variables ZED would set:

sudo ZEVENT_POOL=tank ZEVENT_SUBCLASS=statechange ZEVENT_VDEV_STATE_STR=FAULTED \
  ZEVENT_VDEV_PATH=/dev/sdb1 /etc/zfs/zed.d/all-infrainbox.sh

Disks: a smartd exec script

/usr/local/bin/smart-to-infrainbox.sh

#!/usr/bin/env bash
set -uo pipefail
URL="https://infrainbox.example.com/v1/events"
KEY="iik_src_XXXXXXXXXXXX"
dev="${SMARTD_DEVICE:-unknown}"
msg="${SMARTD_MESSAGE:-SMART failure}"

curl -fsS -m 10 \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d "{\"title\":\"${SMARTD_DEVICESTRING:-$dev}: $msg\",\"severity\":\"CRITICAL\",\"type\":\"smart.failure\",\"resource\":\"disk:$dev\",\"dedupKey\":\"smart:$dev\"}" \
  "$URL"

sudo chmod +x /usr/local/bin/smart-to-infrainbox.sh

Add it to /etc/smartd.conf, either on a specific drive’s line or the catch-all:

DEVICESCAN -a -M exec /usr/local/bin/smart-to-infrainbox.sh

sudo systemctl restart smartd

To test it without waiting for a real failure, add -M test temporarily — it runs the exec script once, at smartd’s next start, as a test:

DEVICESCAN -a -M exec /usr/local/bin/smart-to-infrainbox.sh -M test
sudo systemctl restart smartd
# then remove -M test once you've confirmed the incident arrived

Caution

$SMARTD_MESSAGE can contain characters that break naive JSON like the examples above if you extend them (a stray "). Keep the message short, or build the request body with a proper JSON tool (jq -n --arg msg "$SMARTD_MESSAGE" '...') if you do.

Check it works

• The forced ZED test above opens a CRITICAL incident for pool tank; re-running it with ZEVENT_VDEV_STATE_STR=ONLINE resolves it.
• The smartd -M test run produces one incident per configured disk; disable -M test afterwards so it doesn’t re-alert on every smartd restart.

smartd itself has no clean “back to healthy” signal the way ZED’s statechange does, so a disk’s incident stays open until you resolve it by hand once you’ve dealt with the drive.