Docker and host watcher (preview)

The watcher runs on one host, beside your containers if there are any. It reads Docker read-only, checks the host itself, and reports to InfraInbox over outbound HTTPS. It needs no changes to your applications, and it opens no port and accepts no commands.

What it watches

When	In InfraInbox
A container exits with an error	An incident, such as “immich-server exited with code 137”, with what the exit code means, the restart count and, on an out-of-memory kill, the memory limit. A container you stop on purpose (`docker stop`, a Compose recreate) is not a crash.
A container is killed for running out of memory	An incident: “… was killed: out of memory”.
A container’s `HEALTHCHECK` fails	An incident: “… is unhealthy”.
It starts again and is still running 30 seconds later, or its health check passes	The same incident resolves. A crash loop stays one open incident.
The container is removed	The same incident resolves, unless `--rm` removed it right after it failed.
The host or the watcher stops	The host’s heartbeat monitor goes down: one incident for the whole host, not one per container.
The watcher can’t reach Docker for two minutes	An incident, “web-01: the watcher cannot reach Docker”, which resolves when Docker is back.
A container has no `HEALTHCHECK` (first start)	One informational notice, once: without a health check, a process that is up but not answering is invisible. It closes on its own once the container has one.
A disk is nearly full	An incident, such as “nas: /var/lib/docker is 93% full (7.1 GiB free)”; resolves once space is freed.
A ZFS pool or md RAID array degrades	An incident naming the pool or array; resolves once it’s healthy again.
The host just rebooted	One informational notice, once per boot.
The host’s clock drifts from the server	An incident once the drift is sustained; resolves once it’s back in sync.
Memory pressure is sustained	An incident; resolves once it eases.

Every condition above ends only by a matching recovery event or a manual resolve — never by sitting quiet for 24 hours, even though the watcher reports a condition once when it starts and once when it recovers, with nothing in between.

Events from a Docker Compose project carry compose:<project> and service:<service> tags, so rules and filters can pick out one stack.

The last five rows need no Docker at all. On a Proxmox node, a NAS or anything else with no containers to watch, the watcher runs in host-only mode: just the heartbeat and the host conditions. It switches to that mode by itself when it finds no Docker, or set INFRAINBOX_DOCKER=off.

If InfraInbox is unreachable, events wait in memory (up to 1000) and are retried in order, so a restart never overtakes the crash it resolves. After a reconnect, the watcher re-reads Docker and reports what each container is doing now, so an incident whose recovery happened while it was away still closes.

What it doesn’t do

It only reads. It has no shell, listens on no port, and every write to Docker is refused at the socket proxy in front of it. Nothing is ever run on the host through the watcher. It reports conditions, not metrics: there are no graphs or time series.

Setup

Go to Add source → InfraInbox watcher and name it after the host (nas, web-01, and so on).
InfraInbox issues one ingest key, shown once, and a Compose file with it already filled in.

Docker Compose

The page shows a Compose file with your key and, when you self-host, your server’s address already filled in. It looks like this:

services:
  docker-socket-proxy:
    image: tecnativa/docker-socket-proxy:v0.5.0
    restart: unless-stopped
    environment:
      CONTAINERS: 1   # list and inspect containers
      EVENTS: 1       # the transition stream
      INFO: 1         # the host's name
      POST: 0         # read-only: every write is refused
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost:2375/_ping"]
      interval: 30s

  infrainbox-watcher:
    image: ghcr.io/infrainbox/infrainbox-watcher:0.1
    restart: unless-stopped
    depends_on:
      - docker-socket-proxy
    environment:
      INFRAINBOX_TOKEN: iik_src_…
      INFRAINBOX_URL: https://infrainbox.example.com   # only needed when self-hosted
      DOCKER_HOST: tcp://docker-socket-proxy:2375
      INFRAINBOX_HOST_ROOT: /host
    volumes:
      - /var/lib/docker:/host/var/lib/docker:ro   # read-only: the disk check sees how full it is
    read_only: true
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]

Save it on the host and run:

mkdir -p ~/infrainbox-watcher && cd ~/infrainbox-watcher
nano compose.yml          # paste the file from the page
docker compose up -d

Check it started:

docker compose logs infrainbox-watcher

You should see starting, inventory containers=…, enrolled and sent.

As a systemd service, without Compose

Download the release archive for the host’s architecture (linux amd64 or arm64) and unpack it, then from that folder:

sudo install -m 755 infrainbox-watcher /usr/local/bin/
sudo install -m 644 deploy/systemd/infrainbox-watcher.service /etc/systemd/system/
sudo install -d -m 755 /etc/infrainbox-watcher
sudo install -m 600 deploy/watcher.env.example /etc/infrainbox-watcher/watcher.env

Put this in /etc/infrainbox-watcher/watcher.env (root-only, mode 600):

INFRAINBOX_TOKEN=iik_src_…
INFRAINBOX_URL=https://infrainbox.example.com

Then:

sudo systemctl enable --now infrainbox-watcher

Check it works

Within a minute of starting, a heartbeat monitor named after the host appears on the source’s Heartbeats tab by itself; nobody creates it. To check everything at once without changing anything, run the watcher’s doctor: it tests the configuration, Docker and the proxy’s permissions, the server, the key and the clock, prints a line for each, and exits non-zero if any of them failed.

docker compose run --rm infrainbox-watcher doctor

If the watcher starts reporting 401, its key was rotated, its source was deleted — or the whole workspace was closed, which stops every key in it at once (see Security and privacy). Issue a new key on the source’s page and put it back in watcher.env.

Configuration

Two kinds of setting, kept apart on purpose. Environment variables hold everything about trust and privacy: the key, the server, TLS, how Docker is reached, which host paths are mounted, and crash log lines. They are never command-line flags, so the key never appears in a process list, and a normal install sets only the first one or two. An optional settings file tunes what the watcher detects: see Settings file below.

Variable	Default	Meaning
`INFRAINBOX_TOKEN`	required	The source key (`iik_src_…`).
`INFRAINBOX_URL`	InfraInbox Cloud	Your server’s address. Only a self-hosted server needs it. Must be `https://`, except for a private address.
`DOCKER_HOST`	`unix:///var/run/docker.sock`	Usually `tcp://docker-socket-proxy:2375`.
`INFRAINBOX_HOST`	Docker’s name for the host	How the host is named in InfraInbox; its heartbeat monitor takes this name.
`INFRAINBOX_IGNORE`	none	Comma-separated container names to stay quiet about. Patterns work too: `-backup-`.
`INFRAINBOX_CONFIG`	none	Path to the settings file, for example `/etc/infrainbox-watcher/watcher.json`.
`INFRAINBOX_DOCKER`	on when Docker is found	`off` runs host-only mode.
`INFRAINBOX_DISKS`	none	Extra paths the disk-nearly-full check watches, besides Docker’s own data directory.
`INFRAINBOX_HOST_ROOT`	none	Inside a container, where the host’s filesystems are mounted for the disk check (`/host` in the Compose file).
`INFRAINBOX_CRASH_LOGS`	`0`	How many of a crashed container’s last log lines to attach, up to 100. Off by default, because logs can carry secrets.
`INFRAINBOX_TLS_CA_FILE`	none	A PEM file with your private CA, for a self-hosted server whose certificate isn’t publicly trusted.
`INFRAINBOX_TLS_PIN`	none	`sha256/…` certificate pins, comma-separated.
`INFRAINBOX_HEARTBEAT_EVERY`	`60s`	How often the heartbeat is sent.

Settings file

Every check can be switched off or given its own thresholds, and you choose which containers are watched, in one JSON file. Every field is optional; anything left out keeps its default.

{
  "checks": {
    "disk":   {"warnAt": 85, "errorAt": 95, "resolveBelow": 80, "for": "10m",
               "paths": {"/mnt/backup": {"warnAt": 95, "errorAt": 99, "resolveBelow": 90}}},
    "memory": {"enabled": false},
    "clock":  {"driftSeconds": 120},
    "reboot": {"window": "1h"}
  },
  "containers": {
    "ignore": ["watchtower", "*-backup-*"],
    "healthcheckNotice": false,
    "projects": {"media": {"severity": "warning"}}
  }
}

Checks: disk (space and inodes, with per-path overrides), memory, clock, reboot, zfs and raid. Each has enabled, plus its thresholds and how long a reading has to hold (for). A switched-off check reads nothing and sends nothing.
Containers: ignore and only take name patterns (only empty means all). projects sets ignore, severity or oneshot for every container of one Docker Compose project. healthcheckNotice: false turns off the missing-health-check notice.
Order: a container’s own labels win over its project’s settings, which win over the file’s global ones.

Mount the file read-only and point the watcher at it: in Compose, add ./watcher.json:/etc/infrainbox-watcher/watcher.json:ro to the watcher’s volumes and INFRAINBOX_CONFIG: /etc/infrainbox-watcher/watcher.json to its environment; with systemd, put it at /etc/infrainbox-watcher/watcher.json (mode 644, in the readable directory the install creates — the key stays in the root-only watcher.env) and the same line in watcher.env. A misspelt field or an impossible value (a resolve point above its trigger) stops the watcher at start with the field named, rather than being ignored. doctor prints the settings in effect, and infrainbox-watcher settings prints them in full.

Settings from the dashboard. Watchers check in with InfraInbox about every 5 minutes, with their version and the settings they run with. The same detection settings can also be kept once per source, for every watcher that uses its key, from the Watchers view in the dashboard nav. The host’s own file always wins: a field it sets is never replaced by the source’s settings, and a served setting only retunes what the watcher already watches, it never adds a path or a container.

The Watchers view

The list (/watchers) shows every watcher checking in with the workspace’s sources: host, source, version, when it last checked in, whether it’s stale (three missed check-ins in a row), its mode (Docker or Host only) and how many containers it watches. Filter it to one source, or sort it by host or last check-in. An empty list explains how to set the watcher up and links to Add source → InfraInbox watcher.

A watcher’s own page (/watchers/<id>) says under its name whether it’s current or stale and when it last checked in, then shows its report — version, OS/architecture, when it started, when it last checked in — and the settings it’s running with, in words: one card per check, on or off with its thresholds, and the containers section, not the raw JSON the watcher reports. A check the watcher didn’t report reads Not reported rather than On, and a host-only watcher has no containers section at all — it watches none.

What the host’s own file overrode — the fields its local settings file set, which always win over anything served.
What it refused, and why — a served setting the watcher didn’t apply, most often a disk path override for a filesystem the host doesn’t watch (a served setting only retunes what’s already watched; it never widens it).

From here, “Forget this watcher” removes its row; it has no key of its own to cut off, so a host that’s still running simply checks in again and comes back. “Settings for every watcher of <source>” opens the editor below.

The settings editor (/watchers/settings/<sourceId>, also reachable from a source’s own Heartbeats tab) is a form over the same detection settings the check-in API serves: one card per check, with its enabled toggle and its thresholds, and the containers section — ignore/only patterns, the missing-health-check notice, and a table of per-project overrides (watch, severity, oneshot). It’s sparse: an unset field shows the watcher’s own default as a placeholder, only what you actually change is saved, and every field has its own way back to “default” (a tri-state toggle, or a “Reset”/“Clear” control for a typed-in value). A save applies to every watcher of that source within about 5 minutes — sooner, at each watcher’s next check-in — and a host’s own settings file still always wins. Saving carries the revision last read: if someone else changed it first, the save is refused and the page reloads the latest version rather than overwriting it; an invalid value is rejected with the reason next to the field it’s about. The save bar at the foot of the window appears once you change something, with Discard and Save; a save confirms with Saved and an Undo, and a refused one lists its problems there, each linked to its field.

Per-container labels

A container’s own Docker labels change how the watcher treats it:

Label	Effect
`infrainbox.ignore=true`	Stay quiet about this container.
`infrainbox.severity=warning`, `error` or `critical`	Replaces the container’s usual ERROR severity.
`infrainbox.oneshot=true`	A job: no missing-health-check notice, and a clean exit resolves quietly.
`infrainbox.logs=<n>`	This container’s own log-line count, overriding `INFRAINBOX_CRASH_LOGS` (`0` turns it off).

services:
  backup-job:
    image: my/backup:1
    labels:
      infrainbox.oneshot: "true"
      infrainbox.logs: "20"

Heartbeat monitors — how the host heartbeat the watcher creates behaves
Events API — the plain source key the watcher uses under the hood is the same one any script can use