Docker and host watcher (preview)
The watcher runs on one host, beside your containers if there are any. It reads Docker read-only, checks the host itself, and reports to InfraInbox over outbound HTTPS. It needs no changes to your applications, and it opens no port and accepts no commands.
What it watches
Section titled “What it watches”| When | In InfraInbox |
|---|---|
| A container exits with an error | An incident, such as “immich-server exited with code 137”, with what the exit code means, the restart count and, on an out-of-memory kill, the memory limit. A container you stop on purpose (docker stop, a Compose recreate) is not a crash. |
| A container is killed for running out of memory | An incident: “… was killed: out of memory”. |
A container’s HEALTHCHECK fails |
An incident: “… is unhealthy”. |
| It starts again and is still running 30 seconds later, or its health check passes | The same incident resolves. A crash loop stays one open incident. |
| The container is removed | The same incident resolves, unless --rm removed it right after it failed. |
| The host or the watcher stops | The host’s heartbeat goes down: one incident for the whole host, not one per container. |
| The watcher can’t reach Docker for two minutes | An incident, “web-01: the watcher cannot reach Docker”, which resolves when Docker is back. |
A container has no HEALTHCHECK (first start) |
One informational notice, once: without a health check, a process that is up but not answering is invisible. It closes on its own once the container has one. |
| A disk is nearly full | An incident, such as “nas: /var/lib/docker is 93% full (7.1 GiB free)”; resolves once space is freed. |
| A ZFS pool or md RAID array degrades | An incident naming the pool or array; resolves once it’s healthy again. |
| The host just rebooted | One informational notice, once per boot. |
| The host’s clock drifts from the server | An incident once the drift is sustained; resolves once it’s back in sync. |
| Memory pressure is sustained | An incident; resolves once it eases. |
| The CPUs stay at least 90% busy for 10 minutes | An incident; resolves once they’re under 80%. |
| The hottest sensor stays at 85 °C or more for 5 minutes | An incident naming the sensor; resolves once it’s under 75 °C. |
| The kernel kills a process to free memory | An incident; resolves after a minute with no more kills. A container killed this way also reports on its own. |
The disk, memory, CPU, temperature and clock incidents carry their numbers: the reading that opened, escalated or cleared it, the limit it crossed, the level it clears under and the highest seen, both in the notice and on the incident.
Every condition above ends only by a matching recovery event or a manual resolve — never by sitting quiet for 24 hours, even though the watcher reports a condition once when it starts and once when it recovers, with nothing in between.
Events from a Docker Compose project carry compose:<project> and service:<service> tags, so rules and filters can pick out one stack.
The last eight rows need no Docker at all. On a Proxmox node, a NAS or anything else with no containers to watch, the watcher runs in host-only mode: just the heartbeat and the host conditions. It switches to that mode by itself when it finds no Docker, or set INFRAINBOX_DOCKER=off.
If InfraInbox is unreachable, events wait in memory (up to 1000) and are retried in order, so a restart never overtakes the crash it resolves. After a reconnect, the watcher re-reads Docker and reports what each container is doing now, so an incident whose recovery happened while it was away still closes.
What it doesn’t do
Section titled “What it doesn’t do”It only reads. It has no shell, listens on no port, and every write to Docker is refused at the socket proxy in front of it. Nothing is ever run on the host through the watcher. It reports conditions, not metrics: a reading rides on the event for the change it reports, and there are no graphs or time series.
It doesn’t read SMART data or hardware RAID controllers: those need device access and privileges the watcher doesn’t ask for. For SMART, send smartd’s warnings to InfraInbox instead (ZFS and disk health alerts). Inside an unprivileged Proxmox container (LXC) or Docker Desktop, the host’s own disks, arrays and sensors are out of sight: run the watcher on the host itself.
- Go to Add source → InfraInbox watcher and name it after the host (
nas,web-01, and so on). - InfraInbox issues one ingest key, shown once, and the commands that install the watcher. The key is never written into them: you type or paste it at a hidden prompt. Pick Docker Compose or As a service (no Docker Compose); the page shows one method’s commands at a time, switching without losing the key. Clicking I’ve copied it only hides the key — the method choice and the commands stay on the page, with a placeholder in the key’s place, so you can still finish the install or copy them again. The page never claims the source is ready before a watcher has actually checked in: it says Waiting for watcher… until one does, then Watcher connected. If the check itself fails — a network or server error, not “no watcher yet” — it says Couldn’t check for the watcher with a Retry, rather than implying nothing has reported.
Installing on another host later, or reinstalling this one? The same Compose paste is on the source’s Connect tab — it carries no key either, so it’s the same block every time.
Docker Compose
Section titled “Docker Compose”The page shows one block to paste on the host. It writes compose.yml, asks for the key without showing it, keeps it in watcher.env (readable only by you, never in the Compose file), checks everything with doctor, and starts the watcher:
mkdir -p ~/infrainbox-watcher && cd ~/infrainbox-watchercat > compose.yml <<'EOF'…the Compose file below…EOFbash -c 'read -rsp "Ingest key: " k; echo; umask 077; printf "INFRAINBOX_TOKEN=%s\nINFRAINBOX_URL=https://infrainbox.example.com\n" "$k" > watcher.env' && docker compose run --rm infrainbox-watcher doctor && docker compose up -dThe last line also writes your server’s address into watcher.env as INFRAINBOX_URL, on InfraInbox Cloud too. bash -c is there because zsh’s read -p means something else. Setting up a host from your laptop? Run the last line through ssh -t <host> from that folder, so the prompt stays hidden.
The Compose file holds no secret, so it’s the same on every host:
services: docker-socket-proxy: image: tecnativa/docker-socket-proxy:v0.5.0 restart: unless-stopped environment: CONTAINERS: 1 # list and inspect containers EVENTS: 1 # the stream of container starts, stops and health changes INFO: 1 # the host's name PING: 1 # the health check below POST: 0 # read-only: every write is refused # Everything else is off, even if a future proxy image changes its defaults. AUTH: 0 BUILD: 0 # … every other area, set to 0 the same way volumes: - /var/run/docker.sock:/var/run/docker.sock:ro networks: [socket] security_opt: [no-new-privileges:true] healthcheck: test: ["CMD", "wget", "-q", "--spider", "http://localhost:2375/_ping"] interval: 30s timeout: 5s retries: 3 start_period: 5s
infrainbox-watcher: image: ghcr.io/infrainbox/infrainbox-watcher:0.1.0 restart: unless-stopped env_file: watcher.env # the key, readable only by you environment: DOCKER_HOST: tcp://docker-socket-proxy:2375 INFRAINBOX_HOST_ROOT: /host volumes: - /var/lib/docker:/host/var/lib/docker:ro # read-only: the disk check sees how full it is # Another disk: any folder on it at /host/disks/<name>; only how full it is is read # - /mnt/media/.infrainbox:/host/disks/media:ro depends_on: [docker-socket-proxy] networks: [socket, egress] read_only: true cap_drop: [ALL] security_opt: [no-new-privileges:true] mem_limit: 128m
networks: socket: internal: true # only the watcher reaches the proxy, and the proxy reaches nothing egress: {} # the watcher's outbound HTTPS; nothing is publishedMore disks. Out of the box the watcher checks the disk Docker keeps its data on. To watch another disk, add one line per disk under volumes:, mounting any folder on it at /host/disks/<name>, read-only. For example, - /mnt/media/.infrainbox:/host/disks/media:ro watches the disk behind /mnt/media as media; Docker creates the folder if it’s missing. The watcher asks only how full the disk is, so an empty folder is enough and it never sees the disk’s files. Restart the watcher (docker compose up -d) to pick up a new disk.
Check it started:
docker compose logs infrainbox-watcherYou should see starting, inventory containers=…, enrolled and sent.
On its first start the watcher reports everything it finds: for each container, whether it has a HEALTHCHECK and that it’s running, then each disk and host check that’s fine — often a few dozen events at once. That’s expected. Apart from the one-time notice for a container without a health check, they’re all-clears with nothing open to close: recorded in the event log, and they notify nobody.
As a systemd service, without Compose
Section titled “As a systemd service, without Compose”Download the release archive for the host’s architecture (linux amd64 or arm64) and unpack it, then from that folder:
sudo install -m 755 infrainbox-watcher /usr/local/bin/sudo install -m 644 deploy/systemd/infrainbox-watcher.service /etc/systemd/system/sudo install -d -m 755 /etc/infrainbox-watchersudo install -m 600 deploy/watcher.env.example /etc/infrainbox-watcher/watcher.envPut this in /etc/infrainbox-watcher/watcher.env (root-only, mode 600):
INFRAINBOX_TOKEN=iik_src_…INFRAINBOX_URL=https://infrainbox.example.comThen:
sudo systemctl enable --now infrainbox-watcherCheck it works
Section titled “Check it works”Within a minute of starting, the host appears in the Watchers section of the source’s Checks tab, with the heartbeat it pings by itself; nobody creates it. To check everything at once without changing anything, run the watcher’s doctor: it tests the configuration, Docker and the proxy’s permissions, the server, the key and the clock, prints a line for each, and exits non-zero if any of them failed.
docker compose run --rm infrainbox-watcher doctorIf the watcher starts reporting 401, its key was rotated, its source was deleted — or the whole workspace was closed, which stops every key in it at once (see Security and privacy). Issue a new key on the source’s page and put it back in watcher.env: in the Compose folder, run the bash -c 'read -rsp …' line again.
Configuration
Section titled “Configuration”Two kinds of setting, kept apart on purpose. Environment variables hold everything about trust and privacy: the key, the server, TLS, how Docker is reached, which host paths are mounted, and crash log lines. They are never command-line flags, so the key never appears in a process list, and a normal install sets only the first one or two. An optional settings file tunes what the watcher detects: see Settings file below.
| Variable | Default | Meaning |
|---|---|---|
INFRAINBOX_TOKEN |
required | The source key (iik_src_…). |
INFRAINBOX_URL |
https://dash.infrainbox.app |
Your server’s address: https://dash.infrainbox.app on InfraInbox Cloud, your own when you self-host. Always set it; the install on the source’s page writes it. Must be https://, except for a private address. |
INFRAINBOX_SOURCE_KEY |
none | Another name for INFRAINBOX_TOKEN, read only when that is unset. |
DOCKER_HOST |
unix:///var/run/docker.sock |
Usually tcp://docker-socket-proxy:2375. |
INFRAINBOX_HOST |
Docker’s name for the host | How the host is named in InfraInbox; its heartbeat takes this name. |
INFRAINBOX_IGNORE |
none | Comma-separated container names to stay quiet about. Patterns work too: *-backup-*. |
INFRAINBOX_CONFIG |
none | Path to the settings file, for example /etc/infrainbox-watcher/watcher.json. |
INFRAINBOX_DOCKER |
on when Docker is found | off runs host-only mode. |
INFRAINBOX_DISKS |
none | Extra paths the disk-nearly-full check watches, besides Docker’s own data directory. In Compose, mounting a folder of each disk at /host/disks/<name> is simpler (see More disks). |
INFRAINBOX_HOST_ROOT |
none | Inside a container, where the host’s filesystems are mounted for the disk check (/host in the Compose file). |
INFRAINBOX_CRASH_LOGS |
0 |
How many of a crashed container’s last log lines to attach, up to 100. Off by default, because logs can carry secrets. |
INFRAINBOX_TLS_CA_FILE |
none | A PEM file with your private CA, for a self-hosted server whose certificate isn’t publicly trusted. |
INFRAINBOX_TLS_PIN |
none | sha256/… certificate pins, comma-separated. |
INFRAINBOX_HEARTBEAT_EVERY |
60s |
How often the heartbeat is sent. |
INFRAINBOX_HEARTBEAT_KEY |
none | Leave it unset: the watcher creates its own heartbeat check with the source key. Set, it pings that check (iik_hb_…) instead, for a server too old to create one. |
Settings file
Section titled “Settings file”Every check can be switched off or given its own thresholds, and you choose which containers are watched, in one JSON file. Every field is optional; anything left out keeps its default.
{ "checks": { "disk": {"warnAt": 85, "errorAt": 95, "resolveBelow": 80, "for": "10m", "paths": {"/mnt/backup": {"warnAt": 95, "errorAt": 99, "resolveBelow": 90}}}, "memory": {"enabled": false}, "clock": {"driftSeconds": 120}, "reboot": {"window": "1h"} }, "containers": { "ignore": ["watchtower", "*-backup-*"], "healthcheckNotice": false, "projects": {"media": {"severity": "warning"}} }}- Checks:
disk(space and inodes, with per-path overrides),memory,clock,reboot,zfsandraid. Each hasenabled, plus its thresholds and how long a reading has to hold (for). A switched-off check reads nothing and sends nothing. - Containers:
ignoreandonlytake name patterns (onlyempty means all).projectssetsignore,severityoroneshotfor every container of one Docker Compose project.healthcheckNotice: falseturns off the missing-health-check notice. - Order: a container’s own labels win over its project’s settings, which win over the file’s global ones.
Mount the file read-only and point the watcher at it: in Compose, add ./watcher.json:/etc/infrainbox-watcher/watcher.json:ro to the watcher’s volumes and INFRAINBOX_CONFIG: /etc/infrainbox-watcher/watcher.json to its environment; with systemd, put it at /etc/infrainbox-watcher/watcher.json (mode 644, in the readable directory the install creates — the key stays in the root-only watcher.env) and the same line in watcher.env. A misspelt field or an impossible value (a resolve point above its trigger) stops the watcher at start with the field named, rather than being ignored. doctor prints the settings in effect, and infrainbox-watcher settings prints them in full.
Settings from the dashboard. Watchers check in with InfraInbox about every 5 minutes, with their version and the settings they run with, and may also list the containers they see (up to 200: name, Compose project, and whether and why each is watched), which the watcher’s page reads back. The same detection settings can also be kept once per source, for every watcher that uses its key, from the Watchers tab of Sources. The host’s own file always wins: a field it sets is never replaced by the source’s settings, and a served setting only retunes what the watcher already watches, it never adds a path or a container.
The Watchers view
Section titled “The Watchers view”The list is the Watchers tab of Sources (/sources/watchers), beside the Sources tab, each with its count. It shows every host running a watcher for one of the workspace’s sources: host, source, version, when it last checked in, and whether it’s Reporting or Stale (three missed check-ins in a row). A row opens the watcher in a drawer on the right; the source’s name opens the source. Filter it to one source, or sort it by host or last check-in. Install on a host (owner or admin) starts Add source → InfraInbox watcher. A source created that way reads InfraInbox watcher from the start, before any host checks in.
On a source’s Checks tab, the Watchers section lists the source’s hosts. The watcher reports its disks and containers about every 5 minutes and pings its own host heartbeat; if the pings stop, that heartbeat opens the incident. A source with a watcher doesn’t offer a second, hand-made host heartbeat.
A watcher’s drawer (a row on the Checks tab or the Watchers tab; the old /watchers/<id> address opens it too) shows its facts once: Reporting or Stale, when it last checked in, its mode, version, OS/architecture and start time. Then its host heartbeat, its container counts and the settings it’s running with, in words. A field its own settings file set is marked its own file where it appears. A check it didn’t report reads Not reported, and a host-only watcher has no container settings. What it refused, and why lists a served setting it didn’t apply, most often a disk path the host doesn’t watch. Forget this watcher removes its row; a host that’s still running checks in again and comes back. Change for all hosts opens the settings drawer.
Watcher settings open in a drawer from the Settings button of the Watchers section (the old /watchers/settings/<sourceId> address opens it too). They apply to every watcher of the source within about 5 minutes; the drawer names the hosts. It has two tabs:
- Containers. Choose All containers, except the ones I ignore or Only the ones I pick. Seen on your hosts lists the Compose projects and containers, so you pick rather than type: every container the watchers last reported, watched or not (a container its own
infrainbox.*label decides shows that and can’t be picked), or — when no watcher of the source reports that list — the containers named in its recent events. The panel says which one it shows. Each project is one line: its name, a summary such as “4 containers · watched · Error”, and its Alerts choice; open the line for its containers and the rest of its settings (projects with settings of their own start open). Click a container name to ignore it (or, in “Only” mode, to pick it); throwaway…-run-<id>containers are grouped as “+N one-off runs”. Each project has:- Alerts: Default follows the rule above and says what that is, for example “Default (watched)”. Watch watches its containers even if a name pattern would ignore them; Ignore drops them all.
- Severity: how serious a crash, out-of-memory kill or unhealthy state is. Recoveries are unchanged.
- Job (it was “Oneshot”): the containers exit when done, like backups or cron jobs. A clean exit is quiet, a failed one still alerts, and they skip the health-check notice. It has no “No”: a container labelled as a job stays one.
- Also ignore names like… takes name patterns (
*matches anything), each showing what it matches. - Tell me about containers without a health check sends one Info notice per running container without a Docker HEALTHCHECK.
- A Preview counts what the settings would watch among the containers seen, beside what the hosts last reported.
- A container’s own
infrainbox.*labels win, then project settings, then name patterns.
- Host checks. One switch per check with a line on what it does, such as “A disk is nearly full: warns at 90% held for 5m, errors at 97% at once, clears below 85%”. A check none of the hosts can run isn’t listed: no ZFS pool, no md RAID array, or no temperature sensor (most VMs and VPSs have none). Each watcher tells InfraInbox what its host can check every time it checks in, so a pool created later shows up within about 5 minutes. A watcher’s own drawer leaves those checks out too. Adjust thresholds shows the numbers, each with its default beside it. Thresholds that contradict each other (a clear point above a warning) are flagged and hold the save. Durations are written like
5mor1h.
What a host’s own file sets is shown where it applies, read-only, for example “docker-prod’s own file: Ignored names watchtower”. The file always wins. The drawer is sparse: only what you change is saved, and Reset to default undoes a setting. Saving carries the revision last read: if someone else changed the settings first, the save is refused and the latest version is shown. The save bar has Discard and Save; a save confirms with Saved and Undo and closes the drawer. ✕, Escape or a click outside asks before dropping unsaved changes.
Per-container labels
Section titled “Per-container labels”A container’s own Docker labels change how the watcher treats it:
| Label | Effect |
|---|---|
infrainbox.ignore=true |
Stay quiet about this container. |
infrainbox.severity=warning, error or critical |
Replaces the container’s usual ERROR severity. |
infrainbox.oneshot=true |
A job: no missing-health-check notice, and a clean exit resolves quietly. |
infrainbox.logs=<n> |
This container’s own log-line count, overriding INFRAINBOX_CRASH_LOGS (0 turns it off). |
services: backup-job: image: my/backup:1 labels: infrainbox.oneshot: "true" infrainbox.logs: "20"- Heartbeats and ping checks — how the host heartbeat the watcher creates behaves
- Events API — the plain source key the watcher uses under the hood is the same one any script can use