# Docker and host watcher (preview)

> A small program that watches Docker and the host it runs on, and pings its own heartbeat. Currently a preview.

Web page: https://infrainbox.app/docs/sources/watcher/

> **Note:** The InfraInbox watcher is in preview: pre-release, not yet a stable, generally available build. The setup below works today, but expect rougher edges than the rest of InfraInbox. It was called the InfraInbox agent until September 2026.

The watcher runs on one host, beside your containers if there are any. It reads Docker read-only, checks the host itself, and reports to InfraInbox over outbound HTTPS. It needs no changes to your applications, and it opens no port and accepts no commands.

## What it watches

| When | In InfraInbox |
|---|---|
| A container exits with an error | An incident, such as "immich-server exited with code 137", with what the exit code means, the restart count and, on an out-of-memory kill, the memory limit. A container you stop on purpose (`docker stop`, a Compose recreate) is not a crash. |
| A container is killed for running out of memory | An incident: "… was killed: out of memory". |
| A container's `HEALTHCHECK` fails | An incident: "… is unhealthy". |
| It starts again and is still running 30 seconds later, or its health check passes | The same incident resolves. A crash loop stays one open incident. |
| The container is removed | The same incident resolves, unless `--rm` removed it right after it failed. |
| The host or the watcher stops | The host's heartbeat monitor goes down: one incident for the whole host, not one per container. |
| The watcher can't reach Docker for two minutes | An incident, "web-01: the watcher cannot reach Docker", which resolves when Docker is back. |
| A container has no `HEALTHCHECK` (first start) | One informational notice, once: without a health check, a process that is up but not answering is invisible. It closes on its own once the container has one. |
| A disk is nearly full | An incident, such as "nas: /var/lib/docker is 93% full (7.1 GiB free)"; resolves once space is freed. |
| A ZFS pool or md RAID array degrades | An incident naming the pool or array; resolves once it's healthy again. |
| The host just rebooted | One informational notice, once per boot. |
| The host's clock drifts from the server | An incident once the drift is sustained; resolves once it's back in sync. |
| Memory pressure is sustained | An incident; resolves once it eases. |

Every condition above ends only by a matching recovery event or a manual resolve — never by sitting quiet for 24 hours, even though the watcher reports a condition once when it starts and once when it recovers, with nothing in between.

Events from a Docker Compose project carry `compose:<project>` and `service:<service>` tags, so rules and filters can pick out one stack.

The last five rows need no Docker at all. On a Proxmox node, a NAS or anything else with no containers to watch, the watcher runs in **host-only mode**: just the heartbeat and the host conditions. It switches to that mode by itself when it finds no Docker, or set `INFRAINBOX_DOCKER=off`.

If InfraInbox is unreachable, events wait in memory (up to 1000) and are retried in order, so a restart never overtakes the crash it resolves. After a reconnect, the watcher re-reads Docker and reports what each container is doing now, so an incident whose recovery happened while it was away still closes.

## What it doesn't do

It only reads. It has no shell, listens on no port, and every write to Docker is refused at the socket proxy in front of it. Nothing is ever run on the host through the watcher. It reports conditions, not metrics: there are no graphs or time series.

## Setup

1. Go to **Add source → InfraInbox watcher** (dashboard screen `/sources/new/watcher`) and name it after the host (`nas`, `web-01`, and so on).
2. InfraInbox issues one ingest key, shown once, and a Compose file with it already filled in.

### Docker Compose

The page shows a Compose file with your key and, when you self-host, your server's address already filled in. It looks like this:

```yaml
services:
  docker-socket-proxy:
    image: tecnativa/docker-socket-proxy:v0.5.0
    restart: unless-stopped
    environment:
      CONTAINERS: 1   # list and inspect containers
      EVENTS: 1       # the transition stream
      INFO: 1         # the host's name
      POST: 0         # read-only: every write is refused
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost:2375/_ping"]
      interval: 30s

  infrainbox-watcher:
    image: ghcr.io/infrainbox/infrainbox-watcher:0.1
    restart: unless-stopped
    depends_on:
      - docker-socket-proxy
    environment:
      INFRAINBOX_TOKEN: iik_src_…
      INFRAINBOX_URL: https://infrainbox.example.com   # only needed when self-hosted
      DOCKER_HOST: tcp://docker-socket-proxy:2375
      INFRAINBOX_HOST_ROOT: /host
    volumes:
      - /var/lib/docker:/host/var/lib/docker:ro   # read-only: the disk check sees how full it is
    read_only: true
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
```

Save it on the host and run:

```bash
mkdir -p ~/infrainbox-watcher && cd ~/infrainbox-watcher
nano compose.yml          # paste the file from the page
docker compose up -d
```

Check it started:

```bash
docker compose logs infrainbox-watcher
```

You should see `starting`, `inventory containers=…`, `enrolled` and `sent`.

### As a systemd service, without Compose

Download the release archive for the host's architecture (`linux amd64` or `arm64`) and unpack it, then from that folder:

```bash
sudo install -m 755 infrainbox-watcher /usr/local/bin/
sudo install -m 644 deploy/systemd/infrainbox-watcher.service /etc/systemd/system/
sudo install -d -m 755 /etc/infrainbox-watcher
sudo install -m 600 deploy/watcher.env.example /etc/infrainbox-watcher/watcher.env
```

Put this in `/etc/infrainbox-watcher/watcher.env` (root-only, mode 600):

```
INFRAINBOX_TOKEN=iik_src_…
INFRAINBOX_URL=https://infrainbox.example.com
```

Then:

```bash
sudo systemctl enable --now infrainbox-watcher
```

## Check it works

Within a minute of starting, a heartbeat monitor named after the host appears on the source's Heartbeats tab by itself; nobody creates it. To check everything at once without changing anything, run the watcher's `doctor`: it tests the configuration, Docker and the proxy's permissions, the server, the key and the clock, prints a line for each, and exits non-zero if any of them failed.

```bash
docker compose run --rm infrainbox-watcher doctor
```

If the watcher starts reporting `401`, its key was rotated, its source was deleted — or the whole workspace was closed, which stops every key in it at once (see [Security and privacy](https://infrainbox.app/docs/self-hosting/security.md)). Issue a new key on the source's page and put it back in `watcher.env`.

## Configuration

Two kinds of setting, kept apart on purpose. **Environment variables** hold everything about trust and privacy: the key, the server, TLS, how Docker is reached, which host paths are mounted, and crash log lines. They are never command-line flags, so the key never appears in a process list, and a normal install sets only the first one or two. An optional **settings file** tunes what the watcher detects: see [Settings file](#settings-file) below.

| Variable | Default | Meaning |
|---|---|---|
| `INFRAINBOX_TOKEN` | required | The source key (`iik_src_…`). |
| `INFRAINBOX_URL` | InfraInbox Cloud | Your server's address. Only a self-hosted server needs it. Must be `https://`, except for a private address. |
| `DOCKER_HOST` | `unix:///var/run/docker.sock` | Usually `tcp://docker-socket-proxy:2375`. |
| `INFRAINBOX_HOST` | Docker's name for the host | How the host is named in InfraInbox; its heartbeat monitor takes this name. |
| `INFRAINBOX_IGNORE` | none | Comma-separated container names to stay quiet about. Patterns work too: `*-backup-*`. |
| `INFRAINBOX_CONFIG` | none | Path to the settings file, for example `/etc/infrainbox-watcher/watcher.json`. |
| `INFRAINBOX_DOCKER` | on when Docker is found | `off` runs host-only mode. |
| `INFRAINBOX_DISKS` | none | Extra paths the disk-nearly-full check watches, besides Docker's own data directory. |
| `INFRAINBOX_HOST_ROOT` | none | Inside a container, where the host's filesystems are mounted for the disk check (`/host` in the Compose file). |
| `INFRAINBOX_CRASH_LOGS` | `0` | How many of a crashed container's last log lines to attach, up to 100. Off by default, because logs can carry secrets. |
| `INFRAINBOX_TLS_CA_FILE` | none | A PEM file with your private CA, for a self-hosted server whose certificate isn't publicly trusted. |
| `INFRAINBOX_TLS_PIN` | none | `sha256/…` certificate pins, comma-separated. |
| `INFRAINBOX_HEARTBEAT_EVERY` | `60s` | How often the heartbeat is sent. |

### Settings file

Every check can be switched off or given its own thresholds, and you choose which containers are watched, in one JSON file. Every field is optional; anything left out keeps its default.

```json
{
  "checks": {
    "disk":   {"warnAt": 85, "errorAt": 95, "resolveBelow": 80, "for": "10m",
               "paths": {"/mnt/backup": {"warnAt": 95, "errorAt": 99, "resolveBelow": 90}}},
    "memory": {"enabled": false},
    "clock":  {"driftSeconds": 120},
    "reboot": {"window": "1h"}
  },
  "containers": {
    "ignore": ["watchtower", "*-backup-*"],
    "healthcheckNotice": false,
    "projects": {"media": {"severity": "warning"}}
  }
}
```

- **Checks:** `disk` (space and inodes, with per-path overrides), `memory`, `clock`, `reboot`, `zfs` and `raid`. Each has `enabled`, plus its thresholds and how long a reading has to hold (`for`). A switched-off check reads nothing and sends nothing.
- **Containers:** `ignore` and `only` take name patterns (`only` empty means all). `projects` sets `ignore`, `severity` or `oneshot` for every container of one Docker Compose project. `healthcheckNotice: false` turns off the missing-health-check notice.
- **Order:** a container's own labels win over its project's settings, which win over the file's global ones.

Mount the file read-only and point the watcher at it: in Compose, add `./watcher.json:/etc/infrainbox-watcher/watcher.json:ro` to the watcher's `volumes` and `INFRAINBOX_CONFIG: /etc/infrainbox-watcher/watcher.json` to its `environment`; with systemd, put it at `/etc/infrainbox-watcher/watcher.json` (mode 644, in the readable directory the install creates — the key stays in the root-only `watcher.env`) and the same line in `watcher.env`. A misspelt field or an impossible value (a resolve point above its trigger) stops the watcher at start with the field named, rather than being ignored. `doctor` prints the settings in effect, and `infrainbox-watcher settings` prints them in full.

**Settings from the dashboard.** Watchers check in with InfraInbox about every 5 minutes, with their version and the settings they run with. The same detection settings can also be kept once per source, for every watcher that uses its key, from the **Watchers** view, linked from the header of **Sources** (dashboard screen `/sources`). The host's own file always wins: a field it sets is never replaced by the source's settings, and a served setting only retunes what the watcher already watches, it never adds a path or a container.

### The Watchers view

**The list** (`/watchers`, crumbed *Sources › Watchers*) shows every watcher checking in with the workspace's sources: host, source, version, when it last checked in, whether it's stale (three missed check-ins in a row), its mode (Docker or Host only) and how many containers it watches. Filter it to one source, or sort it by host or last check-in. An empty list explains how to set the watcher up and links to **Add source → InfraInbox watcher** (dashboard screen `/sources/new/watcher`).

**A watcher's own page** (`/watchers/<id>`) says under its name whether it's current or stale and when it last checked in, then shows its report — version, OS/architecture, when it started, when it last checked in — and the **settings it's running with, in words**: one card per check, on or off with its thresholds, and the containers section, not the raw JSON the watcher reports. A check the watcher didn't report reads **Not reported** rather than On, and a host-only watcher has no containers section at all — it watches none.

- **What the host's own file overrode** — the fields its local settings file set, which always win over anything served.
- **What it refused, and why** — a served setting the watcher didn't apply, most often a disk path override for a filesystem the host doesn't watch (a served setting only retunes what's already watched; it never widens it).

From here, **"Forget this watcher"** removes its row; it has no key of its own to cut off, so a host that's still running simply checks in again and comes back. **"Settings for every watcher of \<source\>"** opens the editor below.

**The settings editor** (`/watchers/settings/<sourceId>`, also reachable from a source's own Heartbeats tab) is a form over the same detection settings the check-in API serves: one card per check, with its enabled toggle and its thresholds, and the containers section — ignore/only patterns, the missing-health-check notice, and a table of per-project overrides (watch, severity, oneshot). It's **sparse**: an unset field shows the watcher's own default as a placeholder, only what you actually change is saved, and every field has its own way back to "default" (a tri-state toggle, or a "Reset"/"Clear" control for a typed-in value). A save applies to every watcher of that source within about 5 minutes — sooner, at each watcher's next check-in — and a host's own settings file still always wins. Saving carries the revision last read: if someone else changed it first, the save is refused and the page reloads the latest version rather than overwriting it; an invalid value is rejected with the reason next to the field it's about. The save bar at the foot of the window appears once you change something, with **Discard** and **Save**; a save confirms with **Saved** and an **Undo**, and a refused one lists its problems there, each linked to its field.

### Per-container labels

A container's own Docker labels change how the watcher treats it:

| Label | Effect |
|---|---|
| `infrainbox.ignore=true` | Stay quiet about this container. |
| `infrainbox.severity=warning`, `error` or `critical` | Replaces the container's usual ERROR severity. |
| `infrainbox.oneshot=true` | A job: no missing-health-check notice, and a clean exit resolves quietly. |
| `infrainbox.logs=<n>` | This container's own log-line count, overriding `INFRAINBOX_CRASH_LOGS` (`0` turns it off). |

```yaml
services:
  backup-job:
    image: my/backup:1
    labels:
      infrainbox.oneshot: "true"
      infrainbox.logs: "20"
```

## Next

- [Heartbeat monitors](https://infrainbox.app/docs/sources/heartbeats.md) — how the host heartbeat the watcher creates behaves
- [Events API](https://infrainbox.app/docs/sources/events-api.md) — the plain source key the watcher uses under the hood is the same one any script can use