# Proxmox VE and Backup Server

> Wire a Proxmox VE cluster or Backup Server instance into InfraInbox with its native notification target.

Web page: https://infrainbox.app/docs/sources/proxmox/

Proxmox VE and Proxmox Backup Server both have a built-in notification system that can call a webhook. InfraInbox gives you a target to paste in, then reads the JSON Proxmox sends.

**Tested with:** Proxmox VE 8.3+ and Proxmox Backup Server 3.3+.

## What you get

A firing notification opens an incident; a matching success closes it. The shipped preset (editable under [Presets](https://infrainbox.app/docs/notifications/presets.md)) already tunes the common cases:

| Event | Behaviour |
|---|---|
| A backup or PBS job (gc, prune, sync, verify, tape) fails | Opens an incident, auto-resolves if nothing recurs in 20 hours |
| Replication fails | Opens an incident, 24-hour quiet expiry |
| HA fencing starts or completes | Opens an incident that re-notifies every 5 minutes (up to 6 times) until acknowledged, and never expires on its own |
| A SMART or ZFS fault is reported | Opens an incident that never expires — it needs a manual resolve |
| A ZFS scrub or resilver reports a problem | Same: manual resolve only |
| Other system mail, an ACME renewal failure, a tape load request | Recorded, but silent by default, with a 24-hour quiet expiry |

Everything else Proxmox sends still becomes an event you can see and route with your own rules — the preset only covers the cases above.

## What Proxmox can't tell you

Proxmox's own notifications can't report everything. The wizard's last step and the source's Heartbeats tab close two of these gaps for you:

- **A node going dark** — power loss, a kernel panic, a severed network — is invisible to Proxmox's own notification system, because the node that would report it is the one that's gone. A heartbeat per node catches it: about 4 minutes of silence becomes an offline incident. Point the heartbeat at an InfraInbox that isn't running on the same cluster, or it can't report the cluster going dark either.
- **A backup job that stops running entirely** — disabled, its node down at the scheduled time, switched back to email — never sends a failure, because nothing runs to report one. Watching the job (daily or weekly cadence) alerts when its expected report doesn't arrive.

Not covered yet, by Proxmox or by InfraInbox: quorum loss, a non-HA guest that crashes, and storage filling up. Closing those needs a read-only connection to the Proxmox API, which isn't built yet.

## Prerequisites

- A Proxmox VE cluster or standalone node, or a Proxmox Backup Server instance, that can reach your InfraInbox server over HTTPS.
- One source per cluster, standalone node or Backup Server instance — a cluster's notification configuration is cluster-wide, so it needs only one.

## Setup

1. Go to **Add source → Proxmox** (dashboard screen `/sources/new/proxmox`), pick **Proxmox VE** or **Proxmox Backup Server (PBS)**, and name it after the cluster, node or instance.
2. InfraInbox generates the exact configuration for your source and shows it on the next screen: an endpoint name, the method and URL, the headers, and the body template, plus a one-time secret. Copy the secret now — it's shown only once.
3. In the Proxmox web UI:
   - **PVE:** Datacenter → Notifications → Notification Targets → add a Webhook target with the fields shown.
   - **PBS:** Configuration → Notifications → add a Webhook target with the fields shown.
4. Type or paste the `Authorization` header exactly as shown — it's Proxmox's own template syntax (`Bearer {{ secrets.token }}`), not a placeholder for you to fill in yourself.
5. Click **Test** on the target. InfraInbox shows "Test received" and the source becomes **Reachable**.
6. Add a matcher that targets the new webhook target (PVE: name it, target the webhook, leave the match rules empty so it catches everything — don't filter out `info`, since successes are what auto-resolve an incident). PBS: Configuration → Notifications → Notification Matchers, same idea.
7. **PVE only:** in every backup job (Datacenter → Backup), switch its notification mode to the notification system (the Notifications tab on PVE 9.x, the General tab on 8.x) — a job still emailing root directly never reaches the target.
   **PBS only:** switch every datastore's Options → Notification mode to "Notification system", and do the same for tape backup jobs.
8. Run a small backup job now (or a guest's Backup → Backup now, with "Send email to" left empty). The source switches to **Receiving events**.

> **Note:** PBS can't report backups that clients push to it directly. Those results come from the matching PVE backup job instead, or from a client script posting to the [Events API](https://infrainbox.app/docs/sources/events-api.md).

## TLS

Proxmox checks certificates against each node's system CA store and has no skip-verify option:

- **Public certificate:** use one (Let's Encrypt behind a reverse proxy, or a Tailscale `*.ts.net` name) — every node trusts it automatically.
- **Private CA:** install the CA certificate on every node (`/usr/local/share/ca-certificates/`, then `update-ca-certificates`), or Proxmox's Test fails with `UnknownIssuer`.

If your instance still allows plain HTTP on private networks, the secret crosses your LAN unencrypted — prefer HTTPS wherever you can.

## Rotating the key

1. Rotate the source's key on its Credentials tab first. The old key keeps working for 24 hours.
2. In Proxmox, open the `infrainbox` target you created and replace the secret with the new key exactly as shown, then click Test.

## If Proxmox's Test fails

| Proxmox shows | Likely cause | Fix |
|---|---|---|
| Certificate error (`UnknownIssuer`) | The node doesn't trust the certificate's CA | Use a publicly trusted certificate, or install the private CA on every node |
| HTTP 401 | The `Authorization` header is mistyped, or the secret is missing or was rotated | Re-type `Bearer {{ secrets.token }}` exactly, then re-paste the secret |
| Timeout or connection refused | An egress firewall, proxy or DNS problem | Allow outbound HTTPS from every node, and check the datacenter's HTTP proxy setting |

## Next

- [Heartbeat monitors](https://infrainbox.app/docs/sources/heartbeats.md) — add a node's liveness check or watch a backup job
- [Presets](https://infrainbox.app/docs/notifications/presets.md) — see or change the rules above