Skip to content

How incidents work

An event is one thing a source reported. An incident is a problem, grouped from its events by fingerprint. Not every event opens or touches an incident — by the shipped Default, CRITICAL, ERROR and WARNING events open or update an incident; INFO, SUCCESS and DEBUG events are logged to History › Events received only. A severity kept in History can still name recipients: each then gets a one-off message, held back by silences, quiet hours and a delay like any other. You can change this per rule — see Routing: one list of rules.

Events group onto the same incident by a fingerprint computed, in priority order, from:

  1. The sender’s own dedupKey, compared verbatim and case-sensitively.
  2. Failing that, an integration’s own correlation key (built into its adapter or built-in rules).
  3. Failing that, the event’s type, resource and a normalized title.

A fingerprint is always scoped to its source: the same fingerprint on two different sources never merges into one incident. Every event that matches an active incident’s fingerprint adds to its occurrence count rather than opening a new one.

An incident that has been resolved for less than 30 minutes reopens on the next matching event instead of opening a new one — its state goes back to open, its severity and peak severity reset to the new event’s, and its reopen count goes up by one. Past 30 minutes, a matching event opens a fresh incident instead. An event that arrived (by its own timestamp) before the resolution is attached to the resolved incident as a late occurrence and doesn’t reopen it.

A resolved notice’s “open for” time counts the stretch it ends: from the reopen, not from when the incident first opened. An incident that was down 2 minutes, fine for 23, then down again for 15 reports 15 minutes.

An incident that reopens 3 times within 30 minutes is marked flapping. While flapping, its open, reopen and resolve notices are held back — you get one “unstable” notice instead of a burst of them. Flapping ends once the incident has gone 30 minutes with no further reopen or resolve, at which point one final notice reports its actual state: if it resolved meanwhile, that is the resolved notice it held back, sent silently; if someone acknowledged or snoozed it, the message is updated to say so. See Tame a flapping check.

A route can set expire_after: how long an incident may sit with no new occurrence before InfraInbox resolves it on its own, silently (no “resolved” alert, just a quiet correction to anyone already told). The shipped default is 24 hours; a route may set anything from a few seconds up to 30 days, or never to keep the incident open until something explicitly resolves it. This exists for event types with no separate recovery signal — a failure-only alert must not still read “open” a week later.

These are two different things:

  • Cooldown rate-limits repeat occurrences of an already-open incident: by default, the same destination isn’t told about a reoccurring event more than once every 10 minutes. A rule can change this per route.
  • Re-notify is a separate, opt-in repeat of the whole alert for an incident nobody has acknowledged. It’s off unless a rule turns it on. The shipped Default turns it on for CRITICAL (every 5 minutes, up to 6 times) and ERROR (every 15 minutes, up to 4 times, only while the incident is still at least ERROR); WARNING opens an incident but has no destinations or re-notify configured by default.

Re-notify pauses while an incident is acknowledged or snoozed, and resumes if you unacknowledge it, unsnooze it, or it escalates. It stops for good once the incident is resolved.

Action State change Re-notify Sent to already-told destinations
Acknowledge open/snoozed → acknowledged Pauses An “update” notice
Unacknowledge acknowledged → open Resumes An “update” notice
Snooze (until a time you pick) open/acknowledged → snoozed Pauses An “update” notice
Unsnooze, or the snooze time is reached snoozed → open Resumes An “update” notice (or, on expiry, a snooze_expired notice)
Resolve any non-resolved state → resolved Stops See cross-channel sync below

Acknowledging is incident-global (any member’s ack counts for everyone, not just for them) and clears any active snooze — an ack is the stronger statement. Resolving an incident owned by a heartbeat that’s currently DOWN is refused (409): the host is unreachable, so nothing on it can be trusted to have recovered. The dashboard offers Pause heartbeat instead, which resolves the incident with that as its reason.

Not every destination can be told about a state change the same way:

  • Telegram and Discord edit the original alert message in place to show the new state.
  • Webhook destinations get a new event of the new type (incident.acknowledged, incident.snoozed, and so on) — there’s no message to edit, but the event itself is the update.
  • Slack and InfraInbox Push cannot edit a past message. For an ack, unack, snooze or unsnooze, they simply get nothing — their last message keeps showing the state it showed when it was sent, until the incident is later resolved or reoccurs. For a resolution, they do get a fresh, silent follow-up message.

An incident that resolves from a real recovery event (the source itself said so) always gets a genuine “resolved” notice, subject to the destination’s own recovery setting; a resolution by hand, by quiet expiry, or by pausing a check is treated the same way as any other state change for cross-channel sync purposes.

Every incident, in the Inbox and on its full page, has a Who was told, and why section. It starts with three boxes, left to right: the source it came from, what decided where it went — one of your rules, the integration’s built-in rules, the source’s rule for everything else, or Default — and how many destinations were told. The box that decided is highlighted.

Each box opens what it names: the source; a rule opens its editor; built-in rules or a rule for everything else opens the source’s Notifications tab. Every link brings you back to the incident.

Below the boxes, one row per destination (and per paired phone), from its delivery history: Sent with the time, Failed and why, Skipped and why, or Held by quiet hours, which says when quiet hours end. A held notice isn’t sent when they end; the next notice about the incident is. The destination’s name opens its page. An incident that began before InfraInbox recorded what decided says Not recorded in the middle box. See Triage in the inbox and History › Notices sent.

Every notice about an incident — Telegram, Slack, Discord, webhook or a phone push — carries a link straight back to this page, so opening one from any channel lands you here.