# Glances threshold alerts

> A small script Glances runs when memory, load or a disk crosses its critical threshold.

Web page: https://infrainbox.app/docs/cookbook/glances-thresholds/

**You'll get:** an incident when [Glances](https://nicolargo.github.io/glances/) sees memory, load or a disk cross its warning or critical threshold, one per host and check.

## Before you start

- An ingest key from a generic InfraInbox source, shaped like `iik_src_…` (see [Events API: curl and scripts](https://infrainbox.app/docs/sources/events-api.md)).
- Glances running as a service on the host, with its configuration file (`/etc/glances/glances.conf`).

## 1. Write the reporting script

Glances runs an action as a plain command, not through a shell, so the JSON is built in a script:

```bash
#!/bin/sh
# /usr/local/bin/infrainbox-glances
# usage: infrainbox-glances <check> <warning|critical> <value> [item]
INFRAINBOX_URL=https://infrainbox.example.com/v1/events
INFRAINBOX_KEY=iik_src_…
host=$(hostname)
what="$1${4:+ $4}"
curl -fsS -m 10 "$INFRAINBOX_URL" \
  -H "Authorization: Bearer $INFRAINBOX_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"title\":\"$host: $what at $3\",\"severity\":\"$2\",\"type\":\"glances.$1\",\"resource\":\"host:$host\",\"dedupKey\":\"glances:$host:$what\"}"
```

```bash
sudo chmod 700 /usr/local/bin/infrainbox-glances
```

The key is in the script, so keep it readable only by the user Glances runs as.

## 2. Add the actions to Glances

```ini
# /etc/glances/glances.conf
[mem]
warning=80
critical=90
warning_action=/usr/local/bin/infrainbox-glances mem warning {{percent}}%
critical_action=/usr/local/bin/infrainbox-glances mem critical {{percent}}%

[load]
critical=5.0
critical_action=/usr/local/bin/infrainbox-glances load critical {{min5}}

[fs]
critical=95
critical_action=/usr/local/bin/infrainbox-glances fs critical {{percent}}% {{mnt_point}}
```

Restart Glances to load it.

## Check it works

Run the script by hand: `/usr/local/bin/infrainbox-glances mem critical 99%`. An incident opens on the source.

## Limitations

- Glances has no action for the return to normal, so these incidents close after 24 hours without a repeat, or when you resolve them. Add `critical_action_repeat=true` beside a `critical_action` to report on every refresh while it lasts, which keeps the incident open for as long as the problem does.
- Glances runs the action on the host, so a host that is down reports nothing. Pair it with a [heartbeat](https://infrainbox.app/docs/cookbook/host-down.md).

## Next

- [Know when a host goes dark](https://infrainbox.app/docs/cookbook/host-down.md) — the check this can't do
- [Routing rules](https://infrainbox.app/docs/notifications/routing.md) — route the `glances.*` types
