Glances threshold alerts
You’ll get: an incident when Glances sees memory, load or a disk cross its warning or critical threshold, one per host and check.
Before you start
Section titled “Before you start”- An ingest key from a generic InfraInbox source, shaped like
iik_src_…(see Events API: curl and scripts). - Glances running as a service on the host, with its configuration file (
/etc/glances/glances.conf).
1. Write the reporting script
Section titled “1. Write the reporting script”Glances runs an action as a plain command, not through a shell, so the JSON is built in a script:
#!/bin/sh# usage: infrainbox-glances <check> <warning|critical> <value> [item]INFRAINBOX_URL=https://infrainbox.example.com/v1/eventsINFRAINBOX_KEY=iik_src_…host=$(hostname)what="$1${4:+ $4}"curl -fsS -m 10 "$INFRAINBOX_URL" \ -H "Authorization: Bearer $INFRAINBOX_KEY" \ -H "Content-Type: application/json" \ -d "{\"title\":\"$host: $what at $3\",\"severity\":\"$2\",\"type\":\"glances.$1\",\"resource\":\"host:$host\",\"dedupKey\":\"glances:$host:$what\"}"sudo chmod 700 /usr/local/bin/infrainbox-glancesThe key is in the script, so keep it readable only by the user Glances runs as.
2. Add the actions to Glances
Section titled “2. Add the actions to Glances”[mem]warning=80critical=90warning_action=/usr/local/bin/infrainbox-glances mem warning {{percent}}%critical_action=/usr/local/bin/infrainbox-glances mem critical {{percent}}%
[load]critical=5.0critical_action=/usr/local/bin/infrainbox-glances load critical {{min5}}
[fs]critical=95critical_action=/usr/local/bin/infrainbox-glances fs critical {{percent}}% {{mnt_point}}Restart Glances to load it.
Check it works
Section titled “Check it works”Run the script by hand: /usr/local/bin/infrainbox-glances mem critical 99%. An incident opens on the source.
Limitations
Section titled “Limitations”- Glances has no action for the return to normal, so these incidents close after 24 hours without a repeat, or when you resolve them. Add
critical_action_repeat=truebeside acritical_actionto report on every refresh while it lasts, which keeps the incident open for as long as the problem does. - Glances runs the action on the host, so a host that is down reports nothing. Pair it with a heartbeat.
- Know when a host goes dark — the check this can’t do
- Routing rules — route the
glances.*types