Observability
Alertmanager Routing
1.Two halves
- Prometheus evaluates alerting rules and sends the firing alerts. Loki's ruler can send too (Kubernetes Events as a Signal).
- Alertmanager deduplicates, groups, inhibits, silences and routes them to receivers (Slack, PagerDuty, webhook, ...).
RememberAlertmanager never runs PromQL. If an alert is wrong, fix the rule in Prometheus.
2.Alerting rules
| Field | Default | Meaning |
|---|---|---|
expr | required | PromQL; one alert per output series |
for | 0s | time in pending before firing |
keep_firing_for | 0s | keep firing after it clears (anti-flap); 2.42.0+ |
labels | none | identity and routing keys |
annotations | none | text; may change freely |
- No
$valuein labels : each evaluation is a new alert, sofornever completes. foronincrease(x[15m]) > 0is not "bad for that long" : one failure holds it true.
- alert: PodCrashLooping
expr: increase(
kube_pod_container_status_restarts_total[15m])
> 3
for: 5m
keep_firing_for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.namespace }}/\
{{ $labels.pod }} restarting"
On Kubernetes it lives in a PrometheusRule, selected by ruleSelector (Prometheus and ServiceMonitors). Restarts : kube-state-metrics.
3.The send loop
- Resent at most every 1m (
--rules.) withalert. resend-delay EndsAt = now + 4 x max(eval interval, resend-delay). - Pending is never sent; resolved is resent for 15m. Send to every replica, never via a load balancer.
TrapNo more sends, so alerts resolve as
EndsAt passes : a dead Prometheus looks like "all clear". A Watchdog catches it.4.Inside Alertmanager
- The dispatcher runs once per alert. A new group waits
group_wait. - Each flush runs the notifier stages in order. Inhibited and silenced alerts are dropped there (red).
- A sent group goes into the nflog, so other replicas dedup against it.
5.The three timers
| Timer | Default | What it does |
|---|---|---|
group_wait | 30s | delay the first message of a new group so related alerts join |
group_interval | 5m | pace for updates; also the send timeout, max( plus the peer wait |
repeat_interval | 4h | resend an unchanged group; rounded up to a multiple of group_interval |
amtool check-config says SUCCESS for repeat_interval < group_interval; only the server logs a warning at load. resolve_timeout does nothing for Prometheus alerts (they always carry EndsAt).
In short : Prometheus fires and keeps re-sending alerts. Alertmanager groups them by labels, waits
group_wait, updates every group_interval, repeats every repeat_interval.1.The routing tree
- Every alert enters the root
route: no matchers allowed, areceiverrequired. - It walks the children depth-first and stops at the first match, unless that child sets
continue: true. No child matches : the current node handles it. - Unset settings inherit from the parent.
group_by: ['...']means every label (no grouping). - First match wins : a catch-all above specific routes eats everything.
2.Matchers
- Operators
=,!=,=~,!~.matchers:replaces deprecatedmatchandmatch_re(still parsed). - RE2 regexes are anchored :
=~ "prod"missesproduction. - In YAML, quote a matcher holding
{ } [ ] , # | > : - Since 0.27 a UTF-8 parser runs in fallback mode; strict mode is opt-in (
utf8-strict-mode).
3.Inhibition and silences
- Inhibition mutes target alerts while a source alert fires and the
equallabels match. A label missing on both sides counts as equal. - The source must arrive first, so a very short
group_waitcan leak the target. - A silence is matchers plus a time window, made with
amtoolor the UI. Silences and the notification log are saved to disk every 15m and kept 120h.
| Way to mute | Trigger | Lifetime | Use for |
|---|---|---|---|
| Inhibit rule | another alert firing | config | critical mutes warning, node down mutes its Pods |
| Silence | matchers, by hand | start and end time, saved | maintenance, known issues |
mute_ | the clock | config | quiet hours on a route |
null receiver | route match | config | drop on purpose (InfoInhibitor) |
4.Receivers
- A receiver holds integrations; one with none is a
nullreceiver. - No RESOLVED in Slack or email unless
send_resolved: true. *_filefields (api_url_file,routing_key_file) read a mounted Secret (ConfigMaps and Secrets).- Go text templates (
.CommonLabels,.Alerts.Firing) :check-configparses but never runs them, so render them.
| Integration | send_resolved default |
|---|---|
| Slack | false |
| false | |
| webhook | true |
| PagerDuty | true |
Rocket.Chat is false in code, though the docs say true.
| Command | What it does |
|---|---|
amtool check-config alertmanager.yml | parse the config, print SUCCESS |
amtool config routes test --config. | receivers a label set reaches |
amtool alert add TestAlert severity=warning --annotation=summary=test | inject a test alert |
amtool silence add alertname=X -d 2h -c "maintenance" | silence for 2h (default 1h) |
amtool silence expire $(amtool silence query -q alertname=X) | end those silences now |
amtool template render --template. | render against example data |
In short : Labels decide everything : the first matching route picks the receiver, inhibit rules and silences mute by matchers, and
amtool tests a tree before any rule fires.1.Operator objects
AlertmanagerCR (CRDs) : StatefulSetalertmanager-<name>, Servicealertmanager-operated(web9093,tcp-meshandudp-mesh9094). One replica : gossip off.- Config : Secret
alertmanager-<name>or AlertmanagerConfigs, merged intoalertmanager-<name>-generated(Operators). - AlertmanagerConfig is
v1alpha1(API Machinery), validated by a webhook (Admission). Namespace selector nil = own namespace.
Operator traps
- Operator v0.94.1 defaults to Alertmanager v0.34.0, the inhibition un-mute bug. Set
spec.version. limits.passes a flag Alertmanager lacks : crash-loop.maxPerSilenceBytes
2.Team routes : AlertmanagerConfig
- The operator appends
namespace="<ns>"to each top route and both inhibit sides (removes nothing from routes), forcescontinue: true, renames receivers, prepends the routes. - Alerts with no
namespace(node, cluster-wide) never reach a team route.
| Strategy | OnNamespace (default) | OnNamespace | None |
|---|---|---|---|
| Effect | namespace matcher added | same, except objects in Alertmanager's own namespace see all alerts | nothing injected |
3.High availability
- Replicas gossip silences and the nflog on 9094, TCP and UDP (TCP only with
--cluster.).tls-config - Each waits
position x 15sand skips if a peer already sent. Broken gossip fails open : duplicates. - Allow 9093 from Prometheus, 9094 TCP and UDP between replicas (Network Policy in Depth).
4.Watch the watcher
Watchdogalways fires (vector(1)). Route it to a heartbeat service : when pings stop, the pipeline is dead.- Both distributions ship it unwired. Set
group_intervalon its route too, orrepeat_interval: 1mbecomes 5m. - Self-monitoring :
alertmanager_,notifications_ failed_ total{reason} alertmanager_,config_ last_ reload_ successful alertmanager_; on Prometheus,notifications_ suppressed_ total prometheus_.notifications_ alertmanagers_ discovered
| Setting | kube-prometheus v0.19.0 | kube-prometheus-stack 92.1.1 |
|---|---|---|
| replicas | 3 | 1 (emptyDir) |
root group_by | [namespace] | ['namespace'] |
repeat_interval | 12h | 12h |
| receiver | Default | 'null' |
| Watchdog | empty receiver | routed to null |
| Command | Does |
|---|---|
kubectl -n monitoring get secret alertmanager-main-generated -o jsonpath='{. | merged config |
amtool cluster show | gossip status |
curl -XPOST localhost:9093/-/ | reload |
Seen in practiceA plain Deployment, a PVC for silences and nflog, Slack receivers by severity, URLs from a SOPS Secret (Secrets in Git with SOPS) via Flux CD v2. Pitfalls : no Watchdog, nothing scrapes Alertmanager, rules set
runbook but the template prints runbook_url. Do better : a Watchdog to a dead man's switch, scrape it, one label name everywhere.In short : The operator runs Alertmanager as a StatefulSet and merges team AlertmanagerConfigs scoped to their namespace. Run replicas, send to all of them, wire a Watchdog.