Observability

Alertmanager Routing

1.Two halves

  • Prometheus evaluates alerting rules and sends the firing alerts. Loki's ruler can send too (Kubernetes Events as a Signal).
  • Alertmanager deduplicates, groups, inhibits, silences and routes them to receivers (Slack, PagerDuty, webhook, ...).
RememberAlertmanager never runs PromQL. If an alert is wrong, fix the rule in Prometheus.

2.Alerting rules

FieldDefaultMeaning
exprrequiredPromQL; one alert per output series
for0stime in pending before firing
keep_firing_for0skeep firing after it clears (anti-flap); 2.42.0+
labelsnoneidentity and routing keys
annotationsnonetext; may change freely
  • No $value in labels : each evaluation is a new alert, so for never completes.
  • for on increase(x[15m]) > 0 is not "bad for that long" : one failure holds it true.
- alert: PodCrashLooping
  expr: increase(
    kube_pod_container_status_restarts_total[15m])
    > 3
  for: 5m
  keep_firing_for: 10m
  labels: { severity: warning }
  annotations:
    summary: "{{ $labels.namespace }}/\
      {{ $labels.pod }} restarting"

On Kubernetes it lives in a PrometheusRule, selected by ruleSelector (Prometheus and ServiceMonitors). Restarts : kube-state-metrics.

3.The send loop

  • Resent at most every 1m (--rules.alert.resend-delay) with EndsAt = now + 4 x max(eval interval, resend-delay).
  • Pending is never sent; resolved is resent for 15m. Send to every replica, never via a load balancer.
TrapNo more sends, so alerts resolve as EndsAt passes : a dead Prometheus looks like "all clear". A Watchdog catches it.

4.Inside Alertmanager

dispatcher, once per alert alert in route walk tree group group_by wait group_wait each flush gossipsettle inhibitedyes: drop timeintervals silencedyes: drop peerwait dedup(nflog) sendretry receiver
  • The dispatcher runs once per alert. A new group waits group_wait.
  • Each flush runs the notifier stages in order. Inhibited and silenced alerts are dropped there (red).
  • A sent group goes into the nflog, so other replicas dedup against it.

5.The three timers

TimerDefaultWhat it does
group_wait30sdelay the first message of a new group so related alerts join
group_interval5mpace for updates; also the send timeout, max(group_interval, 10s) plus the peer wait
repeat_interval4hresend an unchanged group; rounded up to a multiple of group_interval

amtool check-config says SUCCESS for repeat_interval < group_interval; only the server logs a warning at load. resolve_timeout does nothing for Prometheus alerts (they always carry EndsAt).

In short : Prometheus fires and keeps re-sending alerts. Alertmanager groups them by labels, waits group_wait, updates every group_interval, repeats every repeat_interval.

1.The routing tree

  • Every alert enters the root route : no matchers allowed, a receiver required.
  • It walks the children depth-first and stops at the first match, unless that child sets continue: true. No child matches : the current node handles it.
  • Unset settings inherit from the parent. group_by: ['...'] means every label (no grouping).
  • First match wins : a catch-all above specific routes eats everything.
alert severity=critical, namespace=shop root route receiver: default group_by: [namespace, alertname] 1st 2nd 3rd alertname = Watchdog alertname = InfoInhibitor severity = critical heartbeat "null" pager no match : skip, match : stop

2.Matchers

  • Operators =, !=, =~, !~. matchers: replaces deprecated match and match_re (still parsed).
  • RE2 regexes are anchored : =~ "prod" misses production.
  • In YAML, quote a matcher holding { } [ ] , # | > :
  • Since 0.27 a UTF-8 parser runs in fallback mode; strict mode is opt-in (utf8-strict-mode).

3.Inhibition and silences

  • Inhibition mutes target alerts while a source alert fires and the equal labels match. A label missing on both sides counts as equal.
  • The source must arrive first, so a very short group_wait can leak the target.
  • A silence is matchers plus a time window, made with amtool or the UI. Silences and the notification log are saved to disk every 15m and kept 120h.
Way to muteTriggerLifetimeUse for
Inhibit ruleanother alert firingconfigcritical mutes warning, node down mutes its Pods
Silencematchers, by handstart and end time, savedmaintenance, known issues
mute_time_intervalsthe clockconfigquiet hours on a route
null receiverroute matchconfigdrop on purpose (InfoInhibitor)

4.Receivers

  • A receiver holds integrations; one with none is a null receiver.
  • No RESOLVED in Slack or email unless send_resolved: true.
  • *_file fields (api_url_file, routing_key_file) read a mounted Secret (ConfigMaps and Secrets).
  • Go text templates (.CommonLabels, .Alerts.Firing) : check-config parses but never runs them, so render them.
Integrationsend_resolved default
Slackfalse
emailfalse
webhooktrue
PagerDutytrue

Rocket.Chat is false in code, though the docs say true.

CommandWhat it does
amtool check-config alertmanager.ymlparse the config, print SUCCESS
amtool config routes test --config.file=am.yml --verify.receivers=pager severity=criticalreceivers a label set reaches
amtool alert add TestAlert severity=warning --annotation=summary=testinject a test alert
amtool silence add alertname=X -d 2h -c "maintenance"silence for 2h (default 1h)
amtool silence expire $(amtool silence query -q alertname=X)end those silences now
amtool template render --template.glob='t/*.tmpl' --template.text='{{ template "slack.default.title" . }}'render against example data
In short : Labels decide everything : the first matching route picks the receiver, inhibit rules and silences mute by matchers, and amtool tests a tree before any rule fires.

1.Operator objects

  • Alertmanager CR (CRDs) : StatefulSet alertmanager-<name>, Service alertmanager-operated (web 9093, tcp-mesh and udp-mesh 9094). One replica : gossip off.
  • Config : Secret alertmanager-<name> or AlertmanagerConfigs, merged into alertmanager-<name>-generated (Operators).
  • AlertmanagerConfig is v1alpha1 (API Machinery), validated by a webhook (Admission). Namespace selector nil = own namespace.
Operator traps
  • Operator v0.94.1 defaults to Alertmanager v0.34.0, the inhibition un-mute bug. Set spec.version.
  • limits.maxPerSilenceBytes passes a flag Alertmanager lacks : crash-loop.

2.Team routes : AlertmanagerConfig

  • The operator appends namespace="<ns>" to each top route and both inhibit sides (removes nothing from routes), forces continue: true, renames receivers, prepends the routes.
  • Alerts with no namespace (node, cluster-wide) never reach a team route.
AlertmanagerConfig ns team-a severity=critical to team-a-slack operator append namespace, force continue, rename generated route severity="critical", namespace="team-a" continue: true, receiver team-a/team-a/team-a-slack placed before the main routes
StrategyOnNamespace (default)OnNamespaceExceptForAlertmanagerNamespaceNone
Effectnamespace matcher addedsame, except objects in Alertmanager's own namespace see all alertsnothing injected

3.High availability

  • Replicas gossip silences and the nflog on 9094, TCP and UDP (TCP only with --cluster.tls-config).
  • Each waits position x 15s and skips if a peer already sent. Broken gossip fails open : duplicates.
  • Allow 9093 from Prometheus, 9094 TCP and UDP between replicas (Network Policy in Depth).
Prometheus am-0wait 0s am-1wait 15s am-2wait 30s gossip :9094 Slack nflog : already sent nflog : already sent

4.Watch the watcher

  • Watchdog always fires (vector(1)). Route it to a heartbeat service : when pings stop, the pipeline is dead.
  • Both distributions ship it unwired. Set group_interval on its route too, or repeat_interval: 1m becomes 5m.
  • Self-monitoring : alertmanager_notifications_failed_total{reason}, alertmanager_config_last_reload_successful, alertmanager_notifications_suppressed_total; on Prometheus, prometheus_notifications_alertmanagers_discovered.
Settingkube-prometheus
v0.19.0
kube-prometheus-stack
92.1.1
replicas31 (emptyDir)
root group_by[namespace]['namespace']
repeat_interval12h12h
receiverDefault'null'
Watchdogempty receiverrouted to null
CommandDoes
kubectl -n monitoring get secret alertmanager-main-generated -o jsonpath='{.data.alertmanager\.yaml\.gz}' | base64 -d | gunzipmerged config
amtool cluster showgossip status
curl -XPOST localhost:9093/-/reloadreload
Seen in practiceA plain Deployment, a PVC for silences and nflog, Slack receivers by severity, URLs from a SOPS Secret (Secrets in Git with SOPS) via Flux CD v2. Pitfalls : no Watchdog, nothing scrapes Alertmanager, rules set runbook but the template prints runbook_url. Do better : a Watchdog to a dead man's switch, scrape it, one label name everywhere.
In short : The operator runs Alertmanager as a StatefulSet and merges team AlertmanagerConfigs scoped to their namespace. Run replicas, send to all of them, wire a Watchdog.