Observability

Kubernetes Events as a Signal

1.The Event object

  • A namespaced API object that says why another object changed.
  • The API reference : "informative, best-effort, supplemental data". Reasons and messages "may evolve with time".
  • One stored object, two API views : core v1 and events.k8s.io/v1 (GA 1.19). v1beta1 is gone since 1.25.
  • Short name ev. Events about Nodes or PVs live in default.
core/v1events.k8s.io/v1Meaning
involved­Objectregardingthe object it is about
messagenotetext, max 1024 bytes if eventTime is set
reasonreasonCamelCase cause, max 128
typetypeNormal or Warning
reporting­Componentreporting­Controllerwho reported it (source is deprecated­Source)
countseries.counthow many repeats
last­Timestampseries.last­ObservedTimelast seen

2.Who writes Events

  • Legacy recorder (core/v1) : kubelet and kube-controller-manager. A repeat bumps count.
  • Newer recorder (events.k8s.io/v1) : kube-scheduler and kube-proxy. Repeats become a series.
  • Flux 2.9.x writes core/v1. Controllers record Events while they reconcile (who is who).
Not an audit logAny client with create on events can write one (RBAC). For "who did what", use the API server audit log.

3.Folded, throttled, dropped

kubelet / scheduler recorder (client-go) same event : bump count ~10 similar / 10 min : combined spam bucket : 25, then 1 per 5 min queue of 1000 : drop when full lost lost kube-apiserver validate EventRateLimit : alpha, off etcd lease 1h expired watchers kubectl events, exporters list / watch from etcd, no cache
WhatRecorder numbers
Legacy combinefrom the 10th similar message within a sliding 600 s, prefix (combined from similar events):
Legacy spamburst 25, then 1 per 300 s, per source, object and type
New series2nd repeat starts it; later counts written on a 30 min heartbeat or after 6 quiet min
Both1000-event queue per sink, 12 write tries
Server sideThe admission plugin EventRateLimit can limit Events per server, namespace, user or source and object. It is still alpha (since 1.13) and off by default.

4.Retention

  • --event-ttl defaults to 1h. It is an etcd lease set on every create and update, so an Event vanishes an hour after its last write.
  • "No events" on an old Pod means expired, not healthy.
  • No watch cache for Events : lists and watches hit etcd. Big clusters give Events their own etcd (--etcd-servers-overrides=/events#...).
Best effortNever build billing or audit on Events. A missing Event proves nothing.
In short : An Event is a best-effort note about why an object changed. Recorders fold and drop repeats before the API server, and the API server forgets each Event an hour after its last write.

1.Three ways to look

  • kubectl events (stable 1.28) : sorted by time, --for kind/name, --types=Warning, --watch.
  • kubectl get events is not sorted : add --sort-by=.lastTimestamp.
  • kubectl describe (1.36+) shows Events for one exactly named object only. Add --show-events for several.

2.Field selectors

  • A closed list : involvedObject.kind, .name, .namespace (and four more), reason, reportingComponent, source, type, metadata.namespace, metadata.name.
  • Operators =, ==, != only. A comma means AND.
  • Anything else, message=... included, returns "field label not supported".

3.Reasons worth knowing

ReasonFromTypeNearest metric
FailedSchedulingschedulerWarningkube_pod_status_unschedulable
Pulling / PulledkubeletNormalnone
Failed (start or pull)kubeletWarning..._waiting_reason
BackOffkubeletWarning (restart), Normal (image pull)..._waiting_reason (CrashLoopBackOff or ImagePullBackOff)
Unhealthykubelet proberWarningnone (readiness : kube_pod_status_ready)
FailedMountkubeletWarningnone
EvictedkubeletWarningkube_pod_status_reason{reason="Evicted"}
SystemOOMkubelet, on the NodeWarningnone
OOMKillingnode-problem-detector onlyWarning..._last_terminated_reason{reason="OOMKilled"}

Metrics come from kube-state-metrics. More on the reasons : scheduling, image pulls, probes, volume failures.

4.Reading a failure story

scheduler kubelet FailedScheduling Warning Scheduled Pulling Pulled Created Started Unhealthy x3 Killing BackOff x12 a healthy start stops at Started yellow = Warning

5.Traps

  • BackOff and Failed each mean two things : read the message. A type=Warning filter misses image-pull BackOff.
  • Scheduler Events read through core/v1 show count: null. Read series.count, which can lag up to 30 min.
  • Messages change between releases : 1.35 dropped the container name from Created / Started, 1.37 added preemption text to FailedScheduling. Match the reason, not the message.
CommandWhat it does
kubectl events -A --types=WarningAll warnings, sorted by time
kubectl events -n shop --for deployment/web --watchOne object, streaming
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestampWarnings, sorted
kubectl get events -n shop --field-selector involvedObject.kind=Pod,involvedObject.name=web-0One Pod
kubectl get events -A --field-selector reason=FailedSchedulingOne reason
kubectl describe pods -l app=web --show-eventsSeveral Pods (1.36+)
In short : Use kubectl events for a sorted view and field selectors to narrow it. Read the reason, not the message, and remember that some reasons mean two things and one of them is Normal.

1.Why export ?

The 1h TTL means no history. kube-state-metrics deliberately exports no Event metrics (high cardinality, "transition signals"). If you want Events tomorrow, ship them somewhere today.

2.Three exporters

kubernetes-event-exporter v1.7Alloy loki.source.kubernetes_eventsOTel k8s_events / k8s_objects
Maintenanceno release since 2024-02, forks, no clear successoractive, v1.20.1active, alpha / beta
Count updatesignored : first occurrence onlyre-sent with the new countnew and updated
Late eventsdropped after 5 s by defaultkeptkept
Restartstartup replay discardedpositions filestorage extension
One instance byleader electionsingle Deployment, or clustering (1.16+)k8s_leader_elector
Sinksabout 20 receiversLoki via loki.writeany OTel exporter

The two Collector receivers go in a deployment-mode Collector (OpenTelemetry Operator). k8s_objects was k8sobjects before v0.151.0.

3.One collector, never a DaemonSet

  • Each replica of a per-node agent that watches Events ships every Event.
  • Use one Deployment, leader election, or Alloy clustering.
  • Clustered on all namespaces, one Alloy peer collects : failover, not scale-out.
  • Alloy syntax : Loki and Alloy.
Alloy : all Events to Loki
loki.source.kubernetes_events "cluster" {
  job_name   = "integrations/kubernetes/eventhandler"
  log_format = "logfmt"
  clustering { enabled = true }
  forward_to = [loki.write.default.receiver]
}

4.Alerting : metric first, Event second

  • Prefer the state metric (Prometheus) : kube_pod_status_unschedulable == 1.
  • Events for causes with no status field : FailedMount, probe flaps, FailedScheduling detail, Flux errors.
  • Loki ruler (LogQL, Prometheus rule format) with a count threshold : Events are lossy.
  • No ruler.alertmanager_url, no alerts (Alertmanager).
Events API exporter / Alloy / OTel Loki Loki ruler needs alertmanager_url Alertmanager Slack Flux controller Event POST notification-controller drops repeats for 5 min Alert / Provider Slack
# ruler rule shape only, untested
count_over_time({job=...} | logfmt
  | type="Warning"
  | reason="FailedMount" [10m]) > 0
WantUse
CrashLoopBackOff..._waiting_reason
OOMKilled..._last_terminated_reason
why a volume failed to mountFailedMount Event

5.Flux events

  • Each Flux controller writes a Kubernetes Event and POSTs it to notification-controller.
  • That drops repeats for 5 min, then sends via Alert, Provider : notification.toolkit.fluxcd.io/v1beta3.
  • Normal is info, Warning is error. Read : flux events --for Kustomization/<name>.
Seen in practicePitfall : one exporter prints Events as JSON to its Pod log for a collector to ship. Its 5 s age limit plus throttled lookups can quietly discard more Events than it sends, and nothing alerts on them. Do better : alert on events_discarded, raise the age limit and QPS.
In short : To keep Events, export them with one collector into a log store. Alert on the state metric first, and on exported Events with a count threshold when the reason is the only signal.