Observability
Kubernetes Events as a Signal
1.The Event object
- A namespaced API object that says why another object changed.
- The API reference : "informative, best-effort, supplemental data". Reasons and messages "may evolve with time".
- One stored object, two API views : core
v1andevents.k8s.io/v1(GA 1.19).v1beta1is gone since 1.25. - Short name
ev. Events about Nodes or PVs live indefault.
| core/v1 | events. | Meaning |
|---|---|---|
involvedObject | regarding | the object it is about |
message | note | text, max 1024 bytes if eventTime is set |
reason | reason | CamelCase cause, max 128 |
type | type | Normal or Warning |
reportingComponent | reportingController | who reported it (source is deprecatedSource) |
count | series.count | how many repeats |
lastTimestamp | series. | last seen |
2.Who writes Events
- Legacy recorder (core/v1) : kubelet and kube-controller-manager. A repeat bumps
count. - Newer recorder (events.k8s.io/v1) : kube-scheduler and kube-proxy. Repeats become a
series. - Flux 2.9.x writes core/v1. Controllers record Events while they reconcile (who is who).
Not an audit logAny client with
create on events can write one (RBAC). For "who did what", use the API server audit log.3.Folded, throttled, dropped
| What | Recorder numbers |
|---|---|
| Legacy combine | from the 10th similar message within a sliding 600 s, prefix (combined from similar events): |
| Legacy spam | burst 25, then 1 per 300 s, per source, object and type |
| New series | 2nd repeat starts it; later counts written on a 30 min heartbeat or after 6 quiet min |
| Both | 1000-event queue per sink, 12 write tries |
Server sideThe admission plugin
EventRateLimit can limit Events per server, namespace, user or source and object. It is still alpha (since 1.13) and off by default.4.Retention
--event-ttldefaults to 1h. It is an etcd lease set on every create and update, so an Event vanishes an hour after its last write.- "No events" on an old Pod means expired, not healthy.
- No watch cache for Events : lists and watches hit etcd. Big clusters give Events their own etcd (
--etcd-servers-overrides=/).events#...
Best effortNever build billing or audit on Events. A missing Event proves nothing.
In short : An Event is a best-effort note about why an object changed. Recorders fold and drop repeats before the API server, and the API server forgets each Event an hour after its last write.
1.Three ways to look
kubectl events(stable 1.28) : sorted by time,--for kind/name,--types=Warning,--watch.kubectl get eventsis not sorted : add--sort-by=.lastTimestamp.kubectl describe(1.36+) shows Events for one exactly named object only. Add--show-eventsfor several.
2.Field selectors
- A closed list :
involvedObject.,kind .name,.namespace(and four more),reason,reportingComponent,source,type,metadata.,namespace metadata.name. - Operators
=,==,!=only. A comma means AND. - Anything else,
message=...included, returns "field label not supported".
3.Reasons worth knowing
| Reason | From | Type | Nearest metric |
|---|---|---|---|
FailedScheduling | scheduler | Warning | kube_ |
Pulling / Pulled | kubelet | Normal | none |
Failed (start or pull) | kubelet | Warning | ..._ |
BackOff | kubelet | Warning (restart), Normal (image pull) | ..._ (CrashLoopBackOff or ImagePullBackOff) |
Unhealthy | kubelet prober | Warning | none (readiness : kube_) |
FailedMount | kubelet | Warning | none |
Evicted | kubelet | Warning | kube_ |
SystemOOM | kubelet, on the Node | Warning | none |
OOMKilling | node-problem-detector only | Warning | ..._ |
Metrics come from kube-state-metrics. More on the reasons : scheduling, image pulls, probes, volume failures.
4.Reading a failure story
5.Traps
BackOffandFailedeach mean two things : read the message. Atype=Warningfilter misses image-pullBackOff.- Scheduler Events read through core/v1 show
count: null. Readseries.count, which can lag up to 30 min. - Messages change between releases : 1.35 dropped the container name from
Created/Started, 1.37 added preemption text toFailedScheduling. Match the reason, not the message.
| Command | What it does |
|---|---|
kubectl events -A --types=Warning | All warnings, sorted by time |
kubectl events -n shop --for deployment/web --watch | One object, streaming |
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | Warnings, sorted |
kubectl get events -n shop --field-selector involvedObject. | One Pod |
kubectl get events -A --field-selector reason=FailedScheduling | One reason |
kubectl describe pods -l app=web --show-events | Several Pods (1.36+) |
In short : Use
kubectl events for a sorted view and field selectors to narrow it. Read the reason, not the message, and remember that some reasons mean two things and one of them is Normal.1.Why export ?
The 1h TTL means no history. kube-state-metrics deliberately exports no Event metrics (high cardinality, "transition signals"). If you want Events tomorrow, ship them somewhere today.
2.Three exporters
| kubernetes- | Alloy loki.source. | OTel k8s_events / k8s_objects | |
|---|---|---|---|
| Maintenance | no release since 2024-02, forks, no clear successor | active, v1.20.1 | active, alpha / beta |
| Count updates | ignored : first occurrence only | re-sent with the new count | new and updated |
| Late events | dropped after 5 s by default | kept | kept |
| Restart | startup replay discarded | positions file | storage extension |
| One instance by | leader election | single Deployment, or clustering (1.16+) | k8s_ |
| Sinks | about 20 receivers | Loki via loki.write | any OTel exporter |
The two Collector receivers go in a deployment-mode Collector (OpenTelemetry Operator). k8s_objects was k8sobjects before v0.151.0.
3.One collector, never a DaemonSet
- Each replica of a per-node agent that watches Events ships every Event.
- Use one Deployment, leader election, or Alloy clustering.
- Clustered on all namespaces, one Alloy peer collects : failover, not scale-out.
- Alloy syntax : Loki and Alloy.
Alloy : all Events to Loki
loki.source.kubernetes_events "cluster" {
job_name = "integrations/kubernetes/eventhandler"
log_format = "logfmt"
clustering { enabled = true }
forward_to = [loki.write.default.receiver]
}
4.Alerting : metric first, Event second
- Prefer the state metric (Prometheus) :
kube_.pod_ status_ unschedulable == 1 - Events for causes with no status field :
FailedMount, probe flaps,FailedSchedulingdetail, Flux errors. - Loki ruler (LogQL, Prometheus rule format) with a count threshold : Events are lossy.
- No
ruler., no alerts (Alertmanager).alertmanager_ url
# ruler rule shape only, untested
count_over_time({job=...} | logfmt
| type="Warning"
| reason="FailedMount" [10m]) > 0
| Want | Use |
|---|---|
| CrashLoopBackOff | ..._ |
| OOMKilled | ..._ |
| why a volume failed to mount | FailedMount Event |
5.Flux events
- Each Flux controller writes a Kubernetes Event and POSTs it to notification-controller.
- That drops repeats for 5 min, then sends via
Alert,Provider:notification..toolkit. fluxcd. io/ v1beta3 - Normal is
info, Warning iserror. Read :flux events --for Kustomization/<name>.
Seen in practicePitfall : one exporter prints Events as JSON to its Pod log for a collector to ship. Its 5 s age limit plus throttled lookups can quietly discard more Events than it sends, and nothing alerts on them. Do better : alert on
events_discarded, raise the age limit and QPS.In short : To keep Events, export them with one collector into a log store. Alert on the state metric first, and on exported Events with a count threshold when the reason is the only signal.