Scaling and metrics
kube-state-metrics
1.What is kube-state-metrics (KSM) ?
A small service that watches the Kubernetes API and turns the state of objects (Pods, Deployments, Nodes...) into Prometheus metrics on /metrics.
- Metrics come from object state, not usage
- Read only : needs just list + watch RBAC
- Stateless Go binary, keeps objects in memory
- Metrics on
:8080, self metrics +/readyzon:8081
NoteKSM never writes to the cluster. It only reads.
2.Why do we need it ?
- metrics-server gives only live CPU / memory
kubectlshows state only right now, no history- We want alerts like CrashLooping Pod or PVC Pending
Fix : KSM exposes object state as time series that Prometheus can scrape, graph and alert on. It shows raw API data, so values match the objects.
Key Points
- Sits in kube-system or monitoring
- Runs as Deployment + Service + RBAC
- Pods, Deployments, Nodes, Jobs, PVCs, HPAs...
- Secrets : metadata only, never the data
- Ships no CRDs of its own
3.KSM vs metrics-server
Usage vs state. They complement each other, not replace.
4.Sharding and Scale
One instance holds all watched objects in memory, so memory grows with the cluster.
--shard=N --total-shards=M, total 1 means off- Auto sharding : a StatefulSet, ordinal = shard index
- Scrape filter :
/metrics?resources=pods - 2 replicas without sharding = duplicate metrics
In short : KSM reads the API and exposes object state as Prometheus metrics. Pair it with metrics-server for usage, and shard it when the cluster grows.
5.Handy Metrics Cheat Sheet
| Metric | Meaning |
|---|---|
| kube_ | Pod phase (Pending / Running / Failed...) |
| kube_ | Restart counter, CrashLoop detection |
| kube_ | Why waiting (e.g. CrashLoopBackOff) |
| kube_ | Desired replicas |
| kube_ | Available replicas (lower than desired = stuck rollout) |
| kube_ | Node Ready, MemoryPressure... |
| kube_ | PVC Bound / Pending |
| kube_ | Info metric with K8s labels (allowlisted ones only) |
6.Gotchas
- Labels missing ? Set
--metric-labels-allowlist(e.g. pods=[app,team]) - No CPU / memory here, use metrics-server or cAdvisor
- Custom Resource State is feature-frozen, needs RBAC on the CRD and the custom resource
- Big cluster ? Use shards,
--resources, namespace filters - Helm chart lives in prometheus-community, versioned separately
Custom Resource State
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
group: myteam.io
version: v1
kind: Foo
metrics:
- name: active_count
each:
type: Gauge
gauge:
path: [status, uptime]
Passed with --custom-resource-state-config-file.
7.KSM Commands
| Command | Description | Example |
|---|---|---|
| helm install kube-state-metrics prometheus-community/kube-state-metrics -n monitoring --create-namespace | Install with the prometheus-community chart | helm repo add prometheus-community https://prometheus-community.github.io/helm-charts |
| kubectl apply -f examples/standard | Install from the repo manifests (uses kube-system) | git clone https://github.com/kubernetes/kube-state-metrics |
| kubectl -n monitoring port-forward svc/kube-state-metrics 8080:8080 | Reach the metrics endpoint locally | curl -s localhost:8080/metrics | grep kube_deployment_status_replicas |
| curl 'localhost:8080/metrics?resources=pods' | Scrape only some resource types | curl -s 'localhost:8080/ |
| kubectl -n monitoring logs deploy/kube-state-metrics | Debug RBAC forbidden errors on list / watch | kubectl auth can-i list pods --as=system:serviceaccount:monitoring:kube-state-metrics |
| kubectl -n monitoring get pod -l app.kubernetes.io/name=kube-state-metrics | Check the pod, /readyz on :8081 is the probe | curl -s localhost:8081/readyz |
In short : Watch the kube_* metrics for restarts, replicas and PVC phase, tune labels and shards in the flags, and use the commands above to install and debug KSM.