Scaling and metrics

kube-state-metrics

1.What is kube-state-metrics (KSM) ?

A small service that watches the Kubernetes API and turns the state of objects (Pods, Deployments, Nodes...) into Prometheus metrics on /metrics.

  • Metrics come from object state, not usage
  • Read only : needs just list + watch RBAC
  • Stateless Go binary, keeps objects in memory
  • Metrics on :8080, self metrics + /readyz on :8081
NoteKSM never writes to the cluster. It only reads.
Kubernetes API list + watch kube-state-metrics(in-memory store) gauges /metrics :8080 scrape ~30s Prometheus Alertmanager Grafana

2.Why do we need it ?

  • metrics-server gives only live CPU / memory
  • kubectl shows state only right now, no history
  • We want alerts like CrashLooping Pod or PVC Pending

Fix : KSM exposes object state as time series that Prometheus can scrape, graph and alert on. It shows raw API data, so values match the objects.

Key Points
  • Sits in kube-system or monitoring
  • Runs as Deployment + Service + RBAC
  • Pods, Deployments, Nodes, Jobs, PVCs, HPAs...
  • Secrets : metadata only, never the data
  • Ships no CRDs of its own

3.KSM vs metrics-server

Usage vs state. They complement each other, not replace.

kube-state-metricsmetrics-server
AnswersWhat is the object state?How much CPU / memory now?
SourceKubernetes API objectsKubelet metrics endpoint
OutputPrometheus /metricsmetrics.k8s.io API
Used byAlerts, dashboardskubectl top, HPA, VPA
HistoryVia PrometheusNone, latest only
Usage kubelet cpu, mem metrics-server usage kubectl top/ HPA State API server objects kube-state-metrics kube_* Prometheusalerts

4.Sharding and Scale

One instance holds all watched objects in memory, so memory grows with the cluster.

  • --shard=N --total-shards=M, total 1 means off
  • Auto sharding : a StatefulSet, ordinal = shard index
  • Scrape filter : /metrics?resources=pods
  • 2 replicas without sharding = duplicate metrics
API server slice by UID hash shard 0KSM shard 1KSM shard 2KSM one perslice
In short : KSM reads the API and exposes object state as Prometheus metrics. Pair it with metrics-server for usage, and shard it when the cluster grows.

5.Handy Metrics Cheat Sheet

MetricMeaning
kube_pod_status_phasePod phase (Pending / Running / Failed...)
kube_pod_container_status_restarts_totalRestart counter, CrashLoop detection
kube_pod_container_status_waiting_reasonWhy waiting (e.g. CrashLoopBackOff)
kube_deployment_spec_replicasDesired replicas
kube_deployment_status_replicas_availableAvailable replicas (lower than desired = stuck rollout)
kube_node_status_conditionNode Ready, MemoryPressure...
kube_persistentvolumeclaim_status_phasePVC Bound / Pending
kube_*_labelsInfo metric with K8s labels (allowlisted ones only)

6.Gotchas

  • Labels missing ? Set --metric-labels-allowlist (e.g. pods=[app,team])
  • No CPU / memory here, use metrics-server or cAdvisor
  • Custom Resource State is feature-frozen, needs RBAC on the CRD and the custom resource
  • Big cluster ? Use shards, --resources, namespace filters
  • Helm chart lives in prometheus-community, versioned separately
Custom Resource State
kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        version: v1
        kind: Foo
      metrics:
        - name: active_count
          each:
            type: Gauge
            gauge:
              path: [status, uptime]

Passed with --custom-resource-state-config-file.

7.KSM Commands

CommandDescriptionExample
helm install kube-state-metrics prometheus-community/kube-state-metrics -n monitoring --create-namespaceInstall with the prometheus-community charthelm repo add prometheus-community https://prometheus-community.github.io/helm-charts
kubectl apply -f examples/standardInstall from the repo manifests (uses kube-system)git clone https://github.com/kubernetes/kube-state-metrics
kubectl -n monitoring port-forward svc/kube-state-metrics 8080:8080Reach the metrics endpoint locallycurl -s localhost:8080/metrics | grep kube_deployment_status_replicas
curl 'localhost:8080/metrics?resources=pods'Scrape only some resource typescurl -s 'localhost:8080/metrics?resources=pods,secrets' | head
kubectl -n monitoring logs deploy/kube-state-metricsDebug RBAC forbidden errors on list / watchkubectl auth can-i list pods --as=system:serviceaccount:monitoring:kube-state-metrics
kubectl -n monitoring get pod -l app.kubernetes.io/name=kube-state-metricsCheck the pod, /readyz on :8081 is the probecurl -s localhost:8081/readyz
In short : Watch the kube_* metrics for restarts, replicas and PVC phase, tune labels and shards in the flags, and use the commands above to install and debug KSM.