Scaling and metrics

metrics-server

1.What is Metrics Server ?

Metrics Server asks every kubelet for current CPU and memory use and keeps it in memory. It serves the numbers through the Metrics API. That is what makes kubectl top and the HPA work.

  • Scrapes kubelets about every 15s
  • Keeps only the latest sample (no history)
  • Only CPU and memory, for Nodes and Pods
In Simple Words Metrics Server tells Kubernetes how much CPU and memory things use right now.
Built for autoscaling, not for monitoring

2.Why it exists ?

Kubernetes core does not ship a source of live Pod and Node usage.

Scraping every kubelet yourself means each consumer redoes auth, discovery and aggregation. Metrics Server is the reference implementation of the Metrics API.

Without it
  • kubectl top fails : "Metrics API not available"
  • HPA shows <unknown> targets
  • No CPU / memory for autoscaling
  • VPA recommender has no live data

3.Resource Metrics Pipeline

Metrics Server pulls from the kubelet. Consumers only talk to the API server, never to kubelets.

  • cAdvisorReads cgroup stats per container (inside kubelet).
  • kubeletExposes /metrics/resource.
  • APIServicev1beta1.metrics.k8s.io points the aggregator at the Service.
  • Servicemetrics-server in kube-system, port 443.
cAdvisor stats kubelet scrape ~15s metrics-serverlatest sample in memory proxied via APIService kube-apiserver (aggregator) kubectl top HPA / VPA

4.kubectl top pods : Request Flow

GET metrics.k8s.io/v1beta1 pods
kubectl topUser asks
kube-apiserverMatches group metrics.k8s.io
APIServiceForwards to Service
metrics-serverTLS 443, reads store
PodMetricsListcpu, memory, window

If the APIService is Available=False, kubectl top fails even when the Pod is Running.

5.Metrics Server vs Others

ToolAnswersData kept
metrics-serverCPU / memory now ?Latest, in memory
PrometheusBehaviour over time ?Time-series on disk
kube-state-metricsState of objects ?Object state, no usage
Remember
  • top memory = working set, not free or docker stats
  • --metric-resolution = scrape interval (15s)
  • Not a monitoring source : never scrape it with Prometheus
In short : Metrics Server scrapes kubelets, keeps the latest CPU and memory in memory, and serves it as metrics.k8s.io for kubectl top and the HPA. No history, so use Prometheus for that.

1.HPA Uses It

The HPA asks the Metrics API for Pod usage, compares it with the target, then scales the Deployment.

kube-apiserver proxy metrics-server GETmetrics.k8s.io HPA scale Deployment Pods kubeletusage
Key Idea utilization = usage / request
ratio = currentUtil / targetUtil
desired = ceil(replicas x ratio)
Every 15s. Requests are the 100%, so no requests means no %.
Scale-up is immediate. Scale-down waits 5 min (stabilization window) so replicas do not flap.
hpa.yaml (CPU 50%)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

2.Sizing and Versions

Metrics Server0.9.x0.8.x0.7.x0.6.x
Kubernetes1.34+1.31+1.27+1.25+
Default100m CPU, 200MiB (~100 nodes)
Per node over 100+1m CPU, +2MiB
HA2 replicas (high-availability.yaml)

3.Commands

CommandWhat it tells you
kubectl apply -f .../components.yamlInstalls Deployment, Service, APIService
helm install metrics-server metrics-server/metrics-server -n kube-systemSame install, with Helm
kubectl top nodes / kubectl top pods -A --sort-by=memoryLive CPU and memory per Node or Pod
kubectl get apiservice v1beta1.metrics.k8s.ioAvailable=True ? If not, describe it for the reason
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodesRaw Metrics API, bypasses kubectl top
kubectl -n kube-system logs deploy/metrics-serverScrape errors : TLS, no route to node

4.Gotchas

  • x509 errors on kind or minikube : --kubelet-insecure-tls is for testing only. Real fix is kubelet certs signed by the cluster CA.
  • "No route to host" : set --kubelet-preferred-address-types (e.g. InternalIP).
  • Aggregation layer must be on. Kubelet needs webhook authn / authz, else 401 / 403.
  • kubectl get hpa shows <unknown>/50% : Pods have no requests, or Metrics Server is down.
  • New Pods have no metrics for a scrape or two. Memory is working set, so it differs from free.
  • Metrics API is metrics.k8s.io/v1beta1 : confirm with kubectl api-versions | grep metrics.
In short : The HPA divides usage by requests, so set requests. Check the APIService first when kubectl top fails, and never keep --kubelet-insecure-tls outside a lab.