Scaling and metrics
metrics-server
1.What is Metrics Server ?
Metrics Server asks every kubelet for current CPU and memory use and keeps it in memory. It serves the numbers through the Metrics API. That is what makes kubectl top and the HPA work.
- Scrapes kubelets about every 15s
- Keeps only the latest sample (no history)
- Only CPU and memory, for Nodes and Pods
In Simple Words
Metrics Server tells Kubernetes how much CPU and memory things use right now.
Built for autoscaling, not for monitoring
2.Why it exists ?
Kubernetes core does not ship a source of live Pod and Node usage.
Scraping every kubelet yourself means each consumer redoes auth, discovery and aggregation. Metrics Server is the reference implementation of the Metrics API.
Without it
kubectl topfails : "Metrics API not available"- HPA shows <unknown> targets
- No CPU / memory for autoscaling
- VPA recommender has no live data
3.Resource Metrics Pipeline
Metrics Server pulls from the kubelet. Consumers only talk to the API server, never to kubelets.
- cAdvisorReads cgroup stats per container (inside kubelet).
- kubeletExposes /metrics/resource.
- APIServicev1beta1.metrics.k8s.io points the aggregator at the Service.
- Servicemetrics-server in kube-system, port 443.
4.kubectl top pods : Request Flow
GET metrics.k8s.io/v1beta1 pods
kubectl topUser asks
kube-apiserverMatches group metrics.k8s.io
APIServiceForwards to Service
metrics-serverTLS 443, reads store
PodMetricsListcpu, memory, window
If the APIService is Available=False, kubectl top fails even when the Pod is Running.
5.Metrics Server vs Others
| Tool | Answers | Data kept |
|---|---|---|
| metrics-server | CPU / memory now ? | Latest, in memory |
| Prometheus | Behaviour over time ? | Time-series on disk |
| kube-state-metrics | State of objects ? | Object state, no usage |
Remember
- top memory = working set, not
freeordocker stats --metric-resolution= scrape interval (15s)- Not a monitoring source : never scrape it with Prometheus
In short : Metrics Server scrapes kubelets, keeps the latest CPU and memory in memory, and serves it as metrics.k8s.io for
kubectl top and the HPA. No history, so use Prometheus for that.1.HPA Uses It
The HPA asks the Metrics API for Pod usage, compares it with the target, then scales the Deployment.
Key Idea
utilization = usage / request
ratio = currentUtil / targetUtil
desired = ceil(replicas x ratio)
ratio = currentUtil / targetUtil
desired = ceil(replicas x ratio)
Every 15s. Requests are the 100%, so no requests means no %.
Scale-up is immediate. Scale-down waits 5 min (stabilization window) so replicas do not flap.
hpa.yaml (CPU 50%)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
2.Sizing and Versions
| Metrics Server | 0.9.x | 0.8.x | 0.7.x | 0.6.x |
|---|---|---|---|---|
| Kubernetes | 1.34+ | 1.31+ | 1.27+ | 1.25+ |
| Default | 100m CPU, 200MiB (~100 nodes) |
| Per node over 100 | +1m CPU, +2MiB |
| HA | 2 replicas (high-availability.yaml) |
3.Commands
| Command | What it tells you |
|---|---|
| kubectl apply -f .../components.yaml | Installs Deployment, Service, APIService |
| helm install metrics-server metrics-server/metrics-server -n kube-system | Same install, with Helm |
| kubectl top nodes / kubectl top pods -A --sort-by=memory | Live CPU and memory per Node or Pod |
| kubectl get apiservice v1beta1.metrics.k8s.io | Available=True ? If not, describe it for the reason |
| kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | Raw Metrics API, bypasses kubectl top |
| kubectl -n kube-system logs deploy/metrics-server | Scrape errors : TLS, no route to node |
4.Gotchas
- x509 errors on kind or minikube :
--kubelet-insecure-tlsis for testing only. Real fix is kubelet certs signed by the cluster CA. - "No route to host" : set
--kubelet-preferred-address-types(e.g. InternalIP). - Aggregation layer must be on. Kubelet needs webhook authn / authz, else 401 / 403.
kubectl get hpashows <unknown>/50% : Pods have no requests, or Metrics Server is down.- New Pods have no metrics for a scrape or two. Memory is working set, so it differs from
free. - Metrics API is metrics.k8s.io/v1beta1 : confirm with
kubectl api-versions | grep metrics.
In short : The HPA divides usage by requests, so set requests. Check the APIService first when
kubectl top fails, and never keep --kubelet-insecure-tls outside a lab.