Observability
Prometheus and ServiceMonitors
1.What is Prometheus ?
A pull-based time-series database. It scrapes /metrics from discovered targets into a local TSDB, answers PromQL and sends alerts to Alertmanager.
- metrics-server : for
kubectl topand HPA. Not a source. - kube-state-metrics : a source of object state.
- Plainscrape 1m, timeout 10s, rules 1m, retention 15d
- Prometheus CRscrape 30s, rules 30s, retention 24h if unset
First query you run : up == 0.
2.Data model
- Series = name (
__name__) + labels. - New label value = new series.
- Targets add
job,instance. Scrapes addup,scrape_*. - Base units, plural suffix, counters end in
_total.
| Type | Behaves | Series | Query with |
|---|---|---|---|
| Counter | only goes up, resets to 0 on restart | _total | rate(), increase() |
| Gauge | goes up and down | the value | the value, avg_over_time() |
| Histogram | cumulative buckets, aggregatable | _bucket{le}, _sum, _count | histogram_ over rate |
| Summary | quantiles computed in the client | {quantile} | read it, cannot be aggregated |
Native histograms : one composite sample, far fewer series. Stable since 3.8.0, still opt-in with scrape_.
3.The scrape loop
relabel_configs pick targets and build __address__. metric_ run on every sample (not up). A breached sample_limit fails the whole scrape. Vanished series get a staleness marker; lookback is 5m.
4.PromQL essentials
xinstant,x[5m]range vector. Rate first, then aggregate :sum(rate(x[5m])), never rate a sum.rateneeds two samples : range of 4x the scrape interval or more (a convention). At 1m,[1m]is usually empty.histogram_must keepquantile leinby.absent(x)alerts on vanished metrics.
| Starter query | Shows |
|---|---|
up == 0 | targets that failed their last scrape |
sum by (job) ( | request rate per job |
histogram_ | p90 latency, classic histogram |
absent( | 1 when the api job has no series at all |
5.Where Kubernetes metrics come from
| Source | Port, path | Measures |
|---|---|---|
| node-exporter (DaemonSet) | :9100/metrics | node CPU, memory, disk, network |
| kube-state-metrics | :8080/metrics | object state from the API |
| kubelet (no Service : the operator makes one) | :10250/ | the kubelet's own metrics |
| cAdvisor (in the kubelet) | :10250/ | per-container usage |
| kube-apiserver | :6443/metrics | request rates and latencies |
| metrics-server | reads kubelet /metrics/ | latest CPU, memory : not a source |
rate first, then aggregate.1.Why an operator ?
- Config becomes CRDs (Operators).
- Instance CRDs become StatefulSets.
- Scrape CRDs : one gzipped config in Secret
prometheus-<name>. config-reloader:/-/reload, no restart.
Install : bundle.yaml (kubectl create, into default), kube-prometheus-stack (Helm), kube-prometheus (jsonnet).
Also : Probe, ThanosRuler, AlertmanagerConfig (v1alpha1).
2.The CRDs
| Kind | Version | Short | Role |
|---|---|---|---|
| Prometheus | v1 | prom | instance |
| PrometheusAgent | v1alpha1 | promagent | scrape, remote write only |
| Alertmanager | v1 | am | instance |
| ServiceMonitor | v1 | smon | scrape Services |
| PodMonitor | v1 | pmon | scrape Pods |
| ScrapeConfig | v1alpha1 | scfg | raw jobs, outside targets |
| PrometheusRule | v1 | promrule | rules |
3.ServiceMonitor anatomy
selectorpicks Services by label;namespaceSelector: own namespace by default.endpoints[].port= Service port name, not a number or Pod port name. OrtargetPort(container port name or number).path/metrics,intervalfrom the CR.relabelingsbefore the scrape,metricRelabelingsafter.job= Service name (default). Job :serviceMonitor/<ns>/<name>/<i>.- PodMonitor selects Pods directly : DaemonSets, sidecars, no Service (named ports : Services).
kind: ServiceMonitor
metadata: {labels: {release: kps}}
spec:
selector: {matchLabels: {app: web-app}}
endpoints:
- {port: web, interval: 30s}
4.How a ServiceMonitor gets picked
- Two levels :
serviceMonitorSelectormatches labels, andserviceMonitorNamespaceSelectorthe namespace. - kube-prometheus-stack wants
release: <release>. No label : silently ignored. Or setserviceMonitorSelectorNilUsesHelmValues: false(also pods, probes, scrape configs, rules).
| Value | Resource selector | Namespace selector |
|---|---|---|
{} | all objects | all namespaces |
| omitted (null) | nothing | own namespace only |
matchLabels | labelled objects | labelled namespaces |
5."My ServiceMonitor is not picked up"
- Labels match ? Namespace matched ?
- Rejected ?
InvalidConfigurationWarning Event. - In the generated config ?
- Discovered ? None : no Service. Dropped : port.
- RBAC : list/watch Services, Pods, Endpoint(Slice)s.
- Scraped ?
up == 0and its error.
| Command | Does |
|---|---|
kubectl get prometheus,servicemonitors,podmonitors,prometheusrules -A | list objects |
kubectl -n monitoring get secret prometheus-k8s -ojson | jq -r '. | in the config ? |
kubectl get events -n <ns> --field-selector involvedObject. | rejected ? |
kubectl -n monitoring port-forward svc/ | UI : Targets, SD pages |
promtool check config prometheus.yml | validate a config |
kubectl apply --server-side --force-conflicts -f monitoring. | CRDs, server-side |
{} = all, omitted = none.1.Recording rules and PrometheusRule
- Precomputes an expression every group
interval, stored as a new series. - Name :
level:metric:operations, e.g.job:http_.requests:rate5m
- PrometheusRule objects, picked by
ruleSelector+ruleNamespaceSelector. NilruleSelector= no rules. - Rendered into ConfigMaps
prometheus-<name>-rulefiles-<n>.
Test with promtool test rules. Alerting rules (for, keep_firing_for) : Alertmanager Routing.
2.Storage : head, WAL, blocks, retention
- Writes : in-memory head + WAL. Every 2h the head becomes a block.
- Compaction merges blocks (to 10% of retention or 31d). Retention deletes whole blocks, up to 2h late.
- Local disk, no NFS. Size retention at most 80 to 85% of the volume.
- Disk = retention s x samples/s x 1 to 2 bytes.
| Layer | Default retention | Watch out |
|---|---|---|
| Plain Prometheus | 15d | time or size, first to trigger wins |
Prometheus CR | 24h when nothing is set | no volumeClaimTemplate = emptyDir |
| kube-prometheus-stack | 10d | on emptyDir unless a volumeClaimTemplate is set |
Config-file retention since 3.8.0 (reloadable, flags deprecated). The operator uses it only for 3.11.0+ (3.8 to 3.10 : unit bug).
3.Cardinality
- Series count, not sample count, drives memory.
- Most metrics under 10 label combinations; check any over 100. No user IDs or emails as labels.
- Find :
/,api/ v1/ status/ tsdb promtool tsdb analyze. - Limit :
sample_limit, metric relabeldroporlabeldrop.
4.High availability and scale
| Option | What it does |
|---|---|
replicas: 2 | twin Pods, prometheus_; dedupe later |
shards: N | hash of __address__; no resharding |
| Agent | scrape + remote write; no queries, rules |
Remote write : a queue per endpoint reads the WAL (slow receiver, scraping goes on). About 25% more memory; down over about 2h = data lost. RW 2.0 still experimental. Tempo remote-writes span metrics here; KEDA runs PromQL.
5.Three ways to run it
| Way | CRDs | Discovery | Rules |
|---|---|---|---|
plain prometheus chart | no | prometheus.io/* annotations, by relabeling | ConfigMap |
| kube-prometheus-stack (Helm) | yes | ServiceMonitor and friends (13 out of the box) | 35 PrometheusRule objects |
| kube-prometheus (jsonnet) | yes | same | kubernetes-mixin |
- The operator ignores
prometheus.: leaving the plain chart drops those targets. Add a ScrapeConfig.io/ scrape - Helm never upgrades CRDs. Flux defaults :
install.crds: Create,upgrade.crds: Skip. Set bothCreateReplace.
| Command | Does |
|---|---|
promtool check rules rules.yml | lint rules |
promtool test rules tests.yml | unit-test rules |
promtool tsdb analyze <dir> | cardinality |
curl localhost:9090/ | top series |
curl localhost:9090/ | true version |
No operator : a community chart rendered once, applied by Flux. Annotation targets; most app metrics by remote write from the OTel gateway. Rules in a ConfigMap, promtool-tested.
Pitfall : K3s repeats apiserver and etcd series in the kubelet job, so the head grows until an OOMKill. Raising limits for duplicated series is the wrong fix. Do better : drop them at scrape and alert on head series.