Observability

Prometheus and ServiceMonitors

1.What is Prometheus ?

A pull-based time-series database. It scrapes /metrics from discovered targets into a local TSDB, answers PromQL and sends alerts to Alertmanager.

Defaults by layer
  • Plainscrape 1m, timeout 10s, rules 1m, retention 15d
  • Prometheus CRscrape 30s, rules 30s, retention 24h if unset

First query you run : up == 0.

2.Data model

  • Series = name (__name__) + labels.
  • New label value = new series.
  • Targets add job, instance. Scrapes add up, scrape_*.
  • Base units, plural suffix, counters end in _total.
TypeBehavesSeriesQuery with
Counteronly goes up, resets to 0 on restart_totalrate(), increase()
Gaugegoes up and downthe valuethe value, avg_over_time()
Histogramcumulative buckets, aggregatable_bucket{le}, _sum, _counthistogram_quantile() over rate
Summaryquantiles computed in the client{quantile}read it, cannot be aggregated

Native histograms : one composite sample, far fewer series. Stable since 3.8.0, still opt-in with scrape_native_histograms: true.

3.The scrape loop

relabel_configs pick targets and build __address__. metric_relabel_configs run on every sample (not up). A breached sample_limit fails the whole scrape. Vanished series get a staleness marker; lookback is 5m.

node-exporter :9100 kube-state-metrics :8080 kubelet / cAdvisor :10250 finds targets GET /metrics discovery __meta_* labels relabel keep / drop scrape every interval metric relabel every sample TSDB head + WAL, 2h blocks rules alerts Alertmanager PromQL, Grafana

4.PromQL essentials

  • x instant, x[5m] range vector. Rate first, then aggregate : sum(rate(x[5m])), never rate a sum.
  • rate needs two samples : range of 4x the scrape interval or more (a convention). At 1m, [1m] is usually empty.
  • histogram_quantile must keep le in by. absent(x) alerts on vanished metrics.
Starter queryShows
up == 0targets that failed their last scrape
sum by (job) (rate(http_requests_total[5m]))request rate per job
histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))p90 latency, classic histogram
absent(up{job="api"})1 when the api job has no series at all

5.Where Kubernetes metrics come from

SourcePort, pathMeasures
node-exporter (DaemonSet):9100/metricsnode CPU, memory, disk, network
kube-state-metrics:8080/metricsobject state from the API
kubelet (no Service : the operator makes one):10250/metricsthe kubelet's own metrics
cAdvisor (in the kubelet):10250/metrics/cadvisorper-container usage
kube-apiserver:6443/metricsrequest rates and latencies
metrics-serverreads kubelet /metrics/resourcelatest CPU, memory : not a source
In short : Prometheus scrapes discovered targets into a local TSDB of name-plus-labels series. Query counters with rate first, then aggregate.

1.Why an operator ?

  • Config becomes CRDs (Operators).
  • Instance CRDs become StatefulSets.
  • Scrape CRDs : one gzipped config in Secret prometheus-<name>.
  • config-reloader : /-/reload, no restart.

Install : bundle.yaml (kubectl create, into default), kube-prometheus-stack (Helm), kube-prometheus (jsonnet).
Also : Probe, ThanosRuler, AlertmanagerConfig (v1alpha1).

2.The CRDs

KindVersionShortRole
Prometheusv1prominstance
PrometheusAgentv1alpha1promagentscrape, remote write only
Alertmanagerv1aminstance
ServiceMonitorv1smonscrape Services
PodMonitorv1pmonscrape Pods
ScrapeConfigv1alpha1scfgraw jobs, outside targets
PrometheusRulev1promrulerules

3.ServiceMonitor anatomy

  • selector picks Services by label; namespaceSelector : own namespace by default.
  • endpoints[].port = Service port name, not a number or Pod port name. Or targetPort (container port name or number).
  • path /metrics, interval from the CR. relabelings before the scrape, metricRelabelings after.
  • job = Service name (default). Job : serviceMonitor/<ns>/<name>/<i>.
  • PodMonitor selects Pods directly : DaemonSets, sidecars, no Service (named ports : Services).
kind: ServiceMonitor
metadata: {labels: {release: kps}}
spec:
  selector: {matchLabels: {app: web-app}}
  endpoints:
  - {port: web, interval: 30s}

4.How a ServiceMonitor gets picked

  • Two levels : serviceMonitorSelector matches labels, and serviceMonitorNamespaceSelector the namespace.
  • kube-prometheus-stack wants release: <release>. No label : silently ignored. Or set serviceMonitorSelectorNilUsesHelmValues: false (also pods, probes, scrape configs, rules).
Prometheus CR serviceMonitorSelector + NamespaceSelector ServiceMonitor watches selector Prometheus /-/reload operator Service writes EndpointSlices Secret prometheus-<name> config-reloader Pods port web
ValueResource selectorNamespace selector
{}all objectsall namespaces
omitted (null)nothingown namespace only
matchLabelslabelled objectslabelled namespaces

5."My ServiceMonitor is not picked up"

  1. Labels match ? Namespace matched ?
  2. Rejected ? InvalidConfiguration Warning Event.
  3. In the generated config ?
  1. Discovered ? None : no Service. Dropped : port.
  2. RBAC : list/watch Services, Pods, Endpoint(Slice)s.
  3. Scraped ? up == 0 and its error.
CommandDoes
kubectl get prometheus,servicemonitors,podmonitors,prometheusrules -Alist objects
kubectl -n monitoring get secret prometheus-k8s -ojson | jq -r '.data["prometheus.yaml.gz"]' | base64 -d | gunzip | grep <name>in the config ?
kubectl get events -n <ns> --field-selector involvedObject.name=<name>rejected ?
kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090UI : Targets, SD pages
promtool check config prometheus.ymlvalidate a config
kubectl apply --server-side --force-conflicts -f monitoring.coreos.com_*.yamlCRDs, server-side
In short : A ServiceMonitor picks Services by label and scrapes their Pods on a named port. The Prometheus CR must select it and its namespace : {} = all, omitted = none.

1.Recording rules and PrometheusRule

  • Precomputes an expression every group interval, stored as a new series.
  • Name : level:metric:operations, e.g. job:http_requests:rate5m.
  • PrometheusRule objects, picked by ruleSelector + ruleNamespaceSelector. Nil ruleSelector = no rules.
  • Rendered into ConfigMaps prometheus-<name>-rulefiles-<n>.

Test with promtool test rules. Alerting rules (for, keep_firing_for) : Alertmanager Routing.

2.Storage : head, WAL, blocks, retention

  • Writes : in-memory head + WAL. Every 2h the head becomes a block.
  • Compaction merges blocks (to 10% of retention or 31d). Retention deletes whole blocks, up to 2h late.
  • Local disk, no NFS. Size retention at most 80 to 85% of the volume.
  • Disk = retention s x samples/s x 1 to 2 bytes.
scrape head (memory) every 2h block WAL compaction bigger blocks retention deleted
LayerDefault retentionWatch out
Plain Prometheus15dtime or size, first to trigger wins
Prometheus CR24h when nothing is setno volumeClaimTemplate = emptyDir
kube-prometheus-stack10don emptyDir unless a volumeClaimTemplate is set

Config-file retention since 3.8.0 (reloadable, flags deprecated). The operator uses it only for 3.11.0+ (3.8 to 3.10 : unit bug).

3.Cardinality

  • Series count, not sample count, drives memory.
  • Most metrics under 10 label combinations; check any over 100. No user IDs or emails as labels.
  • Find : /api/v1/status/tsdb, promtool tsdb analyze.
  • Limit : sample_limit, metric relabel drop or labeldrop.

4.High availability and scale

OptionWhat it does
replicas: 2twin Pods, prometheus_replica; dedupe later
shards: Nhash of __address__; no resharding
Agentscrape + remote write; no queries, rules

Remote write : a queue per endpoint reads the WAL (slow receiver, scraping goes on). About 25% more memory; down over about 2h = data lost. RW 2.0 still experimental. Tempo remote-writes span metrics here; KEDA runs PromQL.

5.Three ways to run it

WayCRDsDiscoveryRules
plain prometheus chartnoprometheus.io/* annotations, by relabelingConfigMap
kube-prometheus-stack (Helm)yesServiceMonitor and friends (13 out of the box)35 PrometheusRule objects
kube-prometheus (jsonnet)yessamekubernetes-mixin
  • The operator ignores prometheus.io/scrape : leaving the plain chart drops those targets. Add a ScrapeConfig.
  • Helm never upgrades CRDs. Flux defaults : install.crds: Create, upgrade.crds: Skip. Set both CreateReplace.
CommandDoes
promtool check rules rules.ymllint rules
promtool test rules tests.ymlunit-test rules
promtool tsdb analyze <dir>cardinality
curl localhost:9090/api/v1/status/tsdbtop series
curl localhost:9090/api/v1/status/buildinfotrue version
Seen in practice

No operator : a community chart rendered once, applied by Flux. Annotation targets; most app metrics by remote write from the OTel gateway. Rules in a ConfigMap, promtool-tested.

Pitfall : K3s repeats apiserver and etcd series in the kubelet job, so the head grows until an OOMKill. Raising limits for duplicated series is the wrong fix. Do better : drop them at scrape and alert on head series.

In short : Record expensive queries, watch retention and series count, scale with shards or an agent plus remote write. The operator only uses what its selectors match.