Observability

OpenTelemetry Operator

1.What is the OTel Operator ?

A Kubernetes operator that deploys and manages OpenTelemetry Collectors and injects auto-instrumentation into your pods, all from YAML.

  • Hand-writing Collector Deployments, ConfigMaps and Services drifts.
  • Tracing normally means changing every app, per language.
  • Many Collector replicas double-scrape Prometheus targets.

Operator = controller + CRDs + admission webhooks. The webhooks need TLS certs, so cert-manager is the documented prerequisite.

You (kubectl apply) Operator Controller Webhooks creates Collectors mutates App pods(injected) cert-manager (TLS) install first (webhook cert)

2.The Moving Parts

Collector and Instrumentation CRs do most of the work. The operator keeps them in sync.

Reconcile loop Collector CR Operator reconciles ConfigMap + Workload + Service drift

Drift detected, operator updates it back.

  • ManagerWatches CRs, creates workloads, ConfigMaps, Services.
  • WebhooksValidate CRs and mutate pods for injection.
  • Collector CRMode, image, receivers, processors, exporters, pipelines.
  • Instrumentation CRExporter endpoint, propagators, sampler, language images.
  • Target AllocatorSplits Prometheus targets across replicas.
  • OpAMPBridge CRConnects Collectors to an OpAMP server for remote config and health.

3.Collector Modes

The CR picks how the Collector runs. The operator creates a Service <name>-collector.

Tip : modes map to Deployment, DaemonSet and StatefulSet.
ModeShapeWhen to use
deploymentStateless gatewayCentral OTLP endpoint
daemonsetOne per nodeNode metrics, logs
statefulsetStable pod namesTarget Allocator
sidecarInside app podPer-pod isolation
In short : The OpenTelemetry Operator turns Collectors and auto-instrumentation into CRDs. The Collector CR picks a mode and the operator builds the workload, ConfigMap and Service.

1.Auto-Instrumentation

No image change needed. A pod annotation opts the workload in, and a webhook mutates the pod.

Injection flow
Deploymentinject-java: true
API servercalls the webhook
Operator webhookreads Instrumentation
Mutated Podinit container + OTEL_* env
Appsends OTLP
Collector Svc:4317 / :4318
Annotation (pod template)Value
instrumentation.opentelemetry.io/
inject-java
"true"
...-nodejs / python / dotnet / go"true"
...-apache-httpd / nginx / sdksame pattern
Other values"name", "ns/name"
or "false"
Note : the init container copies the language agent into the app container. The app image never changes.
Env vars injected
  • OTEL_SERVICE_NAMEworkload
  • OTEL_EXPORTER_OTLP_ENDPOINTCR exporter
  • OTEL_PROPAGATORSfrom the CR
  • OTEL_TRACES_SAMPLERCR sampler
  • OTEL_RESOURCE_ATTRIBUTESpod, node, ns
Mutated Pod init container language agent copies app container + OTEL_* env

2.Target Allocator

Shards Prometheus scrape targets across Collector replicas so nothing is scraped twice. It can discover ServiceMonitor and PodMonitor CRs.

  • Enabled with spec.targetAllocator.
  • Usually statefulset mode (stable pod names); daemonset needs the per-node strategy.
  • It rewrites scrape_configs to http_sd_configs.
  • prometheusCR.enabled: true picks up the monitor CRs.
Service / PodMonitor Target Allocator http_sd: a share each Collector 0 Collector 1 splits targets, no double scrape scrape only own targets
allocationStrategy :
consistent-hashing (default)
least-weighted (fewest targets)
per-node (daemonset mode)
Tip : keep a prometheus receiver in the Collector config. The allocator only hands out targets, the Collectors still scrape.
In short : Annotate the pod template and the webhook injects the agent and OTEL_* env from the Instrumentation CR. The Target Allocator splits Prometheus targets across Collectors.

1.Handy Commands

CommandWhat it does
kubectl apply -f opentelemetry-operator.yamlInstall via manifest from GitHub releases
helm install opentelemetry-operator open-telemetry/opentelemetry-operatorInstall via Helm (repo open-telemetry)
kubectl get opentelemetrycollectors -AList Collector CRs
kubectl get instrumentations -AList Instrumentation CRs
kubectl describe pod <app-pod>Check init container + OTEL_* env
Install order
cert-manager
Operator
Collector CR
Instr. CR
annotate pod template
restart pods

2.Quick YAML

# Collector (v1beta1)
kind: OpenTelemetryCollector
spec:
  mode: deployment
  config:
    receivers: { otlp: ... }
    exporters: { debug: {} }
    service: { pipelines: ... }
# Instrumentation (v1alpha1)
kind: Instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators: [tracecontext, baggage]
  sampler: { type: parentbased_traceidratio }
  java:
    image: <registry>/autoinstrumentation-java
  • Collector CR: mode plus a normal Collector config.
  • Instrumentation CR: where to send, how to propagate, how to sample.
  • Endpoint is the <name>-collector Service.
  • Apply both, then annotate the pod template.
Gotchas
  • Annotation goes on the pod template, not the Deployment metadata.
  • Create the Instrumentation CR first. Restart existing pods.
  • Webhook cert not ready blocks CR and pod creation.
  • Versions are 0.x. Read release notes before upgrading.
Where the data lands
  • Metrics : remote write or scrape into Prometheus.
  • Logs : OTLP to Loki at /otlp.
  • Traces : OTLP to Tempo on 4317 (gRPC) or 4318 (HTTP).
  • Profiles : Pyroscope (OTLP profiles are alpha).
In short : Install cert-manager, then the operator. Apply the Collector and Instrumentation CRs, then annotate the pod template.