Observability
OpenTelemetry Operator
1.What is the OTel Operator ?
A Kubernetes operator that deploys and manages OpenTelemetry Collectors and injects auto-instrumentation into your pods, all from YAML.
- Hand-writing Collector Deployments, ConfigMaps and Services drifts.
- Tracing normally means changing every app, per language.
- Many Collector replicas double-scrape Prometheus targets.
Operator = controller + CRDs + admission webhooks. The webhooks need TLS certs, so cert-manager is the documented prerequisite.
2.The Moving Parts
Collector and Instrumentation CRs do most of the work. The operator keeps them in sync.
Reconcile loop
Drift detected, operator updates it back.
- ManagerWatches CRs, creates workloads, ConfigMaps, Services.
- WebhooksValidate CRs and mutate pods for injection.
- Collector CRMode, image, receivers, processors, exporters, pipelines.
- Instrumentation CRExporter endpoint, propagators, sampler, language images.
- Target AllocatorSplits Prometheus targets across replicas.
- OpAMPBridge CRConnects Collectors to an OpAMP server for remote config and health.
3.Collector Modes
The CR picks how the Collector runs. The operator creates a Service <name>-collector.
Tip : modes map to Deployment, DaemonSet and StatefulSet.
| Mode | Shape | When to use |
|---|---|---|
| deployment | Stateless gateway | Central OTLP endpoint |
| daemonset | One per node | Node metrics, logs |
| statefulset | Stable pod names | Target Allocator |
| sidecar | Inside app pod | Per-pod isolation |
In short : The OpenTelemetry Operator turns Collectors and auto-instrumentation into CRDs. The Collector CR picks a mode and the operator builds the workload, ConfigMap and Service.
1.Auto-Instrumentation
No image change needed. A pod annotation opts the workload in, and a webhook mutates the pod.
Injection flow
Deploymentinject-java: true
API servercalls the webhook
Operator webhookreads Instrumentation
Mutated Podinit container + OTEL_* env
Appsends OTLP
Collector Svc:4317 / :4318
| Annotation (pod template) | Value |
|---|---|
| instrumentation.opentelemetry.io/ inject-java | "true" |
| ...-nodejs / python / dotnet / go | "true" |
| ...-apache-httpd / nginx / sdk | same pattern |
| Other values | "name", "ns/name" or "false" |
Note : the init container copies the language agent into the app container. The app image never changes.
Env vars injected
- OTEL_SERVICE_NAMEworkload
- OTEL_EXPORTER_OTLP_ENDPOINTCR exporter
- OTEL_PROPAGATORSfrom the CR
- OTEL_TRACES_SAMPLERCR sampler
- OTEL_RESOURCE_ATTRIBUTESpod, node, ns
2.Target Allocator
Shards Prometheus scrape targets across Collector replicas so nothing is scraped twice. It can discover ServiceMonitor and PodMonitor CRs.
- Enabled with
spec..targetAllocator - Usually statefulset mode (stable pod names); daemonset needs the per-node strategy.
- It rewrites scrape_configs to
http_sd_configs. prometheusCR.picks up the monitor CRs.enabled: true
allocationStrategy :
consistent-hashing (default)
least-weighted (fewest targets)
per-node (daemonset mode)
consistent-hashing (default)
least-weighted (fewest targets)
per-node (daemonset mode)
Tip : keep a
prometheus receiver in the Collector config. The allocator only hands out targets, the Collectors still scrape.In short : Annotate the pod template and the webhook injects the agent and OTEL_* env from the Instrumentation CR. The Target Allocator splits Prometheus targets across Collectors.
1.Handy Commands
| Command | What it does |
|---|---|
kubectl apply -f opentelemetry-operator. | Install via manifest from GitHub releases |
helm install opentelemetry-operator open-telemetry/opentelemetry-operator | Install via Helm (repo open-telemetry) |
kubectl get opentelemetrycollectors -A | List Collector CRs |
kubectl get instrumentations -A | List Instrumentation CRs |
kubectl describe pod <app-pod> | Check init container + OTEL_* env |
Install order
cert-manager
Operator
Collector CR
Instr. CR
annotate pod template
restart pods
2.Quick YAML
# Collector (v1beta1)
kind: OpenTelemetryCollector
spec:
mode: deployment
config:
receivers: { otlp: ... }
exporters: { debug: {} }
service: { pipelines: ... }
# Instrumentation (v1alpha1)
kind: Instrumentation
spec:
exporter:
endpoint: http://demo-collector:4318
propagators: [tracecontext, baggage]
sampler: { type: parentbased_traceidratio }
java:
image: <registry>/autoinstrumentation-java
- Collector CR: mode plus a normal Collector config.
- Instrumentation CR: where to send, how to propagate, how to sample.
- Endpoint is the
<name>-collectorService. - Apply both, then annotate the pod template.
Gotchas
- Annotation goes on the pod template, not the Deployment metadata.
- Create the Instrumentation CR first. Restart existing pods.
- Webhook cert not ready blocks CR and pod creation.
- Versions are 0.x. Read release notes before upgrading.
Where the data lands
- Metrics : remote write or scrape into Prometheus.
- Logs : OTLP to Loki at
/otlp. - Traces : OTLP to Tempo on 4317 (gRPC) or 4318 (HTTP).
- Profiles : Pyroscope (OTLP profiles are alpha).
In short : Install cert-manager, then the operator. Apply the Collector and Instrumentation CRs, then annotate the pod template.