Observability
Tracing with Tempo
1.Traces, spans, context
A span is one timed piece of work. A trace is the tree of spans that share one 16-byte trace ID.
- Span : 8-byte span ID, parent span ID (none for the root), name, start and end.
- Kind (server, client, producer, consumer, internal), status (unset, ok, error), attributes, events, links.
- Resource says who made it :
service.name,k8s.pod.name.
00 - 4bf92f3577b34da6a3ce929d0e0e4736 - 00f067aa0ba902b7 - 01
version - trace ID (16 bytes) - parent ID, the caller's span (8 bytes) - flags
Lowest flag bit = sampled. tracestate carries vendor keys. SDKs default to propagators tracecontext,baggage.
2.The pipeline
- The app makes spans : SDK, or an agent injected from an
InstrumentationCR (OTel Operator). - Each outgoing call carries
traceparent. The next service adds a child span. - Head sampling happens here, in the SDK.
Alloy syntax lives in Loki and Alloy. Platform parts emit spans too : kgateway, Knative, CoreDNS (Cluster DNS).
- The SDK batches and exports OTLP to a node agent (DaemonSet Collector or Alloy) that adds
k8sattributesmetadata. - A gateway tier can tail-sample, then exports OTLP to Tempo's distributor on 4317.
- Tempo stores the spans. Grafana queries it on 3200.
3.Ports and protocols
| Port | What | Trap |
|---|---|---|
| 4317 | OTLP gRPC | Receivers bind localhost since Tempo 2.7 : set 0.0.0.0:4317 |
| 4318 | OTLP HTTP /v1/traces | Injected Java, Node.js, Python agents (operator v0.160.0) send http/protobuf here |
| 3200 | HTTP API (queries, /ready) | Was 80 before 2.8 |
| 9095 | Internal gRPC | server. |
Pitfall : an Instrumentation CR points agents at :4317. The Java agent defaults to http/protobuf and the operator passes the endpoint, not a protocol. HTTP at a gRPC port : no crash, no failed probe, no telemetry, for months. Do better : 4318 for HTTP, and alert when spans stop.
4.Sampling
| Head | Tail | |
|---|---|---|
| Decides | At the root, from the trace ID (parentbased_) | After seeing most spans : keep errors and slow traces |
| Cost | Cheap, but blind to which traces will fail | Every span of a trace must reach one instance |
| Where | SDK or Instrumentation CR | Collector tail_sampling processor (beta) |
- Tail defaults :
decision_wait: 30s,num_traces: 50000. Policies likestatus_code,latency,probabilistic. - Scale with two tiers : a
loadbalancingexporter withrouting_key: traceIDin front. - Sampling breaks links : an exemplar or log line can point at a dropped trace.
- It shrinks span metrics : with 10% sampling, expect roughly 10% of the request rate.
traceparent, apps export spans over OTLP to a collector, and the collector samples and forwards them to Tempo. Mind the ports : 4317 is gRPC, 4318 is HTTP.1.Components
Tempo stores only traces, indexes nothing up front, and keeps Parquet blocks in object storage : cheap.
| Component | Job | Modes |
|---|---|---|
| Distributor | Receives OTLP, Jaeger, Zipkin. Applies limits, shards by trace ID | both |
| Kafka-compatible log | Durable write-ahead log | microservices |
| Block-builder | Kafka to Parquet blocks in object storage | microservices |
| Live-store | Recent traces, queryable within seconds | both |
| Query frontend | Shards a query into jobs | both |
| Querier | Runs jobs on live-stores and object storage | both |
| Backend scheduler, worker | Compaction and retention. Only one scheduler should run | both |
| Metrics-generator | Span metrics and service graphs | optional |
2.Write path and read path
- Kafka acks before the distributor answers : it is the durability point, so RF is 1 (2.x clusters used 3).
- Three consumers read Kafka. Queriers read live-stores for recent data and object storage for the rest.
- Monolithic (
-target=all, the default) needs no Kafka and runs everything in one process. The generator remote-writes to Prometheus.
3.Storage and retention
- 3.1 writes vParquet5, and will not start if told to write vParquet3.
localstorage on a PVC (Volumes) is for tests and monolithic mode.- Retention key :
backend_, default 336h (14 days).worker. compaction. block_ retention - The chart's
tempo.retention: 24hreaches only the scheduler key, which deletes nothing : a default chart 3.1.0 install keeps 14 days. Settempo.too.backendWorker. compaction. block_ retention
4.From 2.x to 3.x
| 2.x | 3.x |
|---|---|
| ingester | live-store plus block-builder |
| compactor | backend scheduler plus worker |
| RF3 in clusters | RF1, Kafka for durability |
local-blocks processor | live-store serves recent data |
| flat overrides | refused unless overrides. |
No downgrade from 3.x. TraceQL metrics history starts at the upgrade.
5.Installing on Kubernetes
grafana-community/: a monolithic StatefulSet, 3.x since chart 3.0.0, persistence off by default.tempo tempo-distributedneeds Kafka. The Tempo Operator v0.22.0 still targets 2.x.- Tempo has no auth : put a proxy in front if anything outside the cluster can reach it.
| Command | What it does |
|---|---|
helm install tempo grafana-community/ | Monolithic Tempo |
kubectl -n observability port-forward svc/tempo 3200 | Reach the API |
curl -s localhost:3200/ | Readiness (probes) |
curl -s -G localhost:3200/ | TraceQL search |
tempo-cli migrate config --mode=monolithic old.yaml > new.yaml | 2.x to 3.x config |
1.TraceQL
{ conditions }selects spans,|chains a pipeline. A trace comes back if a spanset survives.- Intrinsics use a colon :
span:duration,trace:duration,span:status.
- Attributes use a dot :
span.,http. status_ code resource.. Bareservice. name durationmeansspan:duration. - Regex is fully anchored. Structure :
>>descendant,>child,~sibling.
| Query | Finds |
|---|---|
{ resource. | Failed spans in frontend |
{ resource. | Errors anywhere below frontend |
{ trace:duration > 5s } | Slow traces |
{ span. | Server errors |
{ resource. | Request rate by status (metrics) |
{ span:name = "GET /:endpoint" } | quantile_ | p99 latency (metrics) |
({ status = error } | rate()) / ({ } | rate()) | Error ratio (3.1 arithmetic) |
2.TraceQL metrics
- Spans to series :
rate(),quantile_,over_ time() by(). - GA in 3.0. Alerting on them is experimental. 3.1 adds arithmetic.
3.Metrics from traces
The metrics-generator turns spans into series and remote-writes them to Prometheus (needs --web.). Its processors do nothing until listed in overrides.. Spans that ended over 30s ago are not counted.
| Series | Labels | Answers |
|---|---|---|
traces_ | service, span_name, span_kind, status_code | Request and error rate |
traces_ | same | Latency histogram |
traces_ | client, server, connection_type | Calls between services |
traces_ | same | Failed calls |
Cardinality : span_name is usually the biggest driver; raw URLs or user IDs as dimensions explode it. The docs list ..._ and ..._ service-graph series that Tempo never emits : do not alert on them.
4.Correlations in Grafana
- Each jump is set on its own data source.
serviceMapdraws the service graph from Prometheus series. Trace to profiles (tracesToProfiles) is in Pyroscope. - Trace to logs : default time shift is 0, start with
-2s/2s. With Loki, the link shows only when a configured tag is on the span. - Exemplars need three switches :
send_exemplars: trueon the generator, Prometheus--enable-feature=exemplar-storage(else silently dropped), GrafanaexemplarTraceIdDestinations. - Trace IDs go in Loki structured metadata, not labels (Loki and Alloy).
A monolithic Tempo from a rendered chart (Flux) on a local volume, a two-tier OTel Collector, no tail sampling, the generator feeding a RED dashboard. Pitfall : exemplars are sent, but Prometheus runs without exemplar-storage and Grafana has no exemplarTraceIdDestinations, so metric to trace is quietly broken. Do better : set all three switches and click one exemplar through to its trace.