Observability

Tracing with Tempo

1.Traces, spans, context

A span is one timed piece of work. A trace is the tree of spans that share one 16-byte trace ID.

  • Span : 8-byte span ID, parent span ID (none for the root), name, start and end.
  • Kind (server, client, producer, consumer, internal), status (unset, ok, error), attributes, events, links.
  • Resource says who made it : service.name, k8s.pod.name.
gateway api db traceparent traceparent one trace ID, three spans GET /order (root, no parent) api /order SELECT parent ID time start
The traceparent header

00 - 4bf92f3577b34da6a3ce929d0e0e4736 - 00f067aa0ba902b7 - 01

version - trace ID (16 bytes) - parent ID, the caller's span (8 bytes) - flags

Lowest flag bit = sampled. tracestate carries vendor keys. SDKs default to propagators tracecontext,baggage.

2.The pipeline

  1. The app makes spans : SDK, or an agent injected from an Instrumentation CR (OTel Operator).
  2. Each outgoing call carries traceparent. The next service adds a child span.
  3. Head sampling happens here, in the SDK.

Alloy syntax lives in Loki and Alloy. Platform parts emit spans too : kgateway, Knative, CoreDNS (Cluster DNS).

  1. The SDK batches and exports OTLP to a node agent (DaemonSet Collector or Alloy) that adds k8sattributes metadata.
  2. A gateway tier can tail-sample, then exports OTLP to Tempo's distributor on 4317.
  3. Tempo stores the spans. Grafana queries it on 3200.

3.Ports and protocols

PortWhatTrap
4317OTLP gRPCReceivers bind localhost since Tempo 2.7 : set 0.0.0.0:4317
4318OTLP HTTP /v1/tracesInjected Java, Node.js, Python agents (operator v0.160.0) send http/protobuf here
3200HTTP API (queries, /ready)Was 80 before 2.8
9095Internal gRPCserver.grpc_listen_port
Seen in practice

Pitfall : an Instrumentation CR points agents at :4317. The Java agent defaults to http/protobuf and the operator passes the endpoint, not a protocol. HTTP at a gRPC port : no crash, no failed probe, no telemetry, for months. Do better : 4318 for HTTP, and alert when spans stop.

4.Sampling

HeadTail
DecidesAt the root, from the trace ID (parentbased_traceidratio)After seeing most spans : keep errors and slow traces
CostCheap, but blind to which traces will failEvery span of a trace must reach one instance
WhereSDK or Instrumentation CRCollector tail_sampling processor (beta)
  • Tail defaults : decision_wait: 30s, num_traces: 50000. Policies like status_code, latency, probabilistic.
  • Scale with two tiers : a loadbalancing exporter with routing_key: traceID in front.
  • Sampling breaks links : an exemplar or log line can point at a dropped trace.
  • It shrinks span metrics : with 10% sampling, expect roughly 10% of the request rate.
In short : Every hop passes the trace ID in traceparent, apps export spans over OTLP to a collector, and the collector samples and forwards them to Tempo. Mind the ports : 4317 is gRPC, 4318 is HTTP.

1.Components

Tempo stores only traces, indexes nothing up front, and keeps Parquet blocks in object storage : cheap.

ComponentJobModes
DistributorReceives OTLP, Jaeger, Zipkin. Applies limits, shards by trace IDboth
Kafka-compatible logDurable write-ahead logmicro­services
Block-builderKafka to Parquet blocks in object storagemicro­services
Live-storeRecent traces, queryable within secondsboth
Query frontendShards a query into jobsboth
QuerierRuns jobs on live-stores and object storageboth
Backend scheduler, workerCompaction and retention. Only one scheduler should runboth
Metrics-generatorSpan metrics and service graphsoptional

2.Write path and read path

  • Kafka acks before the distributor answers : it is the durability point, so RF is 1 (2.x clusters used 3).
  • Three consumers read Kafka. Queriers read live-stores for recent data and object storage for the rest.
  • Monolithic (-target=all, the default) needs no Kafka and runs everything in one process. The generator remote-writes to Prometheus.
write path Collector / Alloy distributor Kafka metrics-generator block-builder live-store remote write Parquet Prometheus object storage read path Grafana query frontend querier recent older blocks

3.Storage and retention

  • 3.1 writes vParquet5, and will not start if told to write vParquet3.
  • local storage on a PVC (Volumes) is for tests and monolithic mode.
  • Retention key : backend_worker.compaction.block_retention, default 336h (14 days).
  • The chart's tempo.retention: 24h reaches only the scheduler key, which deletes nothing : a default chart 3.1.0 install keeps 14 days. Set tempo.backendWorker.compaction.block_retention too.

4.From 2.x to 3.x

2.x3.x
ingesterlive-store plus block-builder
compactorbackend scheduler plus worker
RF3 in clustersRF1, Kafka for durability
local-blocks processorlive-store serves recent data
flat overridesrefused unless overrides.enable_legacy_overrides: true

No downgrade from 3.x. TraceQL metrics history starts at the upgrade.

5.Installing on Kubernetes

  • grafana-community/tempo : a monolithic StatefulSet, 3.x since chart 3.0.0, persistence off by default.
  • tempo-distributed needs Kafka. The Tempo Operator v0.22.0 still targets 2.x.
  • Tempo has no auth : put a proxy in front if anything outside the cluster can reach it.
CommandWhat it does
helm install tempo grafana-community/tempo --version 3.1.0 -n observability -f values.yamlMonolithic Tempo
kubectl -n observability port-forward svc/tempo 3200Reach the API
curl -s localhost:3200/readyReadiness (probes)
curl -s -G localhost:3200/api/search --data-urlencode 'q={status=error}'TraceQL search
tempo-cli migrate config --mode=monolithic old.yaml > new.yaml2.x to 3.x config
In short : Tempo 3.x writes through Kafka (or in-process when monolithic), keeps recent traces in live-stores and the rest as Parquet in object storage. Set retention on the backend worker.

1.TraceQL

  • { conditions } selects spans, | chains a pipeline. A trace comes back if a spanset survives.
  • Intrinsics use a colon : span:duration, trace:duration, span:status.
  • Attributes use a dot : span.http.status_code, resource.service.name. Bare duration means span:duration.
  • Regex is fully anchored. Structure : >> descendant, > child, ~ sibling.
QueryFinds
{ resource.service.name = "frontend" && status = error }Failed spans in frontend
{ resource.service.name = "frontend" } >> { status = error }Errors anywhere below frontend
{ trace:duration > 5s }Slow traces
{ span.http.status_code >= 500 }Server errors
{ resource.service.name = "api" } | rate() by (span.http.status_code)Request rate by status (metrics)
{ span:name = "GET /:endpoint" } | quantile_over_time(span:duration, .99)p99 latency (metrics)
({ status = error } | rate()) / ({ } | rate())Error ratio (3.1 arithmetic)

2.TraceQL metrics

  • Spans to series : rate(), quantile_over_time(), by().
  • GA in 3.0. Alerting on them is experimental. 3.1 adds arithmetic.

3.Metrics from traces

The metrics-generator turns spans into series and remote-writes them to Prometheus (needs --web.enable-remote-write-receiver). Its processors do nothing until listed in overrides.defaults.metrics_generator.processors. Spans that ended over 30s ago are not counted.

SeriesLabelsAnswers
traces_spanmetrics_calls_totalservice, span_name, span_kind, status_codeRequest and error rate
traces_spanmetrics_latencysameLatency histogram
traces_service_graph_request_totalclient, server, connection_typeCalls between services
traces_service_graph_request_failed_totalsameFailed calls

Cardinality : span_name is usually the biggest driver; raw URLs or user IDs as dimensions explode it. The docs list ..._unpaired_spans_total and ..._dropped_spans_total service-graph series that Tempo never emits : do not alert on them.

4.Correlations in Grafana

  • Each jump is set on its own data source. serviceMap draws the service graph from Prometheus series. Trace to profiles (tracesToProfiles) is in Pyroscope.
  • Trace to logs : default time shift is 0, start with -2s / 2s. With Loki, the link shows only when a configured tag is on the span.
  • Exemplars need three switches : send_exemplars: true on the generator, Prometheus --enable-feature=exemplar-storage (else silently dropped), Grafana exemplarTraceIdDestinations.
  • Trace IDs go in Loki structured metadata, not labels (Loki and Alloy).
Metrics (Prometheus) Traces (Tempo) Logs (Loki) Profiles (Pyroscope) exemplar traceID tracesToLogsV2 tracesToMetrics derived field tracesToProfiles each arrow is set on one data source
Seen in practice

A monolithic Tempo from a rendered chart (Flux) on a local volume, a two-tier OTel Collector, no tail sampling, the generator feeding a RED dashboard. Pitfall : exemplars are sent, but Prometheus runs without exemplar-storage and Grafana has no exemplarTraceIdDestinations, so metric to trace is quietly broken. Do better : set all three switches and click one exemplar through to its trace.

In short : Find traces with TraceQL, turn spans into RED metrics with the metrics-generator, and wire each jump in Grafana on its own data source. Exemplars need all three switches.