Observability

Continuous Profiling with Pyroscope

1.The fourth signal

  • Continuous profiling samples running code all the time, in production, and stores the stacks. Later you ask : which function used the CPU or memory at 03:12 ?
  • A one-off pprof grab only answers "now, on this one instance".
  • A hot Pod in kubectl top (Metrics Server) is the cue to open its profile.

Overhead : the docs say about 2 to 5%. That is the vendor's own estimate, with no published method.

QuestionSignal
Is it slow or failing ?metrics, Prometheus
What happened ?logs, Loki
Which request, which hop ?traces, Tempo
Which line of code ?profiles, Pyroscope

2.Profiles and flame graphs

  • A pprof profile is gzip-compressed protobuf : a list of samples. Each is a stack of locations plus one value per sample type (CPU nanoseconds, bytes allocated).
  • Pyroscope names a type <name>:<sample_type>:<sample_unit>:<period_type>:<period_unit>. CPU time : process_cpu:cpu:nanoseconds:cpu:nanoseconds.
Reading a flame graph root (all samples) handler gc json encode db query depth = stack
  • Width is the share of the total. Depth is the call stack. The x axis is not time.
  • Self = time in the function itself. Total = with its callees.
  • Alloc rising and never falling suggests a leak.

3.Profile types

TypeAnswersGo endpoint
CPUwhere CPU time goes/debug/pprof/profile?seconds=N
wallwhere time goes, waiting includednone (Java, .NET, Node.js)
allocwho allocates, and how much/debug/pprof/allocs
inusewhat is live right now/debug/pprof/heap
goroutinehow many goroutines, and where/debug/pprof/goroutine
mutex, blocklock and channel contentionoff until runtime.Set­Mutex­Profile­Fraction / Set­Block­Profile­Rate

4.Four ways in

Pod /debug/pprof node kernel app + SDK OTel Collectorprofiling receiver Alloy pyroscope.scrape Alloy pyroscope.ebpf(DaemonSet) pyroscope.receive_http pullevery 15s samples push pyroscope.write Pyroscope :4040 push, direct OTLP, alpha
  • Code change : pull needs pprof exposed, SDK push adds a library, eBPF and OTLP need none.
  • Privileges : eBPF needs root and host PID, OTLP needs root for eBPF. Pull and SDK push need none.
  • Maturity : pull, eBPF and SDK push are GA. OTLP is Alpha.
PulleBPFSDK pushOTLP
Profile typesCPU, memory, goroutine, mutex, blockCPU onlyrichest per languageCPU
Custom labelsPod labelsPod labelsyes, dynamicresource attributes
In short : Profiles are sampled stacks with values, drawn as flame graphs where width is the share, not time. Pull, eBPF, SDK push and OTLP are four ways to get stacks to Pyroscope with a good service_name.

1.v2 storage

distributor segment-writer metastore (Raft)the only stateful part register compaction-worker jobs query-frontend which blocks ? query-backend plan object storage (segments, then blocks) segment every 500ms merge read directly

Pyroscope 2.0 (2026-04-20) made v2 storage the default : no ingesters and no local disks on the write path. Durability comes from object storage.

Ports4040 HTTP API, UI, ingest and OTLP (gRPC and HTTP; the gRPC client must speak cleartext HTTP/2, and it is not 4317 as in Tempo). 9095 internal gRPC. 7946 memberlist. 9099 metastore Raft.

2.Components

ComponentStateJob
distributorstatelessvalidates, routes by tenant, then by service_name
segment-writerstatelessbuffers, writes a segment every 500ms, client waits until durable and indexed. A StatefulSet, no volumes
metastorestatefulRaft index in 6-hour windows. Run 3 (1 failure) or 5 (2 failures)
compaction-workerstatelessmerges segments into blocks
query-frontendstatelessplans the read from metastore metadata
query-backendstatelessreads object storage directly, no cache layer

3.Single binary vs microservices

  • Single binary (-target=all, the default) runs every role, with a one-member metastore. It may use the local filesystem backend on a PVC.
  • Microservices need object storage; filesystem is single node only.
v2 or dual ? The chart renders plain v2 since chart 2.0.2. The bare binary still defaults to v1-v2-dual : writes go to v2, old v1 data stays readable. v1 (ingesters) is deprecated, and v1 knobs like pyroscopedb.max-block-duration do nothing on v2.

4.Defaults that bite

SettingDefaultDo
pyroscope.persistence.enabledfalse : profiles lost on restartenable it
extraArgs.log.leveldebugset info
retention vs querier.max-query-lookback31 days kept, 7 queryable, older clippedraise the lookback
querier.max-query-length1done query covers a day at most
ingestion limit4 MB/s, burst 2 MB, then HTTP 429use Alloy 1.19+ (retries 429)
CommandWhat it does
helm -n observability install pyroscope grafana/pyroscope --set pyroscope.persistence.enabled=truesingle binary with a PVC
kubectl -n observability port-forward svc/pyroscope 4040open the UI
curl -s localhost:4040/readyreadiness
profilecli query series --query='{service_name="checkout"}'list series
profilecli upload --extra-labels=service_name=payments ./cpu.pprofupload a pprof file
In short : Pyroscope v2 writes segments straight to object storage and keeps only its Raft metastore stateful. Turn on persistence, quiet the logs, and remember profiles are kept 31 days but only 7 are queryable.

1.Annotation pull

  • Annotate the Pod template. .port or .port_name is required : the Helm doc's named-port fallback is not in the template.
  • These keys are the chart's embedded Alloy. Plain Alloy reads pyroscope.io/service_name.
  • service_name is the main label (UI, distributor); a missing one is unspecified.
# checkout Deployment, pod template
annotations:
  profiles.grafana.com/cpu.scrape: "true"
  profiles.grafana.com/cpu.port_name: http-metrics
  profiles.grafana.com/memory.scrape: "true"
  profiles.grafana.com/memory.port_name: http-metrics
  profiles.grafana.com/service_name: "checkout"
# containers[].ports: name http-metrics, 8080
# the app imports net/http/pprof

2.eBPF DaemonSet

  • pyroscope.ebpf samples every CPU at 19 Hz (since Alloy 1.11.0; 97 before) and sends every 15s.
  • Root in the host PID namespace (privileged and hostPID, or a capability list), so its namespace needs the privileged level (Pod Security).
  • One Pod per node (DaemonSet), local Pods by spec.nodeName, tolerations for control-plane nodes.
  • CPU only (off-CPU sampling exists, off by default).
  • Stripped binaries show module names only; very shallow stacks mean no frame pointers.
  • Never set a CPU limit on a profiler : throttling drops samples when the node is busiest.
  • Alloy syntax is taught in Loki and Alloy.

3.From a slow span to its code

  • Wrap the tracer provider (otelpyroscope.NewTracerProvider(tp) in Go). Samples in the local root span get span_id, trace_id; the span gets pyroscope.profile.id.
  • CPU (Go, Ruby, .NET, Python); CPU or wall (Java). Spans under about 10ms may have no samples.
  • The Tempo data source's tracesToProfiles (datasourceUid, profileTypeId, tags) adds Profiles for this span (span start minus 60s to end plus 60s).
  • Map service.name (from the OTel Operator or SDK) to service_name. No id or tags, no link (others : Tempo).
trace in Tempo span (pyroscope.profile.id) Profiles for this span Pyroscope, filtered by span_id flame graph of that span

4.OTel profiles and the gotchas

OTel Profiles : public Alpha since 2026-03-26, proto still v1development; Pyroscope calls OTLP ingest "suitable for development and testing". Relabel the eBPF profiler's process.executable.name to service_name at ingest.

SymptomCauseFix
no profiles from an annotated Podno .port or .port_nameadd one
a new "service" every rolloutservice_name built from the Pod nameuse namespace and container
no memory profileseBPF is CPU onlyuse pull or an SDK
empty mutex and blockoff in the Go runtimecall Set­Mutex­Profile­Fraction / Set­Block­Profile­Rate
nothing older than 7 days7-day query lookback (data kept 31)raise max-query-lookback
Seen in practice Single-binary Pyroscope on a local-path volume (one node), the chart's Alloy scraping only itself, and a privileged eBPF Alloy DaemonSet profiling every Pod's CPU, all rendered charts under Flux CD v2. Pitfalls : CPU limits throttle both Alloys; an eBPF service_name built from Pod names churns per rollout; traces-to-profiles left unwired. Do better : no CPU limit on profilers, a stable service_name per workload, span profiles.
In short : Annotate Pods for pull, run eBPF as a privileged DaemonSet with no CPU limit for CPU everywhere, and use an SDK when you need more types or span profiles. Map service.name to service_name so a slow span leads to its flame graph.