Observability
Continuous Profiling with Pyroscope
1.The fourth signal
- Continuous profiling samples running code all the time, in production, and stores the stacks. Later you ask : which function used the CPU or memory at 03:12 ?
- A one-off
pprofgrab only answers "now, on this one instance". - A hot Pod in
kubectl top(Metrics Server) is the cue to open its profile.
Overhead : the docs say about 2 to 5%. That is the vendor's own estimate, with no published method.
| Question | Signal |
|---|---|
| Is it slow or failing ? | metrics, Prometheus |
| What happened ? | logs, Loki |
| Which request, which hop ? | traces, Tempo |
| Which line of code ? | profiles, Pyroscope |
2.Profiles and flame graphs
- A pprof profile is gzip-compressed protobuf : a list of samples. Each is a stack of locations plus one value per sample type (CPU nanoseconds, bytes allocated).
- Pyroscope names a type
<name>:<sample_. CPU time :type>:<sample_ unit>:<period_ type>:<period_ unit> process_.cpu:cpu:nanoseconds:cpu:nanoseconds
Reading a flame graph
- Width is the share of the total. Depth is the call stack. The x axis is not time.
- Self = time in the function itself. Total = with its callees.
- Alloc rising and never falling suggests a leak.
3.Profile types
| Type | Answers | Go endpoint |
|---|---|---|
| CPU | where CPU time goes | / |
| wall | where time goes, waiting included | none (Java, .NET, Node.js) |
| alloc | who allocates, and how much | / |
| inuse | what is live right now | /debug/pprof/heap |
| goroutine | how many goroutines, and where | / |
| mutex, block | lock and channel contention | off until runtime. / SetBlockProfileRate |
4.Four ways in
- Code change : pull needs pprof exposed, SDK push adds a library, eBPF and OTLP need none.
- Privileges : eBPF needs root and host PID, OTLP needs root for eBPF. Pull and SDK push need none.
- Maturity : pull, eBPF and SDK push are GA. OTLP is Alpha.
| Pull | eBPF | SDK push | OTLP | |
|---|---|---|---|---|
| Profile types | CPU, memory, goroutine, mutex, block | CPU only | richest per language | CPU |
| Custom labels | Pod labels | Pod labels | yes, dynamic | resource attributes |
In short : Profiles are sampled stacks with values, drawn as flame graphs where width is the share, not time. Pull, eBPF, SDK push and OTLP are four ways to get stacks to Pyroscope with a good
service_name.1.v2 storage
Pyroscope 2.0 (2026-04-20) made v2 storage the default : no ingesters and no local disks on the write path. Durability comes from object storage.
Ports4040 HTTP API, UI, ingest and OTLP (gRPC and HTTP; the gRPC client must speak cleartext HTTP/2, and it is not 4317 as in Tempo). 9095 internal gRPC. 7946 memberlist. 9099 metastore Raft.
2.Components
| Component | State | Job |
|---|---|---|
| distributor | stateless | validates, routes by tenant, then by service_name |
| segment-writer | stateless | buffers, writes a segment every 500ms, client waits until durable and indexed. A StatefulSet, no volumes |
| metastore | stateful | Raft index in 6-hour windows. Run 3 (1 failure) or 5 (2 failures) |
| compaction-worker | stateless | merges segments into blocks |
| query-frontend | stateless | plans the read from metastore metadata |
| query-backend | stateless | reads object storage directly, no cache layer |
3.Single binary vs microservices
- Single binary (
-target=all, the default) runs every role, with a one-member metastore. It may use the localfilesystembackend on a PVC. - Microservices need object storage;
filesystemis single node only.
v2 or dual ?
The chart renders plain
v2 since chart 2.0.2. The bare binary still defaults to v1-v2-dual : writes go to v2, old v1 data stays readable. v1 (ingesters) is deprecated, and v1 knobs like pyroscopedb.max-block-duration do nothing on v2.
4.Defaults that bite
| Setting | Default | Do |
|---|---|---|
pyroscope. | false : profiles lost on restart | enable it |
extraArgs. | debug | set info |
retention vs querier. | 31 days kept, 7 queryable, older clipped | raise the lookback |
querier. | 1d | one query covers a day at most |
| ingestion limit | 4 MB/s, burst 2 MB, then HTTP 429 | use Alloy 1.19+ (retries 429) |
| Command | What it does |
|---|---|
helm -n observability install pyroscope grafana/pyroscope --set pyroscope. | single binary with a PVC |
kubectl -n observability port-forward svc/pyroscope 4040 | open the UI |
curl -s localhost:4040/ | readiness |
profilecli query series --query='{service_ | list series |
profilecli upload --extra-labels=service_ | upload a pprof file |
In short : Pyroscope v2 writes segments straight to object storage and keeps only its Raft metastore stateful. Turn on persistence, quiet the logs, and remember profiles are kept 31 days but only 7 are queryable.
1.Annotation pull
- Annotate the Pod template.
.portor.port_nameis required : the Helm doc's named-port fallback is not in the template. - These keys are the chart's embedded Alloy. Plain Alloy reads
pyroscope..io/ service_ name service_nameis the main label (UI, distributor); a missing one isunspecified.
# checkout Deployment, pod template annotations: profiles.grafana.com/cpu.scrape: "true" profiles.grafana.com/cpu.port_name: http-metrics profiles.grafana.com/memory.scrape: "true" profiles.grafana.com/memory.port_name: http-metrics profiles.grafana.com/service_name: "checkout" # containers[].ports: name http-metrics, 8080 # the app imports net/http/pprof
2.eBPF DaemonSet
pyroscope.ebpfsamples every CPU at 19 Hz (since Alloy 1.11.0; 97 before) and sends every 15s.- Root in the host PID namespace (
privilegedandhostPID, or a capability list), so its namespace needs theprivilegedlevel (Pod Security). - One Pod per node (DaemonSet), local Pods by
spec.nodeName, tolerations for control-plane nodes.
- CPU only (off-CPU sampling exists, off by default).
- Stripped binaries show module names only; very shallow stacks mean no frame pointers.
- Never set a CPU limit on a profiler : throttling drops samples when the node is busiest.
- Alloy syntax is taught in Loki and Alloy.
3.From a slow span to its code
- Wrap the tracer provider (
otelpyroscope.in Go). Samples in the local root span getNewTracerProvider( tp) span_id,trace_id; the span getspyroscope..profile. id - CPU (Go, Ruby, .NET, Python); CPU or wall (Java). Spans under about 10ms may have no samples.
- The Tempo data source's
tracesToProfiles(datasourceUid,profileTypeId,tags) adds Profiles for this span (span start minus 60s to end plus 60s). - Map
service.name(from the OTel Operator or SDK) toservice_name. No id or tags, no link (others : Tempo).
4.OTel profiles and the gotchas
OTel Profiles : public Alpha since 2026-03-26, proto still v1development; Pyroscope calls OTLP ingest "suitable for development and testing". Relabel the eBPF profiler's process. to service_name at ingest.
| Symptom | Cause | Fix |
|---|---|---|
| no profiles from an annotated Pod | no .port or .port_name | add one |
| a new "service" every rollout | service_name built from the Pod name | use namespace and container |
| no memory profiles | eBPF is CPU only | use pull or an SDK |
| empty mutex and block | off in the Go runtime | call SetMutexProfileFraction / SetBlockProfileRate |
| nothing older than 7 days | 7-day query lookback (data kept 31) | raise max-query-lookback |
Seen in practice
Single-binary Pyroscope on a
local-path volume (one node), the chart's Alloy scraping only itself, and a privileged eBPF Alloy DaemonSet profiling every Pod's CPU, all rendered charts under Flux CD v2. Pitfalls : CPU limits throttle both Alloys; an eBPF service_name built from Pod names churns per rollout; traces-to-profiles left unwired. Do better : no CPU limit on profilers, a stable service_name per workload, span profiles.
In short : Annotate Pods for pull, run eBPF as a privileged DaemonSet with no CPU limit for CPU everywhere, and use an SDK when you need more types or span profiles. Map
service.name to service_name so a slow span leads to its flame graph.