Observability
Loki and Log Pipelines with Alloy
1.Labels, streams, chunks
kubectl logsshows one container (Pods). Loki keeps every Pod's lines, found by label.- A stream = one tenant + one unique label set. Its lines are compressed into chunks; a small TSDB index maps label sets to chunks. Both live in object storage.
- A query picks streams by label, then greps, parses and aggregates the lines.
- Cardinality matters : 15 label names per series by default; under 100,000 active streams, even for a huge tenant.
Good labels
cluster, namespace, app, container. Few values, and they do not change per request.
Bad labels
pod name, request ID, trace ID. Every new value is a new stream. Put them in structured metadata or in the line.
2.Write path and read path
- The distributor hashes tenant + labels onto the ring. Quorum
floor(RF / 2) + 1: RF 3 needs 2 writes. - Out-of-order lines only within 1h of the stream's newest line (older docs say 2h), else
too_far_behind. - The compactor keeps one index per tenant per day and applies retention.
3.Labels vs structured metadata vs line
Structured metadata (Loki 3.0+) : kept with each line, not indexed, still queryable. Trace IDs go here (Tempo).
| Index labels | Structured metadata | Line | |
|---|---|---|---|
| indexed | yes (TSDB) | no | no |
| defines a stream | yes | no | no |
| good for | bounded values | pod, trace ID, user ID | the rest |
| query | {app="x"} | | trace_id="abc" after the selector | |= "abc", parsers |
| needs | any schema | tsdb + v13 | any |
4.Deployment modes
| Mode | Targets | Scale | Status |
|---|---|---|---|
| Monolithic | -target=all | up to about 20 GB/day | Helm chart default since 12.0.0 |
| Simple scalable | write, read, backend | about 1 TB/day | deprecated : Loki 4.0 will refuse to start |
| Microservices | one per component | very large | supported |
The chart moved to grafana-community on 2026-03-16. Chart 17.0.0 refuses minio.; the MinIO subchart goes on 2026-10-31.
5.Schema and retention
- New installs :
tsdb,v13,period: 24h. Structured metadata is on, so Loki will not start unless the active period isv13+tsdb. - A schema change cannot be rolled back : add a new period starting at a future date.
- Retention is off by default. The compactor runs it :
retention_enabled,delete_(required),request_ store retention_period. - Markers sit on local disk : a PVC (Storage) or 3.7's
deletion_.marker_ object_ store_ prefix
Bucket lifecycle rule : over retention + 2h, tenant chunk prefix only; an empty prefix also expires
index/.In short : Loki indexes labels, not text. A stream is one label set, its lines are compressed into chunks in object storage, and anything high-cardinality belongs in structured metadata or the line.
1.What is Alloy ?
- Grafana's collector : an OTel Collector distribution with native Prometheus, Loki and Pyroscope pipelines.
- It replaces two retired agents : Grafana Agent (EOL 2025-11-01) and Promtail (EOL 2026-03-02, not part of Loki 3.7).
alloy convert --source-format=promtailmigrates. Positions and metric names change : dashboards break.
2.Alloy syntax in five lines
- Block :
<component..name> "<label>" { attr = value } - Export :
discovery.. References build a graph, so file order does not matter.kubernetes. pods. targets //comments, backtick raw strings for regexes.alloy fmt,alloy validate.--stability.levelgates experimental components.- UI on
127.0.0.1:12345(the chart sets0.0.0.0). Reload :/-/reloador SIGHUP.
3.A Kubernetes log pipeline
loki.source.file "pods" {
targets = discovery.relabel.pods.output
forward_to = [loki.process.pods.receiver]
file_match {
enabled = true // else the glob is literal
}
}
loki.process "pods" {
stage.cri {}
forward_to = [loki.write.default.receiver]
}
Three traps
- A glob
__path__(/) needsvar/ log/ pods/*$1/*. log file_match.alloy validatestays quiet. - Set
alloy.. Persistmounts. varlog: true storagePath:/tmp/alloyloses positions. - No clustering on a log-tailing DaemonSet.
loki.write
Batches 1MiB or 1s; on 5xx or 429 backs off 500ms to 5m, drops after 10 retries; other 4xx drop at once. Watch
loki_write_dropped_entries_total . WAL experimental, off.
4.File tailing vs API tailing
| loki.source. | loki.source. | |
|---|---|---|
| reads | /var/log/pods on the node | kubelet log endpoint, through the API server |
| needs | hostPath, usually root (Pod Security) | read on pods/log (RBAC), no privileges |
| runs as | DaemonSet (DaemonSets) | Deployment, clustered |
| restart safety | positions file of offsets | positions file of last timestamps |
| cost | node disk, inotify | kubelet CPU and network |
| node logs | yes | no |
5.loki.process stages that matter
| Stage | Does |
|---|---|
stage.cri | strip CRI prefix, join partials |
stage.json / logfmt | extract fields |
stage.labels | bounded field to a label |
stage. | high-cardinality field, unindexed |
stage.drop / multiline | drop noise, join stack traces |
6.Other ways in
- Events :
loki., run once or clustered, never per node (Events).source. kubernetes_ events - OTLP :
otelcol.toreceiver. otlp otelcol., endpointexporter. otlphttp http://. It appendsloki:3100/ otlp /v1/logs, so the full path fails. Collector setup : OTel Operator.
In short : Alloy wires components into a graph by reference : discover Pods, relabel them, tail files or the API, turn fields into labels or structured metadata, and push the lines to Loki.
1.Query shape
- A log query is a stream selector, then a pipeline : line filters
|=,!=,|~,!~(this regex is not anchored, unlike label matchers), a parser (json,logfmt,pattern,regexp), label filters,line_format. - A parse failure adds
__error__(filter__error__=""). One matcher must not match empty :{app=~".*"}is rejected.
2.Metric queries
Like PromQL : wrap a log query in rate, count_over_time, bytes_rate; unwrap a field for quantile_; sum by.
| Query | Gives |
|---|---|
{namespace="shop", container="api"} |= "error" != "timeout" | error lines, minus timeouts |
{job="ingress"} | json | status >= 500 | 5xx lines, parsed |
sum by (container) ( | errors per second, per container |
{job="x"} | trace_ | one trace's lines (metadata) |
quantile_ | p99 latency per route |
3.OTLP and its labels
- OTLP logs go to
/otlp/v1/logs. Loki 3.7.8 indexes 18 resource attributes (docs say 17, the code wins), dots as underscores :service_name,k8s_pod_name... The rest is structured metadata. - Grafana now advises demoting
k8s.pod.nameandservice.: replace the global list, or per tenant setinstance. id ignore_defaults: trueand list what to index. - A bare per-tenant
structured_rule loses : defaults are prepended, first match wins.metadata
4.Limits and discard reasons
| Reason | HTTP | Knob (default) |
|---|---|---|
rate_limited | 429 | ingestion_rate_mb 4, burst 6 |
per_ | 429 | 3MB |
stream_limit | 429 | max_ 5000 |
line_too_long | 400 | max_line_size 256KB |
too_far_behind | 400 | the 1h window |
max_ | 400 | 15 |
Counted in loki_. "too many outstanding requests" is the query queue : frontend max_ 2048, scheduler max_ 32000.
5.Running it
| Command | Does |
|---|---|
logcli query --since 1h --limit 100 '{namespace="shop"} |= "error"' | LogQL from a shell |
logcli series '{}' --since=1h --analyze-labels | find high-cardinality labels |
loki -config. | check a config and exit |
curl http:// | readiness, says ready |
alloy validate config.alloy | schema errors, not runtime ones |
kubectl debug -it loki-0 --image=busybox --target=loki | a shell : no shell in the image since 3.6.0 |
Seen in practice
Monolithic Loki, a rendered chart under Flux, chunks in a self-hosted S3 store with SPIRE-issued tokens, no static keys (SPIRE). Logs come from an OTel Collector DaemonSet tailing
/var/log/pods over OTLP (OTel Operator), not Alloy. Pitfall : the OTLP default labels, k8s_pod_name included, against current advice. Do better : demote pod names to structured metadata, keep a few bounded labels.
In short : Pick streams by label, then filter and parse the lines. Wrap a log query in
rate or count_over_time to get numbers, and keep OTLP labels few.