Observability

Loki and Log Pipelines with Alloy

1.Labels, streams, chunks

  • kubectl logs shows one container (Pods). Loki keeps every Pod's lines, found by label.
  • A stream = one tenant + one unique label set. Its lines are compressed into chunks; a small TSDB index maps label sets to chunks. Both live in object storage.
  • A query picks streams by label, then greps, parses and aggregates the lines.
  • Cardinality matters : 15 label names per series by default; under 100,000 active streams, even for a huge tenant.
Good labels cluster, namespace, app, container. Few values, and they do not change per request.
Bad labels pod name, request ID, trace ID. Every new value is a new stream. Put them in structured metadata or in the line.

2.Write path and read path

  • The distributor hashes tenant + labels onto the ring. Quorum floor(RF / 2) + 1 : RF 3 needs 2 writes.
  • Out-of-order lines only within 1h of the stream's newest line (older docs say 2h), else too_far_behind.
  • The compactor keeps one index per tenant per day and applies retention.
Alloy / OTel client push distributor validate, rate limit, hash ingesters replicas, WAL, chunks flush Grafana logcli query frontend split, cache querier runs LogQL last 3h older object storage chunks + TSDB index

3.Labels vs structured metadata vs line

Structured metadata (Loki 3.0+) : kept with each line, not indexed, still queryable. Trace IDs go here (Tempo).

Index labelsStructured metadataLine
indexedyes (TSDB)nono
defines a streamyesnono
good forbounded valuespod, trace ID, user IDthe rest
query{app="x"}| trace_id="abc" after the selector|= "abc", parsers
needsany schematsdb + v13any

4.Deployment modes

ModeTargetsScaleStatus
Monolithic-target=allup to about 20 GB/dayHelm chart default since 12.0.0
Simple scalablewrite, read, backendabout 1 TB/daydeprecated : Loki 4.0 will refuse to start
Microservicesone per componentvery largesupported

The chart moved to grafana-community on 2026-03-16. Chart 17.0.0 refuses minio.enabled=true; the MinIO subchart goes on 2026-10-31.

5.Schema and retention

  • New installs : tsdb, v13, period: 24h. Structured metadata is on, so Loki will not start unless the active period is v13 + tsdb.
  • A schema change cannot be rolled back : add a new period starting at a future date.
  • Retention is off by default. The compactor runs it : retention_enabled, delete_request_store (required), retention_period.
  • Markers sit on local disk : a PVC (Storage) or 3.7's deletion_marker_object_store_prefix.
Bucket lifecycle rule : over retention + 2h, tenant chunk prefix only; an empty prefix also expires index/.
In short : Loki indexes labels, not text. A stream is one label set, its lines are compressed into chunks in object storage, and anything high-cardinality belongs in structured metadata or the line.

1.What is Alloy ?

  • Grafana's collector : an OTel Collector distribution with native Prometheus, Loki and Pyroscope pipelines.
  • It replaces two retired agents : Grafana Agent (EOL 2025-11-01) and Promtail (EOL 2026-03-02, not part of Loki 3.7).
  • alloy convert --source-format=promtail migrates. Positions and metric names change : dashboards break.

2.Alloy syntax in five lines

  • Block : <component.name> "<label>" { attr = value }.
  • Export : discovery.kubernetes.pods.targets. References build a graph, so file order does not matter.
  • // comments, backtick raw strings for regexes. alloy fmt, alloy validate.
  • --stability.level gates experimental components.
  • UI on 127.0.0.1:12345 (the chart sets 0.0.0.0). Reload : /-/reload or SIGHUP.

3.A Kubernetes log pipeline

loki.source.file "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.process.pods.receiver]
  file_match {
    enabled = true // else the glob is literal
  }
}
loki.process "pods" {
  stage.cri {}
  forward_to = [loki.write.default.receiver]
}
discovery.kubernetes Pods on this node targets discovery.relabel labels + __path__ output loki.source.file file_match { enabled = true } forward_to loki.process stage.cri, labels receiver loki.write batch, retry POST Loki /loki/api/v1/push
Three traps
  • A glob __path__ (/var/log/pods/*$1/*.log) needs file_match. alloy validate stays quiet.
  • Set alloy.mounts.varlog: true. Persist storagePath : /tmp/alloy loses positions.
  • No clustering on a log-tailing DaemonSet.
loki.write Batches 1MiB or 1s; on 5xx or 429 backs off 500ms to 5m, drops after 10 retries; other 4xx drop at once. Watch loki_write_dropped_entries_total. WAL experimental, off.

4.File tailing vs API tailing

loki.source.fileloki.source.kubernetes
reads/var/log/pods on the nodekubelet log endpoint, through the API server
needshostPath, usually root (Pod Security)read on pods/log (RBAC), no privileges
runs asDaemonSet (DaemonSets)Deployment, clustered
restart safetypositions file of offsetspositions file of last timestamps
costnode disk, inotifykubelet CPU and network
node logsyesno

5.loki.process stages that matter

StageDoes
stage.cristrip CRI prefix, join partials
stage.json / logfmtextract fields
stage.labelsbounded field to a label
stage.structured_metadatahigh-cardinality field, unindexed
stage.drop / multilinedrop noise, join stack traces

6.Other ways in

  • Events : loki.source.kubernetes_events, run once or clustered, never per node (Events).
  • OTLP : otelcol.receiver.otlp to otelcol.exporter.otlphttp, endpoint http://loki:3100/otlp. It appends /v1/logs, so the full path fails. Collector setup : OTel Operator.
In short : Alloy wires components into a graph by reference : discover Pods, relabel them, tail files or the API, turn fields into labels or structured metadata, and push the lines to Loki.

1.Query shape

  • A log query is a stream selector, then a pipeline : line filters |=, !=, |~, !~ (this regex is not anchored, unlike label matchers), a parser (json, logfmt, pattern, regexp), label filters, line_format.
  • A parse failure adds __error__ (filter __error__=""). One matcher must not match empty : {app=~".*"} is rejected.
{namespace="shop"} index |= "error" | json | status >= 500 line_format on each line : only the first box uses the index

2.Metric queries

Like PromQL : wrap a log query in rate, count_over_time, bytes_rate; unwrap a field for quantile_over_time; sum by.

QueryGives
{namespace="shop", container="api"} |= "error" != "timeout"error lines, minus timeouts
{job="ingress"} | json | status >= 5005xx lines, parsed
sum by (container) (rate({namespace="shop"} |= "error" [5m]))errors per second, per container
{job="x"} | trace_id="0242ac120002"one trace's lines (metadata)
quantile_over_time(0.99, {app="api"} | logfmt | unwrap duration_seconds(latency) [5m]) by (route)p99 latency per route

3.OTLP and its labels

  • OTLP logs go to /otlp/v1/logs. Loki 3.7.8 indexes 18 resource attributes (docs say 17, the code wins), dots as underscores : service_name, k8s_pod_name ... The rest is structured metadata.
  • Grafana now advises demoting k8s.pod.name and service.instance.id : replace the global list, or per tenant set ignore_defaults: true and list what to index.
  • A bare per-tenant structured_metadata rule loses : defaults are prepended, first match wins.

4.Limits and discard reasons

ReasonHTTPKnob (default)
rate_limited429ingestion_rate_mb 4, burst 6
per_stream_rate_limit4293MB
stream_limit429max_global_streams_per_user 5000
line_too_long400max_line_size 256KB
too_far_behind400the 1h window
max_label_names_per_series40015

Counted in loki_discarded_samples_total{reason}. "too many outstanding requests" is the query queue : frontend max_outstanding_per_tenant 2048, scheduler max_outstanding_requests_per_tenant 32000.

5.Running it

CommandDoes
logcli query --since 1h --limit 100 '{namespace="shop"} |= "error"'LogQL from a shell
logcli series '{}' --since=1h --analyze-labelsfind high-cardinality labels
loki -config.file=loki.yaml -verify-config=truecheck a config and exit
curl http://loki:3100/readyreadiness, says ready
alloy validate config.alloyschema errors, not runtime ones
kubectl debug -it loki-0 --image=busybox --target=lokia shell : no shell in the image since 3.6.0
Seen in practice Monolithic Loki, a rendered chart under Flux, chunks in a self-hosted S3 store with SPIRE-issued tokens, no static keys (SPIRE). Logs come from an OTel Collector DaemonSet tailing /var/log/pods over OTLP (OTel Operator), not Alloy. Pitfall : the OTLP default labels, k8s_pod_name included, against current advice. Do better : demote pod names to structured metadata, keep a few bounded labels.
In short : Pick streams by label, then filter and parse the lines. Wrap a log query in rate or count_over_time to get numbers, and keep OTLP labels few.