LabHub

Blog

Grafana Loki and LogQL Master Guide: From Log Pipeline Design to Operations

2026-03-05 · 18 min read · #observability#loki#logql#grafana#2026-03

한국어English日本語

Grafana Loki and LogQL Master Guide: From Log Pipeline Design to Operations

Overview

Logs are the last line of defense in incident analysis. If metrics tell you "what is broken", logs explain "why it broke". But in a large-scale distributed system, collecting and searching hundreds of gigabytes of logs a day is never simple. The Elasticsearch-based ELK stack offers powerful full-text search, but its indexing cost and operational complexity are high.

Grafana Loki takes a fundamentally different approach to this problem. It does not index the log body; it keeps only a label-based index and stores compressed log chunks in object storage. Thanks to this design, storage cost is 70-80% lower than Elasticsearch, and even multi-terabyte log volumes can be operated on cheap object storage such as S3 or GCS.

This article covers the internal architecture of Loki 3.x, the three deployment modes, LogQL query syntax and optimization patterns, a log collection pipeline built on Grafana Alloy, TSDB storage configuration and retention, alert rule composition, real-world troubleshooting cases, and disaster recovery procedures.

Loki Architecture Deep Dive

Loki is designed as a microservice-based architecture in which each component has a clearly defined role. The core philosophy is "treat logs like metrics". It applies the Prometheus label model to logs as-is, so metrics and logs can be correlated through the same label set.

Core Components

Distributor is the entry point that receives push requests from log collection agents such as Alloy and Promtail. It validates the labels of the incoming log streams and routes them to the right Ingester through a consistent hash ring. Rate limiting and validation are handled at this stage, so malformed labels or excessive traffic are blocked before they reach an Ingester.

Ingester accumulates log data in an in-memory buffer, then compresses it into chunks and flushes them to object storage. It uses a WAL (Write-Ahead Log) to prevent data loss when the process terminates abnormally. The Ingester is the component that consumes the most memory, so it deserves the most attention when sizing resources.

Query Frontend receives query requests from clients and spreads their execution across multiple Queriers. It improves query performance through query splitting, which divides a time range into several intervals, and through result caching.

Querier is the worker that actually processes queries. It searches the in-memory data in the Ingesters and the block data in object storage at the same time, then merges the results.

Compactor periodically merges the index files in object storage and deletes expired data according to the retention policy. From Loki 3.6 a horizontally scalable Compactor was introduced, which greatly improved the processing speed of large deletion requests.

Index Gateway handles queries against index data centrally, reducing how often Queriers reach object storage directly. It is especially useful in microservices mode.

Data Flow

[Application] --> [Grafana Alloy] --> [Distributor]
                                           |
                                    [Hash Ring]
                                           |
                                      [Ingester]
                                       /      \
                              [WAL]         [Object Storage (S3/GCS)]
                                                  |
                              [Compactor] <-------+
                                                  |
                              [Query Frontend] ---+---> [Querier]
                                                           |
                                                  [Index Gateway]

Deployment Mode Selection

Loki offers three deployment modes; pick one according to log volume and operational maturity.

Monolithic Mode

All components run in a single process. This suits development environments or small deployments handling a few gigabytes a day or less.

# Helm values - Monolithic mode
loki:
  deploymentMode: SingleBinary
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  schemaConfig:
    configs:
      - from: '2024-01-01'
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h
singleBinary:
  replicas: 1
  persistence:
    size: 50Gi

This is the default mode of the Loki Helm chart, split into three targets: read, write, and backend. It can handle anywhere from a few hundred gigabytes to roughly 1TB of logs a day, which makes it the best choice for most production environments.

# Helm values - Simple Scalable mode
loki:
  deploymentMode: SimpleScalable
  auth_enabled: true
  commonConfig:
    replication_factor: 3
  storage:
    type: s3
    s3:
      endpoint: s3.ap-northeast-2.amazonaws.com
      region: ap-northeast-2
      bucketnames: company-loki-logs
      access_key_id: ${AWS_ACCESS_KEY_ID}
      secret_access_key: ${AWS_SECRET_ACCESS_KEY}
  schemaConfig:
    configs:
      - from: '2024-01-01'
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: index_
          period: 24h
write:
  replicas: 3
  persistence:
    size: 50Gi
  resources:
    requests:
      cpu: '1'
      memory: 2Gi
    limits:
      memory: 4Gi
read:
  replicas: 3
  resources:
    requests:
      cpu: '1'
      memory: 2Gi
    limits:
      memory: 4Gi
backend:
  replicas: 2
  persistence:
    size: 50Gi
gateway:
  replicas: 2

The write target is deployed as a StatefulSet so that it retains WAL data, and the read target is deployed as a Deployment so that it can autoscale. You must place a reverse proxy (gateway) in front to route API requests to the read/write nodes.

Microservices Mode

Each component is deployed as an independent process. Choose this for environments that exceed 1TB a day, or when fine-grained per-component scaling is required. A default Helm deployment creates 3 Distributors, 3 Ingesters, 3 Queriers, 2 Query Frontends, 2 Index Gateways, and 1 Compactor.

Deployment modeDaily log volumeOperational complexityPrimary target
MonolithicA few GB or lessLowDev/test environments
Simple ScalableTens of GB ~ 1TBMediumMost production deployments
MicroservicesOver 1TBHighLarge-scale multi-tenant

Mastering LogQL Queries

LogQL is a Loki-specific query language inspired by the PromQL of Prometheus. It starts with a log stream selector and chains pipeline stages, and it supports two types: log search (Log Query) and metric conversion (Metric Query).

Stream Selectors and Line Filters

Every LogQL query starts with a stream selector. The stream selector is what uses the Loki index, so making it as specific as possible improves query performance.

# Basic stream selector
{namespace="production", app="payment-service"}

# Chaining line filters - filters apply in order, so put the narrowest filter first
{namespace="production", app="payment-service"}
  |= "error"
  != "health-check"
  |~ "timeout|connection refused"

# Pattern match filter in Loki 3.x - 10x faster than regex
{namespace="production"} |> "error <_> timeout"

Filter operators:

Parsers and Label Extraction

Loki provides the json, logfmt, pattern, regexp, and unpack parsers. For JSON or logfmt formats the dedicated parser is the most efficient, and for unstructured logs the pattern parser is faster than regexp.

# JSON parser - extract fields from structured logs
{app="api-gateway"} | json | status >= 500

# logfmt parser
{app="auth-service"} | logfmt | level="error" | duration > 5s

# pattern parser - parsing an nginx access log
{app="nginx"}
  | pattern "<ip> - - [<timestamp>] \"<method> <path> <_>\" <status> <bytes>"
  | status >= 400
  | line_format "{{.ip}} {{.method}} {{.path}} {{.status}}"

# regexp parser - complex unstructured logs
{app="legacy-service"}
  | regexp "(?P<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) \\[(?P<level>\\w+)\\] (?P<message>.*)"
  | level="ERROR"

Parser performance rule of thumb: json/logfmt are the fastest, pattern comes next, and regexp is the slowest. Where possible, place a line filter ahead of the parser so that the number of log lines to be parsed is reduced first.

Metric Queries - Turning Logs into Numbers

LogQL metric queries convert log streams into time series data. This makes it possible to build dashboards and alerts from log data alone, without collecting separate metrics.

# Error log rate per second
rate({namespace="production", app="payment-service"} |= "error" [5m])

# 5xx error count per service
sum by (app) (
  count_over_time(
    {namespace="production"} | json | status >= 500 [5m]
  )
)

# P99 response time computed with unwrap
quantile_over_time(0.99,
  {app="api-gateway"}
    | json
    | unwrap response_time_ms
    | __error__=""
  [5m]
) by (endpoint)

# Average response time trend
avg_over_time(
  {app="api-gateway"}
    | json
    | unwrap duration_seconds
    | __error__=""
  [5m]
) by (service)

# bytes_over_time - monitoring log volume
sum by (namespace) (bytes_over_time({namespace=~".+"} [1h]))

The __error__="" filter is the essential pattern for excluding lines that carry an error from the parsing or unwrap stage. Omit it and non-numeric values slip in, producing inaccurate results.

Practical Query Optimization Tips

  1. Make the stream selector as specific as possible. Instead of {app="payment"}, adding labels as in {namespace="production", app="payment", env="prod"} shrinks the set of streams that has to be searched.
  2. Place line filters ahead of parsers. The order |= "error" | json is faster than | json |= "error", because the line filter runs first and fewer lines are left to parse.
  3. Use pattern matching or string filters instead of regex. |= "error" |= "timeout" is far faster than |~ "error.*timeout".
  4. Keep the time range as narrow as possible. The performance gap between searching the last hour and the last 7 days is dramatic.
  5. Do not extract labels you do not need. | json extracts every JSON field as a label, so when only specific fields are needed, name them explicitly as in | json status, duration.

Log Collection Pipeline Built on Grafana Alloy

Promtail entered LTS in February 2025 and reaches EOL in March 2026. Grafana Alloy is the official successor collector, and it collects not only logs but also metrics, traces, and profiles with a single agent.

Promtail vs Alloy

ItemPromtailGrafana Alloy
Collection scopeLogs onlyLogs, metrics, traces, profiles
Configuration languageYAML (scrape_configs)River (HCL-like DSL)
OTel compatibilityNot supportedNative OTLP support
Kubernetes log collectionFile system based (/var/log/containers)Kubernetes API based (loki.source.kubernetes)
Processing pipelinestages blockloki.process component
StatusEOL (2026-03-02)Actively developed
Migration tooling-alloy convert --source-format=promtail

Alloy Configuration Example - Kubernetes Log Collection

// Kubernetes Pod discovery
discovery.kubernetes "pods" {
  role = "pod"
}

// Label remapping - keep only the labels you need
discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app"]
    target_label  = "app"
  }
}

// Kubernetes log source
loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.process.pipeline.receiver]
}

// Log processing pipeline
loki.process "pipeline" {
  // Parse JSON logs
  stage.json {
    expressions = {
      level   = "level",
      message = "msg",
    }
  }

  // Normalize the level label
  stage.label_drop {
    values = ["filename", "stream"]
  }

  // Drop debug logs - cut volume in production
  stage.match {
    selector = "{level=\"debug\"}"
    action   = "drop"
  }

  forward_to = [loki.write.default.receiver]
}

// Send to Loki
loki.write "default" {
  endpoint {
    url = "http://loki-gateway.monitoring.svc:3100/loki/api/v1/push"
    tenant_id = "production"
  }
  external_labels = {
    cluster = "prod-kr-01",
    region  = "ap-northeast-2",
  }
}

Migrating from Promtail to Alloy

A command is provided that converts an existing Promtail configuration to Alloy.

# Convert a Promtail config to Alloy River syntax
alloy convert --source-format=promtail --output=alloy-config.river promtail-config.yaml

# Verify the conversion result (dry-run)
alloy run --stability.level=generally-available alloy-config.river

# Deploy as a Kubernetes DaemonSet (Helm)
helm upgrade --install alloy grafana/alloy \
  --namespace monitoring \
  --set alloy.configMap=true \
  -f alloy-values.yaml

Storage Design and Retention Management

TSDB Index Store

TSDB, introduced in Loki 2.8, is the currently recommended index store. Query performance improved over the older BoltDB Shipper, and TCO (Total Cost of Ownership) is lower. The index period must be set to 24 hours.

# loki.yaml - TSDB + S3 storage configuration
schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache
  aws:
    s3: s3://ap-northeast-2/company-loki-chunks
    s3forcepathstyle: false

# Loki 3.4+ unified storage configuration (Thanos Object Storage Client)
common:
  storage:
    s3:
      endpoint: s3.ap-northeast-2.amazonaws.com
      region: ap-northeast-2
      bucketnames: company-loki-data
      access_key_id: ${AWS_ACCESS_KEY_ID}
      secret_access_key: ${AWS_SECRET_ACCESS_KEY}

From Loki 3.4 the Thanos Object Storage Client is integrated, so the same storage configuration format is used as for other Grafana databases such as Mimir and Pyroscope. Existing AWS SDK based configuration still works, but the unified configuration is recommended for new deployments.

Configuring the Retention Policy

Retention is handled by the Compactor. retention_enabled: true must be set, and both global retention and per-stream retention are supported.

# loki.yaml - retention configuration
compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

limits_config:
  retention_period: 720h # global: 30 days
  retention_stream:
    - selector: '{namespace="production"}'
      priority: 1
      period: 2160h # production: 90 days
    - selector: '{namespace="staging"}'
      priority: 2
      period: 168h # staging: 7 days
    - selector: '{level="debug"}'
      priority: 3
      period: 72h # debug logs: 3 days

Caution: retention_delete_delay is the delete delay. If you set retention too short by mistake, reverting the setting within that window prevents data loss. Setting it to at least 2 hours is strongly recommended.

Storage Cost Optimization

  1. Chunk compression algorithm: gzip compresses better than snappy (the default) but uses more CPU. If storage cost is the main concern pick gzip; if write performance matters pick snappy.
  2. chunk_target_size: The default is 1.5MB. Raising it reduces the number of objects and therefore the API call cost, but increases Ingester memory usage.
  3. S3 Intelligent-Tiering: Log access patterns fall off over time, so applying S3 Intelligent-Tiering or a lifecycle policy cuts long-term retention cost.

Loki vs Elasticsearch

This section lays out the differences between the two systems most often compared when choosing a log management solution.

Comparison itemGrafana LokiElasticsearch (ELK)
Indexing approachLabels onlyFull-text indexing
Storage costVery low (object storage)High (SSD block storage)
Query languageLogQLKQL / Lucene
Search speed (unstructured)Slow (grep style)Fast (inverted index)
Search speed (label-based)FastFast
Memory requirementLowHigh (JVM heap)
Operational complexityMediumHigh (sharding, rebalancing)
Grafana integrationNativePlugin required
Cost at 100GB/day20-30% of ElasticsearchBaseline
Suitable environmentKubernetes, cloud nativeFull-text search, security analytics (SIEM)

When to choose Loki: you need cost-efficient log management in a Kubernetes environment, you use Grafana as your main dashboard, and label-based structured queries are your primary pattern.

When to keep Elasticsearch: arbitrary full-text search over unstructured logs is frequent, or you use logs for security log analysis (SIEM) or business intelligence.

Configuring Alert Rules

Loki supports LogQL-based alert rules; the Ruler component runs the queries periodically and sends an alert to Alertmanager when the condition holds.

# loki-alert-rules.yaml
groups:
  - name: application-errors
    rules:
      - alert: HighErrorRate
        expr: |
          sum by (app, namespace) (
            rate({namespace="production"} |= "error" [5m])
          ) > 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'High error rate in {{ $labels.app }}'
          description: '{{ $labels.app }} in {{ $labels.namespace }} has error rate {{ $value }}/s for 5 minutes.'

      - alert: SlowResponseTime
        expr: |
          quantile_over_time(0.95,
            {app="api-gateway"}
              | json
              | unwrap response_time_ms
              | __error__=""
            [5m]
          ) by (app) > 3000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'P95 response time exceeds 3s for {{ $labels.app }}'

      - alert: LogVolumeSpike
        expr: |
          sum by (namespace) (bytes_over_time({namespace=~".+"} [5m]))
            / sum by (namespace) (bytes_over_time({namespace=~".+"} [5m] offset 1h))
          > 3
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: 'Log volume spike in {{ $labels.namespace }} (3x increase)'

The LogVolumeSpike alert is particularly useful. A sudden jump in log volume can mean the application has a problem, or that a wrong debug setting was deployed. Left alone it leads to Ingester OOM or an explosion in storage cost.

Practical Troubleshooting

Symptom 1: Ingester OOM (Out of Memory)

Cause: Label cardinality is too high, or a single tenant has an excessive number of active streams. Typically this happens when unique values such as request_id or trace_id are used as labels.

Diagnosis:

# Check the number of active streams
curl -s http://loki:3100/metrics | grep loki_ingester_streams_created_total

# Check the stream count per tenant
curl -s http://loki:3100/loki/api/v1/index/stats \
  --header "X-Scope-OrgID: production" \
  --data-urlencode 'query={namespace="production"}' \
  --data-urlencode 'start=1h'

# Check Ingester memory usage
kubectl top pods -n monitoring -l app.kubernetes.io/component=ingester

Solution:

  1. Move high-cardinality labels to structured metadata.
  2. Set the max_streams_per_user limit appropriately.
  3. Raise the Ingester memory limit or increase replicas.

Symptom 2: Query Timeout

Cause: The query time range is too wide, or the stream selector is too broad.

Solution:

# limits_config tuning
limits_config:
  max_query_length: 721h # maximum query time range
  max_query_parallelism: 32 # query parallelism
  query_timeout: 5m # single query timeout
  split_queries_by_interval: 30m # query split interval
  max_query_series: 500 # maximum number of series

Symptom 3: Out-of-Order Logs Rejected

By default Loki requires logs to arrive in timestamp order. When several agents write to the same stream, or when there is network delay, the order can be reversed.

# Loki 3.4+ out-of-order tolerance setting
limits_config:
  unordered_writes: true
  max_chunk_age: 2h

Symptom 4: Compactor Lag

When the Compactor cannot keep up, index files pile up and query performance degrades.

Solution: Use the horizontally scalable Compactor in Loki 3.6, or lower compaction_interval and raise the CPU/memory resources of the Compactor.

Disaster Recovery Procedures

Ingester Failure Recovery

# 1. Identify the failed Ingester pod
kubectl get pods -n monitoring -l app.kubernetes.io/component=ingester

# 2. Check the WAL state
kubectl exec -n monitoring ingester-0 -- ls -la /loki/wal/

# 3. Restart the failed pod (data is recoverable because of the WAL)
kubectl delete pod -n monitoring ingester-0

# 4. Check the hash ring state after recovery
curl -s http://loki:3100/ring | jq '.shards[] | {addr, state, tokens}'

# 5. Force a flush (if needed)
curl -X POST http://loki:3100/ingester/flush

Object Storage Access Failure

When object storage cannot be reached, data keeps piling up in the Ingester WAL until OOM eventually occurs.

  1. Immediately check the S3/GCS connection state and the IAM permissions.
  2. Temporarily raising the Ingester flush_check_period and chunk_idle_period buys time.
  3. Once storage recovers, flush the accumulated data with POST /ingester/flush.
  4. During a long outage, monitor Ingester disk capacity and expand the PVC if necessary.

Summary of Major Loki 3.x Changes

Bloom filters greatly speed up filter queries that search for a specific text string such as an error message or a UUID. The feature is still experimental, but in large environments it can dramatically reduce grep-style full scans.

Operations Checklist

This section lists the items to confirm before a production Loki deployment.

Pre-deployment

Label Design

In Production

Conclusion

Grafana Loki is the system that proved "large-scale logs can be managed efficiently even without an index". In most operational environments that do not need the full-text indexing of Elasticsearch, Loki delivers the same level of incident analysis capability at 70-80% less cost.

The key is label design. Too many labels cause Ingester OOM and performance degradation; too few turn every query into a full scan. Starting with labels at the namespace, app, env, and level granularity and then adjusting incrementally against real query patterns is the recommended path.

The Bloom filters, pattern match filters, and native OTel support in Loki 3.x are quickly closing the old gap of "search is slow". Moving from Promtail to Alloy and adopting the TSDB index store are no longer optional but essential. Starting with Simple Scalable mode and switching to Microservices mode as log volume grows is the most realistic strategy for reducing failure.

References

Read the original: https://www.youngju.dev/blog/observability/2026-03-05-observability-loki-logql-pipeline-operations.en

Comments

No comments yet.

Sign in to leave a comment