Observability Data Pipeline Cost Optimization: Sampling, Filtering, and Tiering Strategies

Introduction
Cost Structure Analysis: Where Does Cost Occur
Sampling Strategies: Head vs Tail vs Adaptive
Log Filtering Pipeline
Metric Cardinality Management
Storage Tiering Architecture
Integrated Cost Optimization Architecture
Failure Cases and Recovery Procedures
Operational Considerations
Cost Optimization Checklist
Troubleshooting Guide
Conclusion
References

Observability Cost Optimization

Introduction

Observability cost is rising into the top tier of cloud infrastructure spending. The global observability market passed 28.5 billion USD in 2025 and is projected to reach 34.1 billion USD by the end of 2026. According to the Elastic 2026 Observability Survey, 54% of IT decision makers report growing pressure from executives to justify observability spending.

The root cause of the cost explosion is data volume. As microservice architectures have spread, traces account for 60~70% of total observability cost and logs take up 20~30%. When hundreds of services generate millions of spans and logs per second, storage and indexing costs grow exponentially rather than linearly.

However, simply reducing data is not the answer. In the same Elastic survey, 70% of organizations said they prioritize optimizing existing spend over cutting data. The key is to separate signal from noise so that cost falls while the quality of observability holds.

This article covers cost optimization strategies for a telemetry pipeline built around the OpenTelemetry Collector. It explains the difference between Head Sampling and Tail Sampling and how to design policies, how to build a log filtering pipeline, how to manage metric cardinality explosion, and Hot/Warm/Cold storage tiering architecture, centered on production configuration and code examples. It closes with failure cases and recovery procedures plus a cost optimization checklist, so that a platform engineer can apply the guide immediately.

Cost Structure Analysis: Where Does Cost Occur

To optimize observability cost, you first have to understand exactly where and how that cost arises. The cost of a telemetry pipeline can be broadly classified along four axes.

Signal Type	Cost Share	Main Cost Drivers	Optimization Difficulty
Traces	60~70%	High cardinality, large payloads, span count explosion	High
Logs	20~30%	Unstructured data, high volume, full-text indexing	Medium
Metrics	5~15%	Time series cardinality, label combination explosion	Medium
Profiles	1~5%	CPU/memory profile data size	Low

Analysis by Cost Stage

Breaking the lifecycle of telemetry data down into the points where cost arises gives the following.

[Generate] --> [Collect/Send] --> [Process] --> [Index] --> [Store] --> [Query]
    |                |                |            |           |           |
   SDK            Network         Collector     Backend     Storage     Compute
overhead         bandwidth      CPU/memory    index I/O    disk/S3   query cost

The core principle of cost optimization is "drop unnecessary data as early in the pipeline as possible". Dropping at the generation stage saves every cost downstream, whereas dropping at the storage stage means the cost of every earlier stage has already been paid. This is called First-Mile Processing or telemetry optimization.

Observability Cost Formula

A formula for roughly estimating monthly observability cost looks like this.

Monthly cost = (ingest volume x ingest unit price) + (stored volume x storage unit price) + (query count x query unit price)

Ingest volume = number of services x RPS per service x span/log ratio x average payload size x retention period

When a large platform processes billions of data points per day, cardinality grows quickly, and it becomes the single largest cost driver across the whole observability stack. Once an index holds millions of series, RAM and disk usage spike, and ingest lag and data drops follow.

Sampling Strategies: Head vs Tail vs Adaptive

Sampling is the strategy that shows the most immediate effect on observability cost. Instead of collecting 100% of every trace, keeping only meaningful data selectively can cut volume dramatically.

Sampling Strategy Comparison

Item	Head Sampling	Tail Sampling	Adaptive Sampling
Decision point	At trace start (SDK level)	After the trace completes (Collector level)	Dynamic, from live traffic patterns
Decision basis	Probability or trace ID	Analysis of all spans (latency, error, attributes)	Traffic volume and error rate per service
Cost reduction	High (saves network bandwidth too)	Medium (everything is sent as far as the Collector)	Medium~high
Data quality	Low (error traces can be missed)	High (100% of error/high-latency traces kept)	High
Implementation complexity	Low	High (needs memory and routing)	Very high
Memory use	Minimal	High (buffers every span for decision_wait)	Medium
Best fit	Non-critical services with extremely high traffic	Production critical services	Large platforms with volatile traffic

Head Sampling Configuration

Head Sampling decides at the SDK level whether a trace is created. It is the simplest option and saves network bandwidth as well, but it can miss error or high-latency traces. Consistent Probability Sampling guarantees that every service reaches the same sampling decision for the same trace ID.

# OpenTelemetry Collector - Probabilistic Sampler Processor
processors:
  probabilistic_sampler:
    # Sample only 10% of all traces
    sampling_percentage: 10
    # hash_seed: seed for hashing the trace ID (fixed so multiple Collector instances decide identically)
    hash_seed: 22

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler, batch]
      exporters: [otlp/tempo]

Head Sampling is configured in the Python SDK as follows.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBasedTraceIdRatio
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# 10% Head Sampling - ParentBased follows the parent span decision
sampler = ParentBasedTraceIdRatio(0.1)

provider = TracerProvider(sampler=sampler)

# Efficient export through the batch processor
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://otel-collector:4317"),
        max_queue_size=2048,
        max_export_batch_size=512,
        schedule_delay_millis=5000,
    )
)
trace.set_tracer_provider(provider)

Tail Sampling Configuration

Tail Sampling makes the sampling decision after every span of a trace has been collected. It can keep 100% of the traces that errored or ran slow while sampling only normal traces at a low rate, so cost falls without losing data quality.

One caveat: for Tail Sampling to work, every span of the same trace must arrive at the same Collector instance. In a multi-Collector environment a trace-ID-based load balancer or the groupbytrace processor is essential.

# OpenTelemetry Collector - Tail Sampling Processor
# Use this only on a Collector running in Gateway mode
processors:
  tail_sampling:
    # Wait time from the first span received until the decision (waits for the trace to complete)
    decision_wait: 30s
    # Maximum number of traces held in memory
    num_traces: 100000
    # Expected new traces per second (optimizes internal memory allocation)
    expected_new_traces_per_sec: 1000
    policies:
      # Policy 1: keep 100% of traces containing an error
      - name: error-policy
        type: status_code
        status_code:
          status_codes:
            - ERROR

      # Policy 2: keep 100% of traces slower than 500ms
      - name: latency-policy
        type: latency
        latency:
          threshold_ms: 500

      # Policy 3: keep 100% of traces from specific services
      - name: critical-service-policy
        type: string_attribute
        string_attribute:
          key: service.name
          values:
            - payment-service
            - auth-service
            - order-service

      # Policy 4: sample only 5% of the remaining normal traces
      - name: normal-traffic-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

      # Policy 5: composite policy - AND condition
      - name: composite-policy
        type: and
        and:
          and_sub_policy:
            - name: is-health-check
              type: string_attribute
              string_attribute:
                key: http.route
                values:
                  - /healthz
                  - /readyz
                  - /livez
            - name: drop-most
              type: probabilistic
              probabilistic:
                sampling_percentage: 0.1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]

Tail Sampling Architecture: the 2-Tier Collector Pattern

Running Tail Sampling reliably in production requires a 2-Tier architecture that separates the Agent Collector from the Gateway Collector.

[Service A] --OTLP--> [Agent Collector (DaemonSet)]
[Service B] --OTLP--> [Agent Collector (DaemonSet)]  --trace-ID-based routing-->
[Service C] --OTLP--> [Agent Collector (DaemonSet)]

                                                     [Gateway Collector #1] --> [Tempo]
                                                     [Gateway Collector #2] --> [Tempo]
                                                     [Gateway Collector #3] --> [Tempo]
                                                     (performs Tail Sampling)

The Agent Collector uses the loadbalancing exporter to route spans carrying the same trace ID to the same Gateway.

# Agent Collector configuration - LoadBalancing Exporter
exporters:
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-gateway-headless.observability.svc.cluster.local
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loadbalancing]

Log Filtering Pipeline

Logs are the second largest share of observability cost. Debug logs, health check logs and repetitive infrastructure logs in particular often make up most of the volume while carrying little real analytical value. The Filter Processor and Transform Processor of the OpenTelemetry Collector remove unnecessary logs at the pipeline level so that only the information you need reaches the backend.

Log Level Based Filtering

Dropping DEBUG and TRACE level logs at the Collector in a production environment typically cuts log volume by 30~50%.

# OpenTelemetry Collector - Log Level Filtering
processors:
  # Drop DEBUG/TRACE logs
  filter/drop-debug:
    error_mode: ignore
    logs:
      log_record:
        - 'severity_number < SEVERITY_NUMBER_INFO'

  # Drop specific log body patterns (health check, readiness probe)
  filter/drop-healthcheck:
    error_mode: ignore
    logs:
      log_record:
        - 'IsMatch(body, ".*GET /health.*")'
        - 'IsMatch(body, ".*GET /readyz.*")'
        - 'IsMatch(body, ".*GET /livez.*")'
        - 'IsMatch(body, ".*kube-probe.*")'

  # Keep only logs from a specific namespace
  filter/namespace:
    error_mode: ignore
    logs:
      log_record:
        - 'resource.attributes["k8s.namespace.name"] == "kube-system"'

  # Attribute transform - shrink the payload by removing unnecessary fields
  transform/reduce-attributes:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "log.file.path")
          - delete_key(attributes, "log.iostream")
          - truncate_all(attributes, 256)
          - limit(attributes, 20)

service:
  pipelines:
    logs:
      receivers: [otlp, filelog]
      processors:
        - memory_limiter
        - filter/drop-debug
        - filter/drop-healthcheck
        - filter/namespace
        - transform/reduce-attributes
        - batch
      exporters: [otlp/loki]

Log Sampling and Aggregation

Instead of storing every log individually, aggregating repetitive logs and recording only a count is also effective. For example, when the same error message occurs 1,000 times per second, store a single record and add a count attribute.

# OpenTelemetry Collector - aggregate logs with Group By Attributes
processors:
  groupbyattrs:
    keys:
      - service.name
      - severity_text
      - log.template

  # Add a count attribute after aggregating duplicate logs
  transform/aggregate-count:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(attributes["log.dedup_count"], 1) where attributes["log.dedup_count"] == nil

Measuring the Effect of Log Filtering

Once a filtering policy is applied, its effect has to be measured. The Collector own metrics let you track the volume change before and after filtering quantitatively.

#!/bin/bash
# Measure filtering effectiveness from the Collector internal metrics
# Collected from the Prometheus endpoint

# Number of logs received before filtering
RECEIVED=$(curl -s http://localhost:8888/metrics | \
  grep 'otelcol_receiver_accepted_log_records' | \
  awk '{sum += $2} END {print sum}')

# Number of logs exported after filtering
EXPORTED=$(curl -s http://localhost:8888/metrics | \
  grep 'otelcol_exporter_sent_log_records' | \
  awk '{sum += $2} END {print sum}')

# Compute the drop rate
if [ "$RECEIVED" -gt 0 ]; then
  DROP_RATE=$(echo "scale=2; (1 - $EXPORTED / $RECEIVED) * 100" | bc)
  echo "Received logs: $RECEIVED"
  echo "Exported logs: $EXPORTED"
  echo "Drop rate: ${DROP_RATE}%"
  echo "Estimated saving: $(echo "scale=0; $DROP_RATE * 150 / 100" | bc) USD per month (basis: 150 USD/month)"
fi

# Check the drop count per processor
echo ""
echo "=== Drop status by processor ==="
curl -s http://localhost:8888/metrics | \
  grep 'otelcol_processor_dropped_log_records' | \
  sort -t' ' -k2 -rn

Metric Cardinality Management

Metric cardinality explosion is the main reason observability cost becomes unpredictable. Every unique time series requires its own entry in the database index, and once millions of series exist, RAM and disk usage spike, ingest lags, and query performance degrades.

Causes of Cardinality Explosion

Cardinality explosion generally comes from the following label usage patterns.

Risky Pattern	Example	Cardinality Growth	Fix
User ID as a label	`user_id="u12345"`	One series per user	Remove the label, move it to logs/traces
Raw request path	`path="/api/users/12345"`	Unbounded	Normalize the path `path="/api/users/:id"`
Pod name	`pod="web-7f8c9-xk2m"`	Grows with every deploy	Use the Deployment name only
Full error message	`error="Connection refused: 10.0.1.42:3306"`	A series per IP	Classify by error code
Timestamp label	`request_time="1709856000"`	A new series every second	Remove it from the labels

Cardinality Control at the Collector Level

The configuration for cleaning up metric labels and controlling cardinality in the OpenTelemetry Collector is as follows.

# OpenTelemetry Collector - metric cardinality management
processors:
  # Remove high-cardinality attributes
  metricstransform/drop-high-cardinality:
    transforms:
      - include: http.server.request.duration
        action: update
        operations:
          # Remove high-cardinality labels such as user_id and session_id
          - action: delete_label_value
            label: user_id
          - action: delete_label_value
            label: session_id

  # Normalize URL paths
  transform/normalize-paths:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - replace_pattern(attributes["url.path"], "^/api/users/[0-9]+", "/api/users/:id")
          - replace_pattern(attributes["url.path"], "^/api/orders/[0-9]+", "/api/orders/:id")
          - replace_pattern(attributes["url.path"], "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", ":uuid")

  # Limit the number of attributes
  attributes/limit:
    actions:
      - key: http.request.header.x-request-id
        action: delete
      - key: http.request.header.authorization
        action: delete

  # Metric aggregation - drop fine-grained labels and send only aggregated metrics
  filter/metrics-allowlist:
    error_mode: ignore
    metrics:
      metric:
        # Allow only specific metrics (allowlist approach)
        - 'name == "http.server.request.duration"'
        - 'name == "http.server.active_requests"'
        - 'name == "process.runtime.go.goroutines"'
        - 'IsMatch(name, "^system\\..*")'

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors:
        - memory_limiter
        - metricstransform/drop-high-cardinality
        - transform/normalize-paths
        - attributes/limit
        - filter/metrics-allowlist
        - batch
      exporters: [prometheusremotewrite/mimir]

The Observability Budget Pattern

A pattern large organizations have adopted recently is the Observability Budget. It defines up front the maximum volume of telemetry each service may emit, and once the budget is exceeded the Collector drops data automatically.

#!/usr/bin/env python3
"""
Observability Budget Enforcer
Script that monitors the telemetry budget per service and raises warnings
"""

import requests
import yaml
import sys
from datetime import datetime

# Observability Budget per service (daily basis)
BUDGETS = {
    "payment-service": {
        "traces_per_day": 5_000_000,
        "logs_per_day": 10_000_000,
        "metrics_series": 50_000,
        "alert_threshold": 0.8,  # warn once 80% is reached
    },
    "user-service": {
        "traces_per_day": 2_000_000,
        "logs_per_day": 5_000_000,
        "metrics_series": 30_000,
        "alert_threshold": 0.8,
    },
    "default": {
        "traces_per_day": 1_000_000,
        "logs_per_day": 3_000_000,
        "metrics_series": 20_000,
        "alert_threshold": 0.8,
    },
}

PROMETHEUS_URL = "http://prometheus:9090"


def query_prometheus(query: str) -> float:
    """Query the current value from Prometheus."""
    resp = requests.get(
        f"{PROMETHEUS_URL}/api/v1/query",
        params={"query": query},
    )
    result = resp.json().get("data", {}).get("result", [])
    if result:
        return float(result[0]["value"][1])
    return 0.0


def check_budget(service_name: str) -> dict:
    """Check the current telemetry usage of a service against its budget."""
    budget = BUDGETS.get(service_name, BUDGETS["default"])
    today = datetime.now().strftime("%Y-%m-%d")

    # Number of traces created today
    traces_today = query_prometheus(
        f'sum(increase(traces_spanmetrics_calls_total{{service_name="{service_name}"}}[24h]))'
    )

    # Number of logs created today
    logs_today = query_prometheus(
        f'sum(increase(loki_distributor_lines_received_total{{service="{service_name}"}}[24h]))'
    )

    # Current number of active metric series
    active_series = query_prometheus(
        f'count({{service_name="{service_name}"}})'
    )

    usage = {
        "service": service_name,
        "date": today,
        "traces": {
            "current": int(traces_today),
            "budget": budget["traces_per_day"],
            "usage_pct": round(traces_today / budget["traces_per_day"] * 100, 1),
        },
        "logs": {
            "current": int(logs_today),
            "budget": budget["logs_per_day"],
            "usage_pct": round(logs_today / budget["logs_per_day"] * 100, 1),
        },
        "metrics_series": {
            "current": int(active_series),
            "budget": budget["metrics_series"],
            "usage_pct": round(active_series / budget["metrics_series"] * 100, 1),
        },
    }

    # Check whether the budget is exceeded
    for signal_type, data in usage.items():
        if isinstance(data, dict) and "usage_pct" in data:
            if data["usage_pct"] >= budget["alert_threshold"] * 100:
                print(
                    f"[WARNING] {service_name}: {signal_type} budget "
                    f"{data['usage_pct']}% used ({data['current']:,} / {data['budget']:,})"
                )

    return usage


if __name__ == "__main__":
    services = sys.argv[1:] if len(sys.argv) > 1 else list(BUDGETS.keys())
    for svc in services:
        if svc == "default":
            continue
        result = check_budget(svc)
        print(f"\n=== {svc} Observability Budget ===")
        for key, val in result.items():
            if isinstance(val, dict):
                print(f"  {key}: {val['current']:,} / {val['budget']:,} ({val['usage_pct']}%)")

Storage Tiering Architecture

Storing all telemetry data in the same storage layer is inefficient from a cost perspective. Hot/Warm/Cold tiering, which keeps the last 7 days on high-performance storage for fast queries and moves older data to low-cost storage, is the core of cost optimization.

Storage Tier Comparison

Item	Hot Tier	Warm Tier	Cold Tier	Archive
Retention	0~7 days	7~30 days	30~90 days	90 days~1 year+
Storage type	NVMe SSD / EBS gp3	EBS st1 / S3 Standard	S3 Infrequent Access	S3 Glacier
Cost per GB per month	$0.10~0.16	$0.025~0.045	$0.0125	$0.004
Query latency	milliseconds	seconds	minutes	hours
Suitable data	Live monitoring, alerting	Recent incident analysis, trends	Audit, compliance	Legal retention obligations
Cost reduction (vs Hot)	baseline	70~75%	87~92%	97%

Elasticsearch Hot-Warm-Cold Architecture

When Elasticsearch is the log backend, automatic tiering can be configured through ILM (Index Lifecycle Management).

# Elasticsearch ILM Policy - log tiering
# PUT _ilm/policy/observability-logs-policy
{
  'policy':
    {
      'phases':
        {
          'hot':
            {
              'min_age': '0ms',
              'actions':
                {
                  'rollover': { 'max_primary_shard_size': '50gb', 'max_age': '1d' },
                  'set_priority': { 'priority': 100 },
                },
            },
          'warm':
            {
              'min_age': '7d',
              'actions':
                {
                  'shrink': { 'number_of_shards': 1 },
                  'forcemerge': { 'max_num_segments': 1 },
                  'allocate': { 'require': { 'data': 'warm' } },
                  'set_priority': { 'priority': 50 },
                },
            },
          'cold':
            {
              'min_age': '30d',
              'actions':
                {
                  'searchable_snapshot':
                    { 'snapshot_repository': 's3-repo', 'force_merge_index': true },
                  'allocate': { 'require': { 'data': 'cold' } },
                  'set_priority': { 'priority': 0 },
                },
            },
          'delete': { 'min_age': '365d', 'actions': { 'delete': {} } },
        },
    },
}

Grafana Tempo + S3 Tiering

When Grafana Tempo is the trace backend, block-storage-based tiering can be configured. Tempo uses object storage (S3) as its backend by default, so S3 Intelligent-Tiering optimizes cost automatically.

S3 Intelligent-Tiering moves data automatically according to access patterns. After 30 days without access it moves data to the Infrequent Access tier for a 40% saving, and after 90 days without access to the Archive Instant Access tier for a 68% saving. T-Mobile applied this strategy to a 1.87PB data lake and cut S3 cost by 40%.

Grafana Mimir Metric Tiering

When Grafana Mimir is the metric backend, the compactor and store-gateway settings combine data retention with downsampling.

# Mimir configuration - metric retention and downsampling
limits:
  # Keep full-resolution metrics for 14 days
  compactor_blocks_retention_period: 14d

compactor:
  # Enable downsampling
  downsample:
    # Downsample to 5m resolution after 7 days
    - resolution: 5m
      retention: 7d
    # Downsample to 1h resolution after 30 days
    - resolution: 1h
      retention: 30d

store_gateway:
  sharding_ring:
    replication_factor: 3

# S3 backend configuration
blocks_storage:
  backend: s3
  s3:
    bucket_name: mimir-metrics
    endpoint: s3.ap-northeast-2.amazonaws.com
    storage_class: INTELLIGENT_TIERING
  bucket_store:
    sync_dir: /data/mimir-sync
    # Index cache keeps query performance up
    index_cache:
      backend: memcached
      memcached:
        addresses: dns+memcached.observability.svc:11211

Integrated Cost Optimization Architecture

Diagramming the full architecture that integrates the sampling, filtering, cardinality management and storage tiering described so far gives the following.

                    [Microservices Cluster]
                    (SDK Head Sampling 10%)
                            |
                            v
            +-------------------------------+
            |   Agent Collector (DaemonSet)  |
            |   - Memory Limiter            |
            |   - Filter/drop-debug         |
            |   - Filter/drop-healthcheck   |
            |   - Attributes/limit          |
            +-------------------------------+
                     |          |          |
                  Traces     Logs      Metrics
                     |          |          |
                     v          v          v
     +---------------+    +---------+  +-----------+
     | Gateway        |    | Gateway |  | Gateway   |
     | Collector      |    | (Logs)  |  | (Metrics) |
     | (Traces)       |    |         |  |           |
     | - Tail Sampling|    | - Group |  | - Cardina |
     | - LoadBalance  |    |   ByAttr|  |   lity    |
     +-------+--------+    +----+----+  +-----+-----+
             |                   |              |
             v                   v              v
     +-------+--------+  +------+------+  +----+------+
     | Tempo          |  | Loki        |  | Mimir     |
     | (S3 + Intelli- |  | (S3 + ILM)  |  | (S3 +     |
     |  gent Tiering) |  |             |  | Downsample|
     +----------------+  +-------------+  +-----------+
             |                   |              |
             v                   v              v
     +--------------------------------------------------+
     |              Grafana Dashboard                    |
     |  - Live monitoring (Hot Tier)                    |
     |  - Incident analysis (Warm Tier)                 |
     |  - Compliance audit (Cold/Archive Tier)          |
     +--------------------------------------------------+

Applying this architecture can be expected to produce the following cost savings.

Optimization Layer	Technique	Expected Volume Reduction	Cost Effect
SDK Head Sampling	10% probabilistic sampling	Traces down 90%	Large cut in network + storage cost
Agent log filtering	Drop DEBUG/health check	Logs down 30~50%	Cut in ingest + storage cost
Gateway Tail Sampling	Keep error/high latency + 5% of normal	Remaining traces down 95%	Cut in final storage cost
Cardinality control	Remove high-cardinality labels	Series count down 50~70%	Cut in indexing + query cost
Storage tiering	Hot/Warm/Cold + S3 IT	None (same data, cheaper storage)	Storage cost down 40~68%

Taken together, applying all of these strategies can cut total observability cost by 60~80% while keeping observability of error traces and anomalies at 100%.

Failure Cases and Recovery Procedures

Before applying a cost optimization strategy, work out in advance which failure scenarios it can produce and how to recover from each.

Failure Case 1: Tail Sampling OOM (Out of Memory)

Symptom: the Gateway Collector restarts repeatedly with OOM. The number of spans accumulating in memory during decision_wait exceeds the num_traces limit and memory spikes.

Cause: during a traffic spike the num_traces and decision_wait settings exceed memory capacity. For example, with decision_wait: 60s and 10,000 new traces arriving per second, up to 600,000 traces have to be held in memory.

Recovery procedure:

Reduce decision_wait to 30s or less (a trade-off against the trace completion rate).
Adjust num_traces to fit memory capacity. As a rule of thumb, estimate roughly 1~5KB per trace.
Place the memory_limiter processor ahead of Tail Sampling so data is dropped once the memory threshold is passed.
Scale the Gateway Collector resources vertically, or scale the instance count horizontally.

# Memory Limiter + Tail Sampling combination that prevents OOM
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 3800 # 200MB of headroom in a 4GB container
    spike_limit_mib: 800 # allow a temporary burst

  tail_sampling:
    decision_wait: 20s # reduced from 30s to 20s
    num_traces: 50000 # calculated from available memory
    expected_new_traces_per_sec: 2500
    policies:
      - name: error-always
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: latency-always
        type: latency
        latency:
          threshold_ms: 500
      - name: probabilistic-default
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

service:
  pipelines:
    traces:
      receivers: [otlp]
      # memory_limiter must come first
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]

Symptom: an incident occurs in production but there are no related logs or traces, so the cause cannot be analyzed. The filtering policy was set too aggressively and dropped critical data along with the rest.

Cause: the log filter regex is drawn too broadly and matches relevant logs as well, or metric label removal goes so far that an anomaly on a specific instance can no longer be identified.

Recovery procedure:

Apply Shadow Mode first whenever a filtering policy changes. Nothing is actually dropped; only the count of the data that would be dropped is recorded, so the impact can be assessed in advance.
Apply filtering exception rules for critical services (payment, auth, order).
Use error_mode: ignore so that data is not dropped even when condition evaluation fails.

Failure Case 3: Mimir/Prometheus Ingest Failure from Cardinality Explosion

Symptom: Prometheus or Mimir raises a too many active series error and metric ingest is rejected. Empty panels appear on the Grafana dashboard.

Cause: a newly deployed service puts a unique identifier (UUID, timestamp, Pod name and so on) in a label value, and the series count grows explosively.

Recovery procedure:

Immediately drop the offending label with the metricstransform processor in the Collector.
Temporarily raise the max_global_series_per_user limit in Mimir.
Fix the SDK instrumentation code in the service responsible so that the high-cardinality attribute is removed.
Over the longer term, add a cardinality validation step to the CI/CD pipeline.

Failure Case 4: Query Failures During a Tier Transition

Symptom: while the ILM policy moves an index to the Warm or Cold tier, log queries over that time range fail or time out.

Cause: during the tier transition, shard relocation, force merge and snapshot creation occupy cluster resources and query performance degrades.

Recovery procedure:

Schedule ILM transitions for low-traffic hours (2~5 AM).
Use searchable_snapshot so that queries remain possible during the transition.
Provision enough resources on the Warm/Cold nodes that the transition finishes quickly.

Operational Considerations

Here are the points to keep in mind when applying a cost optimization strategy.

On Sampling

If Tail Sampling decision_wait is set too short, late-arriving spans are lost and incomplete traces are stored. Generally set it to at least twice the maximum inter-service latency.
If Head Sampling and Tail Sampling are applied at the same time, traces already dropped in the SDK cannot be recovered by Tail Sampling. When combining the two, set the Head Sampling rate conservatively (50% or more).
When scaling out the Tail Sampling Gateway, trace-ID-based routing (LoadBalancing Exporter or Consistent Hashing) has to be preserved. Wrong routing scatters spans of the same trace across instances and distorts the sampling decision.

On Filtering

Roll filtering policy changes out gradually, without exception. Test in a canary pipeline first, verify the character of the data that would be dropped, then deploy everywhere.
Never filter error level logs. Dropping ERROR and FATAL level logs creates a fatal blind spot during incident response.
Always monitor the Collector own metrics (otelcol_processor_dropped_*). Configure an alert to fire immediately when the drop rate leaves the expected range.

On Cardinality

Run a cardinality impact analysis before deploying a new service. Add a step to the CI/CD pipeline that validates the expected cardinality of metric labels.
Use Recording Rules to pre-aggregate frequently used high-cardinality queries. This cuts both the compute cost and the time spent at query time.

On Tiering

Reading data out of the Cold Tier costs money. Restore requests on S3 Glacier are billed per request, so when frequent Cold reads are expected, use the Archive Instant Access tier of S3 Intelligent-Tiering.
Set the retention policy only after checking legal and regulatory requirements. In regulated industries such as finance and healthcare, retaining data for a specific period is a legal obligation.

Cost Optimization Checklist

Use the checklist below to assess how far the current observability pipeline has been cost-optimized.

Phase 1: Quick Wins You Can Apply Now (1~2 weeks)

Are DEBUG/TRACE level logs dropped at the Collector in the production environment?
Are the logs and traces from health checks and readiness probes filtered out?
Are metric labels free of unique identifiers such as user ID and request ID?
Are URL path labels normalized? (/api/users/:id instead of /api/users/123)
Is the memory_limiter processor configured on the Collector?
Are the Collector internal metrics (drop rate, queue size, memory usage) monitored?

Phase 2: Mid-Term Optimization (2~4 weeks)

Does the Tail Sampling policy keep 100% of error/high-latency traces while limiting normal traffic to 5~10%?
Is a 2-Tier Collector architecture (Agent + Gateway) in place?
Is trace-ID-based routing (LoadBalancing Exporter) configured?
Are unnecessary fields removed from log attributes? (file path, stream type and so on)
Is an Observability Budget defined per service?
Is a cardinality monitoring dashboard in place?

Phase 3: Long-Term Infrastructure Optimization (1~3 months)

Is Hot/Warm/Cold storage tiering configured?
Is S3 Intelligent-Tiering or equivalent automatic tiering enabled?
Is a metric downsampling policy applied? (5m resolution after 7 days, 1h after 30 days)
Do the ILM and retention policies satisfy regulatory requirements?
Does the CI/CD pipeline include a cardinality validation gate?
Does the cost monitoring dashboard track cost trends by signal and by service?
Has a move to Adaptive Sampling been reviewed?

Troubleshooting Guide

When Collector Memory Usage Keeps Growing

# 1. Check Collector memory usage
kubectl top pods -n observability -l app=otel-gateway

# 2. Memory profiling with pprof (the extension has to be enabled)
curl -s http://localhost:1777/debug/pprof/heap > heap.prof
go tool pprof -top heap.prof

# 3. Check pipeline state in zPages
curl -s http://localhost:55679/debug/tracez | jq .

# 4. Check the Tail Sampling queue state
curl -s http://localhost:8888/metrics | grep tail_sampling

When a Filtering Policy Does Not Behave as Expected

Change error_mode to propagate so that errors are printed to the log.
Add the debug exporter to the pipeline and inspect what data survives filtering.
Recheck the syntax of the OTTL (OpenTelemetry Transformation Language) expression. Pay particular attention to the difference between accessing body and accessing attributes.

# Pipeline configuration for debugging
exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

service:
  pipelines:
    logs/debug:
      receivers: [otlp]
      processors: [filter/drop-debug]
      exporters: [debug]

How to Find the Service Behind a Cardinality Explosion

# Check the top-cardinality metrics in Prometheus
curl -s http://prometheus:9090/api/v1/status/tsdb | \
  jq '.data.seriesCountByMetricName | sort_by(-.value) | .[0:10]'

# Analyze label cardinality for a specific metric
curl -s 'http://prometheus:9090/api/v1/query?query=count(http_server_request_duration_seconds_bucket) by (service_name)' | \
  jq '.data.result | sort_by(-.value[1] | tonumber) | .[0:10]'

# Check active series per tenant in Mimir
curl -s http://mimir:8080/api/v1/cardinality/active_series | \
  jq '.data | sort_by(-.active_series) | .[0:10]'

Conclusion

Observability cost optimization is not simply about reducing data. It is an engineering activity that separates signal from noise precisely, so cost efficiency is maximized while the quality of observability holds.

The strategies covered in this article can be summarized as follows.

Sampling: use Head Sampling to cut network cost as well, and apply Tail Sampling on critical services to keep 100% of error and high-latency traces.
Filtering: drop DEBUG and health check logs at the Collector, and shrink payload size by cleaning up attributes.
Cardinality management: remove high-cardinality labels, normalize URL paths, and control per-service usage with an Observability Budget.
Storage tiering: use a Hot/Warm/Cold architecture to move data to low-cost storage automatically as it ages.

Applying these strategies systematically can cut total observability cost by 60~80% while fully preserving the data incident response needs. Cost optimization is not a one-time task but an operational process that has to be adjusted continuously as services grow. The key is to review the checklist regularly and to keep monitoring the Collector internal metrics and cost trends so that the optimal balance holds.

Observability Data Pipeline Cost Optimization: Sampling, Filtering, and Tiering Strategies

Introduction

Cost Structure Analysis: Where Does Cost Occur

Cost Share by Signal

Analysis by Cost Stage

Observability Cost Formula

Sampling Strategies: Head vs Tail vs Adaptive

Sampling Strategy Comparison

Head Sampling Configuration

Tail Sampling Configuration

Tail Sampling Architecture: the 2-Tier Collector Pattern

Log Filtering Pipeline

Log Level Based Filtering

Log Sampling and Aggregation

Measuring the Effect of Log Filtering

Metric Cardinality Management

Causes of Cardinality Explosion

Cardinality Control at the Collector Level

The Observability Budget Pattern

Storage Tiering Architecture

Storage Tier Comparison

Elasticsearch Hot-Warm-Cold Architecture

Grafana Tempo + S3 Tiering

Grafana Mimir Metric Tiering

Integrated Cost Optimization Architecture

Failure Cases and Recovery Procedures

Failure Case 1: Tail Sampling OOM (Out of Memory)

Failure Case 2: Blind Spots from Over-Filtering

Failure Case 3: Mimir/Prometheus Ingest Failure from Cardinality Explosion

Failure Case 4: Query Failures During a Tier Transition

Operational Considerations

On Sampling

On Filtering

On Cardinality

On Tiering

Cost Optimization Checklist

Phase 1: Quick Wins You Can Apply Now (1~2 weeks)

Phase 2: Mid-Term Optimization (2~4 weeks)

Phase 3: Long-Term Infrastructure Optimization (1~3 months)

Troubleshooting Guide

When Collector Memory Usage Keeps Growing

When a Filtering Policy Does Not Behave as Expected

How to Find the Service Behind a Cardinality Explosion

Conclusion

References

Comments