OpenTelemetry Collector Pipeline Design and Operations Guide: From Collection to Backend Integration

• Introduction
• OpenTelemetry Collector Architecture
  • Core Component Structure
  • Core vs Contrib Distributions
  • Data Model and Signal Types
• Receiver Configuration
  • OTLP Receiver
  • Prometheus Receiver
  • Filelog Receiver
  • Receiver Type Comparison
• Processor Pipeline
  • Memory Limiter Processor
  • Batch Processor
  • Attributes Processor
  • Tail Sampling Processor
  • Processor Type Comparison
• Exporter Configuration
  • OTLP Exporter
  • Prometheus Remote Write Exporter
  • Loki Exporter
  • Popular Backend Comparison
• Agent vs Gateway Deployment Patterns
  • Pattern Comparison
  • Agent + Gateway Hybrid Architecture
• Deploying in Kubernetes
  • Agent DaemonSet Manifest
  • Gateway Deployment Manifest
  • Gateway Load Balancing and TraceID-Based Routing
• Memory Management and Backpressure
  • Memory Usage Formula
  • Backpressure Mechanism
  • GOGC Tuning
• Troubleshooting Guide
  • Diagnosing with the Collector's Own Metrics
  • Real-Time Debugging with zPages
  • Common Problems and Fixes
• Operations Checklist
  • Pre-deployment Checklist
  • Monitoring Setup
  • Scaling Strategy
  • Security Checklist
• Failure Cases and Recovery
  • Case 1: A Cascading Failure from Tail Sampling Memory Runaway
  • Case 2: Data Loss from Exporter Queue Saturation
  • Case 3: Missing Pod Metadata from a K8s Attributes Processor Permission Gap
• Configuration Validation and Testing
• Advanced Operations Tips
  • Routing in a Multi-Tenant Environment
  • Key Panels for a Collector Self-Monitoring Dashboard
• References

Introduction

As distributed systems grow more complex, the importance of observability grows exponentially. In an environment where hundreds of microservices interact with one another, designing a pipeline that collects the three telemetry signals - Traces, Metrics, and Logs - in a unified way, transforms them, and routes them to the appropriate backend is a core platform engineering capability.

The OpenTelemetry Collector is a CNCF project that provides a vendor-neutral telemetry pipeline. Without tying you to any particular monitoring solution, it offers a flexible architecture in which Receivers collect data in a variety of formats, Processors carry out transformation and filtering, and Exporters send the result to whichever backend you want. As of 2026 the Collector has matured to v0.120 and above, and its stability in production environments has been amply proven.

In this article, we cover everything you need from an operations perspective: the internal architecture of the OpenTelemetry Collector, practical Receiver, Processor, and Exporter configuration, Agent/Gateway deployment patterns, DaemonSet/Deployment manifests for Kubernetes environments, Tail Sampling strategy, memory management and backpressure mechanisms, a troubleshooting guide, and failure recovery procedures.

OpenTelemetry Collector Architecture

Core Component Structure

The OpenTelemetry Collector's architecture is made up of four core components. Receivers take in telemetry data from external sources, Processors transform the data, and Exporters send it to its final destination. On top of these, Extensions provide supplementary features (health check, authentication, zPages, and so on).

[Application / Infrastructure]
        |
        v
+-------------------+
|    Receivers       |  <-- OTLP, Prometheus, Filelog, Kafka, etc.
+-------------------+
        |
        v
+-------------------+
|    Processors      |  <-- Memory Limiter, Batch, Attributes, Tail Sampling
+-------------------+
        |
        v
+-------------------+
|    Exporters       |  <-- OTLP, Prometheus Remote Write, Loki, Kafka, etc.
+-------------------+

+-------------------+
|    Extensions      |  <-- Health Check, zPages, pprof, Bearer Token Auth
+-------------------+

You can define several pipelines inside a single Collector instance. Each pipeline handles one signal type (traces, metrics, logs) and can have its own combination of Receivers, Processors, and Exporters. Thanks to this design, routing traces to Tempo, metrics to Mimir, and logs to Loki is possible from within a single Collector configuration file.

Core vs Contrib Distributions

The OpenTelemetry Collector comes in two distributions.

In production environments, building a custom binary that contains only the components you need, using the OpenTelemetry Collector Builder (OCB), is recommended on both security and performance grounds.

Data Model and Signal Types

Inside the Collector, data is represented in the pdata (Pipeline Data) format. This internal data model is based on the OTLP (OpenTelemetry Protocol) protocol buffer definitions and supports three signal types.

• Traces: a collection of Spans that represents the execution path of a distributed transaction. TraceID, SpanID, and ParentSpanID express the causal relationships.
• Metrics: time-series data, supporting the Gauge, Sum, Histogram, ExponentialHistogram, and Summary types.
• Logs: timestamp-based event records. They contain SeverityNumber, Body, Attributes, and Resource.

Receiver Configuration

A Receiver is the entry point for telemetry data. It supports both push-style sources (OTLP, Kafka, and so on) and pull-style sources (Prometheus, hostmetrics, and so on), and you can define several Receivers of the same type under different names.

OTLP Receiver

OTLP is OpenTelemetry's native protocol and offers two transports: gRPC and HTTP/protobuf. Most OpenTelemetry SDKs use the OTLP Exporter by default, so this Receiver appears in almost every Collector configuration.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 8 # allow large batches
        max_concurrent_streams: 256 # number of concurrent streams
        keepalive:
          server_parameters:
            max_connection_idle: 30s
            max_connection_age: 60s
            max_connection_age_grace: 10s
          enforcement_policy:
            min_time: 10s
            permit_without_stream: true
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - 'https://*.company.com'
          allowed_headers:
            - 'Content-Type'
            - 'X-Custom-Header'
          max_age: 600

gRPC transport is built on HTTP/2 and supports binary serialization and multiplexing, which makes it efficient for high-volume telemetry. HTTP transport is used for browser-based instrumentation (Web SDK) or in environments with firewall restrictions.

Prometheus Receiver

The Prometheus Receiver provides compatibility with the existing Prometheus ecosystem. You can use Prometheus scrape_configs syntax as-is, so metrics you used to collect with Prometheus can be routed through the Collector to a different backend.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-pods'
          scrape_interval: 30s
          scrape_timeout: 10s
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            # Scrape only Pods that carry the prometheus.io/scrape annotation
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
            # Specify a custom port
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: (.+)
              replacement: '$$1'
            # Add the namespace label
            - source_labels: [__meta_kubernetes_namespace]
              action: replace
              target_label: namespace
            # Add the Pod name label
            - source_labels: [__meta_kubernetes_pod_name]
              action: replace
              target_label: pod
        - job_name: 'node-exporter'
          scrape_interval: 15s
          static_configs:
            - targets: ['node-exporter.monitoring.svc:9100']

Filelog Receiver

The Filelog Receiver collects log files from the file system in real time. It is the key component for collecting container logs in Kubernetes environments, and it performs log parsing, filtering, and transformation through an operator chain.

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    exclude:
      - /var/log/pods/*/otel-collector*/*.log
      - /var/log/pods/kube-system_*/*/*.log
    start_at: end # collect only new logs (beginning starts from the existing logs)
    include_file_path: true
    include_file_name: false
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
    operators:
      # Parse the CRI log format (containerd)
      - type: regex_parser
        id: parser-cri
        regex: '^(?P<time>[^ Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%fZ'
      # Parse the JSON log body
      - type: json_parser
        id: parser-json
        parse_from: attributes.log
        parse_to: body
        on_error: send_quiet # keep the original when parsing fails
      # Severity mapping
      - type: severity_parser
        parse_from: attributes.level
        mapping:
          fatal: [FATAL, fatal, F]
          error: [ERROR, error, E]
          warn: [WARN, warn, W]
          info: [INFO, info, I]
          debug: [DEBUG, debug, D]

Receiver Type Comparison

Processor Pipeline

A Processor is the intermediate layer that transforms data between the Receiver and the Exporter. Order matters: Processors are chained together and run in the order defined in the pipeline. The generally recommended Processor order is as follows.

memory_limiter -> k8sattributes -> resourcedetection -> attributes -> filter -> tail_sampling -> batch

Memory Limiter Processor

The Memory Limiter is the safety mechanism that prevents Collector OOM (Out of Memory). It must be placed first in the Processor chain, and when memory usage reaches the threshold it refuses data in order to protect the process.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1800 # hard limit (80% of the container limit)
    spike_limit_mib: 500 # allowance for sudden spikes
    # limit_percentage: 80     # or a ratio-based setting (cgroup aware)
    # spike_limit_percentage: 25

Set limit_mib to roughly 80% of the container's memory limit. For example, if the container limit is 2Gi, set limit_mib to 1600-1800. The remaining 20% is headroom for the Go runtime and other internal buffers. spike_limit_mib acts as a cushion that allows momentary traffic bursts while still keeping you under limit_mib.

Batch Processor

The Batch Processor gathers individual telemetry records and hands them to the Exporter in batches. This batching reduces the number of network requests and improves compression efficiency, which greatly increases overall pipeline throughput.

processors:
  batch:
    timeout: 5s # maximum wait time
    send_batch_size: 8192 # batch size (number of records)
    send_batch_max_size: 16384 # maximum batch size (split when this size is exceeded)
  # Larger batches on the Gateway
  batch/gateway:
    timeout: 10s
    send_batch_size: 16384
    send_batch_max_size: 32768

The batch is sent as soon as either condition is met, whichever comes first: the timeout or send_batch_size. In low-traffic environments the timeout is usually what fires; in high-traffic environments send_batch_size is.

Attributes Processor

The Attributes Processor adds, modifies, and deletes attributes on telemetry data. It is used for removing sensitive information, tagging environment information, normalizing labels, and similar work.

processors:
  attributes/security:
    actions:
      # Delete sensitive HTTP headers
      - key: http.request.header.authorization
        action: delete
      - key: http.request.header.cookie
        action: delete
      # Hash DB queries (protect sensitive data)
      - key: db.statement
        action: hash
      # Add the environment tag
      - key: deployment.environment
        action: upsert
        value: production
      # Mask IP addresses
      - key: net.peer.ip
        action: extract
        pattern: '^(?P<subnet>\d+\.\d+\.\d+)\.\d+$'
      - key: net.peer.ip
        action: delete
      - key: net.peer.subnet
        from_attribute: subnet
        action: upsert

Tail Sampling Processor

Tail Sampling makes the sampling decision after every Span of the whole trace has been collected. Unlike Head Sampling, it can retain traces that hit an error or responded slowly without missing any, so in production it delivers debugging power and cost savings at the same time.

processors:
  tail_sampling:
    decision_wait: 30s # how long to wait for the trace to complete
    num_traces: 200000 # maximum number of traces held in memory
    expected_new_traces_per_sec: 5000
    policies:
      # Policy 1: retain 100% of traces that contain an error
      - name: error-traces
        type: status_code
        status_code:
          status_codes: [ERROR]
      # Policy 2: retain 100% of traces that took 2 seconds or more
      - name: high-latency
        type: latency
        latency:
          threshold_ms: 2000
          upper_threshold_ms: 0 # 0 means no upper bound
      # Policy 3: retain 50% for core services
      - name: critical-services
        type: and
        and:
          and_sub_policy:
            - name: service-match
              type: string_attribute
              string_attribute:
                key: service.name
                values:
                  - payment-service
                  - auth-service
                  - order-service
            - name: sample-half
              type: probabilistic
              probabilistic:
                sampling_percentage: 50
      # Policy 4: exclude specific HTTP paths (health checks and so on)
      - name: drop-health-checks
        type: string_attribute
        string_attribute:
          key: http.route
          values:
            - /healthz
            - /readyz
            - /livez
          invert_match: true
      # Policy 5: sample only 5% of the remaining traffic
      - name: default-sampling
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

The key caveat with Tail Sampling is that every Span sharing a TraceID has to arrive at the same Collector instance. When you run several Gateways, you must have consistent hashing based on TraceID (consistent hashing at the load balancer).

Processor Type Comparison

Exporter Configuration

An Exporter is responsible for sending transformed telemetry data to its final destination. By defining the same Exporter type under different names you can send to several backends at once, and retry and queue settings guarantee delivery stability.

OTLP Exporter

The OTLP Exporter sends data to another Collector (a Gateway) or to a backend with native OTLP support (Tempo, Jaeger, SigNoz, and so on).

exporters:
  # Traces -> Grafana Tempo
  otlp/tempo:
    endpoint: tempo-distributor.observability.svc:4317
    tls:
      insecure: true # in-cluster communication
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
    timeout: 30s

  # Traces -> Jaeger (native OTLP support)
  otlp/jaeger:
    endpoint: jaeger-collector.observability.svc:4317
    tls:
      cert_file: /certs/client.crt
      key_file: /certs/client.key
      ca_file: /certs/ca.crt

Prometheus Remote Write Exporter

The Prometheus Remote Write Exporter sends metrics to Prometheus-compatible backends (Mimir, Thanos, Cortex, VictoriaMetrics).

exporters:
  prometheusremotewrite/mimir:
    endpoint: https://mimir.observability.svc:9009/api/v1/push
    tls:
      insecure: false
      cert_file: /certs/client.crt
      key_file: /certs/client.key
    headers:
      X-Scope-OrgID: 'tenant-production'
    resource_to_telemetry_conversion:
      enabled: true # convert Resource attributes into metric labels
    external_labels:
      cluster: 'prod-ap-northeast-2'
      region: 'ap-northeast-2'
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 60s
    sending_queue:
      enabled: true
      num_consumers: 5
      queue_size: 10000

Loki Exporter

The Loki Exporter sends log data to Grafana Loki. The label mapping configuration matters here, and excessive label cardinality degrades Loki's performance, so it requires caution.

exporters:
  loki:
    endpoint: https://loki-gateway.observability.svc:3100/loki/api/v1/push
    headers:
      X-Scope-OrgID: 'tenant-production'
    default_labels_enabled:
      exporter: false
      job: true
      instance: true
      level: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
    sending_queue:
      enabled: true
      num_consumers: 5
      queue_size: 5000

Popular Backend Comparison

Agent vs Gateway Deployment Patterns

Ways of deploying the OpenTelemetry Collector fall broadly into the Agent pattern, the Gateway pattern, and a hybrid that combines the two. In production the Agent + Gateway combination is the most widely used, and you should understand the strengths and weaknesses of each pattern and choose according to your traffic scale.

Pattern Comparison

Agent + Gateway Hybrid Architecture

[Node 1]                    [Node 2]                    [Node N]
+----------+               +----------+               +----------+
| App Pods |               | App Pods |               | App Pods |
+----+-----+               +----+-----+               +----+-----+
     |                          |                          |
+----+-----+               +----+-----+               +----+-----+
| OTel     |               | OTel     |               | OTel     |
| Agent    |               | Agent    |               | Agent    |
| (DaemonSet)              | (DaemonSet)              | (DaemonSet)
+----+-----+               +----+-----+               +----+-----+
     |                          |                          |
     +------------+-------------+-----------+--------------+
                  |                         |
           +------+------+          +------+------+
           | OTel Gateway |          | OTel Gateway |
           | (Deployment) |          | (Deployment) |
           +------+------+          +------+------+
                  |                         |
     +------------+-------------------------+
     |             |              |
+----+----+  +----+----+  +-----+-----+
|  Tempo  |  |  Mimir  |  |   Loki    |
+---------+  +---------+  +-----------+

The Agent performs only lightweight processing (memory limiting, basic batching, K8s metadata injection), while the Gateway takes on Tail Sampling, advanced filtering, and final backend routing. This separation keeps the Agent's resource usage to a minimum while still allowing sophisticated data processing on the Gateway.

Deploying in Kubernetes

Agent DaemonSet Manifest

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-agent
  namespace: observability
spec:
  mode: daemonset
  image: otel/opentelemetry-collector-contrib:0.120.0
  serviceAccount: otel-collector-agent
  env:
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: K8S_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi
  volumeMounts:
    - name: varlogpods
      mountPath: /var/log/pods
      readOnly: true
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
  volumes:
    - name: varlogpods
      hostPath:
        path: /var/log/pods
    - name: varlibdockercontainers
      hostPath:
        path: /var/lib/docker/containers
  tolerations:
    - operator: Exists
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      filelog:
        include:
          - /var/log/pods/*/*/*.log
        exclude:
          - /var/log/pods/observability_otel-*/*/*.log
        start_at: end
        include_file_path: true
        operators:
          - type: regex_parser
            id: parser-cri
            regex: '^(?P<time>[^ Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
            timestamp:
              parse_from: attributes.time
              layout: '%Y-%m-%dT%H:%M:%S.%fZ'
      hostmetrics:
        collection_interval: 30s
        scrapers:
          cpu: {}
          memory: {}
          disk: {}
          network: {}
          load: {}
          filesystem:
            exclude_mount_points:
              mount_points: ['/dev/*', '/proc/*', '/sys/*']
              match_type: regexp

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 400
        spike_limit_mib: 100
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.statefulset.name
            - k8s.daemonset.name
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.node.name
            - k8s.container.name
          labels:
            - tag_name: app.label.team
              key: team
              from: pod
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.ip
          - sources:
              - from: connection
      batch:
        timeout: 5s
        send_batch_size: 4096

    exporters:
      otlp/gateway:
        endpoint: otel-gateway.observability.svc.cluster.local:4317
        tls:
          insecure: true
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_interval: 30s
        sending_queue:
          enabled: true
          queue_size: 2000

    extensions:
      health_check:
        endpoint: 0.0.0.0:13133

    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp/gateway]
        metrics:
          receivers: [otlp, hostmetrics]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp/gateway]
        logs:
          receivers: [otlp, filelog]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp/gateway]
      telemetry:
        logs:
          level: warn
        metrics:
          address: 0.0.0.0:8888

Gateway Deployment Manifest

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: deployment
  replicas: 3
  image: otel/opentelemetry-collector-contrib:0.120.0
  serviceAccount: otel-collector-gateway
  resources:
    requests:
      cpu: '1'
      memory: 2Gi
    limits:
      cpu: '2'
      memory: 4Gi
  autoscaler:
    minReplicas: 3
    maxReplicas: 10
    targetCPUUtilization: 70
    targetMemoryUtilization: 80
  podDisruptionBudget:
    minAvailable: 2
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            max_recv_msg_size_mib: 16

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 3200
        spike_limit_mib: 800
      resourcedetection:
        detectors: [env, system, gcp, aws, azure]
        timeout: 5s
        override: false
      attributes/security:
        actions:
          - key: http.request.header.authorization
            action: delete
          - key: db.statement
            action: hash
      filter/metrics:
        metrics:
          exclude:
            match_type: regexp
            metric_names:
              - 'go_.*'
              - 'process_.*'
              - 'promhttp_.*'
      tail_sampling:
        decision_wait: 30s
        num_traces: 200000
        expected_new_traces_per_sec: 5000
        policies:
          - name: error-traces
            type: status_code
            status_code:
              status_codes: [ERROR]
          - name: high-latency
            type: latency
            latency:
              threshold_ms: 2000
          - name: default-sampling
            type: probabilistic
            probabilistic:
              sampling_percentage: 10
      batch:
        timeout: 10s
        send_batch_size: 16384
        send_batch_max_size: 32768

    exporters:
      otlp/tempo:
        endpoint: tempo-distributor.observability.svc:4317
        tls:
          insecure: true
        sending_queue:
          enabled: true
          num_consumers: 10
          queue_size: 10000
      prometheusremotewrite/mimir:
        endpoint: http://mimir-distributor.observability.svc:8080/api/v1/push
        headers:
          X-Scope-OrgID: 'production'
        resource_to_telemetry_conversion:
          enabled: true
        sending_queue:
          enabled: true
          num_consumers: 5
          queue_size: 10000
      loki:
        endpoint: http://loki-gateway.observability.svc:3100/loki/api/v1/push
        headers:
          X-Scope-OrgID: 'production'
        sending_queue:
          enabled: true
          queue_size: 5000

    extensions:
      health_check:
        endpoint: 0.0.0.0:13133
      zpages:
        endpoint: 0.0.0.0:55679
      pprof:
        endpoint: 0.0.0.0:1777

    service:
      extensions: [health_check, zpages, pprof]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, resourcedetection, attributes/security, tail_sampling, batch]
          exporters: [otlp/tempo]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, resourcedetection, filter/metrics, batch]
          exporters: [prometheusremotewrite/mimir]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, resourcedetection, attributes/security, batch]
          exporters: [loki]
      telemetry:
        logs:
          level: info
        metrics:
          address: 0.0.0.0:8888

Gateway Load Balancing and TraceID-Based Routing

When several Gateways run Tail Sampling and Spans sharing a TraceID are spread across different Gateway instances, the sampling decision becomes incomplete. To solve this, use the loadbalancing Exporter when sending from the Agent to the Gateway.

# Agent-side Exporter configuration (when sending to the Gateway)
exporters:
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
        timeout: 10s
    resolver:
      dns:
        hostname: otel-gateway-headless.observability.svc.cluster.local
        port: 4317
      # Or use the Kubernetes resolver
      # k8s:
      #   service: otel-gateway
      #   ports:
      #     - 4317
    routing_key: traceID # consistent hashing based on TraceID

With this configuration every Span carrying the same TraceID is routed to the same Gateway Pod, which guarantees the accuracy of Tail Sampling.

Memory Management and Backpressure

The problem that comes up most often when operating the OpenTelemetry Collector is memory-related. Surges in telemetry traffic, the Tail Sampling wait buffer, and accumulation in the sending queue all combine to push memory usage up.

Memory Usage Formula

When estimating the Collector's memory usage, take the following factors into account.

Total memory = Go runtime baseline (about 50MB)
          + Receiver buffers (number of connections x message size)
          + Processor buffers
            - Batch: send_batch_max_size x average record size
            - Tail Sampling: num_traces x average trace size
          + Exporter queue: queue_size x batch size x average record size
          + Internal overhead (about 10-20%)

Tail Sampling is the single largest consumer of memory. It holds traces in memory for the length of decision_wait, so if decision_wait is 30 seconds and 5,000 new traces arrive per second, roughly 150,000 traces sit in memory at the same time.

Backpressure Mechanism

The Collector's backpressure works in three stages.

In stage 1, when the Exporter's sending queue fills up, the Exporter passes pressure back to the Processor. In stage 2, when memory_limiter sees memory usage reach limit_mib, it sends a data-refusal signal to the Receiver. In stage 3, when the Receiver refuses data, it returns an error to the client (SDK or Agent), and the client's retry logic kicks in.

For this backpressure chain to work properly, memory_limiter must sit first in the Processor chain. Otherwise another Processor can exhaust memory before the memory limit takes effect, and OOM follows.

GOGC Tuning

Tuning the Go runtime's garbage collector also matters for memory management. The default GOGC value is 100, which triggers a GC when the heap grows 100% over the previous GC cycle. In environments with little memory headroom, you can lower GOGC to induce more frequent GC.

env:
  - name: GOGC
    value: '80' # lowered from the default 100 to 80
  - name: GOMEMLIMIT
    value: '3600MiB' # soft memory ceiling (90% of the limit)

Troubleshooting Guide

Diagnosing with the Collector's Own Metrics

The Collector exposes its own telemetry metrics on port 8888 by default. Scraping these metrics with Prometheus and monitoring them on a Grafana dashboard is essential.

# Receive metrics
otelcol_receiver_accepted_spans          # Number of Spans accepted by the Receiver
otelcol_receiver_refused_spans           # Number of Spans refused by the Receiver (backpressure)
otelcol_receiver_accepted_metric_points  # Number of metric points accepted
otelcol_receiver_accepted_log_records    # Number of log records accepted

# Processor metrics
otelcol_processor_dropped_spans          # Number of Spans dropped by a Processor
otelcol_processor_batch_batch_send_size  # Actual batch size sent

# Exporter metrics
otelcol_exporter_sent_spans              # Number of Spans the Exporter sent successfully
otelcol_exporter_send_failed_spans       # Number of Spans that failed to send
otelcol_exporter_queue_size              # Number of items currently waiting in the queue
otelcol_exporter_queue_capacity          # Maximum queue capacity

The key alert formulas are as follows.

• Data loss alert: rate(otelcol_receiver_refused_spans[5m]) > 0 -- if the Receiver is refusing data then backpressure is active, and any data the upstream does not resend can be lost.
• Exporter failure alert: rate(otelcol_exporter_send_failed_spans[5m]) > 0 -- this means the backend connection is failing, so check the network or the state of the backend.
• Queue saturation alert: otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 0.8 -- once the queue is more than 80% full, data drops may start soon.

Real-Time Debugging with zPages

Enabling the zPages extension lets you inspect the Collector's internal state in real time from a browser.

• /debug/servicez -- Collector service information
• /debug/pipelinez -- pipeline configuration and state
• /debug/extensionz -- extension state
• /debug/tracez -- samples of recently processed traces

Common Problems and Fixes

Problem 1: the Collector restarts repeatedly from OOM

The cause is usually that Tail Sampling's num_traces is too large or decision_wait is too long, so an excessive number of traces piles up in memory. Reduce num_traces or shorten decision_wait to 10-15 seconds, and lower memory_limiter's limit_mib to 75% of the container limit.

Problem 2: a "context deadline exceeded" error from the Exporter

This happens when the backend does not respond within the timeout. Increase the Exporter's timeout value, or raise sending_queue's num_consumers to increase concurrent sending. Fundamentally, you need to scale up the backend's processing capacity.

Problem 3: Spans are missing from a trace

This happens when there are several Tail Sampling Gateways but TraceID-based routing has not been configured. Use the loadbalancing Exporter on the Agent to apply consistent hashing based on TraceID.

Problem 4: a "context canceled" error from the Prometheus Receiver

This happens when scrape_timeout is greater than or equal to scrape_interval. Set scrape_timeout to 50-80% of scrape_interval.

Operations Checklist

A checklist for running the OpenTelemetry Collector reliably in a production environment.

Pre-deployment Checklist

• Confirm memory_limiter is the first Processor in every pipeline
• Confirm the batch Processor is the last Processor in every pipeline
• Confirm memory_limiter's limit_mib is at or below 75-80% of the container memory limit
• Confirm the Exporter's sending_queue is enabled and appropriately sized
• Confirm the Exporter's retry_on_failure is enabled
• Confirm the health_check Extension is configured and wired to the liveness/readiness probes
• Confirm certificates are mounted for any Exporter that requires TLS
• Confirm RBAC permissions are set correctly (a ClusterRole is required when using k8sattributes)

Monitoring Setup

• Prometheus scrape configuration for the Collector's own metrics (port 8888) is complete
• Grafana dashboard built (received/dropped/sent Span counts, queue utilization, memory usage)
• Key alert rules configured (refused spans, send failed, queue saturation, OOM)
• zPages or the pprof Extension enabled (for troubleshooting)

Scaling Strategy

• Appropriate resource requests/limits set on the Agent DaemonSet
• HPA configured on the Gateway Deployment (CPU 70%, Memory 80% target)
• PodDisruptionBudget configured on the Gateway (minAvailable or maxUnavailable)
• Gateway headless Service and loadbalancing Exporter configured when Tail Sampling is in use
• Confirm Pod Anti-Affinity spreads Gateway Pods across different nodes

Security Checklist

• Confirm sensitive attributes (Authorization headers, DB queries, and so on) are deleted or hashed in the attributes Processor
• Confirm the Collector endpoint is not needlessly exposed externally
• Confirm the principle of least privilege is applied to the ServiceAccount
• Confirm there are no hardcoded secrets in the configuration file (use environment variables or a Secret)

Failure Cases and Recovery

Case 1: A Cascading Failure from Tail Sampling Memory Runaway

Situation: Tail Sampling was running across 3 Gateways when a Black Friday traffic surge pushed incoming traces to 5 times the usual volume. decision_wait was set to 30 seconds and num_traces to 500,000, but the number of traces actually held in memory exceeded num_traces and memory usage shot up.

Symptom: Gateway Pods were OOM-killed one after another and kept restarting. Traffic concentrated on the restarted Pods, producing a domino effect of cascading OOMs.

Recovery procedure:

1. As an emergency measure, Tail Sampling was disabled and switched to probabilistic head sampling (10%), which immediately cut the volume of incoming traces.
2. The Gateway memory limit was raised from 4Gi to 8Gi and replicas were scaled from 3 to 6.
3. decision_wait was shortened from 30 seconds to 15 seconds and num_traces was adjusted to 200,000, after which Tail Sampling was re-enabled.

Lesson: Tail Sampling's memory usage scales linearly with traffic. You must run load tests against a peak-traffic scenario and set num_traces and decision_wait conservatively.

Case 2: Data Loss from Exporter Queue Saturation

Situation: The Tempo backend's Ingester slowed down because its disk filled up. The Collector's otlp/tempo Exporter began throwing timeout errors and the sending queue filled quickly.

Symptom: As the Exporter queue filled, new trace data began to drop. The otelcol_exporter_send_failed_spans metric shot up, and otelcol_exporter_queue_size reached queue_capacity.

Recovery procedure:

1. The Tempo Ingester's disk was expanded and the affected Ingester Pod was restarted.
2. The Collector's sending_queue size was temporarily raised from 5,000 to 20,000 to buy some buffering headroom.
3. retry_on_failure's max_elapsed_time was raised to 600 seconds so retries continued while the backend recovered.

Lesson: the sending_queue only acts as a cushion against a temporary backend failure. During a prolonged backend outage the queue will inevitably saturate, so backend monitoring and fast incident response are the real fix. For critical data, also consider an architecture that uses Kafka as an intermediate buffer to guarantee durability.

Case 3: Missing Pod Metadata from a K8s Attributes Processor Permission Gap

Situation: The k8sattributes Processor was configured, but metadata such as namespace and pod name was not being injected into the telemetry.

Symptom: Attributes such as k8s.namespace.name and k8s.pod.name were empty on traces and logs. The message "error": "forbidden" was printed in the Collector logs.

Fix: Binding a ClusterRole with get, list, and watch permissions on Pods, Namespaces, and ReplicaSets to the Collector's ServiceAccount resolved it.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  - apiGroups: ['']
    resources: ['pods', 'namespaces', 'nodes']
    verbs: ['get', 'list', 'watch']
  - apiGroups: ['apps']
    resources: ['replicasets', 'deployments', 'statefulsets', 'daemonsets']
    verbs: ['get', 'list', 'watch']
  - apiGroups: ['batch']
    resources: ['jobs', 'cronjobs']
    verbs: ['get', 'list', 'watch']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
subjects:
  - kind: ServiceAccount
    name: otel-collector-agent
    namespace: observability
roleRef:
  kind: ClusterRole
  name: otel-collector
  apiGroup: rbac.authorization.k8s.io

Configuration Validation and Testing

Validating the Collector configuration file before deploying to production matters. A Collector that fails to start because of a configuration error takes the entire telemetry pipeline down with it.

# Validate configuration file syntax
otelcol validate --config=config.yaml

# Start-up test in dry-run mode
otelcol --config=config.yaml --dry-run

# Local test using Docker
docker run --rm \
  -v $(pwd)/config.yaml:/etc/otelcol/config.yaml \
  otel/opentelemetry-collector-contrib:0.120.0 \
  validate --config=/etc/otelcol/config.yaml

Including a configuration validation step in the CI/CD pipeline heads off a bad configuration reaching production. If you use a Helm Chart, run validate against the configuration file in the ConfigMap produced after helm template rendering.

Advanced Operations Tips

Routing in a Multi-Tenant Environment

When several teams share a Collector and data has to be separated per tenant and sent to different backends, use the routing Connector.

connectors:
  routing:
    table:
      - statement: route() where attributes["team"] == "platform"
        pipelines: [traces/platform]
      - statement: route() where attributes["team"] == "payments"
        pipelines: [traces/payments]
    default_pipelines: [traces/default]

service:
  pipelines:
    traces/ingress:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [routing]
    traces/platform:
      receivers: [routing]
      processors: [batch]
      exporters: [otlp/tempo-platform]
    traces/payments:
      receivers: [routing]
      processors: [batch]
      exporters: [otlp/tempo-payments]
    traces/default:
      receivers: [routing]
      processors: [batch]
      exporters: [otlp/tempo-default]

Key Panels for a Collector Self-Monitoring Dashboard

A list of the key panels an operational Grafana dashboard must include.

• Throughput panel: Spans, Metric Points, and Log Records received/sent per second
• Drop rate panel: the refused + dropped / accepted ratio (0% is normal)
• Exporter queue panel: the queue_size to capacity ratio for each Exporter
• Memory panel: process RSS memory and the memory_limiter threshold
• Exporter latency panel: P50, P95, and P99 of send duration
• Retry panel: retry counts and failure rate

References

• OpenTelemetry Collector Configuration -- the official configuration reference. Every option for Receivers, Processors, and Exporters can be looked up here.
• OpenTelemetry Collector Architecture -- the official architecture document. It explains the internal data flow and how components interact.
• OpenTelemetry Collector Complete Guide - SigNoz -- a practical guide from SigNoz. It includes a range of configuration examples and benchmarks.
• OpenTelemetry Collector Guide - Dash0 -- a Collector operations guide from Dash0. It covers deployment patterns and production tips.
• Build Logs-to-Metrics Pipelines - Last9 -- a guide to building pipelines that generate metrics from logs. It explains signal conversion using Connectors.