LabHub

Blog

OpenTelemetry Collector Complete Operations Guide - From Pipeline Configuration to Backend Integration

한국어English日本語

OpenTelemetry Collector Operations Guide

Introduction

As microservice architectures have become the norm, observability has stopped being optional and turned into a necessity. In a distributed environment where dozens to hundreds of services talk to one another, collecting and processing the three telemetry signals - Traces, Metrics, and Logs - efficiently is a basic precondition for running a service reliably.

The OpenTelemetry Collector is the core component of OpenTelemetry, a CNCF (Cloud Native Computing Foundation) graduated project, and it provides a telemetry pipeline that is not tied to any vendor. Its job is to receive the telemetry data an application produces (Receiver), transform it (Processor), and export it to the backend you want (Exporter).

As of 2026, OpenTelemetry has become the de facto industry standard for observability instrumentation. Jaeger v2 has been internally redesigned on top of the OpenTelemetry Collector, and the Zipkin Exporter was officially moved to deprecated status from 2025. Against that backdrop, understanding how to operate the Collector correctly is a core competency for platform engineers and SREs.

In this article, we pull together everything operations needs, with practical examples throughout: the OpenTelemetry Collector's architecture, pipeline configuration, integration with a variety of backends, Kubernetes deployment strategy, sampling techniques, performance tuning, troubleshooting, and failure recovery.


OpenTelemetry Architecture and the Collector's Role

Overall Architecture Overview

The OpenTelemetry framework is broadly made up of three layers.

  1. SDK/API layer: the SDK for each language produces telemetry data from application code.
  2. Collector layer: receives, transforms, and routes the telemetry data that was produced.
  3. Backend layer: the final store (Jaeger, Tempo, Datadog, and so on) saves and visualizes the data.
[Application + OTel SDK]
        |
        | OTLP (gRPC/HTTP)
        v
+========================+
|   OpenTelemetry        |
|   Collector            |
|                        |
|  Receiver -> Processor |
|          -> Exporter   |
+========================+
    |         |         |
    v         v         v
 [Jaeger]  [Tempo]  [Datadog]

The Collector acts as the central hub in this architecture. The application only has to send data over OTLP (OpenTelemetry Protocol), and which backend is used is decided in the Collector configuration. That means swapping backends needs only a Collector configuration change, with no application code change.

Core vs Contrib Distributions

The OpenTelemetry Collector comes in two distributions.

ItemCoreContrib
Component scopeCore Receivers/Processors/Exporters onlyIncludes many community-contributed components
Binary sizeAbout 50MBAbout 200MB or more
Security exposureSmallWide
Suitable forProduction built on a custom buildPoC, testing, early adoption

In production environments, building a custom binary that contains only the components you need, using the OpenTelemetry Collector Builder (OCB), is recommended on both security and resource-efficiency grounds.


Distributed Tracing Tool Comparison: OpenTelemetry vs Jaeger vs Zipkin

Before getting into Collector configuration, let us lay out how the standing of the major distributed tracing tools has shifted.

ItemOpenTelemetryJaegerZipkin
Project statusCNCF Graduated (actively developed)CNCF Graduated (v2 transition complete)Independent project (maintenance mode)
Main languageGo (Collector), SDKs in many languagesGoJava
RoleInstrumentation framework + collection pipelineTrace storage/query/visualization backendTrace storage/query/visualization backend
ProtocolOTLP (gRPC, HTTP/protobuf)OTLP, Thrift (Legacy)HTTP/JSON, Thrift
Telemetry scopeTraces + Metrics + LogsTraces onlyTraces only
Kubernetes supportOperator, Helm Chart, DaemonSet/DeploymentHelm Chart, OperatorHelm Chart
Backend integrationVendor neutral (supports every backend)Own UI + Elasticsearch/Cassandra/ClickHouseOwn UI + Elasticsearch/Cassandra
SamplingHead-based + Tail-based (Collector)Remote Sampling APIProbabilistic sampling
2026 recommendationUse as the instrumentation standardMigration to OTLP-based v2 recommendedNot recommended for new adoption (deprecation planned)

Key summary: OpenTelemetry is the standard for instrumentation and collection, Jaeger v2 is an OTLP-native backend, and Zipkin is a legacy migration target. Since the OpenTelemetry SDK's Zipkin Exporter moved to officially planned deprecation from 2025, new projects are better off adopting an OTLP-based architecture.


Building the Receiver-Processor-Exporter Pipeline

Understanding the Pipeline Structure

An OpenTelemetry Collector configuration file is made up of four top-level sections.

The important point is that even if you define a component in the receivers, processors, or exporters section, it is not activated unless service.pipelines references it.

Basic Pipeline Configuration Example

The following is a basic configuration that receives traces with the OTLP Receiver and, after batch processing, sends them with the OTLP Exporter.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    send_batch_size: 8192
    timeout: 200ms
    send_batch_max_size: 0

exporters:
  otlp:
    endpoint: tempo.monitoring.svc.cluster.local:4317
    tls:
      insecure: true
  debug:
    verbosity: detailed

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  zpages:
    endpoint: 0.0.0.0:55679

service:
  extensions: [health_check, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp, debug]

The critical part of this configuration is the order of the processors. memory_limiter must always be placed as the first processor. The point is to refuse data early when the memory limit is exceeded, preventing an OOM (Out of Memory) crash of the Collector. batch must be placed after memory_limiter and the sampling processors, so that batches are assembled only once data dropping is complete.


Receiver Configuration in Detail

OTLP Receiver

This is the most basic and the recommended Receiver. It supports two protocols: gRPC (4317) and HTTP (4318).

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 4
        keepalive:
          server_parameters:
            max_connection_idle: 11s
            max_connection_age: 30s
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - 'https://*.example.com'

Jaeger Receiver

Used for compatibility with legacy Jaeger clients. It is handy during the transition to Jaeger v2.

receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268
      thrift_compact:
        endpoint: 0.0.0.0:6831

Prometheus Receiver

Collects metrics through Prometheus scraping. It can integrate with Kubernetes service discovery.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector-internal'
          scrape_interval: 15s
          static_configs:
            - targets: ['0.0.0.0:8888']
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: (.+)
              replacement: $$1

Processor Configuration in Detail

Memory Limiter Processor

This is the essential processor that controls the Collector's memory usage to prevent an OOM crash. It must be placed as the first processor in every pipeline.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024 # hard limit (1GB)
    spike_limit_mib: 256 # spike allowance (about 20% of the hard limit)

A guide to choosing the values:

Batch Processor

This processor groups telemetry data into batches, reducing the number of network requests and lowering serialization CPU overhead.

processors:
  batch:
    send_batch_size: 8192 # send immediately once this count is reached
    timeout: 200ms # send once this much time passes, regardless of size
    send_batch_max_size: 16384 # upper bound on batch size (0 means unlimited)

Filter Processor

Drops unnecessary telemetry data early to cut backend cost.

processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
        - 'name == "health_check"'
    metrics:
      metric:
        - 'name == "rpc.server.duration" and resource.attributes["service.name"] == "debug-svc"'

The configuration above filters out traces on health check paths and unnecessary metrics from specific services, so they are never sent to the backend. In high-traffic environments this can cut telemetry cost substantially.

Attributes Processor

This processor adds, modifies, and deletes attributes on Spans and metrics.

processors:
  attributes:
    actions:
      - key: environment
        value: production
        action: upsert
      - key: db.statement
        action: delete
      - key: http.request.header.authorization
        action: delete

It is especially useful for deleting sensitive information (DB queries, authentication tokens, and so on).


Integrating with Various Backends

Jaeger (OTLP)

Jaeger v2 supports OTLP natively, so you use the OTLP Exporter rather than a separate Jaeger Exporter.

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector.monitoring.svc.cluster.local:4317
    tls:
      insecure: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000

Grafana Tempo

Tempo also supports OTLP natively. You can pick either the gRPC or the HTTP protocol.

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.monitoring.svc.cluster.local:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s

Datadog

The Datadog Exporter is included in collector-contrib and authenticates with an API Key.

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: datadoghq.com
    traces:
      span_name_as_resource_name: true
    metrics:
      histograms:
        mode: distributions
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000

Building a Multi-Backend Pipeline

This is a configuration that sends data from a single Collector to several backends at once.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter, attributes, batch]
      exporters: [otlp/tempo, datadog]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]

Each Exporter receives its own copy of the data, so a failure in one Exporter does not affect the others. That said, since every Exporter handles the same data, the Receiver and Processor load is shared.


Kubernetes Deployment Strategy

DaemonSet vs Deployment

There are two main patterns for deploying the Collector in a Kubernetes environment.

ItemDaemonSet (Agent)Deployment (Gateway)
Deployment unit1 on every nodeAs many as the specified Replica count
RoleLocal telemetry collection, initial transformationCentralized processing, aggregation, final sending
Network loadIn-node communication (low)Cross-node communication (high)
Tail SamplingUnsuitable (traces are spread out)Suitable (all Spans concentrated)
Cluster metricsRisk of duplicate data when collectingCan be collected without duplication
Blast radiusThat node onlyThe entire pipeline
ScalingAutomatic as nodes are addedManual or automatic scaling with HPA

The recommended production pattern is a DaemonSet (Agent) + Deployment (Gateway) tiered structure. The Agent performs local collection and first-pass transformation on each node, and the Gateway handles aggregation, Tail Sampling, and final sending to the backend.

DaemonSet Agent Manifest

# otel-agent-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector-agent
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: otel-collector-agent
  template:
    metadata:
      labels:
        app: otel-collector-agent
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.120.0
          args: ['--config=/etc/otelcol/config.yaml']
          env:
            - name: GOMEMLIMIT
              value: '400MiB'
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          ports:
            - containerPort: 4317 # OTLP gRPC
              hostPort: 4317
              protocol: TCP
            - containerPort: 4318 # OTLP HTTP
              hostPort: 4318
              protocol: TCP
            - containerPort: 13133 # Health Check
              protocol: TCP
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /
              port: 13133
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /
              port: 13133
            initialDelaySeconds: 5
            periodSeconds: 5
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol
      volumes:
        - name: config
          configMap:
            name: otel-agent-config

Gateway Deployment Manifest

# otel-gateway-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector-gateway
  namespace: monitoring
spec:
  replicas: 3
  selector:
    matchLabels:
      app: otel-collector-gateway
  template:
    metadata:
      labels:
        app: otel-collector-gateway
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.120.0
          args: ['--config=/etc/otelcol/config.yaml']
          env:
            - name: GOMEMLIMIT
              value: '3200MiB'
          ports:
            - containerPort: 4317
              protocol: TCP
            - containerPort: 13133
              protocol: TCP
          resources:
            requests:
              cpu: 1000m
              memory: 2Gi
            limits:
              cpu: 2000m
              memory: 4Gi
          livenessProbe:
            httpGet:
              path: /
              port: 13133
          readinessProbe:
            httpGet:
              path: /
              port: 13133
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol
      volumes:
        - name: config
          configMap:
            name: otel-gateway-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector-gateway
  namespace: monitoring
spec:
  selector:
    app: otel-collector-gateway
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
    - name: otlp-http
      port: 4318
      targetPort: 4318
  type: ClusterIP

In the Agent configuration, the Gateway Service is specified as the Exporter endpoint.

# Agent exporter configuration
exporters:
  otlp/gateway:
    endpoint: otel-collector-gateway.monitoring.svc.cluster.local:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      queue_size: 2000
    retry_on_failure:
      enabled: true

When connecting to the Gateway you must go through a Kubernetes Service. Referencing Pod IPs directly breaks the connection whenever a Pod restarts, and it does no load balancing either.


Sampling Strategy

Head-based Sampling

This approach makes the sampling decision at the SDK level, at the moment the trace starts.

Advantage: it is simple to implement, and the resource saving is immediate.

Drawback: because the decision is made without seeing the whole trace, it can miss traces that contain an error.

# Use the TraceIdRatioBased Sampler in the SDK configuration
# Sample only 10% of traces
processors:
  probabilistic_sampler:
    sampling_percentage: 10

Tail-based Sampling

This approach makes the sampling decision after all (or most) Spans of a trace have arrived. It is performed in the Collector's Tail Sampling Processor.

Advantage: it can retain 100% of the traces that meet specific conditions, such as errors or high latency.

Drawback: since every Span of a trace has to gather in one place, it can only be used on a Gateway Collector, and it consumes a lot of memory.

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      # Policy 1: sample 100% of traces that contain an error
      - name: errors-policy
        type: status_code
        status_code:
          status_codes:
            - ERROR
      # Policy 2: sample 100% of traces whose latency exceeds 500ms
      - name: latency-policy
        type: latency
        latency:
          threshold_ms: 500
      # Policy 3: sample only 5% of the remaining traces
      - name: probabilistic-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

Operational cautions for Tail Sampling:

EnvironmentRecommended strategyDescription
Low-traffic serviceCollect everything (100%)Retain every trace to keep debugging easy
Typical productionHead 10% + Tail on errors/latencyBalances cost saving against keeping important traces
High-traffic serviceHead 1% + Tail on errors/latencyCost optimization under heavy traffic
Regulated or auditedCollect everything + long-term storageMeets compliance requirements

Performance Tuning and High Availability

Key Performance Parameters

# Example of a performance-optimized configuration
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1638 # ~80% of the 2GB container memory
    spike_limit_mib: 328 # ~20% of limit_mib
  batch:
    send_batch_size: 10000
    timeout: 500ms
    send_batch_max_size: 20000

exporters:
  otlp/backend:
    endpoint: backend:4317
    sending_queue:
      enabled: true
      num_consumers: 20 # number of parallel sending workers
      queue_size: 10000 # queue size (in batches)
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    timeout: 30s

GOMEMLIMIT Configuration

Set the GOMEMLIMIT environment variable to improve the Go runtime's GC efficiency.

# When the container memory limit is 2GB
# GOMEMLIMIT = limit_mib * 0.8 = 1638 * 0.8 ~ 1310MiB
export GOMEMLIMIT=1310MiB
Container memorylimit_mibspike_limit_mibGOMEMLIMIT
512Mi41082328MiB
1Gi820164656MiB
2Gi16383281310MiB
4Gi32766552621MiB

High Availability (HA) Configuration

The key strategies for Gateway Collector high availability are as follows.

  1. At least 3 Replicas: set the Deployment's replicas to at least 3.
  2. Pod Anti-Affinity: configure it so Gateway Pods do not pile up on the same node.
  3. PDB (PodDisruptionBudget): limit how many Pods can terminate at the same time.
  4. HPA: configure automatic scaling based on CPU and memory.
# PodDisruptionBudget configuration
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: otel-gateway-pdb
  namespace: monitoring
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: otel-collector-gateway

Troubleshooting Guide

Common Problems and How to Fix Them

1. Data is not reaching the backend

# Step 1: check the Collector logs
kubectl logs -n monitoring deploy/otel-collector-gateway --tail=100

# Step 2: check pipeline state in zPages
kubectl port-forward -n monitoring svc/otel-collector-gateway 55679:55679
# Open http://localhost:55679/debug/tracez in a browser

# Step 3: temporarily add the debug exporter to observe the data flow
# after adding debug: { verbosity: detailed } under exporters
# add the debug exporter to the pipeline

2. Collector memory usage spikes

3. Exporter queue saturation

# Check queue state with the Collector's internal metrics
curl http://localhost:8888/metrics | grep otelcol_exporter_queue
# otelcol_exporter_queue_size: current queue size
# otelcol_exporter_queue_capacity: maximum queue capacity
# when queue_size approaches capacity, the queue is saturated

Cause: the backend cannot process data fast enough, or too much traffic is flowing into the Collector.

Solution: raise num_consumers, check the backend's processing capacity, or drop unnecessary data early with a filter processor.

4. Processor order mistakes

The wrong processor order produces subtle problems.

MistakeSymptomCorrect order
batch placed before memory_limiterMemory spikes, then OOMmemory_limiter always first
attributes placed after batchAttribute changes are not appliedattributes before batch
tail_sampling placed on a DaemonSetIncomplete sampling decisionsuse tail_sampling on the Gateway only
resourcedetection placed lastResource attributes are missingplace resourcedetection near the front

Key Monitoring Metrics

To monitor the Collector's own operational health, track the following internal metrics.

MetricDescriptionAlert criterion
otelcol_receiver_accepted_spansSpans received successfullyAlert on a sharp drop
otelcol_receiver_refused_spansSpans refused on receiptAlert if a value other than 0 persists
otelcol_exporter_sent_spansSpans sent successfullyAlert on a large gap versus accepted
otelcol_exporter_send_failed_spansSpans that failed to sendAlert if a value other than 0 persists
otelcol_exporter_queue_sizeCurrent queue sizeAlert above 80% of capacity
otelcol_processor_dropped_spansSpans dropped by a processorAlert when above expectations

Failure Cases and Recovery Procedures

Case 1: Total Gateway Failure (All Replicas Down)

Symptom: all telemetry data is lost. The Agent's Exporter queue saturates and data starts dropping.

Cause: deploying a bad configuration change, a cascade of OOMs, a Node failure, and so on.

Recovery procedure:

  1. Roll back immediately to the last known-good ConfigMap.
  2. Check that the Gateway Deployment has enough replicas.
  3. Verify that Pod Anti-Affinity is configured.
  4. Check the Agent-side sending_queue and retry_on_failure settings. Data sitting in the queue is resent automatically once the Gateway recovers.
  5. To prevent a recurrence, configure a PDB and apply Canary deployment for ConfigMap changes.
# ConfigMap rollback
kubectl rollout undo configmap/otel-gateway-config -n monitoring
# Or apply the previous version of the ConfigMap directly
kubectl apply -f otel-gateway-config-backup.yaml

# Restart the Gateway Deployment
kubectl rollout restart deployment/otel-collector-gateway -n monitoring

# Check recovery status
kubectl rollout status deployment/otel-collector-gateway -n monitoring

Case 2: Data Loss Caused by a Backend Failure

Symptom: Exporter queue saturation, and a sharp rise in otelcol_exporter_send_failed_spans.

Cause: the backend (Tempo, Jaeger, and so on) went down or became slow to respond.

Recovery procedure:

  1. Check the backend's state first and recover it.
  2. The Collector retries automatically thanks to the retry_on_failure setting.
  3. Data dropped because of queue saturation cannot be recovered, so in the longer term increase the queue size or introduce a buffer layer such as Kafka.

Case 3: OOM Caused by an Unset memory_limiter

Symptom: the Collector Pod repeatedly restarts in an OOMKilled state.

Cause: the memory_limiter processor is not configured, or is configured with unsuitable values.

Recovery procedure:

  1. Add memory_limiter as the first processor of every pipeline.
  2. Set limit_mib to 80% of the container memory limit.
  3. Set the GOMEMLIMIT environment variable.
  4. Review whether the container resource limit is too small for the workload.

Operations Checklist

Initial Deployment Checklist

Processor Order Checklist

Monitoring Checklist

Security Checklist


Conclusion

The OpenTelemetry Collector is core infrastructure for a modern observability pipeline. Thanks to its vendor-neutral design you can integrate flexibly with any backend - Jaeger, Grafana Tempo, Datadog, and so on - and swapping backends requires no change to application code.

The three most important things for stable operation can be summarized as follows.

  1. Set memory_limiter as the very first processor and configure GOMEMLIMIT appropriately.
  2. Deploy as a DaemonSet (Agent) + Deployment (Gateway) tiered structure to separate local collection from centralized processing.
  3. Enable the Exporter's sending_queue and retry_on_failure to gain resilience against temporary backend failures.

The Collector keeps evolving, and new features continue to arrive, such as remote configuration management through OpAMP (Open Agent Management Protocol). It is worth checking the official documentation and release notes periodically and keeping up with the latest changes.


References

Comments

No comments yet.

Sign in to leave a comment