- Introduction
- OpenTelemetry Architecture and the Collector's Role
- Distributed Tracing Tool Comparison: OpenTelemetry vs Jaeger vs Zipkin
- Building the Receiver-Processor-Exporter Pipeline
- Receiver Configuration in Detail
- Processor Configuration in Detail
- Integrating with Various Backends
- Kubernetes Deployment Strategy
- Sampling Strategy
- Performance Tuning and High Availability
- Troubleshooting Guide
- Failure Cases and Recovery Procedures
- Operations Checklist
- Conclusion
- References

Introduction
As microservice architectures have become the norm, observability has stopped being optional and turned into a necessity. In a distributed environment where dozens to hundreds of services talk to one another, collecting and processing the three telemetry signals - Traces, Metrics, and Logs - efficiently is a basic precondition for running a service reliably.
The OpenTelemetry Collector is the core component of OpenTelemetry, a CNCF (Cloud Native Computing Foundation) graduated project, and it provides a telemetry pipeline that is not tied to any vendor. Its job is to receive the telemetry data an application produces (Receiver), transform it (Processor), and export it to the backend you want (Exporter).
As of 2026, OpenTelemetry has become the de facto industry standard for observability instrumentation. Jaeger v2 has been internally redesigned on top of the OpenTelemetry Collector, and the Zipkin Exporter was officially moved to deprecated status from 2025. Against that backdrop, understanding how to operate the Collector correctly is a core competency for platform engineers and SREs.
In this article, we pull together everything operations needs, with practical examples throughout: the OpenTelemetry Collector's architecture, pipeline configuration, integration with a variety of backends, Kubernetes deployment strategy, sampling techniques, performance tuning, troubleshooting, and failure recovery.
OpenTelemetry Architecture and the Collector's Role
Overall Architecture Overview
The OpenTelemetry framework is broadly made up of three layers.
- SDK/API layer: the SDK for each language produces telemetry data from application code.
- Collector layer: receives, transforms, and routes the telemetry data that was produced.
- Backend layer: the final store (Jaeger, Tempo, Datadog, and so on) saves and visualizes the data.
[Application + OTel SDK]
|
| OTLP (gRPC/HTTP)
v
+========================+
| OpenTelemetry |
| Collector |
| |
| Receiver -> Processor |
| -> Exporter |
+========================+
| | |
v v v
[Jaeger] [Tempo] [Datadog]
The Collector acts as the central hub in this architecture. The application only has to send data over OTLP (OpenTelemetry Protocol), and which backend is used is decided in the Collector configuration. That means swapping backends needs only a Collector configuration change, with no application code change.
Core vs Contrib Distributions
The OpenTelemetry Collector comes in two distributions.
| Item | Core | Contrib |
|---|---|---|
| Component scope | Core Receivers/Processors/Exporters only | Includes many community-contributed components |
| Binary size | About 50MB | About 200MB or more |
| Security exposure | Small | Wide |
| Suitable for | Production built on a custom build | PoC, testing, early adoption |
In production environments, building a custom binary that contains only the components you need, using the OpenTelemetry Collector Builder (OCB), is recommended on both security and resource-efficiency grounds.
Distributed Tracing Tool Comparison: OpenTelemetry vs Jaeger vs Zipkin
Before getting into Collector configuration, let us lay out how the standing of the major distributed tracing tools has shifted.
| Item | OpenTelemetry | Jaeger | Zipkin |
|---|---|---|---|
| Project status | CNCF Graduated (actively developed) | CNCF Graduated (v2 transition complete) | Independent project (maintenance mode) |
| Main language | Go (Collector), SDKs in many languages | Go | Java |
| Role | Instrumentation framework + collection pipeline | Trace storage/query/visualization backend | Trace storage/query/visualization backend |
| Protocol | OTLP (gRPC, HTTP/protobuf) | OTLP, Thrift (Legacy) | HTTP/JSON, Thrift |
| Telemetry scope | Traces + Metrics + Logs | Traces only | Traces only |
| Kubernetes support | Operator, Helm Chart, DaemonSet/Deployment | Helm Chart, Operator | Helm Chart |
| Backend integration | Vendor neutral (supports every backend) | Own UI + Elasticsearch/Cassandra/ClickHouse | Own UI + Elasticsearch/Cassandra |
| Sampling | Head-based + Tail-based (Collector) | Remote Sampling API | Probabilistic sampling |
| 2026 recommendation | Use as the instrumentation standard | Migration to OTLP-based v2 recommended | Not recommended for new adoption (deprecation planned) |
Key summary: OpenTelemetry is the standard for instrumentation and collection, Jaeger v2 is an OTLP-native backend, and Zipkin is a legacy migration target. Since the OpenTelemetry SDK's Zipkin Exporter moved to officially planned deprecation from 2025, new projects are better off adopting an OTLP-based architecture.
Building the Receiver-Processor-Exporter Pipeline
Understanding the Pipeline Structure
An OpenTelemetry Collector configuration file is made up of four top-level sections.
- receivers: defines how data is received
- processors: defines the rules for transforming, converting, and filtering data
- exporters: defines where data is sent
- service: combines the components above to actually activate a pipeline
The important point is that even if you define a component in the receivers, processors, or exporters section, it is not activated unless service.pipelines references it.
Basic Pipeline Configuration Example
The following is a basic configuration that receives traces with the OTLP Receiver and, after batch processing, sends them with the OTLP Exporter.
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128
batch:
send_batch_size: 8192
timeout: 200ms
send_batch_max_size: 0
exporters:
otlp:
endpoint: tempo.monitoring.svc.cluster.local:4317
tls:
insecure: true
debug:
verbosity: detailed
extensions:
health_check:
endpoint: 0.0.0.0:13133
zpages:
endpoint: 0.0.0.0:55679
service:
extensions: [health_check, zpages]
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp, debug]
The critical part of this configuration is the order of the processors. memory_limiter must always be placed as the first processor. The point is to refuse data early when the memory limit is exceeded, preventing an OOM (Out of Memory) crash of the Collector. batch must be placed after memory_limiter and the sampling processors, so that batches are assembled only once data dropping is complete.
Receiver Configuration in Detail
OTLP Receiver
This is the most basic and the recommended Receiver. It supports two protocols: gRPC (4317) and HTTP (4318).
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
max_recv_msg_size_mib: 4
keepalive:
server_parameters:
max_connection_idle: 11s
max_connection_age: 30s
http:
endpoint: 0.0.0.0:4318
cors:
allowed_origins:
- 'https://*.example.com'
Jaeger Receiver
Used for compatibility with legacy Jaeger clients. It is handy during the transition to Jaeger v2.
receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_http:
endpoint: 0.0.0.0:14268
thrift_compact:
endpoint: 0.0.0.0:6831
Prometheus Receiver
Collects metrics through Prometheus scraping. It can integrate with Kubernetes service discovery.
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector-internal'
scrape_interval: 15s
static_configs:
- targets: ['0.0.0.0:8888']
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: (.+)
replacement: $$1
Processor Configuration in Detail
Memory Limiter Processor
This is the essential processor that controls the Collector's memory usage to prevent an OOM crash. It must be placed as the first processor in every pipeline.
processors:
memory_limiter:
check_interval: 1s
limit_mib: 1024 # hard limit (1GB)
spike_limit_mib: 256 # spike allowance (about 20% of the hard limit)
A guide to choosing the values:
limit_mib: set it to about 80% of the container memory limit. If the container memory limit is 2GB,limit_mib: 1638is about right.spike_limit_mib: set it to about 20% oflimit_mib. The soft limit is calculated aslimit_mib - spike_limit_mib.GOMEMLIMITenvironment variable: setting the Collector container'sGOMEMLIMITto 80% of the hard limit (that is, about 64% of the container memory limit) makes the Go runtime's GC work more efficiently.
Batch Processor
This processor groups telemetry data into batches, reducing the number of network requests and lowering serialization CPU overhead.
processors:
batch:
send_batch_size: 8192 # send immediately once this count is reached
timeout: 200ms # send once this much time passes, regardless of size
send_batch_max_size: 16384 # upper bound on batch size (0 means unlimited)
send_batch_sizeacts as a trigger, not as an upper bound on batch size. To actually cap the batch size you must setsend_batch_max_size.- When memory is under pressure, reduce
send_batch_sizeandtimeoutso batches flush sooner.
Filter Processor
Drops unnecessary telemetry data early to cut backend cost.
processors:
filter:
error_mode: ignore
traces:
span:
- 'attributes["http.route"] == "/healthz"'
- 'attributes["http.route"] == "/readyz"'
- 'name == "health_check"'
metrics:
metric:
- 'name == "rpc.server.duration" and resource.attributes["service.name"] == "debug-svc"'
The configuration above filters out traces on health check paths and unnecessary metrics from specific services, so they are never sent to the backend. In high-traffic environments this can cut telemetry cost substantially.
Attributes Processor
This processor adds, modifies, and deletes attributes on Spans and metrics.
processors:
attributes:
actions:
- key: environment
value: production
action: upsert
- key: db.statement
action: delete
- key: http.request.header.authorization
action: delete
It is especially useful for deleting sensitive information (DB queries, authentication tokens, and so on).
Integrating with Various Backends
Jaeger (OTLP)
Jaeger v2 supports OTLP natively, so you use the OTLP Exporter rather than a separate Jaeger Exporter.
exporters:
otlp/jaeger:
endpoint: jaeger-collector.monitoring.svc.cluster.local:4317
tls:
insecure: true
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
sending_queue:
enabled: true
num_consumers: 10
queue_size: 5000
Grafana Tempo
Tempo also supports OTLP natively. You can pick either the gRPC or the HTTP protocol.
exporters:
otlp/tempo:
endpoint: tempo-distributor.monitoring.svc.cluster.local:4317
tls:
insecure: true
sending_queue:
enabled: true
num_consumers: 10
queue_size: 10000
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
Datadog
The Datadog Exporter is included in collector-contrib and authenticates with an API Key.
exporters:
datadog:
api:
key: ${env:DD_API_KEY}
site: datadoghq.com
traces:
span_name_as_resource_name: true
metrics:
histograms:
mode: distributions
sending_queue:
enabled: true
num_consumers: 10
queue_size: 5000
Building a Multi-Backend Pipeline
This is a configuration that sends data from a single Collector to several backends at once.
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, filter, attributes, batch]
exporters: [otlp/tempo, datadog]
metrics:
receivers: [otlp, prometheus]
processors: [memory_limiter, batch]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [loki]
Each Exporter receives its own copy of the data, so a failure in one Exporter does not affect the others. That said, since every Exporter handles the same data, the Receiver and Processor load is shared.
Kubernetes Deployment Strategy
DaemonSet vs Deployment
There are two main patterns for deploying the Collector in a Kubernetes environment.
| Item | DaemonSet (Agent) | Deployment (Gateway) |
|---|---|---|
| Deployment unit | 1 on every node | As many as the specified Replica count |
| Role | Local telemetry collection, initial transformation | Centralized processing, aggregation, final sending |
| Network load | In-node communication (low) | Cross-node communication (high) |
| Tail Sampling | Unsuitable (traces are spread out) | Suitable (all Spans concentrated) |
| Cluster metrics | Risk of duplicate data when collecting | Can be collected without duplication |
| Blast radius | That node only | The entire pipeline |
| Scaling | Automatic as nodes are added | Manual or automatic scaling with HPA |
The recommended production pattern is a DaemonSet (Agent) + Deployment (Gateway) tiered structure. The Agent performs local collection and first-pass transformation on each node, and the Gateway handles aggregation, Tail Sampling, and final sending to the backend.
DaemonSet Agent Manifest
# otel-agent-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: otel-collector-agent
namespace: monitoring
spec:
selector:
matchLabels:
app: otel-collector-agent
template:
metadata:
labels:
app: otel-collector-agent
spec:
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:0.120.0
args: ['--config=/etc/otelcol/config.yaml']
env:
- name: GOMEMLIMIT
value: '400MiB'
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
ports:
- containerPort: 4317 # OTLP gRPC
hostPort: 4317
protocol: TCP
- containerPort: 4318 # OTLP HTTP
hostPort: 4318
protocol: TCP
- containerPort: 13133 # Health Check
protocol: TCP
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
livenessProbe:
httpGet:
path: /
port: 13133
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 13133
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: config
mountPath: /etc/otelcol
volumes:
- name: config
configMap:
name: otel-agent-config
Gateway Deployment Manifest
# otel-gateway-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector-gateway
namespace: monitoring
spec:
replicas: 3
selector:
matchLabels:
app: otel-collector-gateway
template:
metadata:
labels:
app: otel-collector-gateway
spec:
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:0.120.0
args: ['--config=/etc/otelcol/config.yaml']
env:
- name: GOMEMLIMIT
value: '3200MiB'
ports:
- containerPort: 4317
protocol: TCP
- containerPort: 13133
protocol: TCP
resources:
requests:
cpu: 1000m
memory: 2Gi
limits:
cpu: 2000m
memory: 4Gi
livenessProbe:
httpGet:
path: /
port: 13133
readinessProbe:
httpGet:
path: /
port: 13133
volumeMounts:
- name: config
mountPath: /etc/otelcol
volumes:
- name: config
configMap:
name: otel-gateway-config
---
apiVersion: v1
kind: Service
metadata:
name: otel-collector-gateway
namespace: monitoring
spec:
selector:
app: otel-collector-gateway
ports:
- name: otlp-grpc
port: 4317
targetPort: 4317
- name: otlp-http
port: 4318
targetPort: 4318
type: ClusterIP
In the Agent configuration, the Gateway Service is specified as the Exporter endpoint.
# Agent exporter configuration
exporters:
otlp/gateway:
endpoint: otel-collector-gateway.monitoring.svc.cluster.local:4317
tls:
insecure: true
sending_queue:
enabled: true
queue_size: 2000
retry_on_failure:
enabled: true
When connecting to the Gateway you must go through a Kubernetes Service. Referencing Pod IPs directly breaks the connection whenever a Pod restarts, and it does no load balancing either.
Sampling Strategy
Head-based Sampling
This approach makes the sampling decision at the SDK level, at the moment the trace starts.
Advantage: it is simple to implement, and the resource saving is immediate.
Drawback: because the decision is made without seeing the whole trace, it can miss traces that contain an error.
# Use the TraceIdRatioBased Sampler in the SDK configuration
# Sample only 10% of traces
processors:
probabilistic_sampler:
sampling_percentage: 10
Tail-based Sampling
This approach makes the sampling decision after all (or most) Spans of a trace have arrived. It is performed in the Collector's Tail Sampling Processor.
Advantage: it can retain 100% of the traces that meet specific conditions, such as errors or high latency.
Drawback: since every Span of a trace has to gather in one place, it can only be used on a Gateway Collector, and it consumes a lot of memory.
processors:
tail_sampling:
decision_wait: 30s
num_traces: 100000
expected_new_traces_per_sec: 1000
policies:
# Policy 1: sample 100% of traces that contain an error
- name: errors-policy
type: status_code
status_code:
status_codes:
- ERROR
# Policy 2: sample 100% of traces whose latency exceeds 500ms
- name: latency-policy
type: latency
latency:
threshold_ms: 500
# Policy 3: sample only 5% of the remaining traces
- name: probabilistic-policy
type: probabilistic
probabilistic:
sampling_percentage: 5
Operational cautions for Tail Sampling:
- Never use it on a DaemonSet Agent. The Spans of a trace arrive spread across several nodes, so each Agent sees only part of the trace and cannot make a correct sampling decision.
decision_waitis the amount of time allowed for every Span of a trace to arrive. Too short and the decision is made on an incomplete trace; too long and memory consumption climbs sharply.num_tracesis the upper bound on the number of traces held in memory at once. Beyond that, the oldest trace is forcibly decided.
Recommended Sampling Strategy Combinations
| Environment | Recommended strategy | Description |
|---|---|---|
| Low-traffic service | Collect everything (100%) | Retain every trace to keep debugging easy |
| Typical production | Head 10% + Tail on errors/latency | Balances cost saving against keeping important traces |
| High-traffic service | Head 1% + Tail on errors/latency | Cost optimization under heavy traffic |
| Regulated or audited | Collect everything + long-term storage | Meets compliance requirements |
Performance Tuning and High Availability
Key Performance Parameters
# Example of a performance-optimized configuration
processors:
memory_limiter:
check_interval: 1s
limit_mib: 1638 # ~80% of the 2GB container memory
spike_limit_mib: 328 # ~20% of limit_mib
batch:
send_batch_size: 10000
timeout: 500ms
send_batch_max_size: 20000
exporters:
otlp/backend:
endpoint: backend:4317
sending_queue:
enabled: true
num_consumers: 20 # number of parallel sending workers
queue_size: 10000 # queue size (in batches)
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
timeout: 30s
GOMEMLIMIT Configuration
Set the GOMEMLIMIT environment variable to improve the Go runtime's GC efficiency.
# When the container memory limit is 2GB
# GOMEMLIMIT = limit_mib * 0.8 = 1638 * 0.8 ~ 1310MiB
export GOMEMLIMIT=1310MiB
| Container memory | limit_mib | spike_limit_mib | GOMEMLIMIT |
|---|---|---|---|
| 512Mi | 410 | 82 | 328MiB |
| 1Gi | 820 | 164 | 656MiB |
| 2Gi | 1638 | 328 | 1310MiB |
| 4Gi | 3276 | 655 | 2621MiB |
High Availability (HA) Configuration
The key strategies for Gateway Collector high availability are as follows.
- At least 3 Replicas: set the Deployment's replicas to at least 3.
- Pod Anti-Affinity: configure it so Gateway Pods do not pile up on the same node.
- PDB (PodDisruptionBudget): limit how many Pods can terminate at the same time.
- HPA: configure automatic scaling based on CPU and memory.
# PodDisruptionBudget configuration
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: otel-gateway-pdb
namespace: monitoring
spec:
minAvailable: 2
selector:
matchLabels:
app: otel-collector-gateway
Troubleshooting Guide
Common Problems and How to Fix Them
1. Data is not reaching the backend
# Step 1: check the Collector logs
kubectl logs -n monitoring deploy/otel-collector-gateway --tail=100
# Step 2: check pipeline state in zPages
kubectl port-forward -n monitoring svc/otel-collector-gateway 55679:55679
# Open http://localhost:55679/debug/tracez in a browser
# Step 3: temporarily add the debug exporter to observe the data flow
# after adding debug: { verbosity: detailed } under exporters
# add the debug exporter to the pipeline
2. Collector memory usage spikes
- Check that
memory_limiteris configured. - Check that
GOMEMLIMITis set appropriately. - Check that Tail Sampling's
num_tracesvalue is not too large. - Check that the Exporter's
sending_queue.queue_sizeis not excessive.
3. Exporter queue saturation
# Check queue state with the Collector's internal metrics
curl http://localhost:8888/metrics | grep otelcol_exporter_queue
# otelcol_exporter_queue_size: current queue size
# otelcol_exporter_queue_capacity: maximum queue capacity
# when queue_size approaches capacity, the queue is saturated
Cause: the backend cannot process data fast enough, or too much traffic is flowing into the Collector.
Solution: raise num_consumers, check the backend's processing capacity, or drop unnecessary data early with a filter processor.
4. Processor order mistakes
The wrong processor order produces subtle problems.
| Mistake | Symptom | Correct order |
|---|---|---|
| batch placed before memory_limiter | Memory spikes, then OOM | memory_limiter always first |
| attributes placed after batch | Attribute changes are not applied | attributes before batch |
| tail_sampling placed on a DaemonSet | Incomplete sampling decisions | use tail_sampling on the Gateway only |
| resourcedetection placed last | Resource attributes are missing | place resourcedetection near the front |
Key Monitoring Metrics
To monitor the Collector's own operational health, track the following internal metrics.
| Metric | Description | Alert criterion |
|---|---|---|
otelcol_receiver_accepted_spans | Spans received successfully | Alert on a sharp drop |
otelcol_receiver_refused_spans | Spans refused on receipt | Alert if a value other than 0 persists |
otelcol_exporter_sent_spans | Spans sent successfully | Alert on a large gap versus accepted |
otelcol_exporter_send_failed_spans | Spans that failed to send | Alert if a value other than 0 persists |
otelcol_exporter_queue_size | Current queue size | Alert above 80% of capacity |
otelcol_processor_dropped_spans | Spans dropped by a processor | Alert when above expectations |
Failure Cases and Recovery Procedures
Case 1: Total Gateway Failure (All Replicas Down)
Symptom: all telemetry data is lost. The Agent's Exporter queue saturates and data starts dropping.
Cause: deploying a bad configuration change, a cascade of OOMs, a Node failure, and so on.
Recovery procedure:
- Roll back immediately to the last known-good ConfigMap.
- Check that the Gateway Deployment has enough replicas.
- Verify that Pod Anti-Affinity is configured.
- Check the Agent-side sending_queue and retry_on_failure settings. Data sitting in the queue is resent automatically once the Gateway recovers.
- To prevent a recurrence, configure a PDB and apply Canary deployment for ConfigMap changes.
# ConfigMap rollback
kubectl rollout undo configmap/otel-gateway-config -n monitoring
# Or apply the previous version of the ConfigMap directly
kubectl apply -f otel-gateway-config-backup.yaml
# Restart the Gateway Deployment
kubectl rollout restart deployment/otel-collector-gateway -n monitoring
# Check recovery status
kubectl rollout status deployment/otel-collector-gateway -n monitoring
Case 2: Data Loss Caused by a Backend Failure
Symptom: Exporter queue saturation, and a sharp rise in otelcol_exporter_send_failed_spans.
Cause: the backend (Tempo, Jaeger, and so on) went down or became slow to respond.
Recovery procedure:
- Check the backend's state first and recover it.
- The Collector retries automatically thanks to the retry_on_failure setting.
- Data dropped because of queue saturation cannot be recovered, so in the longer term increase the queue size or introduce a buffer layer such as Kafka.
Case 3: OOM Caused by an Unset memory_limiter
Symptom: the Collector Pod repeatedly restarts in an OOMKilled state.
Cause: the memory_limiter processor is not configured, or is configured with unsuitable values.
Recovery procedure:
- Add memory_limiter as the first processor of every pipeline.
- Set limit_mib to 80% of the container memory limit.
- Set the GOMEMLIMIT environment variable.
- Review whether the container resource limit is too small for the workload.
Operations Checklist
Initial Deployment Checklist
- Is the memory_limiter processor configured as the first processor in every pipeline?
- Is the GOMEMLIMIT environment variable set?
- Is the health_check Extension enabled?
- Are livenessProbe and readinessProbe configured?
- Are sending_queue and retry_on_failure enabled on the Exporter?
- Are the container resource requests/limits set appropriately?
- Is a PodDisruptionBudget configured?
Processor Order Checklist
- Is memory_limiter the first processor?
- Does resourcedetection come after memory_limiter (when used)?
- Do data transformation processors such as filter and attributes come before batch?
- Is tail_sampling used only on the Gateway (when used)?
- Is batch the last processor?
Monitoring Checklist
- Are the Collector's internal metrics (port 8888) being collected by Prometheus?
- Is an Exporter queue saturation alert configured?
- Is an alert on the refused metric configured?
- Is an alert on the send_failed metric configured?
- Are alerts on the Collector Pod's memory and CPU usage configured?
Security Checklist
- Are unnecessary ports kept from being exposed externally?
- Is TLS applied on the communication paths that need it?
- Are sensitive values such as API Keys managed as environment variables or Secrets?
- Are sensitive attributes (DB queries, authentication tokens) deleted with the attributes processor?
Conclusion
The OpenTelemetry Collector is core infrastructure for a modern observability pipeline. Thanks to its vendor-neutral design you can integrate flexibly with any backend - Jaeger, Grafana Tempo, Datadog, and so on - and swapping backends requires no change to application code.
The three most important things for stable operation can be summarized as follows.
- Set memory_limiter as the very first processor and configure GOMEMLIMIT appropriately.
- Deploy as a DaemonSet (Agent) + Deployment (Gateway) tiered structure to separate local collection from centralized processing.
- Enable the Exporter's sending_queue and retry_on_failure to gain resilience against temporary backend failures.
The Collector keeps evolving, and new features continue to arrive, such as remote configuration management through OpAMP (Open Agent Management Protocol). It is worth checking the official documentation and release notes periodically and keeping up with the latest changes.
References
- OpenTelemetry Collector Official Documentation
- OpenTelemetry Collector Configuration
- OpenTelemetry Collector Architecture
- OpenTelemetry Sampling Concepts
- OpenTelemetry Collector Scaling Guide
- OpenTelemetry Collector Troubleshooting
- OpenTelemetry Kubernetes Helm Chart
- Tail Sampling Processor (GitHub)
- Memory Limiter Processor (GitHub)
- Batch Processor (GitHub)