Five Decisive Strategies for Opening Up the Black Box of Microservices with OpenTelemetry

• 1. Introduction: Why Is Our Distributed System Still a Labyrinth?
  • 1.1 The Microservices Paradox: We Split Things Apart and Got More Complexity
  • 1.2 The Real Cost of Vendor Lock-In
  • 1.3 OpenTelemetry: "Instrument Once, Export Anywhere"
• 2. [Takeaway 1] W3C Baggage: The Secret Weapon for Propagating Business Context
  • 2.1 Distinguishing Trace Context from Baggage
  • 2.2 Business Use Cases: Why Do You Need Baggage?
  • 2.3 Implementation Examples in Several Languages
    • Java (Spring Boot + OTel SDK)
    • Python (FastAPI + OTel SDK)
    • Go (Gin + OTel SDK)
  • 2.4 Security and Performance Considerations for Baggage
    • Security Risks
    • Performance Overhead
• 3. [Takeaway 2] The Limits of Automatic Instrumentation and the Need for a Hybrid Strategy
  • 3.1 What Zero-Code Instrumentation Covers
  • 3.2 Level of Automatic Instrumentation Support by Language
  • 3.3 What Automatic Instrumentation Misses: The Business Logic Black Box
  • 3.4 A Hybrid Strategy: Combining Automatic and Manual Instrumentation
    • Java: Automatic Instrumentation Plus Manual Spans
    • Python: Automatic Instrumentation Plus Manual Spans
  • 3.5 Guidelines for Applying the Hybrid Strategy
• 4. [Takeaway 3] Tail-Based Sampling: The Intelligent Filter That Picks Real Problems Out of the Flood
  • 4.1 Why Do You Need Sampling?
  • 4.2 Head-Based vs Tail-Based Sampling
  • 4.3 The Key Precondition for Tail-Based Sampling: Trace-ID-Based Load Balancing
  • 4.4 OTel Collector Configuration: Tier 1 (Agent)
  • 4.5 OTel Collector Configuration: Tier 2 (Gateway with Tail Sampling)
  • 4.6 A Memory Sizing Formula for Tail Sampling
  • 4.7 A Guide to Choosing a Sampling Strategy
• 5. [Takeaway 4] Semantic Conventions: The Value of Standardized Data and Collaboration
  • 5.1 The Chaos of a World Without Standards
  • 5.2 The Structure of Semantic Conventions
    • Resource Attributes
    • HTTP Attributes (Span Attributes)
    • Database Attributes
  • 5.3 The Practical Effect of Standardization: Before vs After
  • 5.4 Cross-Signal Correlation: Unifying Logs, Metrics, and Traces
  • 5.5 Collector Configuration for Applying Semantic Conventions
  • 5.6 A Naming Guide for Custom Business Attributes
• 6. [Takeaway 5] Choosing an OTLP Transport: gRPC vs HTTP
  • 6.1 What Is OTLP (OpenTelemetry Protocol)?
  • 6.2 Performance Comparison
  • 6.3 gRPC's Advantages and Where It Fits
  • 6.4 HTTP's Advantages and Where It Fits
  • 6.5 A Decision Tree for Choosing a Transport
  • 6.6 A Hybrid Setup: Receiving gRPC and HTTP at Once
• 7. OTel Collector Deployment Topologies: Agent vs Gateway
  • 7.1 An Overview of the Deployment Patterns
  • 7.2 A Detailed Comparison of the Deployment Patterns
  • 7.3 A Kubernetes DaemonSet Deployment Example
  • 7.4 A Gateway Deployment Example
  • 7.5 Automating with the OpenTelemetry Operator
• 8. Comparing Observability Backends
  • 8.1 Open Source vs Commercial Backends
  • 8.2 Criteria for Choosing a Backend
• 9. A Roadmap for Production Adoption
  • 9.1 A Phased Adoption Checklist
    • Phase 1: Laying the Foundation (2-4 weeks)
    • Phase 2: Rollout and Standardization (4-8 weeks)
    • Phase 3: Going Deeper (8-12 weeks)
    • Phase 4: Optimization and Operational Maturity (ongoing)
  • 9.2 A Complete Pipeline Configuration Example
• 10. Conclusion: Escaping the Observability Cartel
  • 10.1 How OpenTelemetry Changes the Rules of the Game
  • 10.2 The Practical Effect of Strategic Visibility
  • 10.3 Next Steps to Get Started
• References

1. Introduction: Why Is Our Distributed System Still a Labyrinth?

1.1 The Microservices Paradox: We Split Things Apart and Got More Complexity

Microservice architecture (MSA) displaced the monolith on the strength of appealing promises: independent deployment, technical heterogeneity, team autonomy. But in a reality where a single user request starts at the API Gateway and cascades through 10 to 30 or more services — authentication, product catalog, inventory, payment, notification — tracking down the root cause when something goes wrong feels like wandering a labyrinth.

The journey of a user request (a typical e-commerce flow)
============================================

[Client] ──▶ [API Gateway] ──▶ [Auth Service]
                  │
                  ├──▶ [Product Service] ──▶ [Search Engine]
                  │         │
                  │         └──▶ [Recommendation Service] ──▶ [ML Model]
                  │
                  ├──▶ [Cart Service] ──▶ [Redis Cache]
                  │
                  ├──▶ [Order Service] ──▶ [Inventory Service] ──▶ [Warehouse DB]
                  │         │
                  │         └──▶ [Payment Service] ──▶ [External PG API]
                  │
                  └──▶ [Notification Service] ──▶ [Email/SMS/Push]

A single "order complete" involves at least 12 services.
Want to find the cause of p99 latency? Where do you even start?

Traditional APM (Application Performance Monitoring) tools have solved this problem partly. But most commercial APMs depend on their own agent and a proprietary protocol. The Datadog Agent only sends data to Datadog; the New Relic Agent only sends data to New Relic. That is exactly what vendor lock-in means.

1.2 The Real Cost of Vendor Lock-In

Vendor lock-in goes beyond technical inconvenience and turns into real business cost.

1.3 OpenTelemetry: "Instrument Once, Export Anywhere"

OpenTelemetry (OTel) is the second most active project in the CNCF (Cloud Native Computing Foundation), after Kubernetes, and was born in 2019 from the merger of OpenTracing and OpenCensus. Its core philosophy is simple and revolutionary at the same time.

"Instrument once, and export anywhere you like."

OTel provides three things, broadly speaking.

1. A standard wire protocol (OTLP): the transport specification for telemetry data. It supports both gRPC and HTTP.
2. SDKs and APIs: instrumentation libraries available in every major language (Java, Python, Go, .NET, Node.js, Rust, C++, PHP, Ruby, and more).
3. The OpenTelemetry Collector: a vendor-neutral pipeline that receives, processes, and routes telemetry data.

An overview of the OpenTelemetry architecture
============================================

  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
  │  Service A  │  │  Service B  │  │  Service C  │
  │  (Java SDK) │  │ (Python SDK)│  │  (Go SDK)   │
  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘
         │ OTLP           │ OTLP           │ OTLP
         ▼                ▼                ▼
  ┌─────────────────────────────────────────────────┐
  │           OpenTelemetry Collector               │
  │  ┌──────────┐  ┌───────────┐  ┌──────────────┐ │
  │  │Receivers │→ │Processors │→ │  Exporters   │ │
  │  └──────────┘  └───────────┘  └──────────────┘ │
  └────────┬──────────────┬──────────────┬──────────┘
           │              │              │
           ▼              ▼              ▼
    ┌───────────┐  ┌───────────┐  ┌───────────┐
    │  Jaeger   │  │   Tempo   │  │  Datadog  │
    │ (Traces)  │  │ (Traces)  │  │ (All-in-1)│
    └───────────┘  └───────────┘  └───────────┘

Once you adopt OTel, switching vendors is finished by changing the exporter configuration in the Collector. You can move from Jaeger to Tempo, or from Datadog to Grafana Cloud, without touching a single line of code.

This article analyzes, from an architect's point of view, the five decisive strategies for using OpenTelemetry effectively in practice. It is not a "Getting Started" walkthrough; it aims for the depth you need to make production-level decisions.

2. [Takeaway 1] W3C Baggage: The Secret Weapon for Propagating Business Context

2.1 Distinguishing Trace Context from Baggage

In distributed tracing, context propagation fundamentally divides into two kinds.

Key point: Baggage works independently of Trace Context. Baggage can propagate even when tracing is disabled, and conversely you can leave Baggage unused even when tracing is on. That independence is what makes Baggage such a flexible tool.

2.2 Business Use Cases: Why Do You Need Baggage?

Baggage's real value is that it can carry business context across service boundaries, beyond the infrastructure level. Let us look at concrete uses.

Multi-tenant isolation (tenant ID propagation)

In a SaaS environment, propagating which tenant a request belongs to all the way downstream lets each service independently apply per-tenant rate limiting, data isolation, and resource allocation.

Feature flag propagation

A feature flag state decided in the frontend can be applied consistently across the entire backend service chain. In an A/B test, every service can know whether the user is in the treatment or the control group.

QoS (Quality of Service) based on user tier

You can give premium users' requests higher priority, set more generous timeouts for them, or apply a finer sampling rate.

Cost attribution

Attaching a cost-center tag to a request lets you track which department or project consumes how much infrastructure cost.

2.3 Implementation Examples in Several Languages

Java (Spring Boot + OTel SDK)

import io.opentelemetry.api.baggage.Baggage;
import io.opentelemetry.api.baggage.BaggageEntryMetadata;
import io.opentelemetry.context.Scope;
import io.opentelemetry.api.trace.Span;

// ---- Setting Baggage at the API Gateway (the request entry point) ----
@RestController
public class GatewayController {

    @PostMapping("/api/orders")
    public ResponseEntity<?> createOrder(
            @RequestHeader("X-Tenant-Id") String tenantId,
            @RequestHeader("X-User-Tier") String userTier,
            HttpServletRequest request) {

        // Set W3C Baggage - it propagates to every downstream service
        Baggage baggage = Baggage.builder()
            .put("tenantId", tenantId,
                 BaggageEntryMetadata.create("tenant context"))
            .put("userTier", userTier,
                 BaggageEntryMetadata.create("qos context"))
            .put("entryPoint", "order-api",
                 BaggageEntryMetadata.create("routing context"))
            .put("requestRegion", determineRegion(request),
                 BaggageEntryMetadata.create("geo context"))
            .build();

        // Attach the Baggage to the current Context
        try (Scope scope = baggage.makeCurrent()) {
            // Every downstream service called within this scope
            // automatically receives the baggage as an HTTP header
            return orderService.processOrder(request.getBody());
        }
    }
}

// ---- Reading Baggage in the downstream Order Service ----
@Service
public class OrderService {

    public void processOrder(OrderRequest order) {
        // Extract the business context from the propagated Baggage
        String tenantId = Baggage.current().getEntryValue("tenantId");
        String userTier = Baggage.current().getEntryValue("userTier");

        // Add it to the current Span as business attributes (for search/filtering)
        Span currentSpan = Span.current();
        currentSpan.setAttribute("business.tenant_id", tenantId);
        currentSpan.setAttribute("business.user_tier", userTier);

        // Branch on QoS according to the user tier
        if ("premium".equals(userTier)) {
            processWithPriority(order);
        } else {
            processNormally(order);
        }
    }
}

Python (FastAPI + OTel SDK)

from opentelemetry import baggage, trace, context
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.context.context import Context
from fastapi import FastAPI, Request, Header
from typing import Optional

app = FastAPI()
tracer = trace.get_tracer("order-service")

# ---- Setting Baggage at the API Gateway ----
@app.post("/api/orders")
async def create_order(
    request: Request,
    x_tenant_id: Optional[str] = Header(None),
    x_user_tier: Optional[str] = Header(None, alias="X-User-Tier"),
):
    # Set the Baggage
    ctx = baggage.set_baggage("tenantId", x_tenant_id or "unknown")
    ctx = baggage.set_baggage("userTier", x_user_tier or "standard", context=ctx)
    ctx = baggage.set_baggage("featureFlag", "new-checkout-v2", context=ctx)

    # Activate the Context so it propagates automatically to downstream calls
    token = context.attach(ctx)
    try:
        with tracer.start_as_current_span("process-order") as span:
            # Record the Baggage values as Span attributes too
            tenant = baggage.get_baggage("tenantId")
            tier = baggage.get_baggage("userTier")
            span.set_attribute("business.tenant_id", tenant)
            span.set_attribute("business.user_tier", tier)

            result = await order_processor.process(request)
            return {"status": "created", "order_id": result.id}
    finally:
        context.detach(token)


# ---- Reading Baggage in the downstream Inventory Service ----
@app.get("/api/inventory/{product_id}")
async def check_inventory(product_id: str, request: Request):
    # The OTel SDK extracts the Baggage from the HTTP headers automatically
    tenant_id = baggage.get_baggage("tenantId")
    user_tier = baggage.get_baggage("userTier")

    with tracer.start_as_current_span("check-inventory") as span:
        span.set_attribute("business.tenant_id", tenant_id)

        # Access the datasource isolated for this tenant
        inventory = await get_tenant_inventory(tenant_id, product_id)
        return {"available": inventory.quantity > 0}

Go (Gin + OTel SDK)

package main

import (
    "context"
    "net/http"

    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/baggage"
    "go.opentelemetry.io/otel/attribute"
)

var tracer = otel.Tracer("order-service")

// Setting Baggage at the API Gateway
func CreateOrderHandler(c *gin.Context) {
    tenantID := c.GetHeader("X-Tenant-Id")
    userTier := c.GetHeader("X-User-Tier")

    // Create W3C Baggage members
    tenantMember, _ := baggage.NewMember("tenantId", tenantID)
    tierMember, _ := baggage.NewMember("userTier", userTier)
    flagMember, _ := baggage.NewMember("featureFlag", "new-checkout-v2")

    bag, _ := baggage.New(tenantMember, tierMember, flagMember)

    // Attach the Baggage to the Context
    ctx := baggage.ContextWithBaggage(c.Request.Context(), bag)

    // Propagated automatically on downstream calls
    ctx, span := tracer.Start(ctx, "process-order")
    defer span.End()

    span.SetAttributes(
        attribute.String("business.tenant_id", tenantID),
        attribute.String("business.user_tier", userTier),
    )

    result, err := processOrder(ctx, c.Request.Body)
    if err != nil {
        span.RecordError(err)
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }
    c.JSON(http.StatusCreated, result)
}

// Reading Baggage downstream
func processOrder(ctx context.Context, body io.ReadCloser) (*OrderResult, error) {
    bag := baggage.FromContext(ctx)
    tenantID := bag.Member("tenantId").Value()
    userTier := bag.Member("userTier").Value()

    ctx, span := tracer.Start(ctx, "validate-order")
    defer span.End()

    span.SetAttributes(
        attribute.String("business.tenant_id", tenantID),
    )

    // Business logic goes here...
    return &OrderResult{ID: "ord-12345"}, nil
}

2.4 Security and Performance Considerations for Baggage

Baggage is powerful, but it calls for a careful approach on both security and performance.

Security Risks

Baggage's propagation path and the security risk
============================================

[Internal service] ──HTTP header──▶ [Internal service] ──HTTP header──▶ [External API]
     │                              │                             │
     │  baggage: tenantId=acme,     │  baggage: tenantId=acme,    │  ⚠️ Baggage
     │  userId=12345,               │  userId=12345,              │  leaks to the
     │  email=user@acme.com         │  email=user@acme.com        │  outside world!
     │                              │                             │
     └──────────────────────────────┘                             │
        Inside the trust boundary        Outside the trust boundary ───┘

⚠️ Baggage travels as a plaintext HTTP header!
⚠️ Never put PII (personally identifiable information) in it!

Mandatory security measures:

1. No PII: never put personally identifiable information such as email addresses, phone numbers, or national ID numbers into Baggage.
2. Trust boundary sanitization: strip or filter Baggage when calling an external service.
3. Allowlist-based propagation: configure a filter in the Collector or SDK so that only approved keys propagate.
4. Value size limits: the W3C Baggage spec recommends a total of 8,192 bytes, but in practice keep it as small as you can.

# An example of filtering Baggage in the OTel Collector
processors:
  # An attributes processor that allows only specific baggage keys
  attributes/baggage-filter:
    actions:
      # Keep only the approved business keys
      - key: baggage.tenantId
        action: upsert
      - key: baggage.userTier
        action: upsert
      # Delete the sensitive keys
      - key: baggage.email
        action: delete
      - key: baggage.userId
        action: delete

Performance Overhead

Because Baggage is included in the headers of every inter-service HTTP request, it adds network overhead.

Best practice: put only short identifiers (IDs) in Baggage and design services to look up the actual data by that ID. For example, propagate just tenantId=acme instead of a whole user profile, and let each service read the tenant configuration from a cache when it needs it.

3. [Takeaway 2] The Limits of Automatic Instrumentation and the Need for a Hybrid Strategy

3.1 What Zero-Code Instrumentation Covers

OpenTelemetry's automatic instrumentation adds observability to an application without changing a single line of code. The mechanism differs by language, but the core principle is the same: it intercepts the entry and exit points of frameworks and libraries and creates Spans automatically.

What automatic instrumentation captures:

• HTTP requests/responses: inbound (server) and outbound (client) HTTP calls
• Database queries: DB driver calls through JDBC, SQLAlchemy, database/sql, and the like
• Message queues: produce/consume operations on Kafka, RabbitMQ, SQS, and others
• gRPC calls: unary and streaming RPC
• Cache access such as Redis and Memcached
• Framework-internal routing: Spring MVC, Express, Django, and so on

That alone can cover 80-90% of the infrastructure layer. It gets you most latency problems, error-rate spikes, and an understanding of inter-service dependencies.

3.2 Level of Automatic Instrumentation Support by Language

3.3 What Automatic Instrumentation Misses: The Business Logic Black Box

The fundamental limit of automatic instrumentation is that it cannot see inside your business logic. Consider the following scenario.

What automatic instrumentation alone shows you vs. what you actually need to know
============================================

The Span that automatic instrumentation captures:
  [POST /api/orders] ──▶ [SELECT * FROM products] ──▶ [POST /payments]
       200 OK, 1.2s          50ms                       800ms

What you can see: "the order API took 1.2s, the DB took 50ms, payment took 800ms"
What you cannot see: where did the remaining 350ms go?

What actually happens inside that 350ms:
  ├── Inventory availability check logic (50ms)
  ├── Discount coupon rule engine (120ms)      ← this is the bottleneck!
  ├── Shipping cost calculation (30ms)
  ├── Fraud detection scoring (80ms)
  └── Order event serialization (70ms)

Automatic instrumentation shows those 350ms as one opaque lump
labeled "business logic".

3.4 A Hybrid Strategy: Combining Automatic and Manual Instrumentation

The best strategy is to cover the infrastructure layer with automatic instrumentation and instrument business-critical logic precisely by hand.

Java: Automatic Instrumentation Plus Manual Spans

# Step 1: automatic instrumentation (the JVM agent approach)
# Download the OTel Java Agent
curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

# Attach the agent when starting the JVM
java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=order-service \
  -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.resource.attributes=service.namespace=ecommerce,deployment.environment=production \
  -jar order-service.jar

// Step 2: add manual Spans to business-critical logic
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.common.Attributes;

@Service
public class OrderProcessingService {

    // Use the global Tracer initialized by the automatic instrumentation agent
    private static final Tracer tracer =
        GlobalOpenTelemetry.getTracer("order-processing", "1.0.0");

    public OrderResult processOrder(OrderRequest request) {
        // Automatic instrumentation: the HTTP entry Span already exists
        // Manual instrumentation: explicitly instrument the steps inside the business logic

        // Instrument the discount logic
        Span discountSpan = tracer.spanBuilder("apply-discount-rules")
            .setAttribute("business.coupon_code", request.getCouponCode())
            .setAttribute("business.original_amount", request.getTotalAmount())
            .startSpan();

        try (var scope = discountSpan.makeCurrent()) {
            DiscountResult discount = discountEngine.calculate(request);
            discountSpan.setAttribute("business.discount_amount", discount.getAmount());
            discountSpan.setAttribute("business.discount_type", discount.getType());
            discountSpan.setAttribute("business.rules_evaluated", discount.getRulesCount());
        } catch (Exception e) {
            discountSpan.setStatus(StatusCode.ERROR, e.getMessage());
            discountSpan.recordException(e);
            throw e;
        } finally {
            discountSpan.end();
        }

        // Instrument fraud detection
        Span fraudSpan = tracer.spanBuilder("fraud-detection")
            .setAttribute("business.user_id", request.getUserId())
            .setAttribute("business.order_amount", request.getTotalAmount())
            .startSpan();

        try (var scope = fraudSpan.makeCurrent()) {
            FraudScore score = fraudDetector.evaluate(request);
            fraudSpan.setAttribute("business.fraud_score", score.getValue());
            fraudSpan.setAttribute("business.fraud_decision", score.getDecision());

            if (score.getValue() > 0.8) {
                fraudSpan.addEvent("high-fraud-risk-detected",
                    Attributes.of(
                        AttributeKey.doubleKey("score"), score.getValue(),
                        AttributeKey.stringKey("reason"), score.getReason()
                    ));
            }
        } finally {
            fraudSpan.end();
        }

        // ...the rest of the business logic
    }
}

Python: Automatic Instrumentation Plus Manual Spans

# Step 1: install the automatic instrumentation packages
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install  # auto-installs instrumentation packages for detected libraries

# Step 2: run the application with automatic instrumentation
OTEL_SERVICE_NAME=order-service \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_RESOURCE_ATTRIBUTES=service.namespace=ecommerce,deployment.environment=production \
opentelemetry-instrument python -m uvicorn main:app --host 0.0.0.0 --port 8000

# Step 2: add manual Spans to the business logic
from opentelemetry import trace
from opentelemetry.trace import StatusCode

tracer = trace.get_tracer("order-processing", "1.0.0")

class OrderProcessor:
    async def process(self, order: OrderRequest) -> OrderResult:
        # Manually instrument the discount logic
        with tracer.start_as_current_span(
            "apply-discount-rules",
            attributes={
                "business.coupon_code": order.coupon_code,
                "business.original_amount": order.total_amount,
            }
        ) as discount_span:
            try:
                discount = await self.discount_engine.calculate(order)
                discount_span.set_attribute("business.discount_amount", discount.amount)
                discount_span.set_attribute("business.rules_evaluated", discount.rules_count)
            except DiscountError as e:
                discount_span.set_status(StatusCode.ERROR, str(e))
                discount_span.record_exception(e)
                raise

        # Manually instrument fraud detection
        with tracer.start_as_current_span("fraud-detection") as fraud_span:
            score = await self.fraud_detector.evaluate(order)
            fraud_span.set_attribute("business.fraud_score", score.value)
            fraud_span.set_attribute("business.fraud_decision", score.decision)

            if score.value > 0.8:
                fraud_span.add_event("high-fraud-risk", attributes={
                    "score": score.value,
                    "reason": score.reason,
                })

        return await self._finalize_order(order, discount)

3.5 Guidelines for Applying the Hybrid Strategy

Adding manual instrumentation everywhere turns into noise. What matters is having a criterion for deciding where to add manual Spans.

4. [Takeaway 3] Tail-Based Sampling: The Intelligent Filter That Picks Real Problems Out of the Flood

4.1 Why Do You Need Sampling?

What happens if you collect 100% of every Span of every request in a production microservice environment?

An example telemetry volume calculation
============================================

Number of services: 30
Average Spans per service: 5 per request
Requests per second (RPS): 10,000
Average size per Span: 1 KB

Spans per second: 30 x 5 x 10,000 = 1,500,000 spans/sec
Data per second: 1,500,000 x 1 KB = 1.5 GB/sec
Data per day:    1.5 GB x 86,400 = ~130 TB/day

Annual storage cost (on S3): ~$35,000/month = ~$420,000/year
Annual Datadog cost (by ingestion): tens of times higher

Collecting 100% is unrealistic. Sampling is mandatory. The question is how to sample.

4.2 Head-Based vs Tail-Based Sampling

The core trade-off: head-based is light but blind; tail-based is intelligent but heavy.

4.3 The Key Precondition for Tail-Based Sampling: Trace-ID-Based Load Balancing

For tail-based sampling to work correctly, every Span that makes up a single trace has to arrive at the same Collector instance. Only then can the Collector see the whole trace and make a sampling decision.

That calls for a 2-tier Collector architecture.

A 2-tier Collector architecture for tail-based sampling
============================================

  [Service A]   [Service B]   [Service C]   [Service D]
       │              │              │              │
       │  OTLP        │  OTLP        │  OTLP        │  OTLP
       ▼              ▼              ▼              ▼
  ┌──────────────────────────────────────────────────────┐
  │              Tier 1: Agent Collectors                │
  │            (DaemonSet, one per node)                  │
  │                                                      │
  │  ┌──────────┐  ┌──────────┐  ┌──────────┐           │
  │  │ Agent 1  │  │ Agent 2  │  │ Agent 3  │           │
  │  │ (Node 1) │  │ (Node 2) │  │ (Node 3) │           │
  │  └────┬─────┘  └────┬─────┘  └────┬─────┘           │
  │       │              │              │                 │
  │       │   Load Balancing Exporter   │                 │
  │       │  (hashed on the Trace ID)     │                 │
  └───────┼──────────────┼──────────────┼─────────────────┘
          │              │              │
          ▼              ▼              ▼
  ┌──────────────────────────────────────────────────────┐
  │           Tier 2: Gateway Collectors                 │
  │       (Deployment/StatefulSet, scales horizontally)   │
  │                                                      │
  │  ┌──────────────┐  ┌──────────────┐                  │
  │  │  Gateway 1   │  │  Gateway 2   │                  │
  │  │              │  │              │                  │
  │  │ Trace ID     │  │ Trace ID     │                  │
  │  │ a1xx → here  │  │ b2xx → here  │                  │
  │  │              │  │              │                  │
  │  │ tail_sampling │  │ tail_sampling │                  │
  │  │ processor    │  │ processor    │                  │
  │  └──────┬───────┘  └──────┬───────┘                  │
  └─────────┼──────────────────┼──────────────────────────┘
            │                  │
            ▼                  ▼
      ┌───────────┐     ┌───────────┐
      │  Backend  │     │  Backend  │
      │ (Tempo)   │     │ (Jaeger)  │
      └───────────┘     └───────────┘

4.4 OTel Collector Configuration: Tier 1 (Agent)

# otel-collector-agent.yaml
# Tier 1: the Agent Collector, deployed as a DaemonSet on every node

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # A limiter to protect memory (mandatory!)
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  # Add base attributes (node information and so on)
  resource:
    attributes:
      - key: k8s.node.name
        value: '${K8S_NODE_NAME}'
        action: upsert
      - key: deployment.environment
        value: 'production'
        action: upsert

  # Batching (for network efficiency)
  batch:
    send_batch_size: 1024
    send_batch_max_size: 2048
    timeout: 5s

exporters:
  # Trace-ID-based load balancing → forwards to the Tier 2 Gateway
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-gateway-headless.observability.svc.cluster.local
        port: 4317

  # Metrics and Logs go straight to the backend (no sampling needed)
  otlp/metrics:
    endpoint: mimir.observability.svc.cluster.local:4317
    tls:
      insecure: true

  otlp/logs:
    endpoint: loki.observability.svc.cluster.local:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [loadbalancing] # Traces are routed to the Gateway

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/metrics] # Metrics are sent directly

    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/logs] # Logs are sent directly

4.5 OTel Collector Configuration: Tier 2 (Gateway with Tail Sampling)

# otel-collector-gateway.yaml
# Tier 2: the Gateway Collector that performs tail-based sampling

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4096 # The Gateway needs more memory
    spike_limit_mib: 1024

  # ⚠️ Important: do not use batch before tail_sampling!
  # batch can split Spans belonging to the same trace.

  # Tail-based sampling policies
  tail_sampling:
    # How long to wait for a trace to complete
    # The maximum expected trace duration in your system + a network margin
    decision_wait: 30s
    # A grace period for waiting on additional Spans after the decision
    num_traces: 100000 # Maximum number of traces tracked concurrently
    expected_new_traces_per_sec: 1000

    policies:
      # Policy 1: keep 100% of traces that contain an error (highest priority)
      - name: errors-always-keep
        type: status_code
        status_code:
          status_codes:
            - ERROR

      # Policy 2: keep high-latency traces (p99 and above)
      - name: high-latency
        type: latency
        latency:
          threshold_ms: 5000 # Traces that took 5 seconds or more

      # Policy 3: always keep traces from particular services (critical services)
      - name: critical-services
        type: string_attribute
        string_attribute:
          key: service.name
          values:
            - payment-service
            - order-service
          enabled_regex_matching: false

      # Policy 4: keep a higher proportion of premium users' traces
      - name: premium-users
        type: and
        and:
          and_sub_policy:
            - name: is-premium
              type: string_attribute
              string_attribute:
                key: business.user_tier
                values: ['premium', 'enterprise']
            - name: premium-rate
              type: probabilistic
              probabilistic:
                sampling_percentage: 50 # keep 50%

      # Policy 5: keep only 5% of the remaining healthy traces (cost optimization)
      - name: baseline-probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

  # Apply batch after tail_sampling
  batch:
    send_batch_size: 2048
    timeout: 10s

exporters:
  otlp/tempo:
    endpoint: tempo.observability.svc.cluster.local:4317
    tls:
      insecure: true

  otlp/jaeger:
    endpoint: jaeger-collector.observability.svc.cluster.local:4317
    tls:
      insecure: true

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888 # Monitors the Collector's own metrics
    logs:
      level: info

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]

4.6 A Memory Sizing Formula for Tail Sampling

Because tail sampling has to hold every Span in memory until it makes a decision, sizing that memory properly matters.

A formula for calculating the memory requirement
============================================

Required Memory (GB) =
  traces_per_second
  x decision_wait_seconds
  x avg_spans_per_trace
  x bytes_per_span
  / 1,000,000,000
  x safety_factor

An example calculation:
  traces_per_second     = 1,000
  decision_wait_seconds = 30
  avg_spans_per_trace   = 15
  bytes_per_span        = 1,000 (1 KB)
  safety_factor         = 2.0

  = 1,000 x 30 x 15 x 1,000 / 1,000,000,000 x 2.0
  = 450,000,000 / 1,000,000,000 x 2.0
  = 0.45 x 2.0
  = 0.9 GB

→ Allocate at least 1 GB; 2 GB recommended

4.7 A Guide to Choosing a Sampling Strategy

5. [Takeaway 4] Semantic Conventions: The Value of Standardized Data and Collaboration

5.1 The Chaos of a World Without Standards

What happens when 20 teams each run their own microservice and each team names its telemetry attributes however it likes?

Attributes named freely by each team, with no standard
============================================

Team A (order service):        user_id="12345", status_code=200, method="POST"
Team B (payment service):      userId="12345",  httpStatus=200,  httpMethod="POST"
Team C (inventory service):    uid="12345",     response_code=200, req_method="POST"
Team D (notification service): customer_id="12345", http_code=200, verb="POST"
Team E (search service):       user="12345",    code=200,        http.method="POST"

The same request from the same user, described with five different attribute names.
→ No correlation across services!
→ No unified dashboard!
→ No automated alert rules!

OpenTelemetry Semantic Conventions are what solve this problem.

5.2 The Structure of Semantic Conventions

OpenTelemetry Semantic Conventions (currently v1.40.0) define standard attribute names and their meaning for telemetry data. Let us look at the standard attributes in the main areas.

Resource Attributes

These are the metadata that identify the service itself.

HTTP Attributes (Span Attributes)

These are the standard attributes for HTTP requests and responses.

Database Attributes

5.3 The Practical Effect of Standardization: Before vs After

Before (no standard):
  Different attribute names per team → no unified query
  ─────────────────────────────────────
  How do you build a "5xx error rate across all services" dashboard in Grafana?

  Panel 1 (orders):    rate({status_code=~"5.."})
  Panel 2 (payment):   rate({httpStatus=~"5.."})
  Panel 3 (inventory): rate({response_code=~"5.."})
  → A separate query per service. Every new service means editing the dashboard.


After (Semantic Conventions applied):
  Every team uses the same attribute names → one query covers everything
  ─────────────────────────────────────
  The "5xx error rate across all services" dashboard in Grafana:

  One query: rate({http.response.status_code=~"5.."}) by (service.name)
  → Every service is included automatically. No change when a new service appears.

5.4 Cross-Signal Correlation: Unifying Logs, Metrics, and Traces

Another powerful benefit of Semantic Conventions is cross-signal correlation. When you use the same attribute names, you can confirm an anomaly found in a trace against a metric and pull up the related logs immediately.

The cross-signal correlation workflow
============================================

1. An alert fires:
   metric: http_server_request_duration_seconds{service.name="order-service"} > 5s

2. Trace the cause in a trace:
   trace: service.name="order-service"
          AND http.response.status_code >= 500
          → In the Span, confirm db.operation.name="SELECT",
            db.collection.name="orders"

3. Look up the related logs:
   log: service.name="order-service"
        AND trace_id="abc123"
        → Find the "Connection pool exhausted" error log

Because every signal uses the same attribute names — service.name, trace_id, and so on —
this workflow connects together naturally.

5.5 Collector Configuration for Applying Semantic Conventions

If your teams already use different attribute names, you can normalize them centrally with the Collector's attributes processor.

# Normalize attribute names to the Semantic Conventions in the Collector
processors:
  # Convert legacy attribute names to the standard ones
  attributes/normalize:
    actions:
      # Normalize HTTP attributes
      - key: http.method
        action: upsert
        from_attribute: httpMethod # team B's attribute name
      - key: http.method
        action: upsert
        from_attribute: req_method # team C's attribute name
      - key: http.method
        action: upsert
        from_attribute: verb # team D's attribute name
      # Delete the legacy keys
      - key: httpMethod
        action: delete
      - key: req_method
        action: delete
      - key: verb
        action: delete

      # Normalize the user ID
      - key: enduser.id
        action: upsert
        from_attribute: user_id
      - key: enduser.id
        action: upsert
        from_attribute: userId
      - key: enduser.id
        action: upsert
        from_attribute: uid
      - key: enduser.id
        action: upsert
        from_attribute: customer_id
      # Delete the legacy keys
      - key: user_id
        action: delete
      - key: userId
        action: delete
      - key: uid
        action: delete
      - key: customer_id
        action: delete

      # Normalize the HTTP status code
      - key: http.response.status_code
        action: upsert
        from_attribute: status_code
      - key: http.response.status_code
        action: upsert
        from_attribute: httpStatus
      - key: http.response.status_code
        action: upsert
        from_attribute: response_code
      - key: http.response.status_code
        action: upsert
        from_attribute: http_code

  # Add defaults when a required resource attribute is missing
  resource:
    attributes:
      - key: service.namespace
        value: 'default'
        action: insert # does not overwrite if it already exists
      - key: deployment.environment.name
        value: 'production'
        action: insert

5.6 A Naming Guide for Custom Business Attributes

When adding business attributes that the Semantic Conventions do not define, follow a consistent naming rule.

6. [Takeaway 5] Choosing an OTLP Transport: gRPC vs HTTP

6.1 What Is OTLP (OpenTelemetry Protocol)?

OTLP is the standard protocol for transporting telemetry data defined by OpenTelemetry. As of OTLP spec 1.9.0, it can carry Traces, Metrics, and Logs (plus the recently added Profiles) over a single protocol. OTLP supports three transport variants.

6.2 Performance Comparison

6.3 gRPC's Advantages and Where It Fits

When gRPC is the better choice:

• High-throughput environments: services generating more than 10,000 spans per second
• Internal service-to-service communication: trusted networks such as the inside of a Kubernetes cluster
• Infrastructure that supports HTTP/2: service meshes, internal load balancers
• Bidirectional streaming: when you need continuous telemetry streaming
• When saving bandwidth matters: binary encoding plus HPACK compression

# An example gRPC exporter configuration (SDK)
# The environment variable approach
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_COMPRESSION=gzip
OTEL_EXPORTER_OTLP_TIMEOUT=10000

# Collector receiver configuration (gRPC)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16 # Maximum message size to receive
        max_concurrent_streams: 100 # Number of concurrent streams
        keepalive:
          server_parameters:
            max_connection_idle: 60s
            max_connection_age: 300s
            time: 30s
            timeout: 10s
        tls:
          cert_file: /certs/server.crt
          key_file: /certs/server.key
          client_ca_file: /certs/ca.crt # mTLS

6.4 HTTP's Advantages and Where It Fits

When HTTP is the better choice:

• Firewall/proxy compatibility: when a corporate network only allows HTTP/1.1
• Browser environments: sending telemetry directly from a web frontend (CORS is supported)
• Serverless/edge: environments where gRPC is hard to support, such as AWS Lambda and Cloudflare Workers
• Debugging: sending in JSON keeps the payload human-readable
• General-purpose infrastructure: compatible with HTTP load balancers, CDNs, and reverse proxies

# An example HTTP exporter configuration (SDK)
# The environment variable approach (Protobuf)
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_COMPRESSION=gzip
OTEL_EXPORTER_OTLP_TIMEOUT=10000

# HTTP JSON (for debugging)
OTEL_EXPORTER_OTLP_PROTOCOL=http/json
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

# Collector receiver configuration (HTTP)
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - 'https://*.example.com' # Allow browser CORS
          allowed_headers:
            - 'Content-Type'
            - 'X-Custom-Header'
          max_age: 7200
        tls:
          cert_file: /certs/server.crt
          key_file: /certs/server.key

6.5 A Decision Tree for Choosing a Transport

A decision tree for choosing an OTLP transport
============================================

Start: you have to choose a telemetry transport

Q1. Is this a browser or serverless environment?
  ├── YES → use HTTP/Protobuf
  │         (gRPC is not supported in browsers or Lambda)
  │
  └── NO ──▶ Q2. Does a firewall block HTTP/2 or gRPC?
                ├── YES → use HTTP/Protobuf
                │         (it can fall back to HTTP/1.1)
                │
                └── NO ──▶ Q3. More than 10,000 spans per second?
                              ├── YES → use gRPC
                              │         (HTTP/2 multiplexing, best performance)
                              │
                              └── NO ──▶ Q4. Is debugging the main purpose?
                                            ├── YES → use HTTP/JSON
                                            │         (human-readable)
                                            │
                                            └── NO → use HTTP/Protobuf
                                                      (the most universal, safe choice)

6.6 A Hybrid Setup: Receiving gRPC and HTTP at Once

In practice it is common to configure the Collector to receive gRPC and HTTP simultaneously, so that it accommodates every kind of client.

# A hybrid receiver configuration (gRPC and HTTP at the same time)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
        keepalive:
          server_parameters:
            max_connection_idle: 60s
            time: 30s
            timeout: 10s
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins: ['*']
          allowed_headers: ['*']
# Recommended configuration by service type
# ┌─────────────────────────┬──────────────────────────┐
# │ Service type             │ Recommended transport      │
# ├─────────────────────────┼──────────────────────────┤
# │ Backend (Java, Go)       │ gRPC (port 4317)           │
# │ Frontend (browser JS)    │ HTTP/Protobuf (port 4318)  │
# │ Serverless (Lambda)      │ HTTP/Protobuf (port 4318)  │
# │ Debugging/testing        │ HTTP/JSON (port 4318)      │
# │ IoT/Edge                 │ HTTP/Protobuf (port 4318)  │
# └─────────────────────────┴──────────────────────────┘

7. OTel Collector Deployment Topologies: Agent vs Gateway

7.1 An Overview of the Deployment Patterns

The OpenTelemetry Collector supports three core deployment patterns, each with its own trade-offs and its own best fit.

A comparison of OTel Collector deployment patterns
============================================

Pattern 1: Agent (DaemonSet)
  ┌─────────────────────────────────┐
  │          Kubernetes Node        │
  │  ┌─────────┐  ┌─────────┐      │
  │  │Service A│  │Service B│      │
  │  └────┬────┘  └────┬────┘      │
  │       │ localhost   │          │
  │       ▼             ▼          │
  │  ┌──────────────────────┐      │
  │  │   OTel Collector     │      │
  │  │   (DaemonSet Pod)    │      │
  │  └──────────┬───────────┘      │
  └─────────────┼──────────────────┘
                ▼
          ┌──────────┐
          │ Backend  │
          └──────────┘

Pattern 2: Sidecar
  ┌──────────────────────────────┐
  │      Application Pod         │
  │  ┌──────────┐ ┌───────────┐  │
  │  │App       │ │OTel       │  │
  │  │Container │→│Collector  │  │
  │  │          │ │(Sidecar)  │  │
  │  └──────────┘ └─────┬─────┘  │
  └─────────────────────┼────────┘
                        ▼
                  ┌──────────┐
                  │ Backend  │
                  └──────────┘

Pattern 3: Gateway (Deployment)
  [Service A]  [Service B]  [Service C]
       │            │            │
       └────────────┼────────────┘
                    ▼
          ┌──────────────────┐
          │  OTel Collector  │
          │  (Deployment,    │
          │   replicas: 3+)  │
          │  + HPA           │
          └────────┬─────────┘
                   ▼
             ┌──────────┐
             │ Backend  │
             └──────────┘

7.2 A Detailed Comparison of the Deployment Patterns

7.3 A Kubernetes DaemonSet Deployment Example

# otel-collector-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector-agent
  namespace: observability
  labels:
    app: otel-collector
    component: agent
spec:
  selector:
    matchLabels:
      app: otel-collector
      component: agent
  template:
    metadata:
      labels:
        app: otel-collector
        component: agent
    spec:
      serviceAccountName: otel-collector
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.120.0
          args:
            - '--config=/conf/otel-collector-config.yaml'
          ports:
            - containerPort: 4317 # gRPC
              hostPort: 4317
              protocol: TCP
            - containerPort: 4318 # HTTP
              hostPort: 4318
              protocol: TCP
            - containerPort: 8888 # Prometheus metrics
              protocol: TCP
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: K8S_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: GOMEMLIMIT
              value: '460MiB' # Go runtime memory limit
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: 1000m
              memory: 512Mi
          volumeMounts:
            - name: config
              mountPath: /conf
          livenessProbe:
            httpGet:
              path: /
              port: 13133 # health_check extension
            initialDelaySeconds: 15
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /
              port: 13133
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: config
          configMap:
            name: otel-agent-config

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-agent-config
  namespace: observability
data:
  otel-collector-config.yaml: |
    extensions:
      health_check:
        endpoint: 0.0.0.0:13133

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

      # Collect node-level metrics (host metrics)
      hostmetrics:
        collection_interval: 30s
        scrapers:
          cpu: {}
          memory: {}
          disk: {}
          network: {}

      # Collect kubelet metrics
      kubeletstats:
        collection_interval: 30s
        auth_type: "serviceAccount"
        endpoint: "https://${K8S_NODE_NAME}:10250"
        insecure_skip_verify: true

    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 400
        spike_limit_mib: 100

      batch:
        send_batch_size: 1024
        timeout: 5s

      resource:
        attributes:
          - key: k8s.node.name
            value: "${K8S_NODE_NAME}"
            action: upsert

      # Add Kubernetes metadata automatically
      k8sattributes:
        auth_type: "serviceAccount"
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.node.name
          labels:
            - tag_name: app.label.team
              key: team
              from: pod
          annotations:
            - tag_name: app.annotation.version
              key: app-version
              from: pod

    exporters:
      # Traces → Gateway (for tail sampling)
      loadbalancing:
        protocol:
          otlp:
            tls:
              insecure: true
        resolver:
          dns:
            hostname: otel-gateway-headless.observability.svc.cluster.local
            port: 4317

      # Metrics → sent directly to Prometheus/Mimir
      prometheusremotewrite:
        endpoint: http://mimir.observability.svc.cluster.local:9009/api/v1/push

      # Logs → sent directly to Loki
      otlp/logs:
        endpoint: loki.observability.svc.cluster.local:4317
        tls:
          insecure: true

    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, resource, batch]
          exporters: [loadbalancing]
        metrics:
          receivers: [otlp, hostmetrics, kubeletstats]
          processors: [memory_limiter, k8sattributes, resource, batch]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, resource, batch]
          exporters: [otlp/logs]

7.4 A Gateway Deployment Example

# otel-collector-gateway.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector-gateway
  namespace: observability
spec:
  replicas: 3
  selector:
    matchLabels:
      app: otel-collector
      component: gateway
  template:
    metadata:
      labels:
        app: otel-collector
        component: gateway
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.120.0
          args:
            - '--config=/conf/otel-collector-config.yaml'
          ports:
            - containerPort: 4317
              protocol: TCP
          env:
            - name: GOMEMLIMIT
              value: '3600MiB'
          resources:
            requests:
              cpu: 1000m
              memory: 2Gi
            limits:
              cpu: 4000m
              memory: 4Gi
          volumeMounts:
            - name: config
              mountPath: /conf
      volumes:
        - name: config
          configMap:
            name: otel-gateway-config

---
# A headless Service (for the load balancing exporter's DNS resolver)
apiVersion: v1
kind: Service
metadata:
  name: otel-gateway-headless
  namespace: observability
spec:
  clusterIP: None
  selector:
    app: otel-collector
    component: gateway
  ports:
    - port: 4317
      targetPort: 4317
      protocol: TCP

---
# HPA (horizontal autoscaling)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: otel-gateway-hpa
  namespace: observability
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: otel-collector-gateway
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 75

7.5 Automating with the OpenTelemetry Operator

In a Kubernetes environment, the OpenTelemetry Operator lets you manage Collector deployment and automatic instrumentation injection declaratively.

# Declarative deployment through the OpenTelemetryCollector CRD
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-agent
  namespace: observability
spec:
  mode: daemonset # daemonset | deployment | sidecar | statefulset
  image: otel/opentelemetry-collector-contrib:0.120.0
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 1000m
      memory: 512Mi
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 400
      batch:
        send_batch_size: 1024
        timeout: 5s
    exporters:
      otlp:
        endpoint: otel-gateway.observability.svc.cluster.local:4317
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [otlp]

---
# Automatic instrumentation injection (installs the OTel Agent into Java applications automatically)
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
  namespace: production
spec:
  exporter:
    endpoint: http://otel-agent.observability.svc.cluster.local:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: '0.25' # Head-based 25% (tail sampling filters further)
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
    env:
      - name: OTEL_JAVAAGENT_DEBUG
        value: 'false'

---
# Adding this annotation to a Pod/Deployment enables automatic instrumentation
# metadata:
#   annotations:
#     instrumentation.opentelemetry.io/inject-java: "java-instrumentation"

8. Comparing Observability Backends

Choosing the right backend decides whether an observability strategy succeeds or fails. Thanks to OpenTelemetry's vendor neutrality, you are free to choose a backend and to replace it later.

8.1 Open Source vs Commercial Backends

8.2 Criteria for Choosing a Backend

9. A Roadmap for Production Adoption

9.1 A Phased Adoption Checklist

Adopting OpenTelemetry is not about applying everything at once; it should proceed in phases and incrementally.

Phase 1: Laying the Foundation (2-4 weeks)

Checklist
============================================
[ ] Deploy the OTel Collector as a Kubernetes DaemonSet
[ ] Enable the OTLP receiver (gRPC + HTTP)
[ ] Choose a backend and configure the exporter (Tempo, Jaeger, etc.)
[ ] Configure the memory_limiter processor
[ ] Configure the health_check extension
[ ] Monitor the Collector's own metrics (Prometheus scrape)
[ ] Apply automatic instrumentation to one or two pilot services
[ ] Build the basic dashboard (RED metrics: Rate, Errors, Duration)

Phase 2: Rollout and Standardization (4-8 weeks)

Checklist
============================================
[ ] Write a Semantic Conventions guide and train the teams
[ ] Configure the attributes processor to normalize legacy attribute names
[ ] Roll automatic instrumentation out to every service
[ ] Attach Kubernetes metadata automatically with the k8sattributes processor
[ ] Guarantee the required resource attributes with the resource processor
[ ] Verify required attributes such as service.name and deployment.environment
[ ] Configure alert rules (error rate, latency thresholds)

Phase 3: Going Deeper (8-12 weeks)

Checklist
============================================
[ ] Add manual instrumentation to business-critical logic (the hybrid strategy)
[ ] Introduce business context propagation through W3C Baggage
[ ] Build the 2-tier Collector architecture for tail-based sampling
[ ] Configure the load balancing exporter (keyed on Trace ID)
[ ] Design and tune the tail_sampling processor policies
[ ] Build cross-signal correlation dashboards
[ ] Monitor against SLOs (Service Level Objectives)

Phase 4: Optimization and Operational Maturity (ongoing)

Checklist
============================================
[ ] Keep tuning the sampling policies (optimizing cost vs visibility)
[ ] Monitor Collector resource usage and apply autoscaling (HPA)
[ ] Adopt the OTel Operator (automating instrumentation injection)
[ ] Unify observability across multiple clusters and regions
[ ] Set a Baggage security policy (trust boundary filtering)
[ ] Write self-service dashboard guidelines for each team
[ ] Assess observability maturity on a regular basis

9.2 A Complete Pipeline Configuration Example

Here is a complete OTel Collector pipeline configuration for a production environment. It shows the whole flow through receivers, processors, and exporters.

# production-otel-collector.yaml
# A complete, production-level Collector pipeline

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: 0.0.0.0:1777 # Go pprof profiling
  zpages:
    endpoint: 0.0.0.0:55679 # zPages for debugging

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
      http:
        endpoint: 0.0.0.0:4318

  # Prometheus metric scraping (compatible with existing Prometheus targets)
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true

  # Host metrics
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      disk: {}
      network: {}

processors:
  # 1. Memory protection (always first)
  memory_limiter:
    check_interval: 1s
    limit_mib: 1800
    spike_limit_mib: 400

  # 2. Attach Kubernetes metadata
  k8sattributes:
    auth_type: 'serviceAccount'
    passthrough: false
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.node.name
      labels:
        - tag_name: service.team
          key: team
          from: pod
        - tag_name: service.component
          key: component
          from: pod

  # 3. Guarantee resource attributes
  resource:
    attributes:
      - key: deployment.environment.name
        value: 'production'
        action: insert
      - key: service.namespace
        value: 'default'
        action: insert

  # 4. Normalize attributes (Semantic Conventions)
  attributes/normalize:
    actions:
      - key: http.request.method
        action: upsert
        from_attribute: http.method
      - key: http.response.status_code
        action: upsert
        from_attribute: http.status_code

  # 5. Strip sensitive information
  attributes/redact:
    actions:
      - key: db.query.text
        action: hash # replace the query with a hash
      - key: http.request.header.authorization
        action: delete
      - key: http.request.header.cookie
        action: delete

  # 6. Filter out unnecessary Spans
  filter/drop-health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.target"] == "/health"'
        - 'attributes["http.target"] == "/readyz"'
        - 'attributes["http.target"] == "/livez"'
        - 'attributes["http.route"] == "/metrics"'

  # 7. Batching
  batch:
    send_batch_size: 2048
    send_batch_max_size: 4096
    timeout: 10s

exporters:
  # Traces → Grafana Tempo
  otlp/tempo:
    endpoint: tempo.observability.svc.cluster.local:4317
    tls:
      insecure: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000

  # Metrics → Prometheus Remote Write (Mimir)
  prometheusremotewrite:
    endpoint: http://mimir.observability.svc.cluster.local:9009/api/v1/push
    tls:
      insecure: true
    retry_on_failure:
      enabled: true

  # Logs → Grafana Loki
  otlp/loki:
    endpoint: loki.observability.svc.cluster.local:4317
    tls:
      insecure: true

  # For debugging (enable only in development environments)
  debug:
    verbosity: basic
    sampling_initial: 5
    sampling_thereafter: 200

service:
  extensions: [health_check, pprof, zpages]

  telemetry:
    metrics:
      address: 0.0.0.0:8888
      level: detailed
    logs:
      level: info
      encoding: json

  pipelines:
    traces:
      receivers: [otlp]
      processors:
        - memory_limiter
        - k8sattributes
        - resource
        - attributes/normalize
        - attributes/redact
        - filter/drop-health
        - batch
      exporters: [otlp/tempo]

    metrics:
      receivers: [otlp, prometheus, hostmetrics]
      processors:
        - memory_limiter
        - k8sattributes
        - resource
        - batch
      exporters: [prometheusremotewrite]

    logs:
      receivers: [otlp]
      processors:
        - memory_limiter
        - k8sattributes
        - resource
        - attributes/redact
        - batch
      exporters: [otlp/loki]

10. Conclusion: Escaping the Observability Cartel

10.1 How OpenTelemetry Changes the Rules of the Game

The traditional APM market was structured like an "observability cartel". Once you installed a particular vendor's agent, your data format, query language, dashboards, and alert rules all became tied to that vendor. The cost of switching was so high that you had little choice but to swallow the price increases.

OpenTelemetry changes that structure at the root.

• Separating instrumentation from the backend: instrument once with the OTel SDK and you can swap backends purely through Collector configuration
• A standard wire protocol (OTLP): no vendor can force its own protocol on you
• Community-driven: the CNCF's second most active project, 500+ contributors, with every major vendor taking part

10.2 The Practical Effect of Strategic Visibility

The five strategies covered in this article are not idle technical curiosity; they translate into real business value.

10.3 Next Steps to Get Started

1. Right now: apply OTel automatic instrumentation to one pilot service and send the data through the OTel Collector to an open-source backend (Jaeger or Tempo).
2. Within 2 weeks: share a Semantic Conventions guide with the team and agree to use the standard attribute names.
3. Within 4 weeks: add manual instrumentation to a business-critical service and validate the hybrid strategy.
4. Within 8 weeks: introduce tail-based sampling and start optimizing cost.
5. Within 12 weeks: implement business context propagation with W3C Baggage to reach full observability.

It is time to take back our own observation, not the vendor's. OpenTelemetry is the surest starting point for that journey.

References

• OpenTelemetry official documentation
• OpenTelemetry Collector releases
• The OTLP spec 1.9.0
• OpenTelemetry Semantic Conventions v1.40.0
• The W3C Trace Context standard
• The W3C Baggage standard
• OpenTelemetry sampling concepts
• Tail Sampling Processor (Contrib)
• OpenTelemetry Operator (Kubernetes)
• The OpenTelemetry Baggage API spec
• The OpenTelemetry Kubernetes deployment guide
• OTel Collector deployment patterns (New Relic)
• OTLP gRPC vs HTTP compared (SigNoz)