OpenTelemetry Observability Blueprint: Integrating Metrics, Logs, and Traces

• Why a Blueprint Is Needed
• Overall Architecture
  • Architecture Design Principles
• Collector Configuration: Agent (DaemonSet)
• Collector Configuration: Gateway
• Application Instrumentation: Python
  • Auto-instrumentation + Adding Manual Spans
  • Adding Manual Instrumentation to Business Logic
• Cross-Signal Correlation: How to Connect the Three Signals
  • Correlation Configuration in Grafana
  • A Real-World Correlation Scenario
• Cost Management: Sampling Strategy
  • Sampling Strategy per Signal
  • Managing Metric Cardinality
  • Cost Estimation Model
• Phased Adoption Roadmap
  • Phase 1 (2 weeks): Introducing Traces
  • Phase 2 (2 weeks): Integrating Metrics
  • Phase 3 (2 weeks): Integrating Logs
  • Phase 4 (2 weeks): Optimization and Standardization
• Troubleshooting
  • 1. A Trace Breaks Partway Through (Broken Trace)
  • 2. Collector OOM (Out of Memory)
  • 3. Metric Cardinality Explosion
  • 4. trace_id Is Missing from Logs
• Quiz
• References

Why a Blueprint Is Needed

The most common mistake when designing an observability system is building Metrics, Logs, and Traces as separate tools. If you collect metrics with Prometheus, gather logs with ELK, and store traces with Jaeger, the three kinds of data end up sitting apart from one another. When an incident hits, you have to connect "see the error rate rise in metrics -> search for the error message in logs -> track the slow request in traces" by hand, and that process burns 15-30 minutes on average.

OpenTelemetry (OTel) unifies these three signals under one SDK, one protocol (OTLP), and one attribute system (semantic conventions). This blueprint designs the complete architecture for building Metrics, Logs, and Traces together on an OTel foundation.

Overall Architecture

                    [Application Pods]
                    ┌──────────────────┐
                    │  OTel SDK        │
                    │  (Auto + Manual) │
                    │  ┌─────────────┐ │
                    │  │ Traces      │ │
                    │  │ Metrics     │ │
                    │  │ Logs        │ │
                    │  └──────┬──────┘ │
                    └─────────┼────────┘
                              │ OTLP (gRPC)
                              ▼
                    ┌──────────────────┐
                    │  OTel Collector  │
                    │  (DaemonSet)     │
                    │  ┌────────────┐  │
                    │  │ Receivers  │  │
                    │  │ Processors │  │
                    │  │ Exporters  │  │
                    │  └────────────┘  │
                    └────────┬─────────┘
                             │ OTLP
                             ▼
                    ┌──────────────────┐
                    │  OTel Collector  │
                    │  (Gateway)       │
                    │  - Sampling      │
                    │  - Enrichment    │
                    │  - Routing       │
                    └──┬─────┬──────┬──┘
                       │     │      │
                ┌──────┘     │      └──────┐
                ▼            ▼             ▼
          ┌──────────┐ ┌──────────┐ ┌──────────┐
          │ Tempo    │ │ Mimir    │ │ Loki     │
          │ (Traces) │ │ (Metrics)│ │ (Logs)   │
          └──────────┘ └──────────┘ └──────────┘
                └──────────┼──────────┘
                           ▼
                    ┌──────────────┐
                    │   Grafana    │
                    │  (Unified)   │
                    └──────────────┘

Architecture Design Principles

1. Agent-Gateway 2-tier structure: a DaemonSet Collector (agent) collects locally, and a Gateway Collector processes and routes centrally. Keep the Agent light and let the Gateway take on the heavy processing.
2. A single OTLP protocol: every signal is sent over OTLP, which simplifies the network configuration.
3. Semantic Conventions compliance: apply standard attributes such as service.name, service.version, and deployment.environment identically to every signal, which is what makes cross-signal correlation possible.

Collector Configuration: Agent (DaemonSet)

# otel-collector-agent.yaml
# Deployed as a DaemonSet. Responsible for local collection on each node.
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-agent
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

      # Collect Kubernetes node/pod metrics
      kubeletstats:
        collection_interval: 30s
        auth_type: serviceAccount
        endpoint: "https://${env:NODE_IP}:10250"
        insecure_skip_verify: true
        metric_groups:
          - node
          - pod
          - container

      # Host metrics (CPU, memory, disk)
      hostmetrics:
        collection_interval: 30s
        scrapers:
          cpu:
            metrics:
              system.cpu.utilization:
                enabled: true
          memory:
            metrics:
              system.memory.utilization:
                enabled: true
          disk: {}
          network: {}

    processors:
      # Cap memory usage (keep the Agent light)
      memory_limiter:
        check_interval: 5s
        limit_mib: 512
        spike_limit_mib: 128

      # Automatically add Kubernetes metadata
      k8sattributes:
        auth_type: serviceAccount
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.namespace.name
            - k8s.node.name
            - k8s.deployment.name
          labels:
            - tag_name: app
              key: app.kubernetes.io/name
            - tag_name: version
              key: app.kubernetes.io/version

      # Add resource attributes (applied in common to every signal)
      resource:
        attributes:
          - key: deployment.environment
            value: "${env:DEPLOY_ENV}"
            action: upsert
          - key: cloud.region
            value: "${env:CLOUD_REGION}"
            action: upsert

      batch:
        send_batch_size: 1024
        timeout: 5s

    exporters:
      otlp:
        endpoint: otel-collector-gateway:4317
        tls:
          insecure: false
          ca_file: /etc/ssl/certs/ca.crt

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, resource, batch]
          exporters: [otlp]
        metrics:
          receivers: [otlp, kubeletstats, hostmetrics]
          processors: [memory_limiter, k8sattributes, resource, batch]
          exporters: [otlp]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, resource, batch]
          exporters: [otlp]

Collector Configuration: Gateway

# otel-collector-gateway.yaml
# Deployed as a Deployment. Responsible for central processing and routing.
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-gateway
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      memory_limiter:
        check_interval: 5s
        limit_mib: 4096
        spike_limit_mib: 1024

      # Tail-based sampling (performed on the Gateway only)
      # Storing every trace makes cost explode, so intelligent sampling is essential
      tail_sampling:
        decision_wait: 10s
        num_traces: 100000
        policies:
          # Retain 100% of traces that contain an error
          - name: errors-policy
            type: status_code
            status_code:
              status_codes: [ERROR]
          # Retain 100% of slow requests (P95 and above)
          - name: latency-policy
            type: latency
            latency:
              threshold_ms: 1000
          # Sample 10% of normal requests
          - name: probabilistic-policy
            type: probabilistic
            probabilistic:
              sampling_percentage: 10

      # Remove unnecessary attributes (cost reduction)
      attributes/remove:
        actions:
          - key: http.request.header.authorization
            action: delete
          - key: http.request.header.cookie
            action: delete
          - key: db.statement
            action: hash  # Hash SQL statements to protect personal data

      batch:
        send_batch_size: 2048
        timeout: 10s

    exporters:
      # Traces -> Grafana Tempo
      otlp/tempo:
        endpoint: tempo:4317
        tls:
          insecure: true

      # Metrics -> Grafana Mimir (Prometheus compatible)
      prometheusremotewrite:
        endpoint: http://mimir:9009/api/v1/push
        resource_to_telemetry_conversion:
          enabled: true

      # Logs -> Grafana Loki
      loki:
        endpoint: http://loki:3100/loki/api/v1/push

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, tail_sampling, attributes/remove, batch]
          exporters: [otlp/tempo]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, attributes/remove, batch]
          exporters: [loki]

Application Instrumentation: Python

Auto-instrumentation + Adding Manual Spans

# app/tracing.py
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.semconv.resource import ResourceAttributes
import logging

def setup_observability(
    service_name: str,
    service_version: str,
    otlp_endpoint: str = "http://otel-collector:4317",
):
    """Initialize OpenTelemetry. Called once at application startup."""

    # Define the resource (attributes applied in common to every signal)
    resource = Resource.create({
        ResourceAttributes.SERVICE_NAME: service_name,
        ResourceAttributes.SERVICE_VERSION: service_version,
        ResourceAttributes.DEPLOYMENT_ENVIRONMENT: "production",
        "team.name": "platform",
    })

    # --- Traces ---
    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(endpoint=otlp_endpoint, insecure=True),
            max_queue_size=2048,
            max_export_batch_size=512,
            schedule_delay_millis=5000,
        )
    )
    trace.set_tracer_provider(tracer_provider)

    # --- Metrics ---
    metric_reader = PeriodicExportingMetricReader(
        OTLPMetricExporter(endpoint=otlp_endpoint, insecure=True),
        export_interval_millis=30000,  # export every 30 seconds
    )
    meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
    metrics.set_meter_provider(meter_provider)

    # --- Logs (OTel Logs Bridge) ---
    # Bridge Python logging into OTel
    from opentelemetry.sdk._logs import LoggerProvider
    from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
    from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
    from opentelemetry._logs import set_logger_provider

    logger_provider = LoggerProvider(resource=resource)
    logger_provider.add_log_record_processor(
        BatchLogRecordProcessor(
            OTLPLogExporter(endpoint=otlp_endpoint, insecure=True)
        )
    )
    set_logger_provider(logger_provider)

    # Attach the Python logging handler
    from opentelemetry.sdk._logs import LoggingHandler
    handler = LoggingHandler(logger_provider=logger_provider)
    logging.getLogger().addHandler(handler)

Adding Manual Instrumentation to Business Logic

# app/services/order_service.py
from opentelemetry import trace, metrics
import logging

tracer = trace.get_tracer("order-service", "1.0.0")
meter = metrics.get_meter("order-service", "1.0.0")
logger = logging.getLogger(__name__)

# Define business metrics
order_counter = meter.create_counter(
    name="orders.created",
    description="Number of orders created",
    unit="1",
)
order_amount_histogram = meter.create_histogram(
    name="orders.amount",
    description="Distribution of order amounts",
    unit="KRW",
)
payment_duration = meter.create_histogram(
    name="payment.duration",
    description="Payment processing time",
    unit="s",
)

async def create_order(user_id: str, items: list, payment_method: str):
    """Create an order - an example where Traces, Metrics, and Logs are all connected"""

    # 1. Create the parent span
    with tracer.start_as_current_span(
        "create_order",
        attributes={
            "user.id": user_id,
            "order.item_count": len(items),
            "payment.method": payment_method,
        },
    ) as span:
        total_amount = sum(item["price"] * item["qty"] for item in items)
        span.set_attribute("order.total_amount", total_amount)

        # 2. Check inventory (child span)
        with tracer.start_as_current_span("check_inventory") as inv_span:
            for item in items:
                available = await check_stock(item["sku"], item["qty"])
                if not available:
                    inv_span.set_attribute("inventory.out_of_stock_sku", item["sku"])
                    # the trace context is automatically included in the log too
                    logger.warning(
                        f"Out of stock: SKU={item['sku']}, requested={item['qty']}",
                        extra={"sku": item["sku"], "requested_qty": item["qty"]},
                    )
                    span.set_status(trace.StatusCode.ERROR, "Out of stock")
                    raise OutOfStockError(item["sku"])

        # 3. Process the payment (child span)
        import time
        payment_start = time.monotonic()
        with tracer.start_as_current_span(
            "process_payment",
            attributes={"payment.method": payment_method},
        ) as pay_span:
            try:
                result = await payment_gateway.charge(total_amount, payment_method)
                pay_span.set_attribute("payment.transaction_id", result.tx_id)
                logger.info(
                    f"Payment succeeded: tx_id={result.tx_id}, amount={total_amount}",
                    extra={"tx_id": result.tx_id, "amount": total_amount},
                )
            except PaymentError as e:
                pay_span.set_status(trace.StatusCode.ERROR, str(e))
                logger.error(f"Payment failed: {e}", exc_info=True)
                raise
            finally:
                elapsed = time.monotonic() - payment_start
                payment_duration.record(elapsed, {"payment.method": payment_method})

        # 4. Record metrics
        order_counter.add(1, {
            "payment.method": payment_method,
            "order.status": "created",
        })
        order_amount_histogram.record(total_amount, {
            "payment.method": payment_method,
        })

        return {"order_id": "ORD-12345", "status": "created"}

Cross-Signal Correlation: How to Connect the Three Signals

The real value of OTel shows up when the three signals are connected. The key point is to include the same trace_id in every signal.

Correlation Configuration in Grafana

# grafana/provisioning/datasources/datasources.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
        filterBySpanID: true
        tags:
          - key: service.name
            value: service_name
      tracesToMetrics:
        datasourceUid: mimir
        tags:
          - key: service.name
            value: service
      serviceMap:
        datasourceUid: mimir

  - name: Loki
    type: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: TraceID
          datasourceUid: tempo
          matcherRegex: "trace_id=(\\w+)"
          url: '$${__value.raw}'
          matcherType: regex

  - name: Mimir
    type: prometheus
    url: http://mimir:9009/prometheus
    jsonData:
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo

A Real-World Correlation Scenario

1. Spot an error-rate spike on the Grafana dashboard
   -> Mimir query: rate(http_requests_total{status="500"}[5m])

2. Click the exemplar carried by an error request in that time window
   -> Move to Tempo using the trace_id contained in the exemplar

3. Inspect the full trace in Tempo
   -> Visualize which span in which service the error occurred in
   -> Check business context such as user.id and order.id in the span attributes

4. Check the logs that carry the same trace_id as that span
   -> Loki query: {service_name="order-service"} | trace_id="abc123..."
   -> Check the error stack trace, the input data, and intermediate state values

5. Total diagnosis time: 2-3 minutes (previously: 15-30 minutes)

Cost Management: Sampling Strategy

The cost of observability data comes mainly from storage and network. Storing 100% of the data can add up to millions of won per month.

Sampling Strategy per Signal

Managing Metric Cardinality

# Remove high-cardinality attributes at the Collector
processors:
  # Drop high-cardinality labels such as user_id and session_id
  # They should not be used on metrics (use them on traces instead)
  transform/metrics:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "user.id")
          - delete_key(attributes, "session.id")
          - delete_key(attributes, "request.id")
          - delete_key(attributes, "http.url") # cardinality explodes when path parameters are included

Cost Estimation Model

def estimate_monthly_cost(
    daily_requests: int,
    avg_spans_per_trace: int = 8,
    avg_log_lines_per_request: int = 5,
    trace_sample_rate: float = 0.10,
    log_sample_rate: float = 0.20,
) -> dict:
    """Estimate the monthly cost of observability data"""
    monthly_requests = daily_requests * 30

    # Traces
    traces_per_month = monthly_requests * trace_sample_rate
    spans_per_month = traces_per_month * avg_spans_per_trace
    trace_storage_gb = spans_per_month * 0.5 / 1e6  # roughly 0.5KB per span

    # Metrics (always 100%, managed through cardinality)
    # 10 services, 50 metric types, cardinality 100 = 50,000 time series
    metric_series = 50_000
    metric_storage_gb = metric_series * 30 * 24 * 2 * 8 / 1e9  # 8B per data point

    # Logs
    log_lines_per_month = monthly_requests * avg_log_lines_per_request * log_sample_rate
    log_storage_gb = log_lines_per_month * 0.2 / 1e6  # roughly 0.2KB per log line

    # Cost calculation (rough unit prices based on Grafana Cloud)
    trace_cost = trace_storage_gb * 2.0   # $2/GB
    metric_cost = metric_series * 0.008   # $8 per 1000 series
    log_cost = log_storage_gb * 0.50      # $0.50/GB

    return {
        "traces": {
            "sampled_per_month": int(traces_per_month),
            "storage_gb": round(trace_storage_gb, 1),
            "cost_usd": round(trace_cost, 2),
        },
        "metrics": {
            "active_series": metric_series,
            "storage_gb": round(metric_storage_gb, 1),
            "cost_usd": round(metric_cost, 2),
        },
        "logs": {
            "sampled_lines_per_month": int(log_lines_per_month),
            "storage_gb": round(log_storage_gb, 1),
            "cost_usd": round(log_cost, 2),
        },
        "total_monthly_usd": round(trace_cost + metric_cost + log_cost, 2),
    }

# Based on a service handling 1 million requests per day
cost = estimate_monthly_cost(daily_requests=1_000_000)
print(f"Estimated total monthly cost: ${cost['total_monthly_usd']}")

Phased Adoption Roadmap

Adopting everything at once fails. Build it up gradually in four phases.

Phase 1 (2 weeks): Introducing Traces

Goal: enable distributed tracing on 3 core services

Tasks:
1. Deploy the OTel Collector DaemonSet + Gateway
2. Add the OTel SDK to core services (auto-instrumentation first)
3. Deploy the Tempo backend
4. Confirm trace search works in Grafana

Success criteria:
- Do traces appear connected across services?
- Can you identify the bottleneck segment of a high-P95-latency request from the trace?

Phase 2 (2 weeks): Integrating Metrics

Goal: consolidate Prometheus metrics into the OTel pipeline

Tasks:
1. Add a Prometheus receiver to the OTel Collector
2. Switch the existing Prometheus server over to Mimir via remote_write
3. Configure exemplar linking (metrics -> traces)
4. Migrate the existing Grafana dashboards

Success criteria:
- Can you jump to a trace by clicking an exemplar on a metrics dashboard?
- Do the existing alerts behave the same way?

Phase 3 (2 weeks): Integrating Logs

Goal: collect structured logs through OTel and link them to traces

Tasks:
1. Apply the OTel Logs Bridge to application logging
2. Verify that trace_id and span_id are inserted into logs automatically
3. Deploy the Loki backend and connect the Collector
4. Configure two-way trace <-> log linking in Grafana

Success criteria:
- Can you move from trace to log and from log to trace in one click?
- Can you follow the whole request flow from the trace context of an error log?

Phase 4 (2 weeks): Optimization and Standardization

Goal: cost optimization, sampling tuning, team onboarding

Tasks:
1. Enable tail-based sampling and confirm a 30% cost reduction
2. Document the semantic conventions standard
3. Distribute per-team dashboard templates
4. Add the observability workflow to the on-call runbook

Success criteria:
- MTTD (Mean Time To Detect) improved by 50%
- MTTR (Mean Time To Resolve) improved by 30%
- Observability data cost within 5% of infrastructure cost

Troubleshooting

1. A Trace Breaks Partway Through (Broken Trace)

Symptom: when you open a trace in Grafana Tempo, spans from some services are missing

Diagnosis:

# 1. Check context propagation - is traceparent being passed in the HTTP header
curl -v http://service-a/api/test 2>&1 | grep -i traceparent
# traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

# 2. Check that service B receives that header and propagates it
# Search for trace_id in the logs of service B

# 3. Check that spans are not being dropped at the Collector
curl -s http://otel-collector:8888/metrics | grep otelcol_exporter_sent_spans

Cause and fix:

• The HTTP client is not forwarding the traceparent header -> check the OTel HTTP instrumentation library
• Context is not propagated across a message queue (Kafka, RabbitMQ) -> add OTel instrumentation to the queue producer/consumer
• Metadata is not passed on in the gRPC service -> check the gRPC interceptor

2. Collector OOM (Out of Memory)

Error: memory usage exceeded limit: 512 MiB

Fix:

# Check and adjust the memory_limiter settings
processors:
  memory_limiter:
    check_interval: 5s
    limit_mib: 512 # keep the Agent within 512MB
    spike_limit_mib: 128
    # data is dropped once the limit is reached, so
    # allocate enough memory, but stay within a range that does not affect the node

  # reduce the batch size
  batch:
    send_batch_size: 512 # reduced from 2048 to 512
    timeout: 5s

3. Metric Cardinality Explosion

Symptom: memory usage on Mimir or Prometheus climbs sharply and query speed degrades

Diagnosis:

# Find the metrics with high cardinality
topk(10, count by (__name__)({__name__=~".+"}))

# Check the label cardinality of a specific metric
count(http_requests_total) by (url)
# path parameters embedded in the url create tens of thousands of time series

Fix: remove or normalize high-cardinality labels in the Collector's transform processor

4. trace_id Is Missing from Logs

Cause: the OTel Logs Bridge is configured, but the logging library's formatter does not output the trace context

Fix (Python):

import logging
from opentelemetry import trace

class TraceContextFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        ctx = span.get_span_context()
        record.trace_id = format(ctx.trace_id, '032x') if ctx.trace_id else ""
        record.span_id = format(ctx.span_id, '016x') if ctx.span_id else ""
        return True

handler = logging.StreamHandler()
handler.addFilter(TraceContextFilter())
handler.setFormatter(logging.Formatter(
    '%(asctime)s %(levelname)s [trace_id=%(trace_id)s span_id=%(span_id)s] %(message)s'
))
logging.getLogger().addHandler(handler)

Quiz

References

• OpenTelemetry Documentation
• OpenTelemetry Collector Configuration
• OpenTelemetry Semantic Conventions
• Grafana - Correlating Signals
• Grafana Tempo Documentation
• OpenTelemetry Tail Sampling Processor