LabHub

Blog

Observability 2026 Complete Guide — OpenTelemetry, Datadog, Grafana Stack (LGTM+Beyla), Honeycomb, Prometheus, Jaeger, eBPF & SLO Deep Dive

2026-05-16 · 18 min read · #observability#opentelemetry#datadog#grafana#honeycomb#prometheus#jaeger#tempo#loki#mimir#pyroscope#ebpf#slo#sli#monitoring

한국어English日本語

Intro — In May 2026, observability defaults to OpenTelemetry

Back in 2024, "which APM should we buy" was still a vendor choice. By May 2026, the upstream question has converged. Instrumentation is OpenTelemetry, and only the storage, visualization, alerting, and root-cause analysis behind it varies by vendor. OTel is the second most active CNCF project after Kubernetes, and OTLP (gRPC/HTTP) is now the de facto wire format. Datadog, New Relic, Splunk, Dynatrace, Honeycomb, Chronosphere, Grafana Cloud, Coralogix, and Logz.io all accept OTLP as a first-class input.

The other major shift is AI for observability. In 2025, Datadog Bits AI, New Relic AI (NRAI), and Dynatrace Davis AI all shipped natural-language root-cause summaries, automatic anomaly detection, and incident timeline generation as baseline features. The interface for "why did p99 spike" is moving from query languages to natural language.

This article is not a marketing matrix. It is an honest walk-through of what goes where in 2026 production, how to compose a stack on top of OTel, how to choose SaaS vs self-hosted, and how to operate SLOs and error budgets.

The three pillars — and what's beyond

The classical three pillars are metrics, logs, traces. In 2026, two more are first-class.

  1. Continuous profiling: 24/7 CPU/memory profiles (Pyroscope, Parca, Polar Signals, Datadog Continuous Profiler).
  2. RUM (Real User Monitoring) & Synthetics: LCP, INP, FCP, long tasks, network waterfalls captured from browsers/mobile.

The industry now also uses the term MELT (Metrics, Events, Logs, Traces), or MELT+P with profiling. More important than the pillars is interconnection: one click should hop from metric to trace, trace to log, log to profile. This "exemplar → trace → log → profile" chain is the 2026 standard observability UX.

OpenTelemetry — the standardized instrumentation layer

OTel has three parts.

A Python auto-instrumentation example is this short:

# requirements.txt
# opentelemetry-distro==0.50b0
# opentelemetry-exporter-otlp==1.30.0

# One-liner auto-instrumentation
# $ opentelemetry-bootstrap -a install
# $ OTEL_SERVICE_NAME=checkout-api \
#   OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
#   opentelemetry-instrument python app.py

from fastapi import FastAPI
from opentelemetry import trace

tracer = trace.get_tracer(__name__)
app = FastAPI()

@app.get("/checkout/{order_id}")
def checkout(order_id: str):
    with tracer.start_as_current_span("validate_order") as span:
        span.set_attribute("order.id", order_id)
        # business logic
        return {"ok": True, "order_id": order_id}

Add opentelemetry-instrument and over 50 libraries — FastAPI, requests, SQLAlchemy, psycopg, Redis, Kafka clients — gain traces and metrics automatically. Manual instrumentation should be reserved for business-domain spans only.

OTel Collector — the heart of the pipeline

The Collector declares a receivers → processors → exporters pipeline in YAML. The 2026 standard deployment is a two-tier "agent + gateway" topology.

# otel-collector-gateway.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 8192
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  resourcedetection:
    detectors: [env, system, eks, gcp]
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-policy
        type: latency
        latency: {threshold_ms: 500}
      - name: baseline-policy
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

exporters:
  otlp/datadog:
    endpoint: api.datadoghq.com:443
    headers: {dd-api-key: "$DD_API_KEY"}
  otlphttp/grafana:
    endpoint: https://otlp-gateway.grafana.net/otlp
  prometheus:
    endpoint: 0.0.0.0:9464

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, tail_sampling, batch]
      exporters: [otlp/datadog, otlphttp/grafana]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [prometheus, otlphttp/grafana]

The critical knob is tail sampling. Keep 100% of error and slow traces and probabilistically sample around 5% of the rest to control cost. Datadog's trace API is mostly head-sampled, while OTel Collector lets you make the tail decision at the gateway, which is far more flexible.

Metrics — Prometheus and beyond

Prometheus remains the metrics standard in 2026. PromQL is the industry lingua franca, and OTel Collector → Prometheus Remote Write is GA. A single Prometheus has clear cardinality and retention limits, though, so it is usually paired with a long-term storage backend.

A typical PromQL recording rule (5xx ratio SLI):

# recording-rules.yaml
groups:
  - name: checkout-sli
    interval: 30s
    rules:
      - record: job:http_request_errors:ratio_5m
        expr: |
          sum(rate(http_server_request_duration_seconds_count{job="checkout-api",http_response_status_code=~"5.."}[5m]))
          /
          sum(rate(http_server_request_duration_seconds_count{job="checkout-api"}[5m]))
      - record: job:http_request_latency:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(http_server_request_duration_seconds_bucket{job="checkout-api"}[5m]))
          )
      - alert: HighErrorRate
        expr: job:http_request_errors:ratio_5m > 0.01
        for: 10m
        labels: {severity: page, service: checkout}
        annotations:
          summary: "checkout 5xx > 1% for 10m"

histogram_quantile and OTel histogram compatibility unified on native (exponential) histograms in mid-2025. Memory use is 5-10x lower than the older le-bucket approach with better accuracy.

Cardinality crisis and cost control

An OTel Collector example to drop unused metrics and attributes:

# cardinality drop rules
processors:
  transform/metrics:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "request_id")
          - delete_key(attributes, "build_sha")
      - context: metric
        statements:
          - drop() where name == "unused_metric_legacy"
  filter/spans:
    spans:
      exclude:
        match_type: regexp
        attributes:
          - key: http.url
            value: "/healthz|/metrics|/readyz"

Since 2024, the industry headache has been the cardinality explosion. Slap user_id, request_id, build_sha onto labels and time series blow up into the millions. Datadog, Splunk Observability, and New Relic all charge per custom metric, so bills doubling quarter over quarter is a common incident.

The 2026 playbook has three prongs.

  1. High-cardinality belongs in traces/logs: keep it out of metric labels and put it on events (Honeycomb's philosophy).
  2. Adaptive sampling + drop rules: use the OTel Collector's transformprocessor or Datadog's metric pipelines to drop unused labels automatically.
  3. Cardinality control SaaS: Chronosphere Control Plane, Grafana Adaptive Metrics, and Cribl Stream identify and remove unused series.

Charity Majors's "wide, structured events" pitch from 2017 became the actual standard in 2026. Metrics are reserved for SLI/SLA-grade counters and gauges; high-dimensional data flows into events/traces.

Datadog — the undisputed unified SaaS

Datadog hit 3 billion-plus dollars in annual revenue in 2026 with 800-plus integrations, and is effectively the category definer. APM, Logs, RUM, Synthetic, Continuous Profiler, Database Monitoring, Network Performance Monitoring, and CWPP/CNAPP all live in one console. Bits AI, GA in 2025, handles natural-language trace analysis, incident summarization, and IaC code generation.

The strengths are clear.

The weaknesses are equally clear.

Grafana Stack — the OSS LGTM+Beyla

Grafana Labs has a full stack with Loki (logs), Mimir (metrics), Tempo (traces), Pyroscope (profiles), and Beyla (eBPF auto-instrumentation). Apache 2.0 + AGPL licensing (edition-dependent) means both self-hosting and Grafana Cloud SaaS are options.

Loki's LogQL feels like PromQL.

# Loki LogQL — checkout 5xx trend over the last 5 minutes
{service_name="checkout-api"} |= "ERROR"
  | json
  | http_response_status_code >= 500
  | rate(5m)

# Tempo TraceQL — checkout traces with p99 over 1 second
{ resource.service.name = "checkout-api" && duration > 1s && status = error }

# Mimir PromQL — node CPU utilization
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Tempo TraceQL, GA in 2024, is widely seen as the most expressive trace query language by 2026. Span attributes, resource attributes, and child-span conditions all fit in a single query.

Beyla is a zero-instrumentation eBPF auto-instrumenter that shipped in 2024. It extracts HTTP/gRPC spans and RED metrics without touching a single line of code. Go, Java, Python, Node.js, and Rust are all supported. The Beyla 1.10 line in 2026 even decodes mTLS traffic.

Honeycomb — wide events and BubbleUp

Honeycomb was founded in 2016 by Charity Majors and Christine Yen, drawing on Facebook Scuba. The unit of work is a "wide event" with dozens to hundreds of dimensions, and every query can group or filter on any dimension on the fly. There is no cardinality limit.

Pricing is event-count based and tends to be cheaper than Datadog, though processing metrics and logs separately adds cost. In 2026 it is the preferred Datadog/New Relic alternative for teams who are serious about debugging and SLOs.

Jaeger / Tempo / Zipkin — the OSS tracing trio

The OSS trace backends boil down to three.

Jaeger v2 + Tempo + OTel Collector is the standard self-hosted stack in 2026. A growing cohort (Uber, Cloudflare) uses ClickHouse directly as the trace backend.

eBPF Observability — instrumentation without code changes

eBPF carved out a new observability category. By intercepting syscall, network, and CPU events directly in the kernel, it extracts RED metrics, distributed traces, and profiles with zero code changes.

The weakness of eBPF is limited parameter extraction in user-space runtimes like Go, which means business-logic spans still need an OTel SDK. The 2026 division of labor is "eBPF for RED and golden signals, OTel SDK for domain spans".

Operating Prometheus — Mimir vs Thanos vs Cortex vs VictoriaMetrics

The long-term storage debate is still alive in 2026.

ItemMimirThanosCortexVictoriaMetrics
LicenseAGPL v3Apache 2.0Apache 2.0Apache 2.0
Multi-tenancyStrongDecentStrongStrong
CompressionGoodGoodGoodExcellent
Single binaryNoNoNoYes
Operational costHighMediumHighLow
Commercial supportGrafanaCommunityCommunityVictoriaMetrics
Cardinality ceiling100M+50M100M+500M+

Large SRE teams gravitate to Mimir, lean ops teams to VictoriaMetrics, and existing Thanos users rarely have a reason to migrate.

Continuous profiling — Pyroscope, Parca, Polar Signals

Continuous profiling crystallized as a category through 2024-2025. CPU/memory/lock profiles are captured 24/7 to catch regressions and optimize cost.

The most common issues teams discover are (1) JSON serialization hotspots, (2) regex compilation in tight loops, (3) untuned Go GC GOGC, and (4) Python GIL contention.

RUM and Synthetic — observability from the user's view

Server metrics alone do not tell you "why the user says it's slow". RUM (Real User Monitoring) and synthetic monitoring close the gap.

When Google promoted INP to a Core Web Vital alongside LCP and CLS in 2025, long-task and input-delay measurement in RUM became mandatory.

SLOs and error budgets — the Google SRE legacy

SLOs (Service Level Objectives) became the industry lingua franca after the Google SRE book in 2018. By 2026 there is a dedicated SLO tooling category.

A canonical SLO definition:

# slo.yaml — Sloth format
version: prometheus/v1
service: checkout-api
slos:
  - name: availability
    objective: 99.9
    description: "checkout 5xx ratio under 0.1%"
    sli:
      events:
        error_query: sum(rate(http_server_request_duration_seconds_count{job="checkout-api",http_response_status_code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_server_request_duration_seconds_count{job="checkout-api"}[{{.window}}]))
    alerting:
      page_alert:
        labels: {severity: page}
      ticket_alert:
        labels: {severity: ticket}

A 99.9% SLO grants a 43.2-minute monthly downtime budget. Tracking that budget via burn rate is far smarter than a static threshold alert. Burn rate of 14.4x over a 5-minute window (consuming a year's budget in an hour) pages immediately; 6x over a 6-hour window opens a ticket.

RED, USE, LETS, Golden Signals — methodology cheat sheet

There are four common methodologies for picking observability metrics.

In practice teams combine "user-facing surface (API, page) = RED + saturation" with "infrastructure resources (node, DB, queue) = USE".

Distributed trace context propagation — W3C TraceContext + Baggage

Distributed tracing requires that trace ID, span ID, and sampling decision propagate across every service hop. The 2026 standard is W3C TraceContext + W3C Baggage.

The OTel SDK handles propagation automatically through default propagators, but non-HTTP paths like gRPC metadata, Kafka message headers, and Lambda contexts need explicit instrumentation.

A typical query when using ClickHouse as the trace backend:

-- p99 traces in the last hour
SELECT
    TraceId,
    SpanName,
    Duration / 1e6 AS duration_ms,
    SpanAttributes['http.url'] AS url,
    ResourceAttributes['service.name'] AS service
FROM otel_traces
WHERE Timestamp > now() - INTERVAL 1 HOUR
  AND ResourceAttributes['service.name'] = 'checkout-api'
  AND StatusCode = 'STATUS_CODE_ERROR'
ORDER BY duration_ms DESC
LIMIT 50;

Log pipelines — Vector, Fluent Bit, Logstash, Elastic, Splunk

Logs are pricier and trickier than metrics or traces. The 2026 standard refines, drops, and extracts fields at the collector tier, with cold storage separated out.

Storage backends split three ways.

Incident management — Alertmanager, PagerDuty, Opsgenie, Incident.io, FireHydrant, Squadcast

The real work begins after the alert fires. The 2026 incident management lineup:

A canonical Alertmanager routing config:

# alertmanager.yml — severity-based routing
route:
  receiver: default
  group_by: [alertname, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: pagerduty-critical
      continue: true
    - matchers: [severity="ticket"]
      receiver: jira-ops
    - matchers: [service="checkout"]
      receiver: slack-checkout
receivers:
  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: '$PAGERDUTY_KEY'
  - name: slack-checkout
    slack_configs:
      - api_url: '$SLACK_WEBHOOK'
        channel: '#checkout-alerts'

Slack-based ChatOps is the default for every tool in 2026: automatic incident channel creation, automatic role (IC, comms, scribe) assignment, and even postmortem template generation.

AI for observability — Bits AI, Davis AI, NRAI

Every major SaaS shipped an LLM-based assistant in 2025.

The honest verdict in 2026 is "prediction was overhyped; search and summarization actually save time". Root-cause analysis is still done by humans.

Korea adoption — PinPoint, Naver D2, Toss, Kakao

Korea's observability scene is unexpectedly strong on the OSS contribution side.

PinPoint remains the dominant agent-based OSS APM on the JVM with strong MSA transaction visualization.

Japan adoption — LINE Promgen, Mercari Datadog, GREE Elastic

Japanese internet companies have deep observability stacks too.

The LINE Engineering and Mercari Engineering blogs are the primary Japanese-language observability source.

SaaS vs self-hosted decision guide

There is no universal answer, but there are patterns.

Data sovereignty demands (EU GDPR, Korea's cloud security certifications, Japanese government cases) can also force self-hosting.

Cost-control checklist

To avoid bill explosions, check these seven.

  1. No high-cardinality labels: don't shove user_id or request_id into metric labels.
  2. Adaptive sampling: 100% on errors, 1-5% on success.
  3. Log tiering: hot 7-14 days, warm 30-90 days, cold 1 year-plus. Use S3 Glacier.
  4. Drop rules: drop unused series and logs automatically via OTel Collector or Cribl.
  5. Explicit TTL: a retention policy on every data type.
  6. Quarterly bill simulation: compute cost at 2x traffic.
  7. Cardinality alarms: auto-alarm when a new label appears.
  1. OTel by default: new projects start with the OTel SDK from day one.
  2. eBPF zero-instrumentation: Beyla and Pixie cover Java/Go/Python RED metrics without code changes.
  3. AI assistants in the daily flow: natural-language to query, incident summarization, runbook generation.
  4. Cardinality SaaS rises: Chronosphere, Adaptive Metrics, and Cribl sharpen their value proposition.
  5. OpenTelemetry Logs GA: the logs signal goes 1.0 alongside metrics and traces. One SDK for all three signals.

References

Read the original: https://www.youngju.dev/blog/culture/2026-05-16-observability-opentelemetry-datadog-grafana-honeycomb-prometheus-jaeger-ebpf-slo-2026-deep-dive.en

Comments

No comments yet.

Sign in to leave a comment