LabHub

Blog

Distributed Tracing & OpenTelemetry 2026 — OTel / Jaeger / Tempo / Zipkin / Honeycomb / Lightstep / SigNoz / SkyWalking / Datadog APM Deep Dive

2026-05-16 · 21 min read · #observability#distributed-tracing#opentelemetry#otelcol#jaeger#tempo#zipkin#honeycomb#charity-majors#lightstep#servicenow#signoz#apache-skywalking#datadog-apm#new-relic-apm#elastic-apm#sentry-performance#dynatrace#appdynamics#splunk#pixie#beyla#w3c-trace-context#b3-propagation#baggage#head-based-sampling#tail-based-sampling#deep-dive#english

한국어English日本語

Prologue — "Why is checkout slow?"

A Friday evening in 2026. The payments-team Slack channel lights up.

"p99 checkout latency 4.2s. Normally 600ms. What got slow?"

Answering that within five minutes is why humanity spent the last decade building distributed tracing. One request fans through the gateway, into auth, into the payments core, out to an external PG, then back through the receipts queue — every hop captured as a single trace.

In 2020, the field was fragmented: OpenTracing, OpenCensus, Jaeger clients, Zipkin libraries, each APM vendor's agent. Even inside one company Java used Datadog, Go used Jaeger, Node used New Relic. In 2026 the answer is simpler — OpenTelemetry.

This post is a map of distributed tracing in 2026. The OTel spec and Collector, propagation standards, OSS backends (Jaeger/Tempo/Zipkin), observability 2.0 (Honeycomb, SigNoz), acquired giants (Lightstep), the APM super-league (Datadog, New Relic, Dynatrace, Splunk), eBPF auto-instrumentation (Pixie, Beyla), plus sampling and cost. Including how Korean and Japanese companies actually migrated.

1. The 2026 Distributed-Tracing Map — Four Camps

The ecosystem clusters into four buckets.

BucketExamplesCharacter
Standard / instrumentationOpenTelemetryShared spec, SDKs, Collector. Almost everyone flows through this
OSS self-hosted backendsJaeger, Tempo, Zipkin, SigNoz, SkyWalkingSelf-operated. No license fees
APM SaaSDatadog, New Relic, Dynatrace, Splunk, AppDynamics, Sentry, ElasticManaged. Unified UI, alerting, AIOps
observability 2.0Honeycomb, Lightstep (ServiceNow)Wide events, high-cardinality analytics
eBPF auto-instrumentationPixie, Beyla, CorootNo code changes — kernel / network level

Three axes in one picture.

              OSS                              SaaS
        +--------------+              +-----------------+
Instr.  | OTel SDK     |  --- shared - | OTel SDK         |
        | (vendor-     |              | (vendor adapter) |
        |  neutral)    |              |                 |
        +------+-------+              +--------+--------+
               |                                |
        +------v-----------------------------------v-----+
        |          OpenTelemetry Collector              |
        |  receivers --> processors --> exporters       |
        +------+-----------------------------------+----+
               |                                  |
       +-------v--------+                +--------v--------+
       | Jaeger / Tempo |                |  Datadog APM    |
       | Zipkin / SigNoz|                |  New Relic APM  |
       | SkyWalking     |                |  Honeycomb etc. |
       +----------------+                +-----------------+

The lesson: unify instrumentation on OTel, keep backends swappable. That is the 2026 default posture.

2. OpenTelemetry — CNCF Graduated (2024)

OpenTelemetry (OTel) was formed in 2019 from the merger of OpenTracing (2016) and OpenCensus (2018). CNCF Incubating in 2021, CNCF Graduated in November 2024. It is now the de facto industry standard — the second most active CNCF project after Kubernetes.

What OTel provides:

Core data model: a trace is a tree of spans. Each span has trace_id, span_id, parent_span_id, start/end, attributes, events, links.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
)

tracer = trace.get_tracer("checkout")
with tracer.start_as_current_span("charge_card") as span:
    span.set_attribute("user.id", user_id)
    span.set_attribute("amount", amount)
    result = pg.charge(...)
    span.set_attribute("pg.response_code", result.code)

OTel's real value is that metrics, logs, and traces share one SDK. Traces are graduate-stable, metrics are stable, logs reached stable across most languages by 2024-2025. OTel is no longer just a tracing standard — it is a unified telemetry standard.

3. OTel Collector — Receivers, Processors, Exporters

OTel SDK lives inside your application; the Collector runs as a separate process. Reasons for that separation:

  1. Apps do not pull in vendor backend SDKs — dependency hygiene.
  2. Sampling, filtering, routing applied centrally.
  3. Backend swap does not require app restarts.
  4. Surge buffering / retry.

Three component types:

ComponentRoleExamples
ReceiversIngest telemetry from outsideotlp, jaeger, zipkin, prometheus, kafka, filelog
ProcessorsTransform / filter / samplebatch, memory_limiter, tail_sampling, attributes
ExportersSend to backendsotlp, jaeger, prometheus, datadog, honeycomb, logging

These three form pipelines.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: sample_10
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }
  datadog:
    api: { site: datadoghq.com, key: ENV_DD_API_KEY }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo, datadog]

You can fan the same trace to an OSS backend (Tempo) and a SaaS (Datadog) at once. That is why the Collector is the migration tool of choice — add a new exporter line, run side by side, validate, then cut the old one.

Two deployment modes:

Large deployments typically run Agent → Gateway → Backend in two stages.

4. Propagation Standards — W3C Trace Context, B3, Baggage

Distributed tracing is "distributed" because the same trace_id crosses service boundaries. Propagation is usually via HTTP headers.

W3C Trace Context (2020 recommendation)

The two headers W3C standardised in 2020 are the de facto default in 2026.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |   |                                |                |
             |   trace-id (16 byte hex)            parent-span-id   trace-flags
             version

tracestate: vendor1=value1,vendor2=value2

OTel SDKs inject these headers into HTTP, gRPC, messaging (Kafka, RabbitMQ headers), even some SQL comments automatically.

B3 Propagation (from Zipkin)

The multi-header format Zipkin created.

X-B3-TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
X-B3-SpanId: 00f067aa0ba902b7
X-B3-ParentSpanId: 05e3ac9a4f6e3b90
X-B3-Sampled: 1

OTel supports both W3C and B3. Where legacy systems mix in, configure to accept both.

Baggage — "Context carried along with the trace"

Baggage carries key-value data alongside trace_id during propagation. Examples: user.tier=gold, tenant=acme, experiment=ab_42.

baggage: user.tier=gold,tenant=acme

Use cases:

Warning: baggage is not a secret. If the next hop is external, anyone can read the header. Never put PII in baggage.

5. Jaeger — CNCF Graduated, Uber Original

Jaeger was created at Uber in 2015. CNCF Incubating in 2017, CNCF Graduated in 2019. It was the de facto standard before OTel.

Characteristics:

Since 2024, the Jaeger project has explicitly pivoted to an "OTel first" stance. The native Jaeger SDK is deprecated; new users are pointed at OTel SDKs and Jaeger becomes a "backend + UI". The v2 backend is built on top of the OTel Collector internally.

Deployment shapes:

Pros: lightweight, OSS, easy install, intuitive UI.

Limits: traces only — no metrics/logs. High-cardinality analysis is weak. UI peaks at trace_id lookup + time-range filter.

6. Grafana Tempo — Parquet-Based, "Object Storage Is the Hot Path"

Grafana Tempo, born at Grafana Labs in 2020, is a trace backend with a precise philosophy — "a trace store without indexes."

Jaeger and Zipkin index every span (service, operation, tags). The indexes are larger and more expensive than the span bodies themselves. Tempo skips that — lookup by trace_id only, with the premise that you pull trace_id out of metrics (Prometheus) and logs (Loki) to navigate. "Click the exemplar and the trace pops up" as a UX pattern.

Storage goes directly to S3 / GCS / Azure Blob style object storage. No indexes means cheap storage. Since 2023 the on-disk format is unified on Parquet — columnar with good compression, and readable by external tools (Athena, DuckDB).

Tempo also has a query language, TraceQL — for finding patterns inside trace bodies.

{ resource.service.name = "checkout"
  && span.http.status_code = 500
  && duration > 1s
}

Pros:

Limits:

If scale and long-term storage dominate, Tempo wins.

7. Zipkin — Twitter Original, Still Alive

Zipkin was open-sourced by Twitter in 2012. It is the most influential OSS implementation of the Google Dapper paper and the starting point for every tracing system that followed.

Characteristics:

Not many fresh adoptions in 2026, but still useful for:

Versus Jaeger, the UI is plainer but setup is simpler.

8. Honeycomb — Charity Majors' "Observability 2.0"

Honeycomb, founded in 2016 by Charity Majors and Christine Yen, is a SaaS observability company. Their phrase "observability 2.0" reshaped the vocabulary of the industry.

Their thesis: classical monitoring (separate metrics/logs/traces) only sees predefined dimensions. Real debugging needs wide events — high-cardinality attributes (user_id, request_id, k8s_pod) that you can slice and dice freely.

Honeycomb's data model: every span is essentially an "event". Attach attributes freely; queries jump between BubbleUp (automatic outlier breakdown), heatmap, and the trace view.

BubbleUp result example:
  73% of slow (p95) requests have user.tier=enterprise AND
  db.host=replica-3 AND
  client_country=JP.
  <- found automatically; no one had to specify the combination.

Differentiators:

Pros: debugging speed is different. "Why is this slow" narrows to one or two minutes without pre-built indexes.

Limits: SaaS only (no self-hosting). Overkill for simple monitoring without high-cardinality questions.

9. Lightstep, Now ServiceNow Cloud Observability

Lightstep was founded in 2014 by Ben Sigelman, one of the original Google Dapper authors. "Statistical analysis" and "merging metrics and traces" were early themes.

ServiceNow acquired Lightstep in 2021. The product was rebranded to ServiceNow Cloud Observability. As a standalone it is fading, but it survives bundled with ServiceNow ITSM/AIOps in enterprise accounts (especially ITIL/CMDB-bound shops).

Technical legacy:

2026 assessment: worth a look if you live in the ServiceNow ecosystem. Otherwise the action is at Honeycomb, Datadog, and the OSS camp.

10. SigNoz — OSS Full-Stack Observability

SigNoz was started in 2020 out of India. OSS APM built on top of ClickHouse, carrying traces, metrics, logs together. Positioned as "the OSS alternative to Datadog."

Characteristics:

docker-compose up brings up the full stack in one line, and OTel compatibility keeps migration relatively painless.

2024-2025 updates:

Pros: looks polished even though OSS, and full-stack. Cloud SaaS is also available if self-hosting is a burden.

Limits: heavy dependency on ClickHouse — operating at scale needs expertise. Enterprise features (SSO, SOC2 reports) skew to the cloud plan.

11. Apache SkyWalking — APM + Tracing, China-Led

Apache SkyWalking entered Apache Incubating in 2017 and graduated in 2019. Contributors come heavily from Huawei, Alibaba, Tencent, and the user base is dense in East Asia.

Characteristics:

2026 positioning: the main OSS choice across China, India, Southeast Asia. Less common in Korea / Japan, but you meet it naturally if your company runs a Chinese subsidiary.

12. Datadog APM, New Relic APM, Elastic APM, Sentry Performance

The commercial APM super-league's tracing lineup.

Datadog APM

New Relic APM

Elastic APM

Sentry Performance

13. Dynatrace, AppDynamics (Splunk), Splunk Observability Cloud

The enterprise heavyweights.

Dynatrace

AppDynamics (now Splunk-acquired)

Splunk Observability Cloud

14. eBPF Tracing — Pixie and Beyla, Zero-Code Auto-Instrumentation

eBPF lets you run safe code inside the Linux kernel. You can capture network packets, syscalls, HTTP requests at the kernel level — without changing a line of application code.

Pixie

Beyla — Grafana's eBPF Auto-Instrumenter

Where eBPF Tracing Fits

Conclusion: run eBPF and OTel SDK together. eBPF gives instant infra/network visibility; OTel SDK adds business spans and high-cardinality attributes.

15. Sampling — Head vs Tail vs Ratio

Trace cost = spans produced * unit price. Eighty per cent of the cost curve is driven by sampling strategy.

Three strategies:

Head-Based Sampling

Ratio Sampling

Tail-Based Sampling

OTel Collector tail_sampling example:

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    policies:
      - name: keep_errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep_slow
        type: latency
        latency: { threshold_ms: 1500 }
      - name: keep_enterprise
        type: string_attribute
        string_attribute: { key: tenant.tier, values: [enterprise] }
      - name: random_2pct
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

This keeps: 100% errors + 100% over 1.5s + 100% enterprise tenants + 2% of the rest.

Which to Pick

SituationRecommendation
Small traffic (~hundreds rps)Head 100% or ratio 50%
Typical SaaS (~thousands rps)Head 10% + errors 100% (branched in SDK)
Large scale (tens of thousands+ rps)Tail-based + errors / slow / specific tenants 100%
Cost spirallingTail-based immediately — usually 5-10x savings

16. Cost — Tail-Based Sampling Proxies and the Bill

Two patterns that make the tracing bill scary.

  1. Bill grows linearly with traffic — 100% sampling at large scale.
  2. High-cardinality attribute explosion — every span tagged with user_id, request_id, k8s_pod, container_id.

Mitigations:

Rules of thumb (author's conservative estimates):

17. Korea and Japan — Real Migrations

Korea

Japan

The common pattern: to avoid vendor lock-in, SDK is OTel, backends are swappable. SaaS pricing negotiation also leverages OTel compatibility.

18. Who Should Pick What

Small Teams / Side Projects

Mid-Size (Dozens of People, Tens to Hundreds of Thousands of rps)

Enterprise

Debugging-First (High Cardinality, Ad Hoc Analysis)

Forced Self-Hosting (Regulation, Security)

Kubernetes Polyglot, Hard to Change Code

19. Ten Anti-Patterns

  1. Skip OTel and hard-code vendor SDK directly — backend change rewrites your code.
  2. Instrumented but sampling still at 100% — bill explodes.
  3. "I want every error" without tail sampling — head 1% loses 99% of errors.
  4. Put user_id / request_id on every span without thought — straight cardinality bill.
  5. Forget to run W3C and B3 side by side — trace breaks at legacy boundaries.
  6. PII in baggage — leaks via headers at the next hop.
  7. Metrics, logs, traces stored in disconnected backends with no trace_id link — exemplar workflow broken.
  8. No Collector — SDK ships straight to SaaS — backend change means full app redeploy.
  9. Ignore trace_id routing for tail sampling — same trace splits across Collectors and decisions go wrong.
  10. Look only at traces, ignore metrics and logs — traces are "the story of one request", metrics are "the trend of the whole". You need both.

Epilogue — Build Freedom on Top of OTel

The 2026 lesson is one sentence.

Unify instrumentation on OTel and keep backends swappable.

OTel is more than a tracing standard — it is the interface that breaks vendor lock-in. On top of it, pick the backend that matches your taste: the OSS freedom of Jaeger / Tempo, the debugging depth of Honeycomb, the unified UX of Datadog, the AIOps of Dynatrace. The moment swap cost shrinks from rewriting code to editing a Collector line, you gain negotiating power, cost control, and better debugging at once.

Next post candidates: OpenTelemetry metrics deep dive — exemplars and trace-metric linking, Linking logs to traces — Loki / OpenSearch / OpenTelemetry Logs, Tail sampling in practice — load-balancing exporter topology and cost curves.

— Distributed Tracing and OpenTelemetry 2026 Deep Dive, fin.

References

Read the original: https://www.youngju.dev/blog/culture/2026-05-16-distributed-tracing-opentelemetry-2026-otel-jaeger-tempo-zipkin-honeycomb-lightstep-signoz-skywalking-datadog-apm-deep-dive.en

Comments

No comments yet.

Sign in to leave a comment