LabHub

Blog

Network & Service Observability 2026 Deep Dive — eBPF · Cilium Hubble · Pixie · Pyroscope · Grafana Loki + Tempo + Mimir · Netdata · OpenTelemetry

2026-05-16 · 21 min read · #observability#ebpf#cilium-hubble#pixie#pyroscope#loki#tempo#netdata#opentelemetry#monitoring#english

한국어English日本語

Prologue — In 2026, Observability Has to Answer "Why?"

In 2015, observability arrived with a three-pillar vocabulary: metrics, logs, traces. In 2023 the CNCF formally added continuous profiling as the fourth pillar. And in 2026 the real shift happened somewhere else entirely: instrumentation disappeared.

Five years ago, attaching distributed tracing to a Java app meant importing the OpenTelemetry SDK, annotating every method, and threading context propagation through the code. The 2026 default is let the kernel do it via eBPF. Cilium Hubble sees every packet in the cluster. Pixie sees every HTTP/gRPC/SQL call after one DaemonSet install. Beyla, Coroot, Caretta pull out the golden signals without a single line of code change.

That does not mean the SDK era is over. OpenTelemetry graduated from the CNCF in 2024 and OTLP is now the de facto wire protocol. Think of it this way: eBPF gives you "80% of visibility for free", and OTel SDKs fill in "the remaining 20% of business meaning". The real shape of the 2026 stack is a hybrid of eBPF and OTel.

This post draws that map — the four pillars, every eBPF tool worth knowing (Cilium / Hubble / Tetragon / Pixie / Inspektor Gadget / Coroot / Beyla / Caretta), Grafana LGTM (Loki + Tempo + Mimir + Pyroscope), SaaS giants like Datadog/New Relic/Dynatrace, network-specific tools (Suzieq, ntopng, ThousandEyes), and how Korean and Japanese companies actually use them in production.

Observability is a superset of monitoring. If monitoring is "watching whether a known metric crosses a threshold", observability is "the property of a system that lets you answer questions you didn't think to ask in advance". The 2026 difference isn't the tools — it's a stack design that lets you ask those questions.

What this post covers:

  1. The four pillars (metrics / logs / traces / profiles) and golden signals
  2. The eBPF revolution — Cilium 1.16, Hubble, Tetragon, Pixie
  3. OpenTelemetry 2026 — Collector, OTLP, auto-instrument
  4. Metrics stack — Prometheus 3.0, VictoriaMetrics, Mimir
  5. Logs stack — Loki 3, Elastic, Vector, Quickwit, OpenObserve, SigNoz
  6. Traces stack — Tempo 2, Jaeger 2, Zipkin, Honeycomb
  7. Continuous profiling — Pyroscope, Parca, Polar Signals
  8. Network observability — Suzieq, Skydive, ntopng, ThousandEyes
  9. RUM & synthetic monitoring — Cloudflare, Checkly, Grafana Synthetic
  10. APM comparison — Datadog, New Relic, Dynatrace, AppDynamics
  11. K8s observability — Prometheus Operator, k9s, Lens
  12. Service mesh + observability — Kiali, Linkerd dashboard
  13. DevSecOps + observability — Falco + OTel
  14. Storage backends — VictoriaMetrics, Mimir, ClickHouse, MinIO
  15. Cost model — Datadog $35-70/host vs self-host LGTM
  16. Korean adoption — NCsoft Pixie, Coupang Datadog, Naver OTel, Kakao Grafana
  17. Japanese adoption — Mercari, LINE Yahoo, CyberAgent
  18. SLO/SLI and error budget operations
  19. Alerting and PagerDuty/Opsgenie/Incident.io
  20. The shape of AI-native observability
  21. Adoption roadmap — where to start
  22. References

1. The Four Pillars and Golden Signals — What to Measure

The starting question for any observability project is "what do we look at?" The 2026 consensus is four pillars.

Mapping these to Google's SRE four golden signals is the canonical operational view.

Golden SignalDefinitionExample Metric
LatencyTime to process a requestp50/p95/p99 response time
TrafficLoad on the systemRPS, QPS, MB/s
ErrorsFailure rate5xx ratio, exception count
SaturationResource fullnessCPU utilisation, queue depth, disk IOPS

USE vs RED, 2023 revisited — Brendan Gregg's USE (Utilisation / Saturation / Errors) is resource-oriented; Tom Wilkie's RED (Rate / Errors / Duration) is request-oriented. Both are cousins of the golden signals — choosing which lens to start with shapes what your first dashboard looks like.

2. The eBPF Revolution — Cilium, Hubble, Tetragon, Pixie

The real inflection point of 2026 observability is eBPF (extended Berkeley Packet Filter). A safe in-kernel VM that intercepts packets, syscalls, and socket events — and what it brought to observability is decisive.

What eBPF changed:

  1. Language-agnostic instrumentation — the kernel sees every syscall whether your app is Go, Java, Python, or Rust.
  2. Zero-code-change — install one DaemonSet and you're done.
  3. Low overhead — kernel-level, typically 1-3% CPU.
  4. L7 visibility — HTTP/gRPC/SQL parsers run inside eBPF, surfacing method, path, and status code.

Key tools:

# Install Cilium Hubble UI via helm
cilium install --version 1.16.0
cilium hubble enable --ui
cilium hubble port-forward
# Visit http://localhost:12000 for real-time flow visualisation

The Pixie magic — typically you have to embed an OTel SDK in every service to get distributed traces. With Pixie, installing one PEM (Pixie Edge Module) DaemonSet is all it takes. In five minutes you have the HTTP/gRPC call graph for all your microservices.

3. OpenTelemetry 2026 — The De Facto Standard

OpenTelemetry (OTel) graduated from the CNCF in 2024. Translation: it's now the second-largest project in the CNCF after Kubernetes. As of 2026, OTel can fairly be called "the wire standard for observation data."

Components:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Why OTel won — five years ago, Datadog Agent, New Relic Agent, Jaeger Client, Zipkin Brave each pushed their own SDK. Migrating a company from Datadog to New Relic was a multi-month project. OTel broke that lock-in with a simple promise: "standardise the data, free the backend." By 2026 even Datadog treats OTLP as a first-class citizen.

4. Metrics Stack — Prometheus 3.0 and Friends

Metrics is the oldest and most mature pillar. The 2026 standard is unambiguously the Prometheus ecosystem.

# Prometheus 3.0 scrape + OTLP receive
global:
  scrape_interval: 15s

# OTLP receiver (new in 3.0)
otlp:
  promote_resource_attributes:
    - service.name
    - service.namespace
    - deployment.environment

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod

VictoriaMetrics vs Mimir:

AspectVictoriaMetricsGrafana Mimir
ArchitectureSingle binaryDistributed microservices
StorageLocal diskS3 / GCS / MinIO
Query languagePromQL + MetricsQLPromQL
CompressionVery highModerate
Operational costLowMedium
Best scaleSmall to midMulti-tenant, large

5. Logs Stack — Loki, Elastic, Vector, Quickwit

Logs is the most expensive and most error-prone pillar. The 2026 direction is clear: less indexing, more column stores.

# Fluent Bit -> Loki pipeline example
[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            docker
    Tag               kube.*

[OUTPUT]
    Name                  loki
    Match                 kube.*
    Host                  loki.observability.svc
    Port                  3100
    Labels                job=fluentbit

Loki vs Elasticsearch, in one sentence each:

6. Traces Stack — Tempo 2, Jaeger 2

Distributed tracing answers the question "where did this user request get slow?"

# One-liner Python OTel auto-instrument
# pip install opentelemetry-distro opentelemetry-exporter-otlp
# opentelemetry-bootstrap -a install
# opentelemetry-instrument --traces_exporter otlp \
#   --exporter_otlp_endpoint http://tempo:4317 python app.py

What Jaeger 2 means — back in the 1.x days Jaeger had its own Cassandra/Elasticsearch backends and its own SDK. In 2024 Jaeger 2 was rebuilt entirely on the OTel Collector. That's a statement: even the trace backend is no longer a lock-in.

7. Continuous Profiling — Pyroscope, Parca

Settling in as the fourth pillar of observability since 2023, continuous profiling captures CPU / memory / lock profiles in production all the time, at 1-5% overhead.

Typical use cases:

# Deploy Pyroscope to Kubernetes (Helm)
helm repo add grafana https://grafana.github.io/helm-charts
helm install pyroscope grafana/pyroscope \
  --set pyroscope.config.scrape_configs[0].job_name=k8s-pods

eBPF + Pyroscope combo — Pyroscope's eBPF profiler runs at the node level and pulls CPU profiles for every process automatically. Without touching code, you get Go, Rust, C++, and Python profiles in one place.

8. Network-Specific Observability Tools

Distinct from service observability is network observability — the visibility of the network itself.

# Suzieq example — find non-established BGP sessions via SQL
$ suzieq-cli
suzieq> bgp show state=NotEstd
namespace  hostname    vrf  peer        state
prod       leaf01      default  10.0.0.2  Active
prod       leaf03      default  10.0.0.6  Connect

Why network observability got important again — five years ago "the cloud vendor handles it" was the answer. By 2026, multi-cloud, multi-region, service mesh, and zero-trust networking are routine, and "who is talking to whom and about what?" is once again the central question.

9. RUM (Real User Monitoring) and Synthetic Monitoring

Just as important as server-side observability is what the user actually sees on screen.

RUM (Real User Monitoring):

Synthetic Monitoring:

// Checkly synthetic monitor example (Playwright)
import { expect, test } from '@playwright/test'

test('homepage loads', async ({ page }) => {
  const res = await page.goto('https://example.com')
  expect(res.status()).toBeLessThan(400)
  await expect(page.locator('h1')).toContainText('Welcome')
})

The 2026 RUM standard — push Core Web Vitals (LCP, INP, CLS) over OTLP into an OTel Collector and view in Grafana. Cloudflare offers this for free.

10. APM Comparison — Datadog vs New Relic vs Dynatrace

The commercial APM Big Three look much like they did a few years ago.

AspectDatadogNew RelicDynatrace
StrengthBreadth, UXPricing, AIDavis AI, auto-detect
Pricing35-70 USD per host0.30 USD per GB (perceived)DPS units (complex)
OTelFirst-classFirst-classFirst-class
Auto-instrVery strongVery strongStrongest (OneAgent)
K8sStrongStrongStrong
AI analysisBits AINew Relic AIDavis AI (the original)

Also-rans worth knowing:

The 2026 trend — fewer companies do single-vendor "all Datadog" lock-ins. More are hybrid: self-hosted Grafana LGTM for the bulk, with Datadog or New Relic only for select areas. Cost pressure is the main driver.

11. K8s Observability — Operator, k9s, Lens

Kubernetes observability is a full sub-category in its own right.

# ServiceMonitor example for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s

12. Service Mesh and Observability

A service mesh is, by virtue of having a sidecar intercept all traffic, essentially an automatic trace and metric generator.

Sidecar-less mesh — with Istio Ambient Mode and Cilium Service Mesh going GA in 2024-2025, the "node-level data plane" is now the standard. From an observability standpoint, the key insight is that even without sidecars you get the same data.

13. DevSecOps Meets Observability

Runtime security and observability share data, but ask different questions.

# Falco rule example — alert when a shell is spawned inside a container
- rule: Terminal shell in container
  desc: A shell was used as the entrypoint/exec point in a container
  condition: spawned_process and container and shell_procs
  output: "A shell was spawned in a container (user=%user.name container=%container.id)"
  priority: WARNING

14. Storage Backends — VictoriaMetrics, Mimir, ClickHouse, MinIO

The volume of observability data is explosive. Where and how long you store it is the central design question.

Retention guidelines:

Data typeHotWarmCold
Metrics15 days90 days13 months+
Logs7 days30 days1 year+ (S3 IA)
Traces3-7 days30 days90 days+
Profiles7 days30 daysusually discarded

15. Cost Model — SaaS vs Self-Host

Observability typically consumes 5-15% of infrastructure cost. At scale, 30% is not unusual.

SaaS pricing (2026 list prices):

Self-hosted LGTM (rough order):

Roughly: 100 hosts, 1 TB/day of logs, 100 GB/day of traces:

Rule of thumb — under 50 hosts, SaaS is almost always cheaper. Over 200 hosts, self-hosted is almost always cheaper. In between it depends on the company. Cost-pressured Korean and Japanese companies are crossing that threshold to self-hosting quickly.

16. Korean Adoption — NCsoft, Coupang, Naver, Kakao

Korea's big-tech observability story tends to follow this arc: Datadog adoption around 2020 → gradual move toward self-hosting from 2024.

The common pattern — Korean companies tend to (1) start with Datadog/New Relic for fast initial visibility, and (2) move the bulk to LGTM once cost crosses a threshold while keeping RUM/Sentry as SaaS.

17. Japanese Adoption — Mercari, LINE Yahoo, CyberAgent

Japan is generally considered one step ahead of Korea in adopting OpenTelemetry and eBPF.

Japanese SRE culture — the SRE community around Mercari and CyberAgent (SRE Lounge, the SRE NEXT conference) plays a major role in spreading observability best practices. That community is one reason Japan adopted OTel a beat ahead of Korea.

18. SLO/SLI and Error Budget Operations

The final destination of observability is SLOs (Service Level Objectives). Define "p99 response time is under 200 ms and availability is 99.9%", monitor it with metrics, and burn down an error budget when you miss.

Tools:

# Sloth SLO definition example
version: prometheus/v1
service: my-api
slos:
  - name: requests-availability
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{code=~"5.."}[5m]))
        total_query: sum(rate(http_requests_total[5m]))
    alerting:
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

19. Alerting and Incident Management

The final stage of observability is alerting and incident response.

The 2026 direction — Slack and Teams have become the standard incident room. Tools like Incident.io spin up a channel automatically, update the status page, and kick off the post-mortem. PagerDuty is evolving in the same direction.

20. The Shape of AI-Native Observability

The biggest topic of 2026 is how AI changes observability.

Observability for AI workloads — LLMs are non-deterministic and expensive. "Did this user request go to GPT-4o or Claude 3.5? How many tokens did it use? Was the query cached?" — these are the new golden signals.

21. Adoption Roadmap — Where to Start

If you were rebuilding the stack from scratch in 2026, this is roughly the recommended order:

  1. Metrics first — Prometheus + Grafana. node-exporter, kube-state-metrics
  2. Standardise logs — Loki or Elastic. Unify shipping with Vector or Fluent Bit
  3. Adopt traces — Tempo + OTel Collector. Start with auto-instrumentation
  4. Fill 80% visibility with eBPF — Cilium Hubble or Pixie, pick one
  5. Add continuous profiling — Pyroscope, integrated into Grafana
  6. Define SLOs — three to five core services in YAML via Sloth
  7. AI/LLM tracing — if you use LLMs, add LangSmith or Phoenix to the OTel pipeline
  8. Automate incident management — Incident.io or PagerDuty, Slack-integrated

Startup (1-20 people):

Mid-size (20-200 people):

Large (200+ people):

22. References

Read the original: https://www.youngju.dev/blog/culture/2026-05-16-network-service-observability-2026-ebpf-cilium-hubble-pixie-pyroscope-grafana-loki-tempo-netdata-deep-dive.en

Comments

No comments yet.

Sign in to leave a comment