Tag: #observability
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 113 posts
Why Average Response Time Lies — How to Read p50, p95, and p99 Properly
A service whose average response time is 80ms can still keep users waiting three seconds, and that happens all the time. This post explains what the average hides in a long-tail distribution, which question each of p50,
2026-07-26 · 11 min read #observability#percentile#prometheus#histogram#sloAlerting That Does Not Wake You at 3AM — Symptom-Based Alerts and Burn Rate in Practice
On-call collapses not because there are too few alerts but because there are far too many. This post starts with the mechanism by which alert fatigue makes you miss real incidents, then covers the principle of paging onl
2026-07-26 · 13 min read #observability#alerting#slo#error-budget#prometheusDesigning Structured Logging — Logs That Actually Help During an Incident
If you have ever given up while grepping logs in the middle of an incident, the log design is wrong. This post covers how to separate logs humans read from logs machines read, which fields must appear on every single lin
2026-07-26 · 12 min read #observability#logging#structured-logging#opentelemetry#incident-responseThe Questions Distributed Tracing Actually Answers — Spans, Sampling, and Where the Time Went
Tracing is what you need when the reports of slowness keep coming in but you have no idea which of ten services is the culprit. Starting from the structure of traces, spans, and context propagation, this post lays out wh
2026-07-26 · 15 min read #observability#distributed-tracing#opentelemetry#tail-sampling#performanceHow AI Agents Fail in Production — 14 Failure Modes, and Why Retrying Is Not Safe
Putting agents in production hurts in three places. First, failures come from system design, not the model — UC Berkeley's MAST study classified 1642 execution traces into 14 failure modes, and 44.2% of them were system-
2026-07-17 · 20 min read #ai#agents#observability#reliability#mcpOTel's Kubernetes Attributes Are Stable Now — What to Do Before k8sattributes Flips Its Default
OpenTelemetry's Kubernetes semantic conventions were promoted to stable in semconv v1.42.0 on June 12, 2026. But the Collector's k8sattributes processor still defaults to the old schema (v0), so most people haven't notic
2026-07-16 · 12 min read #opentelemetry#observability#kubernetes#semantic-conventions#telemetry-pipelineNative Histograms Are Stable Now — So Why Haven't You Turned Them On Yet
Prometheus native histograms showed up as an experimental feature in v2.40 (November 2022), went stable in 3.8.0 (December 2025), and 3.13, released July 1, 2026, is the first LTS to ship the feature in stable state. The
2026-07-16 · 17 min read #prometheus#observability#native-histograms#metrics#monitoringObservability Deep Dive: Logs, Tracing, and LLM Monitoring
From the three pillars — logs, metrics, and traces — and how they correlate through traceid, to the difference between Loki and OpenSearch for logs, OpenTelemetry-centered distributed tracing (Jaeger and Tempo), and the
2026-07-03 · 14 min read #observability#opentelemetry#tracing#logging#llmReading Logs and Stack Traces Like a Detective
A stack trace is a crime scene; logs are witness statements. How to read a stack trace top to bottom (the top is where it blew up, the bottom is the origin), follow the "caused by" chain, use structured/JSON logging, cor
2026-06-28 · 13 min read #logging#debugging#stack-trace#observabilityOperator Observability — Metrics, Events, Logging, and SLOs
A deep dive into observability for Kubebuilder-based Operators across metrics, events, logging, tracing, status/conditions, and SLOs. From the meaning of controller-runtime default metrics to adding custom metrics, wirin
2026-06-15 · 19 min read #kubernetes#operator#observability#prometheus#controller-runtimeIngress Observability — Metrics, Access Logs, and Tracing
Define the golden signals for an Ingress controller and build a practical observability stack: ingress-nginx Prometheus metrics, Grafana dashboards, structured access logs shipped to Loki, OpenTelemetry distributed traci
2026-06-14 · 11 min read #ingress#kubernetes#nginx#observability#networkingeBPF Fundamentals — Programs, Maps, and the World of the Verifier
A ground-up tour of how eBPF changed Linux kernel programming. We cover program types and maps, how the verifier works, and build a first eBPF program with libbpf and CO-RE from scratch.
2026-06-13 · 19 min read #ebpf#linux#kernel#libbpf#observabilityeBPF Observability in Practice — Opening the Black Box with bpftrace
Ten battle-tested bpftrace one-liners and a tour of the BCC toolbox for inspecting live systems without touching application code. Covers latency histogram interpretation, flame graphs, container environments, and three
2026-06-13 · 15 min read #ebpf#bpftrace#observability#linux#profilingHubble and ClusterMesh — Cilium Observability and Multi-Cluster Operations
Covers the architecture of Hubble, the observability layer of Cilium, with practical CLI queries, Prometheus metrics, and long-term flow log retention, then walks through configuring ClusterMesh for multi-cluster service
2026-06-13 · 13 min read #cilium#hubble#clustermesh#observability#multi-clusterKeycloak Observability — Metrics, Audit Logs, and Event-Driven Monitoring
A comprehensive guide to Keycloak observability: the event system and metrics endpoint, shipping events externally with the EventListener SPI, building Prometheus and Grafana dashboards, OpenTelemetry tracing, anomaly de
2026-06-12 · 13 min read #keycloak#observability#monitoring#prometheus#securityeBPF Is Eating Observability
The idea of safely running sandboxed programs inside the kernel is upending observability. Hook anywhere with kprobes, uprobes, tracepoints, and XDP; the verifier guarantees safety; trace without touching a line of appli
2026-06-11 · 13 min read #ebpf#observability#linux#kernelTime Series Databases 2026 Deep Dive - InfluxDB 3 · TimescaleDB · QuestDB · ClickHouse · Prometheus · VictoriaMetrics · Grafana Mimir
A full-stack 2026 dissection of time-series databases. InfluxDB 3.0 Rust + Apache Arrow + DataFusion + Parquet rewrite, TimescaleDB 2.18 hypertables, QuestDB 8.x SIMD ingest, ClickHouse 25.x MergeTree, Prometheus 3.0 and
2026-05-16 · 26 min read #time-series-database#influxdb#timescaledb#questdb#clickhouseSoftware Debugging Tools 2026 Deep Dive — gdb, lldb, rr, Pernosco, Replay.io, ASan, Valgrind, eBPF, perf, Time-Travel Debugging
A full map of the 2026 debugging landscape in one read. From the undefeated reign of printf to interactive debugging with gdb/lldb, time-travel and omniscient debugging via rr/Pernosco/Replay.io, the ASan-vs-Valgrind tra
2026-05-16 · 25 min read #debugging#gdb#lldb#rr#pernoscoObservability 2026 Complete Guide — OpenTelemetry, Datadog, Grafana Stack (LGTM+Beyla), Honeycomb, Prometheus, Jaeger, eBPF & SLO Deep Dive
A thorough look at the 2026 production observability stack on top of the OpenTelemetry standard. Covers SaaS vendors (Datadog, New Relic, Dynatrace, Splunk, Honeycomb, Coralogix, Logz.io, Chronosphere), open-source stack
2026-05-16 · 18 min read #observability#opentelemetry#datadog#grafana#honeycombNetwork & Service Observability 2026 Deep Dive — eBPF · Cilium Hubble · Pixie · Pyroscope · Grafana Loki + Tempo + Mimir · Netdata · OpenTelemetry
In 2026, observability rests on four pillars (metrics, logs, traces, profiles) and eBPF auto-instrumentation is now the default. Cilium Hubble, Pixie, Pyroscope, the Grafana LGTM stack, Netdata, and OpenTelemetry — a sin
2026-05-16 · 21 min read #observability#ebpf#cilium-hubble#pixie#pyroscope