Tag: #observability
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 113 posts
Self-Hosting Langfuse — Deployment Paths, Secrets, and What Catches You on First Boot
Running Langfuse yourself means standing up two containers and four datastores at once. Working from the official documentation, this post lays out the docker compose path and the Helm chart path, which secrets you have
2026-08-14 · 9 min read #observability#langfuse#self-hosting#docker-compose#kubernetesWhen Traces Become Cost — Retention, Sampling, and Masking in Langfuse
Tracing feels free at low traffic and then comes back as storage cost and a privacy problem. This post splits the points where volume turns into cost into four, confirms from the official documentation that sampling deci
2026-08-14 · 9 min read #observability#langfuse#cost#sampling#data-retentionLangfuse SDK Instrumentation — What Is Captured Automatically and What You Add by Hand
Creating traces is not about installing an SDK, it is about drawing boundaries. Working from the Langfuse Python SDK v4, this post lays out what the decorator, the context manager, and manual observation creation each do
2026-08-14 · 8 min read #observability#langfuse#llm-tracing#instrumentation#opentelemetryWhy Langfuse Puts Traces in ClickHouse — How the Storage Layer Splits the Work
The first surprise when you self-host Langfuse is that there is not one datastore but four. ClickHouse, Postgres, Redis, and object storage each take a different share. Working from the official documentation, this post
2026-08-14 · 10 min read #observability#langfuse#clickhouse#architecture#postgresThe Plugin Kernel Architecture of DeepSeek Harness — What Makes an Agent You Can Rewind
Harness, the open source coding agent DeepSeek released, makes the model, the tools, and even the UI into plugins. But the interesting part is not the word plugin. It is two design decisions: representing every execution
2026-08-14 · 7 min read #ai-agent#architecture#plugin-system#observability#open-sourceThe Langfuse Tracing Data Model — How Trace, Observation, and Score Hold One Execution
Looking at the Langfuse UI first tells you nothing about what you are looking at. This post starts from the shape of the data Langfuse collects. What a trace groups, what separates a span from a generation from an event,
2026-08-14 · 8 min read #observability#langfuse#llm-tracing#data-modeling#opentelemetryAfter You Have Collected It — Langfuse Dashboards, the Metrics API, and Scores on Traces
Collecting traces and getting answers out of them are different jobs. This post lays out which metric axes Langfuse offers and which dimensions you have to slice them by, then walks the Metrics API v2 query structure as
2026-08-14 · 9 min read #observability#langfuse#llm-evaluation#metrics-api#dashboardvLLM Metrics — What to Chart and What to Alert On
The series vLLM exposes answer questions GPU metrics cannot: how many requests are running versus waiting right now, how full the KV cache is, how long until the first token. This post reads the official vLLM documentati
2026-08-12 · 7 min read #gpu#kubernetes#vllm#prometheus#observabilityDCGM Exporter — GPU Utilization Is Not What You Think It Is
DCGM Exporter is the standard path for exposing GPU telemetry in Prometheus format, but the utilization-style metric that ends up on nearly every dashboard does not measure what people expect. This post reads the default
2026-08-12 · 9 min read #gpu#kubernetes#dcgm#prometheus#observabilityNVIDIA GPU Operator — The Six Pieces You Used to Install by Hand
Running GPUs on Kubernetes used to mean matching six pieces on every node by hand: the driver, the NVIDIA Container Toolkit, the device plugin, DCGM, GPU Feature Discovery, and Node Feature Discovery. The NVIDIA GPU Oper
2026-08-12 · 6 min read #gpu#kubernetes#gpu-operator#nvidia#dcgmOpen Source Worth Watching Right Now (4) Observability and Security
With observability data, volume is cost, and a security tool that never makes it into the pipeline never gets used. This post introduces 12 open source projects that have genuinely taken hold — instrumentation standards,
2026-08-12 · 5 min read #open-source#observability#security#opentelemetry#ebpfGPU Serving SLOs and Alert Design — What to Promise and What to Page On
Putting an SLO on a GPU inference service starts with deciding which metric represents the user experience. Time to first token and throughput eat each other, so setting a target on only one guarantees the other collapse
2026-08-12 · 7 min read #gpu#kubernetes#slo#alerting#prometheusVisitor Analytics Shows Only 0.5 Percent of Your Traffic — Judge Bots by Origin, Not by Self-Report
Drawing on one year of defending a 1.5-million-page site against scrapers, this post sets out the principles of dealing with bot traffic. JavaScript-based analytics cannot count bots, so you have to read server logs, and
2026-08-09 · 11 min read #network#bot#cloudflare#waf#scrapingPutting Observability Data Into ClickHouse — Schema, Rollups, TTL, and Splitting the Work
When traces and logs grow to multiple terabytes a day, a single search engine or time-series database starts to buckle. This post lays out why ClickHouse fits observability data so well, in terms of columnar storage, com
2026-08-02 · 16 min read #observability#clickhouse#opentelemetry#data-modeling#costDashboards That Get Read and Alerts Worth Paging — Defining Questions, Variables, SLOs, and Alert Fatigue
A dashboard is not useful for looking pretty — it earns its keep by answering a fixed set of questions in order. This post covers how to write down the questions a panel should answer before building it, and how to struc
2026-08-02 · 15 min read #observability#grafana#alerting#slo#dashboardsDesigning Prometheus Metrics That Answer Questions — Choosing Types, Cardinality Budgets, and the Traps in rate and Quantiles
More metrics is not better — a metric earns its keep only by answering a question. This post starts by laying out which questions counters, gauges, and histograms each answer, and which calculations become impossible in
2026-08-02 · 17 min read #observability#prometheus#promql#metrics#cardinalityMaking Logs Searchable, and Not Going Broke Doing It — Structuring, Mapping Explosions, Retention, and Real Cost
The point where log costs overtake compute costs arrives for most organizations. What delays that point isn't the compression ratio — it's the decision about what becomes a field. This post covers field design for struct
2026-08-02 · 15 min read #observability#logging#opensearch#elasticsearch#costInstrumenting Your App With OpenTelemetry — From Auto-Instrumentation to Manual Spans, and Why You Put a Collector in Front
Instrumentation isn't about installing an SDK — it's about following an order of operations. This post builds a skeleton in a day with auto-instrumentation, locks down resource attributes first, and shows the order for a
2026-08-02 · 16 min read #observability#opentelemetry#instrumentation#otel-collector#tracingThe Mean Tells You Nothing — How to Debug Latency Through Distributions
Farid Zakaria's post "The mean means nothing," published July 27, 2026 and reaching the top of Hacker News, covers a case where average latency got 9% worse after a caching layer rollout — from 112ms to 122ms. In the ver
2026-07-31 · 14 min read #observability#latency#performance#histogram#percentileRunning AI Agents in Production — Idempotency, Budgets, and Confidently Wrong Answers
Moving an agent from prototype to production exposes an operational surface you only discover late: idempotency for retried tool calls, budgets and step limits, observability over non-deterministic control flow, per-tool
2026-07-31 · 16 min read #ai#agents#observability#reliability#mcp