Tag: #reliability
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 26 posts
A Sense for Production — What Looks Different After You Have Been Through an Outage
Two people read the same design doc and one of them reads something else entirely: what logs exist when this breaks at three in the morning, how many minutes a rollback takes, and what happens to the data that already la
2026-08-15 · 7 min read #career#skills#sre#production#reliabilityFollowing 657,607 Links, and the Lifespan of a URL — Why Links Die of Connection Failure, Not 404
A study published its results after following, in 2026, every one of roughly 655,000 shortened links created between 2009 and 2014. More important than the headline number is the composition of the failures: most dead li
2026-08-14 · 6 min read #web#data-engineering#archival#reliability#url-designRunning AI Agents in Production — Idempotency, Budgets, and Confidently Wrong Answers
Moving an agent from prototype to production exposes an operational surface you only discover late: idempotency for retried tool calls, budgets and step limits, observability over non-deterministic control flow, per-tool
2026-07-31 · 16 min read #ai#agents#observability#reliability#mcpChoosing a Rate Limiting Algorithm — What Really Separates Fixed Window, Sliding, and Token Bucket
Letting 200 requests through a 100-per-minute limit is not a bug. It is the defined behaviour of the fixed window algorithm. This post compares the memory and accuracy tradeoffs of four approaches — fixed window, sliding
2026-07-26 · 12 min read #web#rate-limiting#api-design#redis#reliabilityTemporal Worker Versioning GA — The Deploy Problem Replay Created, and the Third Answer After Two Discarded Generations
Durable execution engines like Temporal survive failures by replaying an event history — and it is precisely that replay which makes shipping new code while workflows are still running the hardest problem in this model.
2026-07-17 · 13 min read #distributed-systems#durable-execution#temporal#workflow-engine#reliabilityHow AI Agents Fail in Production — 14 Failure Modes, and Why Retrying Is Not Safe
Putting agents in production hurts in three places. First, failures come from system design, not the model — UC Berkeley's MAST study classified 1642 execution traces into 14 failure modes, and 44.2% of them were system-
2026-07-17 · 20 min read #ai#agents#observability#reliability#mcpPayment Idempotency: Preventing Double Charges
The network will eventually drop, and when it drops, clients retry. But in payments, a retry can become a double charge. Idempotency keys, dedup windows, unique constraints, the retry-plus-timeout problem, at-least-once
2026-07-02 · 14 min read #payments#idempotency#reliabilityThe Truth About Exactly-Once Semantics
What the three delivery guarantees (at-most-once, at-least-once, exactly-once) actually mean, why exactly-once delivery is fundamentally impossible while exactly-once processing is achievable through idempotency, dedupli
2026-06-28 · 13 min read #distributed-systems#messaging#reliabilityRate Limiting Algorithms, Explained: Fixed Window, Sliding Window, Token Bucket, and Leaky Bucket
A side-by-side tour of the main rate limiting algorithms — fixed window, sliding window log and counter, token bucket, and leaky bucket. How each handles bursts, how to implement rate limiting across a distributed fleet
2026-06-25 · 11 min read #systems#api#reliabilityIdempotency and Retries: Building Reliable APIs
The network will eventually fail, and when it does you have to retry. The hard case is the request that was "already processed but whose response got lost." This post covers what idempotency is, safe versus unsafe HTTP m
2026-06-21 · 12 min read #api#reliability#distributed-systemsMigration Incident Case Studies and Checklists — Learning from Others Failures
Migrations are among the most common and most painful sources of outages. This article walks through typical failure modes like lock storms, replication lag, data loss, and irreversible changes, derives causes and lesson
2026-06-16 · 14 min read #database#migration#postmortem#incident#reliabilityThe Expand-Contract Pattern — The Right Way to Change Schemas with Zero Downtime
A thorough walkthrough of the Expand-Contract pattern, the standard technique for changing a live database schema without downtime. We cover per-scenario SQL for adding columns, renaming, dropping, and adding constraints
2026-06-16 · 15 min read #database#migration#zero-downtime#expand-contract#reliabilityAn Introduction to Database Migration Strategy — Schema, Data, and Zero Downtime
A systematic overview of database migration types and risks, versioned migrations, and zero-downtime deployment principles. It covers transactional DDL, backups and dry runs, environment promotion, team process, and an i
2026-06-16 · 16 min read #database#migration#schema#devops#reliabilityCode That Fails Well — A Deep Dive into Error Handling and Resilience Design (2026)
Error handling is not decoration you bolt on after the feature is built — it is the design itself. Classify the kinds of failure, decide between exceptions and error values, validate at the edge and trust the core, put a
2026-05-14 · 25 min read #error-handling#resilience#retry#circuit-breaker#timeoutLoad/Performance Testing Tools 2026 — Deep Dive on k6, Locust, Vegeta, Gatling, Artillery, JMeter (Beyond JMeter)
Performance testing is still hard in 2026. But the tools have definitely gotten better. JS-scriptable k6, Python-friendly Locust, one-liner Vegeta, enterprise Gatling, YAML-quick Artillery, micro-benchmark wrk/wrk2 and a
2026-05-14 · 18 min read #load-testing#performance-testing#k6#locust#vegetaDurable Execution Engines in 2026 — A Deep Dive Comparison of Temporal, Restate, Inngest, Trigger.dev, DBOS: Escaping Cron and Retry Hell
Stop writing long-running workflows with cron, queues, and if-else. The answer in 2026 is Durable Execution. This deep dive compares Temporal, Restate, Inngest, Trigger.dev, and DBOS — plus AWS Step Functions, Cadence, a
2026-05-14 · 26 min read #durable-execution#temporal#restate#inngest#trigger-devAgent Evaluation Systems in 2026 — Inspect AI vs Promptfoo vs Phoenix vs LangSmith vs OpenAI Evals (You're Measuring the Agent, Not the Model)
LLM evals measure the model. Agent evals measure whether the model plus the harness plus the tools actually carry a task to completion. They are different problems. This is a map of the 2026 landscape — Inspect AI from U
2026-05-14 · 20 min read #agent-evaluation#inspect-ai#promptfoo#phoenix#langsmithAnatomy of Famous Postmortems — Learning From the Failures of Cloudflare, Fastly, AWS, Knight Capital, and GitLab
Cloudflare 2022 BGP, Fastly taking down half the internet in 2021, the AWS S3 typo of 2017, Knight Capital losing $440M in 8 minutes, the GitLab DB wipe, and more — the patterns and lessons extracted from famous failures
2026-04-15 · 12 min read #postmortem#sre#outage#reliability#cultureChaos Engineering Deep Dive — Netflix Simian Army, LitmusChaos/Chaos Mesh, AWS FIS, Game Day
Why Netflix started randomly killing production servers in 2010. From Chaos Monkey philosophy and the 4 principles, to the full Simian Army, LitmusChaos/Chaos Mesh/AWS FIS comparison, Game Day design, and blameless postm
2026-04-15 · 10 min read #chaos-engineering#sre#reliability#netflix#kubernetesAWS Well-Architected Framework Complete Guide 2025: Six Pillars, Practical Adoption, Cost/Security/Performance
Everything about the AWS Well-Architected Framework: six pillars (Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, Sustainability), practical checklists, Well-Architected Tool usa
2026-04-15 · 14 min read #aws#well-architected#cloud-architecture#security#reliability