Distributed Systems Complete Guide — Clocks, Consensus, Event Sourcing, Saga, CRDT, Failure Patterns (Season 2 Ep 12, 2025)

Intro — The Real Reason Distributed Systems Are Hard

Butler Lampson gave the famous definition:

"A distributed system is one in which the failure of a computer you did not even know existed can render your own computer unusable."

Distributed systems are hard because these three things happen at the same time:

1. The clocks disagree (Clock Drift)
2. Failures are partial (Partial Failure)
3. The network must be distrusted (Unreliable Network)

This post pulls together, in one place, the theoretical tools (clocks, consensus, consistency) and the practical patterns (Event Sourcing, Saga, CRDT) for those three problems.

Part 1 — 8 Fallacies of Distributed Computing

The eight false assumptions about distributed systems that Peter Deutsch catalogued in 1994:

1. The network is reliable ❌
2. Latency is zero ❌
3. Bandwidth is infinite ❌
4. The network is secure ❌
5. Topology does not change ❌
6. There is one administrator ❌
7. Transport cost is zero ❌
8. The network is homogeneous ❌

Every assumption that violates one of these eight eventually blows up in production.

Part 2 — Three Models of Time

2.1 Physical Clock

time.Now() — the OS clock. Synchronized over NTP.

Problems:

• Clock drift (error measured in ppm)
• NTP jumps (time skips forward when it is adjusted)
• Leap seconds
• Error of milliseconds to tens of milliseconds between data centers

2.2 Logical Clock

Lamport Timestamp (1978):

On an event: counter++
On send: transmit (event, counter)
On receive: counter = max(local, received) + 1

Upside: causality is preserved (happens-before) Downside: you cannot tell whether two events were causally related at all (concurrent vs causal)

2.3 Vector Clock

A vector as long as the node count N:

A: [2, 1, 0]
B: [1, 3, 0]

• You can decide whether A and B are concurrent, or which came first
• Size problem: with many nodes the metadata grows

2.4 HLC (Hybrid Logical Clock, 2014)

A combination of physical time plus a logical counter:

HLC = (physical_time, logical_counter)

• Ordering anchored to the physical clock, causality preserved by the logical clock
• The standard in modern distributed DBs such as CockroachDB and YugabyteDB

2.5 TrueTime (Google Spanner)

GPS and atomic clocks guarantee ±7ms → which makes "any event after this instant is definitely later" possible.

• Commit Wait: wait out the uncertainty window (in ms) before completing
• Strong consistency plus performance (Linearizable)
• An ordinary company cannot build this (hardware investment)

Part 3 — The Consistency Model Spectrum

3.1 The Main Consistency Models

In order from strongest to weakest:

1. Linearizable: every operation appears in real-time order
2. Sequential: every process sees the same order, but it is not real-time order
3. Causal: ordering is guaranteed only for causally related operations
4. Eventual: they converge in the end

3.2 CAP Revisited

• Consistency: linearizability
• Availability: every request gets an answer
• Partition Tolerance: it keeps working through a network partition

Network partitions cannot be avoided → so you pick CP or AP.

3.3 PACELC (More Realistic)

• Partition: C or A
• Else (normal operation): L(atency) or C(onsistency)

Most systems are PA/EL (availability under partition, low latency when healthy).

3.4 Choosing DB Consistency in 2025

Part 4 — Consensus: Agreement Algorithms

4.1 Why Consensus

Several distributed nodes agreeing on the same value — leader election, state replication, transaction commit, and so on.

4.2 FLP Impossibility (1985)

In an asynchronous system, deterministic consensus that tolerates even a single node failure is impossible.

Real systems therefore rely on timing assumptions (eventually synchronous) or on probabilistic methods.

4.3 Paxos (1989)

The original, created by Leslie Lamport. Correct, but notorious for being hard to understand and implement.

Roles:

• Proposer (proposes a value)
• Acceptor (votes)
• Learner (learns the decision)

Problem: hard to use in practice, hence variants such as Multi-Paxos, Fast Paxos, and EPaxos.

4.4 Raft (2014)

Designed at Stanford with "understandable consensus" as the goal. Today it is the de facto standard.

Three essentials:

1. Leader Election: pick a single leader
2. Log Replication: the leader replicates its log to the followers
3. Safety: a committed log entry is never overturned

States:

• Follower → Candidate (Election Timeout) → Leader (majority vote)
• When the leader fails to heartbeat → a new election

Where it is used: etcd, Consul, TiKV, CockroachDB, Kafka KRaft (2024 onward).

4.5 PBFT (Practical Byzantine Fault Tolerance, 1999)

With malicious nodes present, f failures are tolerated among 3f+1 nodes. Used in blockchains.

Phases: Pre-prepare → Prepare → Commit. Message complexity O(N²).

4.6 Tendermint and HotStuff (2018 onward)

BFT modernized for blockchains. HotStuff from Libra/Diem, Tendermint from Cosmos.

4.7 Nakamoto Consensus

Bitcoin's Proof-of-Work. Probabilistic agreement — once enough blocks sit on top of it, it counts as confirmed.

Downsides: energy, and slowness. Which is why Ethereum moved to PoS.

Part 5 — Replication

5.1 Four Replication Patterns

1. Single-Leader: writes go to the leader, reads from anywhere. Simple. Most RDBMSs.
2. Multi-Leader: several leaders accept writes. Conflict resolution required. Rarely used.
3. Leaderless: every node is equal. Dynamo-style. Cassandra, Riak, DynamoDB.
4. Quorum-based: strong consistency when R + W > N.

5.2 Replication Lag

Propagation delay from leader to follower causes these problems:

• Read-your-writes: you cannot read what you just wrote
• Monotonic reads: something you saw earlier disappears
• Consistent Prefix: you see an effect without its cause

Fixes: read from the leader, sticky sessions, timestamp-based consistency.

5.3 Conflict Resolution (Multi-Leader/Leaderless)

• LWW (Last Writer Wins): simple, risks losing data
• Version Vectors: accurate, complex
• Application-level Merge: user defined (CRDT)

Part 6 — CRDT (Conflict-free Replicated Data Type)

6.1 What a CRDT Is

A data structure in which conflict is mathematically impossible. Merge in any order and you get the same result.

6.2 Two Flavors

State-based (CvRDT): ship the state itself, then merge Operation-based (CmRDT): ship the operation, then re-apply it

6.3 Representative CRDTs

6.4 Where They Are Used

• Redis CRDT (Enterprise): Active-Active Geo
• Riak: Dynamo plus CRDT
• Automerge (Ink & Switch): collaborative editors
• Yjs: real-time collaboration (the inspiration behind Notion and Linear)
• Figma: its own CRDT for design collaboration

6.5 Limits

• Hard to express arbitrary invariants (uniqueness, for example)
• Metadata size (tombstones)
• The meaning of operation order is lost (which is both the strength and the weakness)

Part 7 — Event Sourcing + CQRS

7.1 Event Sourcing

Store events instead of state. State is derived by replaying the events.

events = [
  AccountCreated(id=1, balance=0),
  MoneyDeposited(account=1, amount=100),
  MoneyWithdrawn(account=1, amount=30),
]

balance = fold(events, 0, apply) // 70

Upsides:

• A complete history (audit and debugging)
• Time travel
• New views built by replaying the events
• Distribution friendly (events are append-only)

Downsides:

• Complexity ↑
• Schema evolution problems (interpreting old events)
• Replay is slow without snapshots

7.2 CQRS (Command Query Responsibility Segregation)

Separate the write model (Command) from the read model (Query).

Write: events → EventStore
  ↓
Projection
  ↓
Read: Materialized View (optimized)

Upside: reads and writes are optimized and scaled independently. Downside: consistency lag (eventual).

7.3 The Event Sourcing + CQRS Stack

• EventStoreDB: a purpose-built DB
• Kafka + Kafka Streams: event log plus projection
• Axon Framework (Java): all-in-one
• MartenDB (.NET): implemented on top of Postgres

7.4 When to Use It and When to Avoid It

Good when:

• Auditability matters, as in finance and trading
• Complex business logic plus reconstruction of the past
• Event-driven architecture

Avoid when:

• Plain CRUD
• The team is mostly beginners
• The consistency requirements are simple

Part 8 — The Saga Pattern

8.1 The Distributed Transaction Problem

2PC (Two-Phase Commit): flawless in theory, but it blocks and is fragile when the coordinator fails. Barely used today.

The alternative: Saga — a chain of local transactions plus compensating transactions on failure.

8.2 Choreography

Each service publishes and subscribes to events:

OrderCreated → InventoryReserved → PaymentCharged → OrderConfirmed
                  ↓ failure
              InventoryReleased → OrderCancelled (compensation)

Upside: low coupling between services. Downside: the flow is hard to follow.

8.3 Orchestration

A central orchestrator manages the flow explicitly:

Orchestrator:
  1. ReserveInventory
  2. ChargePayment
  3. ShipOrder
  on failure ← CompensateAll

Upside: the flow is explicit and easy to debug. Downside: the orchestrator can become a single point of failure.

8.4 Implementation Tools

• Temporal: a workflow engine. The de facto standard for Sagas.
• AWS Step Functions: managed
• Camunda: BPMN based
• Cadence (Uber, the predecessor of Temporal): open source

8.5 A Temporal Example

func ProcessOrder(ctx workflow.Context, order Order) error {
    err := workflow.ExecuteActivity(ctx, ReserveInventory, order).Get(ctx, nil)
    if err != nil { return err }
    
    err = workflow.ExecuteActivity(ctx, ChargePayment, order).Get(ctx, nil)
    if err != nil {
        workflow.ExecuteActivity(ctx, ReleaseInventory, order)
        return err
    }
    // ...
}

Temporal hands you restart safety, timers, and visualization for free.

Part 9 — Twelve Failure Patterns

9.1 Failure Propagation Patterns

1. Cascading Failure: one service slows down → timeouts pile up in its callers → it spreads
2. Thundering Herd: when a cache expires, every request stampedes to the origin
3. Retry Storm: failed requests are amplified by retries
4. Metastable Failure: a temporary problem persists even after conditions return to normal
5. Split-Brain: a network partition produces two leaders
6. Clock Skew Bug: clock differences invert timestamps
7. Gray Failure: not dead, but degraded
8. Poison Pill: one particular message takes down every consumer
9. Queue Backlog: consumers are too slow and the queue grows without bound
10. Hot Partition: traffic concentrates on one shard
11. Fan-out Overload: a single request turns into 100 internal requests
12. Noisy Neighbor: one tenant on a shared resource affects everyone

9.2 Defensive Patterns

9.3 Chaos Engineering

Inject failure deliberately → validate the system.

• Chaos Monkey (Netflix, 2011): kills instances at random
• Chaos Mesh: K8s native
• LitmusChaos: open source
• Gremlin: commercial

Principles:

• In production (staging at first)
• Limit the blast radius
• Automatic rollback
• Run it on a schedule (Game Day)

Part 10 — Ten Distributed Systems Patterns

1. Idempotency: the same request N times = the effect of one
2. Exactly-once is a lie: At-least-once plus an idempotent consumer
3. Outbox Pattern: atomicity between the DB transaction and event publication
4. Inbox Pattern: how you implement an idempotent consumer
5. Circuit Breaker: detect failure and route around it
6. Bulkhead: thread-pool isolation to stop cascading failure
7. Leader Election: lease based
8. Sharding: split the data
9. Sidecar: common concerns (logging, security) in a separate container
10. Service Mesh: the sidecar promoted to a network layer

Part 11 — A Six-Month Distributed Systems Roadmap

Month 1: Theoretical Foundations

• Read DDIA (Designing Data-Intensive Applications) properly
• Two Lamport papers
• Thought experiments on the 8 Fallacies

Month 2: Consensus Practice

• Implement Raft yourself (MIT 6.824)
• Read the internals of etcd and Consul
• Understand the Kafka KRaft architecture

Month 3: Replication + Consistency

• Practice telling the consistency models apart
• The CockroachDB and Spanner papers
• Implement TrueTime and HLC

Month 4: Event Sourcing + Saga

• Build a mini banking system with Event Sourcing
• Implement a Saga with Temporal
• Apply the Outbox pattern for real

Month 5: CRDT + Real Time

• Study Automerge and Yjs
• Prototype a collaborative editor
• Read the Figma engineering blog closely

Month 6: Failures + Operations

• Adopt Chaos Monkey
• Implement defenses for the major failure patterns
• Analyze 10 incident postmortems

Part 12 — A 12-Point Distributed Systems Checklist

1. You can recite the 8 Fallacies
2. You know the difference between Lamport vs Vector vs HLC
3. You know the difference between CAP vs PACELC
4. You can explain the three essentials of Raft
5. You know what FLP Impossibility means
6. You know the trade-offs of Single-Leader vs Leaderless
7. You know the problem CRDTs solve
8. You know the pros and cons of Event Sourcing + CQRS
9. You know the difference between Saga Choreography vs Orchestration
10. You can name three defenses against cascading failure
11. You know the role of the Outbox and Inbox patterns
12. You know the principles of Chaos Engineering

Part 13 — Ten Distributed Systems Antipatterns

1. "The network is reliable": fallacy number 1. Timeouts and retries are mandatory
2. 2PC in microservices: risk of blocking. Use a Saga
3. Assuming event order across Kafka partitions: order is guaranteed only within a partition
4. Exactly-once without a DB transaction: impossible. Use the Outbox, or idempotency
5. Trusting the clock: ordering by time.Now() → bugs. A logical clock is mandatory
6. Unbounded retries: causes a retry storm. Backoff plus Circuit Breaker
7. Assuming a single-node DB plus replication is safe: you must account for split-brain
8. Blocking instead of NACKing when a consumer fails: a DLQ design is mandatory
9. Global transactions: performance and availability ↓. Redesign the boundaries
10. "Eventual consistency converges in the end": from the user's point of view it is still a problem. Solve it in the UX

Closing — A Distributed System Is "a Model in Your Head"

Distributed systems are the field where intuition betrays you most often. The engineer with "powerful intuition" is the most dangerous one.

What you need:

• A suspicious mind: distrust the network, the clocks, and the nodes alike
• A compensating mind: if atomicity is impossible, design the compensation
• A reproducing mind: record it as events → reproduce it whenever you like

For a senior engineer in 2025, distributed systems skill is the axis that doubles your pay. This field is that hard, and that is exactly why it is that valuable.

Next Post — "Database Complete Guide: Internals, Indexes, Query Planner, Partitioning, Vector DB"

Season 2 Ep 13 is the heart of the data: databases in depth. The next post covers:

• The internals of B-Tree, LSM-Tree, and Hash Index
• Reading the query planner and EXPLAIN deeply
• Transaction isolation levels (Read Uncommitted through Serializable)
• Sharding and partitioning strategy
• The dominance of PostgreSQL (2025)
• Vector DB, with pgvector, Qdrant, and Weaviate

How a DB actually processes a query, in the next post.