LabHub

Blog

System Design Fundamentals — Scalability, Availability, Caching, and Message Queues

2026-04-12 · 17 min read · #architecture#system-design#scalability#caching#message-queue#database#2026-04

한국어English日本語

Table of Contents

  1. What Is System Design
  2. Scalability
  3. Load Balancing
  4. Caching Strategies
  5. CDN
  6. Databases
  7. CAP Theorem
  8. Message Queues
  9. API Design
  10. Monitoring and Logging
  11. Practical Example: URL Shortener
  12. Practical Example: Chat System

What Is System Design

System design is the process of defining the architecture of large-scale software systems. It goes beyond writing code, dealing with how to build scalable and reliable systems that serve millions of users.

Why It Matters

Interview Approach

There is no single correct answer in a system design interview. What matters is your thought process.

  1. Clarify requirements: Identify functional and non-functional requirements
  2. Estimate scale: Calculate user count, QPS, and storage needs
  3. High-level design: Place the core components
  4. Detailed design: Identify and address bottlenecks
  5. Discuss tradeoffs: Every decision has pros and cons

Scalability

Scalability is a system's ability to handle increasing load.

Vertical Scaling (Scale Up)

Adding more powerful CPUs, more RAM, or larger disks to a single server.

Pros:

Cons:

Horizontal Scaling (Scale Out)

Adding more servers to distribute the load.

Pros:

Cons:

Stateless Design

The key to horizontal scaling is stateless design. When servers hold no state, any server can handle any request.

Stateful server:
  User A -> Server 1 (session stored)
  User A -> Server 2 (no session -> error!)

Stateless server:
  User A -> Server 1 (looks up session in external store)
  User A -> Server 2 (looks up session in external store -> works)

State such as sessions and caches is stored in external systems like Redis or Memcached.

Load Balancing

A load balancer distributes incoming traffic across multiple servers.

L4 vs L7 Load Balancing

L4 (Transport Layer):

L7 (Application Layer):

Load Balancing Algorithms

Round Robin: Distributes requests sequentially to each server. Simple but does not account for differences in server capacity.

Weighted Round Robin: Assigns weights to servers, sending more requests to higher-capacity machines.

IP Hash: Hashes the client IP to always route to the same server. Useful when session persistence is needed, but requires redistribution when servers are added or removed.

Least Connections: Sends requests to the server with the fewest active connections. Effective when request processing times vary.

Round Robin example:

  Request 1 -> Server A
  Request 2 -> Server B
  Request 3 -> Server C
  Request 4 -> Server A  (cycles back)
  Request 5 -> Server B  (cycles back)

Health Checks

The load balancer periodically checks server health. Servers that fail to respond or return errors are removed from the rotation.

Load Balancer:
  /health -> Server A: 200 OK (healthy)
  /health -> Server B: 503 Error (unhealthy -> removed)
  /health -> Server C: 200 OK (healthy)

  Traffic goes to Server A and C only

Caching Strategies

Caching stores frequently accessed data in a fast storage layer to reduce response times.

Cache Aside (Lazy Loading)

The most widely used pattern.

  1. The application checks the cache first
  2. On a cache hit, it returns the cached data
  3. On a cache miss, it queries the database, stores the result in the cache, and returns it
Read flow:
  Client -> Check cache
    Hit -> Return from cache
    Miss -> Query DB -> Store in cache -> Return

Pros: Only caches data that is actually requested. Falls back to the database if the cache fails. Cons: First request always incurs a cache miss. Stale data is possible.

Write Through

Writes data to both the cache and the database simultaneously.

Write flow:
  Client -> Update cache -> Update DB -> Response

Pros: Cache and database stay consistent. Cons: Increased write latency. Caches data that may never be read.

Write Behind (Write Back)

Writes to the cache first and asynchronously persists to the database later.

Write flow:
  Client -> Update cache -> Response (immediate)
  Background: Cache -> DB sync (delayed)

Pros: Very fast writes. Reduces database load. Cons: Risk of data loss if the cache fails.

TTL and Cache Invalidation

TTL (Time To Live): Sets an expiration time on cache entries. Too short reduces cache efficiency; too long causes stale data.

Invalidation strategies:

TTL examples:
  SET user:123 "Kim" EX 3600        (expires in 1 hour)
  SET product:456 "Laptop" EX 86400 (expires in 24 hours)

Cache Stampede (Thundering Herd)

When a popular cache entry expires, a flood of requests simultaneously hits the database.

Solutions:

CDN

A CDN (Content Delivery Network) serves content from edge servers distributed around the world, delivering data from the location closest to the user.

How It Works

User (Seoul) -> Seoul edge server (cache hit) -> Instant response
User (Seoul) -> Seoul edge server (cache miss) -> Origin server -> Cache at edge -> Response

Static Asset Delivery

CDNs are primarily used for serving static content:

Optimizing Cache Hit Ratio

A higher cache hit ratio means better CDN efficiency.

Optimization techniques:

Pull CDN vs Push CDN

Pull CDN: Fetches content from the origin on the first request and caches it. Suitable for high-traffic sites.

Push CDN: Content is pre-uploaded to the CDN. Suitable when content rarely changes.

Databases

SQL vs NoSQL

SQL (Relational Databases):

NoSQL (Non-Relational Databases):

Selection criteria:

CriterionSQLNoSQL
Data structureStructuredFlexible
ScalingPrimarily verticalPrimarily horizontal
ConsistencyStrong consistencyEventual consistency (typically)
TransactionsACIDBASE
QueriesComplex JOINs possibleOptimized for simple key-value lookups

Sharding

Distributing data across multiple database instances.

Hash-based sharding: Determines the shard by hashing the key. Achieves even distribution but requires expensive redistribution when shards are added or removed.

user_id % 4 = 0 -> Shard A
user_id % 4 = 1 -> Shard B
user_id % 4 = 2 -> Shard C
user_id % 4 = 3 -> Shard D

Range-based sharding: Assigns shards based on key ranges. Good for range queries but can create hotspots.

Consistent Hashing: Uses a hash ring to minimize key redistribution when nodes are added or removed. Widely used in distributed caches and databases.

Replication

Copying data to multiple nodes to improve availability and read performance.

Primary-Replica (Master-Slave):

Multi-Primary (Master-Master):

Primary-Replica architecture:

  Writes -> [Primary]
               |
        +------+------+
        |              |
    [Replica 1]    [Replica 2]
        |              |
    Read requests   Read requests

Partitioning

Logically dividing data.

Horizontal partitioning: Splits by rows. Similar to sharding. Vertical partitioning: Splits by columns. Separates frequently accessed columns from rarely accessed ones.

CAP Theorem

A distributed system cannot simultaneously satisfy all three of the following properties.

The Three Properties

Consistency: Every node has the same data. Reads always return the most recent write.

Availability: Every request receives a response. The system responds even if some nodes are down.

Partition Tolerance: The system continues to operate despite network partitions.

Real-World Choices

Network partitions are unavoidable in distributed systems, so the practical choice is between CP and AP.

CP systems (Consistency + Partition Tolerance):

AP systems (Availability + Partition Tolerance):

CP system (during partition):
  Node A: latest data = "v2"
  Node B: stale data = "v1" -> Request rejected (consistency first)

AP system (during partition):
  Node A: latest data = "v2"
  Node B: stale data = "v1" -> Returns "v1" (availability first)

PACELC

An extension of CAP that also considers the tradeoff between Latency (L) and Consistency (C) when there is no partition (Else).

Message Queues

A message queue is middleware that enables asynchronous communication. It decouples producers from consumers, reducing coupling and improving scalability.

Why They Are Needed

Problems with synchronous communication:

Synchronous (problem):
  Order API -> Payment -> Inventory -> Notification -> Response
  (Total latency = sum of each service's latency)

Asynchronous (solution):
  Order API -> Publish event to queue -> Respond immediately
  Payment Service    <- Consumes from queue
  Inventory Service  <- Consumes from queue
  Notification Service <- Consumes from queue
  (Each service processes independently)

Apache Kafka

A distributed event streaming platform.

Key features:

Kafka architecture:
  Producer -> Topic (Partition 0, 1, 2) -> Consumer Group
                                            Consumer A (P0)
                                            Consumer B (P1)
                                            Consumer C (P2)

Use cases: Log aggregation, event sourcing, real-time stream processing, microservice communication

RabbitMQ

A traditional message broker using the AMQP protocol.

Key features:

Use cases: Task queues, RPC patterns, scenarios requiring complex routing

Amazon SQS

A fully managed message queue service from AWS.

Key features:

Kafka vs RabbitMQ vs SQS Comparison

FeatureKafkaRabbitMQSQS
ThroughputVery highMediumHigh
Message orderingGuaranteed within partitionCan be guaranteedFIFO only
Message retentionConfigurable retention periodDeleted after consumptionUp to 14 days
Operational complexityHighMediumLow (managed)
Best forEvent streamingTask queuesServerless workloads

Event-Driven Architecture

Event-driven architecture built on message queues is a core pattern in microservice environments.

Event Sourcing: Records state changes as events. Current state is reconstructed by replaying events.

CQRS: Separates commands (writes) and queries (reads), optimizing each independently.

Event Sourcing example:
  Event 1: Order Created (OrderCreated)
  Event 2: Payment Completed (PaymentCompleted)
  Event 3: Shipping Started (ShippingStarted)
  Event 4: Delivery Completed (DeliveryCompleted)

  Current state = result of applying Events 1 + 2 + 3 + 4

API Design

REST vs GraphQL vs gRPC

REST:

GET /api/users/123
POST /api/orders
PUT /api/users/123
DELETE /api/orders/456

GraphQL:

query {
  user(id: "123") {
    name
    email
    orders {
      id
      total
    }
  }
}

gRPC:

service UserService {
  rpc GetUser (GetUserRequest) returns (UserResponse);
  rpc ListUsers (ListUsersRequest) returns (stream UserResponse);
}

API Comparison

FeatureRESTGraphQLgRPC
ProtocolHTTP/1.1HTTP/1.1HTTP/2
Data formatJSONJSONProtocol Buffers
Type systemNone (OpenAPI for docs)Built-inBuilt-in
StreamingLimitedSubscriptionsBidirectional streaming
Best forPublic APIsFrontend BFFMicroservices

API Versioning

Strategies for maintaining backward compatibility when APIs change.

Rate Limiting

Limits request frequency to prevent API abuse and protect the system.

Algorithms:

Token Bucket:
  Bucket capacity: 10
  Refill rate: 1/second

  Time 0: 10 tokens (5 requests -> 5 remaining)
  Time 5: 10 tokens (5 refilled, capped at 10)

Monitoring and Logging

Metrics

Quantify the health of the system.

Key metrics:

Tools: Prometheus, Grafana, Datadog, CloudWatch

Alerting

Notifies on-call engineers when metrics exceed thresholds.

Good alerting criteria:

Distributed Tracing

Tracks the complete path of a request across microservices.

User request (trace-id: abc123)
  -> API Gateway (span-1, 5ms)
    -> Auth Service (span-2, 10ms)
    -> Order Service (span-3, 50ms)
      -> Payment Service (span-4, 200ms)
      -> Inventory Service (span-5, 30ms)
    -> Notification Service (span-6, 15ms)
  Total duration: 310ms

Tools: Jaeger, Zipkin, AWS X-Ray, OpenTelemetry

Logging Strategy

Structured logs: Recording logs in JSON format makes them easier to search and analyze.

{
  "timestamp": "2026-04-12T10:30:00Z",
  "level": "ERROR",
  "service": "order-service",
  "trace_id": "abc123",
  "message": "Payment failed",
  "user_id": "user-789",
  "order_id": "order-456",
  "error_code": "INSUFFICIENT_FUNDS"
}

Use the ELK stack (Elasticsearch + Logstash + Kibana) or a Loki + Grafana combination to collect, store, and visualize logs.

Practical Example: URL Shortener

A frequently asked system design interview question.

Requirements

Functional requirements:

Non-functional requirements:

Scale Estimation

High-Level Design

Client -> Load Balancer -> API Server -> DB
                              |
                           Cache (Redis)

Short Key Generation

Method 1: Hash and truncate

Method 2: Counter-based

Method 3: Pre-generation

Detailed Design

Redirect flow:

1. Client requests short.url/abc123
2. Load balancer forwards to API server
3. API server checks Redis cache
4. Cache hit -> 301 Redirect response
5. Cache miss -> Query DB -> Store in cache -> 301 Redirect

301 vs 302 redirect:

Practical Example: Chat System

Requirements

Functional requirements:

Non-functional requirements:

Communication Protocols

HTTP Polling: The client periodically asks the server for new messages. Simple but inefficient.

Long Polling: The server holds the connection until a new message arrives. More efficient than polling but requires timeout management.

WebSocket: Enables real-time bidirectional communication. After an initial HTTP handshake, a persistent connection is maintained. The best fit for real-time chat.

HTTP Polling:
  Client -> Server: New messages? (every 1 second)
  Server -> Client: None
  Client -> Server: New messages? (every 1 second)
  Server -> Client: Yes! (delivers message)

WebSocket:
  Client <-> Server: Persistent connection
  Server -> Client: New message pushed immediately

High-Level Design

Sender -> WebSocket Server -> Message Queue -> WebSocket Server -> Receiver
               |                                     |
           Message DB                        Presence Service
               |
         Message Search (Elasticsearch)

Message Store Selection

Characteristics of chat messages:

Suitable databases:

Message ID Design

Message IDs must be sortable by time to guarantee ordering.

Snowflake ID approach:

| Timestamp (41 bits) | Datacenter (5 bits) | Machine (5 bits) | Sequence (12 bits) |

Presence Management

Heartbeat approach: Clients send a periodic heartbeat (e.g., every 5 seconds). If no heartbeat is received for a set period, the user is marked offline.

User A: Sends heartbeat (every 5 seconds)
  Server: Records last heartbeat time
  No heartbeat for 30 seconds -> Status changed to "offline"

Group Chat

In group chat, messages must be delivered to all group members.

Small groups (tens of members):

Large groups (thousands of members):

Conclusion

The essence of system design is understanding tradeoffs. No system is perfect, and the goal is always to make the best choices for the given requirements.

Key principles:

The concepts covered here form the foundation of system design. Apply them in real projects, study diverse system design case studies, and continue building your skills.

Read the original: https://www.youngju.dev/blog/architecture/2026-04-12-system-design-fundamentals.en

Comments

No comments yet.

Sign in to leave a comment