Season 5 Ep 2 — If Ep 1 was "where data is stored", Ep 2 is "how fast data flows". The real-time frenzy of 2020–2023, and the return to pragmatism in 2024–2025.
- Prologue — "Real time is an option, not a default"
- Chapter 1 · Freshness Tiers
- Chapter 2 · The 2025 Version of Lambda and Kappa
- Chapter 3 · Comparing the Four Major Streaming Engines
- Chapter 4 · CDC (Change Data Capture)
- Chapter 5 · Iceberg v3 and Real-time Upsert
- Chapter 6 · The Rise of Streaming SQL
- Chapter 7 · The Cost and Latency Trade-off
- Chapter 8 · Observability and Debugging
- Chapter 9 · Failure, Recovery and SLA
- Chapter 10 · Practical Patterns for Streaming plus Lakehouse
- Chapter 11 · Three Real-world Cases
- Chapter 12 · Streaming at Korean Companies
- Chapter 13 · Ten Anti-patterns
- 13.1 "Everything in Real Time"
- 13.2 Blind Faith in Exactly-once
- 13.3 Skipping the Initial CDC Snapshot
- 13.4 No Automatic Propagation of Schema Changes
- 13.5 Kafka Retention Too Short
- 13.6 Checkpoint Interval Too Long
- 13.7 Keeping State Without Limits
- 13.8 No Compaction of Delete Files
- 13.9 Under-allocating Memory and CPU
- 13.10 No Observability or Alerting
- Chapter 14 · Checklist — Twelve Points Before Launching Streaming
- Chapter 15 · Next Up — Season 5 Ep 3: "Comparing OLAP Engines in 2025"
Prologue — "Real time is an option, not a default"
Every keynote at a data conference between 2019 and 2022 said "all data must become real time". The reality of 2025 is different:
- Real-time pipelines are 3–10x more expensive than batch
- Most BI dashboards run 15 minutes to 1 hour behind and nobody complains
- What genuinely needs real time is 5–15% of all data
The right answer in 2025:
"Divide data into freshness tiers based on the SLA of each dataset and metric, and use the tool that fits each tier."
This post makes those tiers and tool choices concrete.
Chapter 1 · Freshness Tiers
1.1 A Five-tier Framework
| Tier | Latency | Example | Tools |
|---|---|---|---|
| Real-time | ms–seconds | Transaction monitoring, anomaly detection, fraud | Flink, Kafka Streams |
| Near-real-time | 1–5 min | Operational dashboards, alerts | Flink, RisingWave, Materialize |
| Fresh | 5–60 min | Inventory, ad optimization | Streaming append + rollup |
| Daily | 24 hours | BI, reports | Spark, dbt, SQL warehouse |
| Historical | Weekly/monthly | Analysis, ML training | Batch yearly/monthly |
1.2 Mapping Each Metric and Table to a Tier
- Inventory metrics: Fresh
- Login anomaly detection: Real-time
- Monthly revenue: Daily
- Features for model training: a mix of Fresh and Daily
1.3 Decision Principles
- The business question: "what happens if the metric is an hour late?"
- If the answer is "nothing at all" → Daily or Fresh
- If the answer is "a loss of 100,000 won" → Near-real-time
- If the answer is "someone gets hurt or we have a legal problem" → Real-time
Chapter 2 · The 2025 Version of Lambda and Kappa
2.1 Lambda (2014)
- Batch layer + Speed layer + Serving layer
- The same logic implemented twice, a maintenance burden
- Dominant from 2015 to 2020
2.2 Kappa (2014, Jay Kreps)
- Unified on a single streaming log (Kafka)
- Reprocessing happens through streaming too
- Theoretically simple, practically hard
2.3 2025: Unified on Lakehouse
- On top of Iceberg/Delta tables, batch and streaming write at the same time
- Flink + Iceberg, Spark Structured Streaming + Delta
- Serving is an OLAP engine or a materialized view
2.4 The "Streaming + Materialized" Pattern
- Source → streaming pipeline → Lakehouse
- Precompute frequently used queries as materialized views
- Ad-hoc queries hit the OLAP engine directly
Chapter 3 · Comparing the Four Major Streaming Engines
3.1 Apache Flink
- Genuinely based on event time, with watermarks and exactly-once
- Rich keyed state, proven in large-scale production
- Steep learning curve, complex to operate
- Flink CDC 2.x and Flink SQL matured in 2024–2025
3.2 Spark Structured Streaming
- Micro-batch streaming in the Spark ecosystem
- The Databricks standard
- Latency target of 1–60 seconds or more, not truly millisecond class
- SQL and Python make it easy
3.3 Kafka Streams / ksqlDB
- A library built into Kafka, JVM based
- Lightweight, optimal when you concentrate on data already in Kafka
- Distributed operation is something you implement yourself
3.4 RisingWave
- Open sourced in 2022, specialized in Streaming SQL
- Postgres-compatible SQL plus materialized views
- State lives in separate storage (S3), which keeps operations simple
- Rising fast as an alternative to Flink
3.5 Materialize
- Streaming plus incremental view maintenance
- Complex joins and aggregates are refreshed automatically
- Centered on enterprise SaaS
3.6 Comparison Table
| Engine | Latency | Complexity | SQL | Use in Korea | Characteristics |
|---|---|---|---|---|---|
| Flink | ms–seconds | High | Yes | Common | Industry standard, strong state management |
| Spark SS | Seconds–minutes | Medium | Yes | Very common | Databricks friendly |
| Kafka Streams | Seconds | Medium | ksqlDB | Moderate | Built into Kafka |
| RisingWave | Seconds | Low | Postgres | Growing | Simple to operate, SaaS/OSS |
| Materialize | Seconds | Medium | Yes | Rare | Strong incremental views |
3.7 Selection Guide
- Event time and complex state: Flink
- The Databricks ecosystem: Spark SS
- Kafka centric with simple logic: Kafka Streams / ksqlDB
- Streaming apps in SQL alone: RisingWave or Materialize
Chapter 4 · CDC (Change Data Capture)
4.1 Why CDC Is Central
- Replicates in real time the changes in an operational DB (Postgres/MySQL) into the analytics system
- Dramatically lower latency and load than a batch dump
- Keeps the Silver layer of the Lakehouse up to date automatically
4.2 Implementation Methods
- Log-based: Postgres WAL, MySQL binlog — the most efficient
- Trigger-based: DB triggers generate change events — heavy load on the DB
- Query-based: polling on an updated_at column — easy, but it cannot catch deletes
4.3 Tools
- Debezium (OSS): Postgres/MySQL/SQL Server/Oracle/Mongo → Kafka
- Fivetran, Airbyte: managed ELT, hundreds of source connectors
- Striim, HVR: enterprise CDC
- Flink CDC: Flink integration on top of Debezium
4.4 The CDC → Iceberg Pattern
Postgres → Debezium → Kafka → Flink → Iceberg
- Kafka retains the events, so reprocessing is possible
- Flink upserts into Iceberg with MERGE INTO
- Uses Iceberg v2/v3 row-level delete
4.5 Practical Traps
- Handling schema changes (adding or dropping columns, type changes)
- Harmonizing the initial snapshot with incremental snapshots
- Guaranteeing exactly-once (preventing both duplicates and gaps)
- Backfill and reprocessing strategy
Chapter 5 · Iceberg v3 and Real-time Upsert
5.1 Iceberg Version History
- v1: Append-only
- v2: Row-level delete (position/equality), MERGE INTO
- v3 (2024–2025): Deletion vectors, row lineage, V3 partition transforms
5.2 Two Kinds of Row-level Delete
- Position delete: a delete file that points at the file and position of a row
- Equality delete: condition-based deletion (useful for implementing UPDATE)
5.3 The Real-time Upsert Workflow
- Flink reads the CDC events
- Generates an equality delete plus an insert based on the PK
- Iceberg reflects it in a snapshot
- Periodic compaction cleans up the delete files
5.4 Performance Cautions
- Read performance degrades as delete files pile up
- The compaction interval matters (minutes to hours)
- Running a background job that converts to Copy-on-Write alongside it is recommended
Chapter 6 · The Rise of Streaming SQL
6.1 Why SQL
- The Flink Java API is hard to learn
- A common language that data engineers and analysts can both handle
- Easy to integrate with tools such as dbt and Dagster
6.2 Flink SQL
- Stabilized in 2023–2024
- Event time, windows and state all expressed in SQL
- UDFs and UDAFs are possible
6.3 ksqlDB
- Streaming SQL on top of Kafka
- Clear Table and Stream concepts
- Suited to simple ETL and aggregation
6.4 The Postgres Compatibility of RisingWave
CREATE MATERIALIZED VIEW→ kept up to date in real time- Postgres tools (Grafana, Superset and so on) work as they are
- Low operational complexity, hence rapid growth in 2025
6.5 Materialize
- Based on Incremental View Maintenance (IVM) research
- Even complex joins are refreshed automatically
- SaaS plus a self-hosted option
Chapter 7 · The Cost and Latency Trade-off
7.1 Cost Components
- Compute (a streaming cluster running 24/7)
- State storage (RocksDB/S3)
- Running Kafka/MSK
- Network (cross-region)
7.2 Representative Cost Comparison (monthly, mid-size scale)
| Option | Monthly cost | Latency |
|---|---|---|
| Batch (Airflow + Spark, daily) | Low ($1–5k) | 24 hours |
| Micro-batch (5 min) | Medium ($3–10k) | 5 min |
| Structured Streaming | Medium–high ($5–20k) | Seconds–minutes |
| Flink cluster | High ($10–30k+) | ms–seconds |
| Managed (RisingWave/Confluent) | Medium–high ($7–25k) | Seconds |
7.3 Cost Reduction Techniques
- Keep the real-time layer thin (core metrics only)
- Check whether the near-real-time layer can take its place
- Scale down in off-peak periods
- The trade-off between state on S3 (the RisingWave way) and local RocksDB (Flink)
7.4 Recommended Architecture by Latency Target
- Under 100ms: in-memory stream processor + Redis/RocksDB
- Seconds: Flink/RisingWave + Kafka
- Minutes: Spark Structured Streaming + Delta/Iceberg
- Hours: micro-batch Airflow + dbt
- Days: batch Spark
Chapter 8 · Observability and Debugging
8.1 Key Metrics
- End-to-end latency (from the event occurring to the result being reflected)
- Lag (Kafka consumer lag, Flink checkpoint lag)
- Throughput (eps, rps)
- Back-pressure (per Flink task)
- State size (RocksDB size, checkpoint size)
8.2 Observability Tools
- Flink UI + Prometheus + Grafana
- Kafka Lag Exporter, Burrow, Conduktor
- Streaming pipeline lineage with OpenLineage
- Datadog/NewRelic APM plus streaming extensions
8.3 Debugging
- Reprocessing based on checkpoints and savepoints
- Debugging CEP (Complex Event Processing) rules
- Tracing events with sampled logs
- Tests run on MiniCluster plus embedded Kafka
8.4 Alerting
- Lag over the threshold → Slack/PagerDuty
- End-to-end latency SLO violation
- Consecutive checkpoint failures
Chapter 9 · Failure, Recovery and SLA
9.1 Designing the SLA
- Availability: 99.9% = 43 minutes of downtime allowed per month
- Freshness: queryable within X seconds of the event occurring
- Correctness: eventual accuracy vs exactly-once
9.2 Recovery Strategy
- Flink: checkpoints (scheduled), savepoints (manual)
- Kafka: Replication factor 3, min.insync.replicas 2
- Iceberg: roll back with snapshots and branches
9.3 Reprocessing
- Keep Kafka retention as long as the reprocessing window (7–30 days is common)
- Or replay the streaming job from the Iceberg source
- Prevent duplicates during reprocessing (idempotent design)
9.4 Multi-region
- Kafka MirrorMaker 2 / Confluent Cluster Linking
- Iceberg relies on storage replication (S3 Cross-region)
- Active-Passive is the norm for streaming jobs
Chapter 10 · Practical Patterns for Streaming plus Lakehouse
10.1 Streaming on Medallion
- Bronze: append the raw events
- Silver: cleansing, deduplication, join upserts
- Gold: aggregates and metrics
10.2 CDC → Silver
- DB change events → Flink → Iceberg Silver upsert
- Silver plays the role of "a copy plus history" of the operational DB
- Gold is produced by dbt/Spark batch
10.3 Event Sourcing
- Keep domain events in Kafka permanently
- Reconstruct state by replaying events
- Long-term retention in Iceberg Bronze
10.4 Real-time Feature Store
- Feast/Tecton + Kafka + Iceberg
- Online features (Redis) plus offline features (Iceberg)
- Monitoring online-offline skew
10.5 Streaming ETL Pipeline
- Source → Bronze (append) → Silver (cleansing) → Gold (aggregation)
- Each stage implemented as a Flink/Spark SS job
- Reprocessing only requires resetting the upstream offset
Chapter 11 · Three Real-world Cases
11.1 An E-commerce Order Pipeline
- Order events into Kafka → Flink → inventory update plus anomaly detection plus analytics
- Inventory: real time at second granularity
- Revenue metrics: 5-minute near-real-time
- Monthly reports: daily batch
11.2 Financial Transaction Monitoring
- Transaction events into Kafka → Flink CEP → fraud score
- Requires latency under 100ms
- State: per-user transaction history (Flink keyed state)
- Results: stored in Redis plus Iceberg
11.3 Game Telemetry
- Client events into Kinesis/Kafka → Flink/Spark SS
- Real-time analysis (concurrent users, DAU) runs near-real-time
- Detailed logs pile up in Iceberg Bronze for later analysis
- A/B tests: Gold aggregates
Chapter 12 · Streaming at Korean Companies
12.1 Traditional Patterns
- Finance: a mix of Tibco EMS, IBM MQ and Kafka
- Telecom: real-time streaming for charging and billing
- Gaming: Kafka + Flink or Kafka Streams
- Commerce: Kafka + Spark SS + ELK
12.2 Recent Trends
- Growing adoption of Confluent Cloud and MSK
- Wider adoption of Flink (especially at Toss, Coupang, Naver and Kakao)
- RisingWave and Materialize are at the early adoption stage in 2024–2025
12.3 Regulatory Considerations
- Finance: an in-house Kafka cluster inside a network-separated environment
- Personal information: PII masking of CDC events is mandatory
- Audit logs: long-term retention and immutability
12.4 Obstacles
- A shortage of data engineers pushes preference toward managed services
- Coexisting with legacy DWs
- A 24/7 on-call culture is still taking root
Chapter 13 · Ten Anti-patterns
13.1 "Everything in Real Time"
Streaming even the tables you do not need, so cost and complexity explode.
13.2 Blind Faith in Exactly-once
Guaranteeing end-to-end exactly-once on both the source and the sink is not easy. Idempotent design is mandatory.
13.3 Skipping the Initial CDC Snapshot
Gaps appear and accuracy drops.
13.4 No Automatic Propagation of Schema Changes
Downstream pipelines break.
13.5 Kafka Retention Too Short
Reprocessing becomes impossible.
13.6 Checkpoint Interval Too Long
Recovery cost and the amount to reprocess explode when something fails.
13.7 Keeping State Without Limits
Flink keyed state grows without bound, leading to OOM.
13.8 No Compaction of Delete Files
Iceberg read performance degrades.
13.9 Under-allocating Memory and CPU
A chain reaction of back-pressure.
13.10 No Observability or Alerting
Customers discover the incident before you do.
Chapter 14 · Checklist — Twelve Points Before Launching Streaming
- Freshness tier mapping (SLA per table and metric)
- Rationale for the engine choice (Flink/Spark SS/RisingWave/Materialize)
- CDC source selection plus an initial snapshot strategy
- Kafka retention plus a reprocessing plan
- Iceberg/Delta table design plus automated compaction
- Checkpoint and savepoint policy
- Idempotency and duplicate handling design
- Observability (lag, latency, throughput, back-pressure)
- SLA/SLO definition plus alerting
- Cost dashboard (compute, storage, network)
- Disaster recovery and multi-region plan
- On-call and incident response playbook
Chapter 15 · Next Up — Season 5 Ep 3: "Comparing OLAP Engines in 2025"
Now that streaming and batch share the same storage, the next question is "who queries fastest on top of it?".
- DuckDB: the revolution of single-node OLAP
- ClickHouse: the standard for real-time OLAP
- Snowflake / BigQuery / Redshift: the managed giants
- Databricks SQL / StarRocks / Doris / Pinot / Druid
- Federated queries with Trino / Presto
- The traps in real benchmarks
- Cost vs latency vs operational burden
- The line between MPP and single node
- A selection guide for Korean companies
- Engine placement patterns for the right tool in the right place
Things get genuinely interesting only after you accept the 2025 reality that "a single engine cannot do everything".
See you in the next post.
Summary: Streaming in 2025 was redefined from "everything in real time" to "freshness tiers based on SLA". You place Flink, Spark SS, RisingWave, Materialize and ksqlDB across the five tiers of Real-time, Near-real-time, Fresh, Daily and Historical, use CDC to flow changes from the operational DB into the Lakehouse, and handle real-time upserts with Iceberg v3 row-level delete. "Unified on Lakehouse" rather than Lambda/Kappa is the dominant pattern, and the trade-offs of cost, latency and complexity are designed deliberately. "Real time is an option, not a default" — that is the pragmatism of 2025.