LabHub

Blog

Data Lakehouse & Modern Data Engineering 2026 — Iceberg / Delta / Hudi / Paimon / Tabular (Databricks acquisition) / Trino / Spark 4 / Flink 2 / DataFusion Deep Dive

2026-05-16 · 30 min read · #data-engineering#data-lakehouse#apache-iceberg#delta-lake#apache-hudi#apache-paimon#tabular#databricks#snowflake#bigquery#dbt#trino#presto#spark#flink#clickhouse#duckdb#apache-arrow#apache-parquet#apache-orc#datafusion#onehouse#deep-dive#english

한국어English日本語

Prologue — The word "data warehouse" has aged in place

In the early 2010s, "where do you put your data" was simple. Structured data went into a data warehouse (Teradata, Oracle Exadata, Vertica). Logs and semi-structured data went into Hadoop HDFS. ETL tools (Informatica, Talend) bridged them. Analysts used SQL; data engineers used MapReduce and Spark.

In 2026 the picture is completely different.

This essay maps that landscape — the three table format heavyweights (Iceberg, Delta, Hudi) and the challenger Paimon, the engines (Spark 4, Flink 2, Trino, DuckDB), the cloud data platforms (Databricks, Snowflake, BigQuery, ClickHouse), and the actual stacks at Korean and Japanese companies — all in one go.

1 · The 2026 Data Lakehouse Map — Three Axes

Start with a picture. The 2026 data lakehouse can be understood as three orthogonal axes.

                        [ Catalog / Governance ]
                  Unity Catalog · Polaris · BigLake
                   Glue · Nessie · Snowflake Horizon
                                |
                                |
   [ Compute / Query Engines ] -+- [ Table Formats / Storage ]
   Spark 4 · Flink 2 · Trino    |  Iceberg · Delta · Hudi · Paimon
   Presto · DuckDB              |  Parquet · ORC · Avro
   ClickHouse · DataFusion      |  S3 · GCS · ADLS
                                |
                                |
                       [ Transform / Orchestration ]
                       dbt · SQLMesh · Coalesce
                       Airflow · Dagster · Prefect

That these three axes can be decoupled is the heart of the lakehouse. Iceberg for the table format, Spark/Trino/DuckDB picked per workload for the engine, Unity for the catalog — that kind of mix-and-match is now natural.

Traditional warehouses (Snowflake, BigQuery) had all three axes locked inside one company. The lakehouse unlocks them. That's why Snowflake also built Polaris, and BigQuery extended BigLake to read external Iceberg tables. Unlocking itself has become the 2026 default.

2 · Apache Iceberg — Winner of the Table Format War

2.1 Why a "table format" is needed at all

Parquet files are great. Columnar, compressed, with statistics pushdown. But that alone is not enough.

A table format is the metadata layer that solves this. JSON and Avro files on top of Parquet hold information like "the current snapshot of this table is X, the schema is Y, the next transaction is Z."

Iceberg, Delta, and Hudi are different ways of solving the same problem.

2.2 Iceberg's data model

Iceberg has a three-tier metadata hierarchy.

   [ Catalog ] ─ catalog (Glue, Nessie, Polaris, REST)
        |
        v
   [ Metadata file v0.json, v1.json ... ]
        |
        v
   [ Snapshot ] ─── state of the table at a point in time
        |
        v
   [ Manifest list ]
        |
        v
   [ Manifest ]  ─ which data files exist where, with statistics
        |
        v
   [ Data files (Parquet/ORC/Avro) ]

Core ideas:

2.3 Why Iceberg won

As recently as 2023, "Iceberg vs Delta vs Hudi" was genuinely a contest. By late 2025, the industry center of gravity had clearly shifted to Iceberg. The reason can be summarized in one sentence.

Iceberg is a standard, Delta is a product.

Snowflake building Polaris Catalog in June 2024 and donating it to Apache in 2025, and AWS S3 Tables (December 2024) supporting Iceberg as first-class — all part of the same current.

2.4 Try Iceberg yourself — PyIceberg

The smallest possible example with PyIceberg (the Python-native Iceberg client).

from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, StringType, TimestampType, NestedField

# 1. Load a catalog (Glue, REST, SQL, Hive all supported)
catalog = load_catalog(
    "my_catalog",
    **{
        "type": "rest",
        "uri": "http://localhost:8181",
        "warehouse": "s3://my-bucket/warehouse",
    }
)

# 2. Define the schema
schema = Schema(
    NestedField(1, "event_id", LongType(), required=True),
    NestedField(2, "user_id", StringType()),
    NestedField(3, "event_time", TimestampType()),
    NestedField(4, "event_type", StringType()),
)

# 3. Create the table
table = catalog.create_table(
    identifier="analytics.events",
    schema=schema,
    partition_spec=...  # day(event_time)
)

# 4. Append data (as an Arrow Table)
import pyarrow as pa
data = pa.table({
    "event_id": [1, 2, 3],
    "user_id": ["u1", "u2", "u3"],
    "event_time": [...],
    "event_type": ["click", "view", "purchase"],
})
table.append(data)

# 5. Read (scan)
result = table.scan(
    row_filter="event_type == 'purchase'",
    selected_fields=("event_id", "user_id"),
).to_pandas()

# 6. Time travel
old_snapshot = table.history()[-2].snapshot_id
old_data = table.scan(snapshot_id=old_snapshot).to_pandas()

That's it. Without any distributed cluster, in pure Python, you create an Iceberg table, read it, and query historical snapshots. This makes it concrete that Iceberg is a "spec," not an "engine."

3 · Delta Lake (Databricks) — Still a Strong Contender

3.1 What makes Delta different

Delta Lake was open-sourced by Databricks in 2019. The essence is the same as Iceberg: a transaction log layered over Parquet files to provide ACID, time travel, and schema management. The difference is the metadata structure and the ecosystem.

   [ Delta Table ]
        |
        v
   _delta_log/
       00000000000000000000.json   ← transaction log (JSON per line)
       00000000000000000001.json
       ...
       00000000000000000010.checkpoint.parquet
       _last_checkpoint
        |
        v
   data files (Parquet)

3.2 UniForm — Delta imitates Iceberg

Databricks's Delta UniForm announcement in 2023 was the moment the market direction became clear.

Take the same Parquet files and write Delta logs and Iceberg metadata at the same time. From the outside, "the table is also an Iceberg table."

This makes it possible to read with Iceberg from Snowflake, Trino, and BigQuery, while writing with Delta inside Databricks. Databricks itself opened the path to making "the Delta camp" meaningless.

After the Tabular acquisition in June 2024, that strategy deepened. By 2025 the direction "do not treat Delta and Iceberg as separate formats — treat them as the same metadata" was set in stone.

3.3 Why Delta still survives

In short: "In a world where Iceberg is the standard, Delta has settled into being Databricks's internal optimized format." That's the 2026 picture.

4 · Apache Hudi (Uber) — The Niche Charm

4.1 Hudi's identity

Hudi was built by Uber in 2016 and donated to Apache in 2019. Its identity is "upsert-first."

From day one it solved "how do we apply frequent updates to a data lake?" Uber's need to reflect CDC (Change Data Capture) events at minute-level latency was the starting point.

   Hudi's two storage types
   
   [ Copy-on-Write (CoW) ]
     Write: rewrite the entire affected Parquet file
     Read: fast (just Parquet)
     Best for: read-heavy, infrequent writes
   
   [ Merge-on-Read (MoR) ]
     Write: stack deltas into separate log files (Avro)
     Read: merge Parquet plus logs (real-time view)
     Best for: frequent updates and deletes (CDC)

4.2 Hudi's strengths — indexing and incremental queries

There are two things Iceberg and Delta still cannot match.

These are exactly the areas Iceberg v3 is trying to catch up on. Iceberg v3 is strengthening row-level deletes, equality deletes, and merge-on-read, and is likely to close the gap in 2026. So Hudi holds the position of "the niche heavyweight."

4.3 Onehouse — Hudi's commercial path

Founded by Hudi co-creator Vinoth Chandar in 2021. Sells a managed service under the "Universal Data Lakehouse" banner, covering Hudi, Iceberg, and Delta. Interestingly, the Hudi company does not sell only Hudi.

Two tools Onehouse released in 2024-25 were significant.

5 · Apache Paimon (Flink Team) — The New Challenger

5.1 Why yet another table format

Iceberg, Delta, and Hudi all started from "batch analytics." Streaming was layered on top.

Paimon starts from "streaming." It began inside the Apache Flink team in 2022 and graduated to a Top-Level Project in 2024. The core difference is the LSM tree (Log-Structured Merge Tree).

   Paimon = "a table format built on an LSM tree"
   
   - The structure LevelDB and RocksDB use
   - Memory → L0 → L1 → L2 ... incremental merge
   - Fast writes + efficient reads + natural compaction
   - Couples naturally with Flink's checkpointing

5.2 What Paimon solves

As of 2025, Paimon is most heavily used inside Chinese giants like Alibaba and ByteDance, and is spreading to the West. If Iceberg is the standard for batch, Paimon aims to be the standard for "streaming lakehouse."

5.3 Iceberg vs Paimon — competition or complement?

Looking at the trend, it's closer to a complement.

In 2026 it'll be normal to keep both formats inside the same data platform. Flink writes to Paimon; once cubes and aggregates form, they harden into Iceberg.

6 · The Tabular Acquisition (Databricks, June 2024, $1B+) — Meaning

6.1 What happened

In June 2024, Databricks acquired Tabular for over $1B. Founded by Iceberg co-creators Ryan Blue, Daniel Weeks, and Jason Reid, Tabular sold a managed Iceberg-based data platform.

Rumors that Snowflake also tried to buy Tabular were strong at the time. Databricks ultimately won, and the price was extremely high relative to Tabular's ARR. What Databricks paid for was not revenue but people and the standard.

6.2 What changed

The event was the signal that the table format war was over. Iceberg becomes the standard; the real competition starts at the catalog, engine, and service layers.

6.3 Ryan Blue's message

At the first conference after the acquisition (Iceberg Summit 2024), Ryan Blue said something that stuck.

"Table formats need to be standards. Once the standard is set, the real competition starts above it."

That's the whole thing. Data engineering in 2026 is no longer "which format do you choose?" but "which engines, catalogs, and UX do you choose on top of Iceberg?"

7 · Onehouse — The Commercialization of the Hudi Camp

Onehouse is the David side of David and Goliath. Where Databricks and Snowflake move tens of billions, Onehouse is a small company just past Series B. But it has a clear position.

The Onehouse hypothesis is simple.

"The era of one format per company is over. Inside one company, Iceberg, Delta, and Hudi will all coexist. Whoever stitches them seamlessly wins."

As of 2026 that hypothesis keeps coming true. In real enterprises it's now common to see "Databricks uses Delta, Snowflake uses Iceberg, in-house analytics uses Hudi" all inside one company.

8 · dbt — The Standard for Transformation

8.1 What dbt changed

dbt (data build tool) was started by Fishtown Analytics (now dbt Labs) in 2016. The core idea is simple.

"Write your models as SQL. We'll solve the dependencies. Tests are SQL too. Docs come out of SQL."

Previously, analysts hand-wrote SQL and embedded it in BI tools. Transformation logic was scattered — where it lived, what depended on what, how it was tested — all in different places.

What dbt organized:

   models/
     staging/
       stg_orders.sql      ← raw → cleaned
       stg_customers.sql
     marts/
       core/
         dim_customers.sql
         fct_orders.sql    ← references stg_orders, dim_customers
   
   tests/
     not_null_dim_customers_id.sql
   
   dbt_project.yml         ← project config

8.2 The Analytics Engineer

The biggest change dbt brought is the rise of the "Analytics Engineer" role. Someone who owns transformation, modeling, testing, and documentation using SQL as their tool. In between data engineer and analyst.

By 2026 this role is the majority of many data teams. Data engineers (with Python and Spark) own ingestion and infrastructure; analytics engineers model with dbt; data analysts build BI on top.

8.3 dbt's competitors

After dbt became the standard, challengers arrived.

As of 2026, dbt is dominant, but SQLMesh is catching up fast. Both share the core value: SQL is code, so it needs versioning, testing, and CI/CD.

9 · Trino (formerly PrestoSQL) / Presto — Distributed OLAP Engine

9.1 The Presto fork

Presto was built by Facebook in 2012 as a distributed SQL engine. In 2019 the core developers left and forked into Trino (originally PrestoSQL). The side that stayed with Presto Foundation (Linux Foundation) became PrestoDB, essentially Meta-internal.

As of 2026:

Six years on from the fork, Trino has become Presto's true successor.

9.2 Trino's position

Trino is a "federated query engine." It does not store data itself. It JOINs an Iceberg table on S3, an operational DB in MySQL, and a Kafka topic with a single SQL statement.

-- Iceberg table ⋈ MySQL table ⋈ PostgreSQL table
SELECT
    i.user_id,
    m.user_name,
    p.last_login,
    SUM(i.amount) AS total_spent
FROM iceberg.analytics.purchases i
JOIN mysql.app.users m ON i.user_id = m.id
JOIN postgres.crm.profiles p ON i.user_id = p.user_id
WHERE i.event_date >= DATE '2026-05-01'
GROUP BY 1, 2, 3

That this is one query is Trino's identity. Trino has settled as the OLAP standard for the data lakehouse.

9.3 Who uses Trino

9.4 Trino vs DuckDB — single-node challenge

Interesting nuance: on a single node DuckDB often beats Trino. When data is small or medium (tens to hundreds of GB), DuckDB is enough and often faster. Trino is narrowing into "queries that genuinely need distribution."

10 · Spark 4 (Aug 2024) + Apache Flink 2 — Processing Engines

10.1 What changed in Spark 4

Apache Spark 4.0 GA'd in August 2024. Three key changes.

Spark is no longer "after Hadoop." It's "the standard ETL and batch engine, writing to and reading from any of Iceberg, Delta, Hudi."

Apache Flink 2.0 released in early 2025. The core is cloud-native state management.

Flink 2 plus Paimon is becoming the 2026 standard for "streaming lakehouse."

10.3 Spark vs Flink — what to use when

AxisSparkFlink
Batchstandardpossible
StreamingStructured Streaming (micro-batch)true streaming (event-by-event)
Latencysecondsmilliseconds
State managementweakfirst-class
ML / DataFramevery strongweak
Talent poolvery largemedium

ETL, batch, and ML go to Spark; true streaming goes to Flink. That divide holds in 2026. The nuance is that Spark's Structured Streaming has gotten good enough that "seconds of latency is fine, Spark is enough" applies more often.

11 · Databricks / Snowflake / BigQuery / ClickHouse — Cloud Data Platforms

11.1 Databricks — the lakehouse pioneer

The Databricks message in 2026: "Data plus AI equals one platform." Not just ETL and BI but model training in one place.

11.2 Snowflake — from warehouse to lakehouse

Started as a traditional warehouse, but in 2024-25 moved fast toward the lakehouse.

The Snowflake message: "Even when Iceberg becomes the standard, we are a first-class citizen of that standard."

11.3 BigQuery — Google's answer

Inside GCP, BigQuery remains the smoothest choice. Accepting Iceberg through BigLake was the big 2025 shift.

11.4 ClickHouse Cloud — the real-time OLAP heavyweight

ClickHouse is not an orthodox OLTP/HTAP system but a columnar DB specialized for real-time analytics. Spun out of Yandex in 2022 as ClickHouse Inc.; Cloud grew significantly in 2024-25.

ClickHouse is less "part of the data lakehouse" and more "a separate engine for real-time OLAP." It connects to Iceberg through external tables.

12 · AWS Athena + Glue — Cloud Managed

12.1 Athena — serverless SQL over S3

AWS Athena is a serverless query engine based on Presto/Trino. Query Parquet, ORC, Iceberg, and Delta files directly from S3 with SQL. No clusters to spin up or manage.

-- Iceberg table query (Athena v3 engine)
SELECT
    event_date,
    COUNT(*) AS events,
    COUNT(DISTINCT user_id) AS dau
FROM iceberg_catalog.analytics.events
WHERE event_date BETWEEN DATE '2026-05-01' AND DATE '2026-05-15'
GROUP BY 1
ORDER BY 1

12.2 Glue — catalog plus ETL

The Glue Data Catalog launched an adapter that mimics Iceberg REST in 2024. Inside AWS, "Iceberg plus Athena plus Glue Catalog plus S3" became the most natural combination.

12.3 S3 Tables (Dec 2024)

AWS's S3 Tables, announced December 2024, was a big shift. S3 itself manages Iceberg tables as first-class. Auto-compaction, snapshot expiration, and metadata management — all handled by S3 directly.

S3 (object storage)
   |
   v
S3 Tables (first-class Iceberg)  ← auto compaction, expiration, stats
   |
   v
Athena · EMR · Glue · Redshift · Trino · Spark

The implication is large. A cloud provider started handling Iceberg directly. The final nail in "Iceberg is the standard."

13 · DuckDB — "SQLite for Analytics"

13.1 DuckDB's identity

DuckDB started in 2019 at CWI in the Netherlands as an in-process OLAP database. The slogan is "SQLite for analytics." No server to run — import as a library and run SQL inside your own process.

import duckdb

# Query Parquet directly. No cluster, no server.
result = duckdb.sql("""
    SELECT
        user_id,
        COUNT(*) AS events,
        SUM(amount) AS total
    FROM 's3://my-bucket/events/*.parquet'
    WHERE event_date >= '2026-05-01'
    GROUP BY user_id
    ORDER BY total DESC
    LIMIT 100
""").df()  # → Pandas DataFrame

That's the whole thing. No PostgreSQL client, no Spark cluster, no Snowflake account.

13.2 Why DuckDB exploded

DuckDB's GitHub stars roughly 4xed in 2024-25. The reasons are simple.

13.3 The slot DuckDB occupies

Spark and Trino started getting asked "do you really need distribution?" When data is under 1TB, DuckDB is often enough.

14 · Apache Arrow / Parquet / ORC / DataFusion — Low-Level Standards

14.1 Apache Arrow — in-memory columnar standard

If Parquet is the standard for storing data on disk, Apache Arrow is the standard for handling it in memory.

Since Pandas 2.0 (2023) Arrow became a backend; Polars was Arrow-native from day one. "DataFrames live on Arrow" has become the standard.

14.2 Parquet vs ORC — the disk formats

The difference is small. Compression ratio and scan performance trade places by scenario. But the ecosystem momentum is overwhelmingly on Parquet. In 2026 there is essentially no reason to start a new project with anything other than Parquet.

14.3 Apache DataFusion — Rust query engine

DataFusion is a SQL query engine written in Rust, started as part of the Apache Arrow project. Andy Grove built it, and it joined Apache in 2021.

DataFusion's position is interesting. It doesn't compete directly with DuckDB — instead, it is adopted as the internal engine of other systems.

"When building a Rust DBMS, start your SQL engine from DataFusion" has become the 2026 default.

15 · Korea / Japan — Toss, Kakao KaaP, Mercari, ZOZO

15.1 Toss data — a single data platform

Toss reorganized its data platform in 2025 around Apache Iceberg, dbt, Trino, and Airflow. From its SLASH 25 conference talks:

What Toss emphasized in particular is "data self-service." Data engineers own only ingestion and infrastructure; analysis and modeling are owned by analytics engineers and analysts themselves.

15.2 Kakao KaaP (Kakao as a Platform)

Kakao runs an internal data platform called KaaP. The core is multi-tenant analytics infrastructure.

In 2025-26 KaaP is rapidly increasing Iceberg adoption. Migrating from Hive tables to Iceberg is a major effort.

15.3 Mercari — Japan's data platform

Mercari is Japan's leading C2C marketplace. The data platform centers on BigQuery, dbt, and Looker.

A 2024 post on the Mercari engineering blog about "taking in external Iceberg through BigLake" was striking. Among Japanese companies, Mercari moved fastest toward the lakehouse direction.

15.4 ZOZO — Japanese fashion commerce

ZOZO operates ZOZOTOWN. The data platform centers on BigQuery and Dataform.

ZOZO is among the Japanese companies that tried data mesh early. A 2024 ZOZO Tech Blog post emphasizing "distributed domain responsibility" is often cited.

16 · Who Should Pick What — SMB / Enterprise / Streaming / Analytics

By scale and need.

16.1 SMB and startups (under $5,000 per month)

AreaRecommendation
StorageS3 plus Parquet
Table formatStart with plain Parquet; move to Iceberg when it grows
EngineDuckDB plus (optional) Athena
Transformationdbt-core
Orchestrationdbt schedules plus GitHub Actions
BIMetabase, Lightdash

The core: don't spin up a cluster. DuckDB is enough up to 100GB. Athena just takes SQL. Snowflake and Databricks are expensive until the data is genuinely large and the analyst headcount passes 5.

16.2 Mid-size (5,000to5,000 to 50,000 per month)

AreaRecommendation
StorageS3 / GCS / ADLS
Table formatIceberg
EngineTrino (Starburst Galaxy), managed Spark
Transformationdbt Cloud or SQLMesh
CatalogGlue, Polaris, Unity Catalog
BILooker, Hex, Mode

Iceberg starts to matter here. The zone where you have 2-3 data engineers, 10-plus analysts, and data infrastructure is a meaningful cost line.

16.3 Enterprise

AreaRecommendation
StorageMulti-cloud (S3 plus GCS)
Table formatIceberg (Delta UniForm acceptable in parallel)
EngineDatabricks (company-wide) plus Snowflake (specific orgs) plus Trino (self-service)
Transformationdbt plus an analytics-engineer org
CatalogUnity Catalog or Polaris (set the company standard)
GovernanceAtlan, Collibra, OpenMetadata

The hardest thing at enterprise scale is governance. The catalog has to become a standard before 100-plus teams can see the same data.

16.4 Streaming-focused

AreaRecommendation
IngestionKafka
ProcessingFlink 2
Table formatPaimon (or Hudi)
Downstream analyticsIceberg plus Trino

Streaming materialized views go to Paimon; huge analytic tables go to Iceberg. Run both formats inside the same platform.

16.5 Real-time OLAP (product analytics, observability)

AreaRecommendation
EngineClickHouse Cloud
IngestionKafka into ClickHouse directly
DataReal-time plus daily aggregates

Less "part of the data lakehouse," more "separate engine." Where millisecond response is required.

17 · Closing — What Begins After the Standard

In the late 2010s there was a "data lake vs data warehouse" debate. In the early 2020s came the "Iceberg vs Delta vs Hudi" war. As of 2026, both are over.

Quoting Ryan Blue again after the Tabular acquisition:

"Once the standard is set, the real competition starts above it."

That's the one-line summary of data engineering in 2026. That Iceberg is the standard is no longer up for debate. The real work begins above it — catalogs, governance, UX, AI integration, real-time sync, multi-cloud.

The data platform you build today will still hold the same Iceberg metadata five years from now. That's what "standard" means.

References

Comments

No comments yet.

Sign in to leave a comment