LabHub

Blog

Hadoop Ecosystem & Data Engineering 2026 Deep Dive - Hadoop, Spark, Flink, Trino, Iceberg, Delta Lake, Hudi, Airflow, dbt

2026-05-16 · 25 min read · #hadoop#spark#flink#trino#presto#iceberg#delta-lake#hudi#airflow#dagster#dbt#lakehouse#data-engineering#big-data#kafka

한국어English日本語

Intro — Hadoop is dead, and Hadoop is everywhere

In May 2026, "Is Hadoop dead?" is a catchy headline but the wrong question on the ground. The honest answer is "HDFS has shrunk, MapReduce is effectively gone, the Cloudera/Hortonworks-era full-stack distribution is over. But YARN is alive, the Hive Metastore has changed clothes into Iceberg/Delta, and the lakehouse pattern of 'cheap object storage + metadata catalog + distributed processing engine' is the inheritance Hadoop left behind."

This piece is not a single-tool brag. It walks through every layer a data engineer actually touches in May 2026 — storage, table formats, processing engines, query engines, orchestration, transformation, warehouses, analytics — in one pass. Cloud managed services (EMR, Dataproc, HDInsight, Synapse), fully managed warehouses (Snowflake, Databricks, BigQuery, Redshift), and the on-prem realities of Korea and Japan are all covered honestly.

The 2026 data engineering stack — broken into 8 layers

The big picture first. The standard 2026 data engineering stack splits into these 8 layers.

  1. Storage: HDFS, S3, GCS, ABFS, MinIO
  2. Table format: Iceberg, Delta Lake, Hudi
  3. Catalog: Hive Metastore, Unity Catalog, Polaris, Nessie, AWS Glue
  4. Batch processing: Spark, Trino-on-Spark, Dremio
  5. Streaming: Flink, Spark Structured Streaming, Kafka Streams, Materialize
  6. Query/Federation: Trino, Presto, Dremio, StarRocks
  7. Transformation: dbt, SQLMesh, Coalesce
  8. Orchestration: Airflow 3.0, Dagster, Prefect, Mage, Kestra

On top of this, the analytics/OLAP layer adds ClickHouse, DuckDB, Pinot, Druid, and the fully managed warehouse layer adds Snowflake, Databricks SQL, BigQuery, Redshift. The Cloudera era — when every layer lived inside one vendor distribution — is over. Today you assemble OSS components yourself.

Why Hadoop isn't dead — YARN and Hive Metastore are stubborn survivors

The "Hadoop is dead" claim usually comes from two sources. First, the post-2018 Cloudera + Hortonworks merger and the slide in new-license revenue. Second, the way object storage (S3/GCS) has crushed HDFS, breaking the "HDFS = Hadoop" equation. Both are true, but the "Hadoop ecosystem" is much broader than HDFS.

The Hadoop components that are alive in 2026 include:

MapReduce, in contrast, has essentially vanished from new workloads. Everyone moved to Spark. The MapReduce remnants are legacy Sqoop, some Oozie batch jobs, and code waiting to be migrated.

HDFS vs object storage — who won

As of May 2026, over 90% of new big-data workloads land on object storage: AWS S3, Azure ADLS Gen2, GCP GCS, or on-prem MinIO/Cloudian. HDFS survives in only three cases.

  1. Financial data sovereignty: Environments where data absolutely cannot leave the building. On-prem HDFS clusters paired with Spark/Trino.
  2. Legacy ETL assets: Thousands of Hive tables and Oozie jobs hard-coded to HDFS paths, where migration is a multi-year project.
  3. Ultra-low-latency shuffle: Some graph/join workloads where object-storage PUT/GET latency is a bottleneck. HDFS or an Alluxio-style cache layer wins here.

The reason S3 won is simple. Storage cost is roughly a third, operational overhead is near zero, and compute and storage are decoupled — so the same data is read concurrently by Spark, Trino, Snowflake, and Athena. That is the core premise of the lakehouse.

The lakehouse era — warehouses and lakes converge

The "data lake vs data warehouse" dichotomy is mostly meaningless in 2026. Everyone converged on the lakehouse pattern. The core idea:

Warehouses (Snowflake, Redshift, BigQuery) used to lock data into proprietary formats, but they have started opening up to Iceberg. As of May 2026, both Snowflake and Databricks treat Iceberg tables as first-class citizens. What this signals is the end of the vendor lock-in era.

The table format big three — Iceberg vs Delta Lake vs Hudi

The heart of the lakehouse is the table format. As of May 2026, three formats dominate.

ItemApache IcebergDelta LakeApache Hudi
OriginNetflix → ASFDatabricks → LFUber → ASF
ACIDYes (Serializable)Yes (Serializable)Yes
Schema evolutionStrong (column-ID based)YesYes
Partition evolutionYesLimitedLimited
MERGE/UPDATEYesYesYes (CoW/MoR)
Time travelYesYesYes
CatalogsREST, Hive, Glue, Polaris, NessieUnity Catalog, HMSHive, Glue
Notable adoptersSnowflake, Confluent, Apple, NetflixDatabricks ecosystemUber, Robinhood

The 2026 trend is the gradual standardization of Iceberg. Snowflake is pushing Iceberg external tables plus Polaris Catalog hard, Databricks made Unity Catalog Iceberg-compatible, and Confluent's Tableflow materializes Kafka topics straight into Iceberg tables. Delta remains the strongest format inside the Databricks ecosystem, but the position of "the general-purpose OSS standard" went to Iceberg.

Hudi is still strong for CDC (Change Data Capture) and incremental write patterns, but new adoption is declining.

Iceberg DDL in practice — REST catalog as the standard

Real-world Iceberg usage produces DDL like the following (Spark SQL).

CREATE TABLE iceberg_catalog.analytics.events (
  event_id BIGINT,
  user_id BIGINT,
  event_type STRING,
  occurred_at TIMESTAMP,
  payload MAP<STRING, STRING>
)
USING iceberg
PARTITIONED BY (days(occurred_at), bucket(16, user_id))
TBLPROPERTIES (
  'format-version'='2',
  'write.parquet.compression-codec'='zstd',
  'write.delete.mode'='merge-on-read',
  'write.update.mode'='merge-on-read'
);

ALTER TABLE iceberg_catalog.analytics.events
ADD COLUMN device_type STRING AFTER event_type;

MERGE INTO iceberg_catalog.analytics.events t
USING staging.events_delta s
ON t.event_id = s.event_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

SELECT count(*) FROM iceberg_catalog.analytics.events
  FOR SYSTEM_TIME AS OF '2026-05-01 00:00:00';

Iceberg's strength is the decoupled catalog. The same table is written from Spark, read from Trino, streamed into from Flink, and queried from Snowflake as an external table. As the REST catalog spec solidifies, vendor lock weakens.

Delta Lake and the Databricks ecosystem

Delta Lake is OSS, but in practice Databricks holds tight control over the format. As of May 2026, the Delta Lake 4.x line headlines:

Typical Delta merges look like this.

MERGE INTO delta.`s3://datalake/customers` AS t
USING (SELECT * FROM staging_customers) AS s
ON t.customer_id = s.customer_id
WHEN MATCHED AND s.updated_at > t.updated_at THEN
  UPDATE SET *
WHEN NOT MATCHED THEN
  INSERT *
WHEN NOT MATCHED BY SOURCE AND t.is_active = true THEN
  UPDATE SET is_active = false;

OPTIMIZE delta.`s3://datalake/customers`
  WHERE updated_at >= current_date() - INTERVAL 7 DAYS;

VACUUM delta.`s3://datalake/customers` RETAIN 168 HOURS;

Delta is weaker outside Databricks because the operational tooling (OPTIMIZE, Z-Order, Predictive Optimization) is automated only inside Databricks. You can run OSS Delta, but the operational gap with the Databricks-managed experience is wide.

Apache Hudi — the CDC and incremental champion

Hudi was built at Uber, so it is optimized for stream-to-batch CDC. Hudi 1.0 went GA, and the two table types remain key.

Hudi also exposes a TIMELINE metadata structure that arranges every commit chronologically and supports incremental queries that read "only the last hour of changes." That makes it powerful inside CDC pipelines.

Adoption is in decline in 2026, but it remains active inside AWS EMR, OneHouse / Onehouse Open Engines, and similar offerings.

Spark 4.0 — still the center of data engineering

"Is Spark dead?" comes up often, and the 2026 answer is clearly no. Spark 4.0 went GA in late 2025, holding its position as the central processing engine for data engineering. Highlights of 4.0:

Photon is Databricks's proprietary vectorized engine, and the OSS equivalent — Apache Gluten plus Velox/ClickHouse backends — is catching up fast. EMR Serverless, Dataproc Serverless, and Synapse Spark all add their own acceleration engines.

Spark Structured Streaming is micro-batch compared to Flink, but as of 2026 Continuous Processing has stabilized enough to reclaim some of Flink's territory.

Spark Structured Streaming example — Kafka to Iceberg

The Spark Structured Streaming plus Iceberg combo is the most common real-time ingestion pattern in May 2026.

from pyspark.sql import SparkSession
from pyspark.sql.functions import from_json, col, current_timestamp
from pyspark.sql.types import StructType, StringType, LongType

spark = (SparkSession.builder
    .appName("kafka-to-iceberg")
    .config("spark.sql.catalog.lakehouse", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.lakehouse.type", "rest")
    .config("spark.sql.catalog.lakehouse.uri", "https://catalog.internal:8181")
    .getOrCreate())

schema = StructType().add("event_id", LongType()).add("user_id", LongType()).add("event_type", StringType())

stream = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "kafka:9092")
    .option("subscribe", "user_events")
    .option("startingOffsets", "latest")
    .load()
    .select(from_json(col("value").cast("string"), schema).alias("d"))
    .select("d.*", current_timestamp().alias("ingested_at")))

(stream.writeStream
    .format("iceberg")
    .option("checkpointLocation", "s3://lakehouse/_checkpoints/user_events")
    .outputMode("append")
    .toTable("lakehouse.analytics.user_events"))

The Iceberg catalog is REST, so the separate Hive Metastore dependency goes away, and checkpoints live in S3. In 2024 this pattern was unusual; in 2026 it is nearly the standard.

Flink — when you need real streaming

Flink remains the first choice when you need true event-time streaming, exactly-once semantics, and complex windowing. As of May 2026, Flink 1.20 / 2.0 leads the trend.

Flink beats Spark Streaming in low latency plus accurate event-time windowing plus backpressure handling. Spark Streaming is friendlier to ETL and easier to reuse batch code, but sub-100ms latency is hard.

Flink SQL expressed for the same Kafka → Iceberg pattern looks like this.

CREATE TABLE kafka_events (
  event_id BIGINT,
  user_id BIGINT,
  event_type STRING,
  occurred_at TIMESTAMP_LTZ(3),
  WATERMARK FOR occurred_at AS occurred_at - INTERVAL '5' SECOND
) WITH (
  'connector' = 'kafka',
  'topic' = 'user_events',
  'properties.bootstrap.servers' = 'kafka:9092',
  'format' = 'json',
  'scan.startup.mode' = 'latest-offset'
);

CREATE TABLE iceberg_events (
  event_id BIGINT,
  user_id BIGINT,
  event_type STRING,
  occurred_at TIMESTAMP_LTZ(3)
) PARTITIONED BY (`occurred_at`) WITH (
  'connector' = 'iceberg',
  'catalog-type' = 'rest',
  'uri' = 'https://catalog.internal:8181',
  'warehouse' = 's3://lakehouse/'
);

INSERT INTO iceberg_events
SELECT event_id, user_id, event_type, occurred_at
FROM kafka_events;

Trino and Presto — the federation SQL standard

Trino (formerly PrestoSQL) became the standard by focusing on one thing: "SQL federation across many sources." As of May 2026 Trino has 450+ connectors, and these patterns are common.

Presto is the original from Meta; Trino is the fork. As of 2026 OSS activity overwhelmingly favors Trino, while Meta continues evolving Presto for its own infrastructure. Starburst provides the commercial Trino distribution.

A typical federated query looks like the following.

WITH iceberg_orders AS (
  SELECT order_id, user_id, total, ordered_at
  FROM iceberg.sales.orders
  WHERE ordered_at >= date '2026-05-01'
),
pg_users AS (
  SELECT id AS user_id, email, country
  FROM postgres.public.users
),
kafka_clicks AS (
  SELECT user_id, count(*) AS clicks
  FROM kafka.events.clickstream
  WHERE timestamp_kafka >= timestamp '2026-05-01 00:00:00'
  GROUP BY user_id
)
SELECT u.country, sum(o.total) AS revenue, sum(c.clicks) AS clicks
FROM iceberg_orders o
JOIN pg_users u USING (user_id)
LEFT JOIN kafka_clicks c USING (user_id)
GROUP BY u.country
ORDER BY revenue DESC;

Trino joins Iceberg, PostgreSQL, and Kafka inside a single query. Ad-hoc analysis without a separate ETL step is a huge value to BI and analyst teams.

dbt 1.9 + Iceberg — the modern ELT stack joins forces

dbt has become the standard for SQL-based transformation. As of May 2026 dbt Core 1.9 / dbt Cloud bring these changes.

A typical dbt model looks like this.

{{ config(
    materialized='incremental',
    incremental_strategy='microbatch',
    event_time='occurred_at',
    batch_size='day',
    file_format='iceberg',
    partition_by=['days(occurred_at)']
) }}

SELECT
  event_id,
  user_id,
  event_type,
  occurred_at,
  date_trunc('day', occurred_at) AS event_date
FROM {{ source('raw', 'events') }}

{% if is_incremental() %}
WHERE occurred_at >= (SELECT max(occurred_at) FROM {{ this }})
{% endif %}

SQLMesh is rising as a dbt alternative with virtual environments, stronger dimensional modeling, and semantic versions. Coalesce is a UI-based SQL transformation tool friendly to non-technical users.

Orchestration — Airflow 3.0 stood back up

Airflow long suffered the criticism of being heavy and outdated, but Airflow 3.0, GA in late 2025, shifted the field.

Where the alternative orchestrators sit:

ToolStrengthAdoption pattern
Airflow 3.0Generality, ecosystem, connectorsEverywhere
DagsterAsset-centric, type-safe, data visibilityNew data platforms
Prefect 3Dynamic DAGs, Python-friendlyData/ML workflows
Mage AINotebook plus pipelineAnalysts, startups
KestraYAML-based, multi-languageEnterprise
Argo WorkflowsK8s-native, container per stepInfra/platform teams

A typical Airflow DAG looks like this.

from airflow.sdk import dag, task
from datetime import datetime, timedelta

@dag(
    schedule="@daily",
    start_date=datetime(2026, 5, 1),
    catchup=False,
    tags=["lakehouse", "iceberg"],
)
def daily_iceberg_compaction():

    @task
    def list_partitions(target_date: str) -> list[str]:
        return [f"event_date={target_date}"]

    @task
    def compact(partition: str) -> str:
        import subprocess
        subprocess.run([
            "spark-submit", "/opt/jobs/compact.py",
            "--table", "iceberg.analytics.events",
            "--partition", partition,
        ], check=True)
        return partition

    partitions = list_partitions("{{ ds }}")
    compact.expand(partition=partitions)

daily_iceberg_compaction()

Thanks to Airflow 3.0's Task SDK, workers run without Airflow dependencies, and dynamic task mapping (expand) parallelizes per-partition compactions.

Dagster — the asset-centric orchestrator

Dagster's philosophy is "DAG of assets," not "DAG of tasks." Datasets are first-class citizens, and tasks are functions that produce assets.

Where Dagster stands in 2026:

The Dagster Cloud (enterprise SaaS) and OSS Core are clearly separated, which makes the business model stable.

Prefect is strong on dynamic workflows, and after Prefect 3.0 in 2024 the perf/UX improvements have been large. Mage is notebook- and data-native, friendly to analysts. Kestra uses a YAML DSL on the JVM, fitting enterprise environments.

ClickHouse, DuckDB, Pinot, Druid — analytics convergence

In the OLAP layer, the 2026 trend is clear. The "one DB does all analytics" era is over, and each tool has settled into what it does best.

The reason DuckDB grew fastest in 2026 is local-side S3 Parquet/Iceberg queries, support for Python/CLI/SQL, and shipping as a single binary. It is increasingly replacing Snowflake on the analyst's laptop.

Snowflake, Databricks, BigQuery, Redshift — the managed big four

The fully managed warehouse/lakehouse landscape is dominated by four:

The traditional "Snowflake vs Databricks" duopoly blurred in 2026 because both companies are converging on the same direction. Snowflake absorbed lakehouse capabilities through Iceberg plus containers, and Databricks entered the warehouse market with Databricks SQL.

BigQuery is the default on Google Cloud, and Redshift is the choice on AWS where cost optimization matters most.

EMR, Dataproc, HDInsight — managed Hadoop today

All three major clouds run managed Hadoop services.

EMR Serverless is widely adopted in 2026 for the "run a Spark job briefly and disappear" pattern. Running batch ETL on EMR Serverless instead of Lambda is now common.

The legacy of Cloudera + Hortonworks — CDP today

Cloudera and Hortonworks, the commercial center of Hadoop, merged in 2018 and unified into a single product, CDP (Cloudera Data Platform). Where CDP stands in 2026:

OSS operational tools like Apache Bigtop, Apache Ambari, and Apache Ranger survived. Ranger in particular is used as the standard for data governance and access control even in the lakehouse era.

Korean big data operations — Naver, Kakao, KT, Coupang, Woowa Bros

Korean big tech runs OSS Hadoop ecosystems deeply.

In the hiring market, positions listing "Iceberg, dbt, Airflow, Snowflake/Databricks" have grown more visible than "Hadoop, Spark, Hive experience." In 2024 Iceberg was rare in Korean data-engineer postings; in May 2026 it is common across large IT and finance firms.

Japanese big data operations — LINE, ZOZO, NTT, Rakuten, Mercari

Japan is historically more conservative and on-prem-heavy than Korea, but the pace of change in 2026 has accelerated.

A distinguishing feature of Japan is the higher on-prem share than Korea. Finance, telecom, and manufacturing approach cloud migration cautiously due to data-sovereignty concerns. The lakehouse pattern itself, however, is being adopted rapidly.

Kafka and the streaming backbone — the artery of data

At the very center of the data engineering stack sits Kafka. As of May 2026:

The streaming SQL layer splits between Flink SQL and ksqlDB, and the 2026 trend is that Flink is the standard. ksqlDB is strong inside Confluent, but OSS activity strongly favors Flink.

CDC (Change Data Capture) is Debezium's domain, with Flink CDC catching up quickly. Airbyte and Fivetran-style ELT SaaS also support CDC, but at high traffic Debezium + Kafka is the common path.

Airbyte, Fivetran, Stitch — ELT automation

The ingestion side of ELT has standardized on SaaS tools.

Airbyte's 2024 Connector Builder + Low-Code CDK lowered the connector-authoring barrier, and 2026 brings experiments in LLM-driven automatic connector generation.

The Modern Data Stack combination of dbt + Airbyte/Fivetran + Snowflake/BigQuery + Looker/Mode/Hex peaked in 2020-2022; in 2026 it is more common to see dbt + Iceberg + Trino/Databricks blended with the lakehouse trend.

Data catalogs and governance — Unity, Polaris, Nessie, OpenMetadata

As data assets explode, catalogs have become first-class infrastructure.

The 2026 headline is the "Iceberg REST catalog standard". With Snowflake Polaris, Databricks Unity Catalog, Nessie, and AWS Glue all conforming, multiple engines hitting the same data through one catalog finally comes true. The multi-engine promise of the lakehouse era is realized.

Data quality and observability — Monte Carlo, Great Expectations, Soda

Tools that guarantee pipeline quality have become their own category.

Inside dbt you run dbt test for NULL/uniqueness checks, layer Great Expectations / Soda on top for business rules, and use Monte Carlo / Elementary for operational SLAs — that combination is the standard pattern in 2026.

Data mesh and data contracts — where org meets infrastructure

The data mesh concept that emerged between 2020 and 2022 became more practical in 2026. The simple definition:

To realize this, the data contract pattern emerged. A contract that specifies schema, SLA, and semantics. Tools include dbt contracts, Open Data Product Standard, Data Contract CLI, Soda Contracts.

Organizations that fit data mesh are (1) ones with independent domain teams, (2) high data-product demand, and (3) large orgs where the central data team is a bottleneck. In small orgs, a single centralized data team is more efficient.

On-prem big data reality — banks, telecom, government, manufacturing

Despite the cloud era, on-prem big data remains a large market in 2026. Key environments:

  1. Banking and securities: Data sovereignty, compliance (GDPR/PIPA), audit trails. Hadoop + Hive + Spark + Ranger.
  2. Telecom: Petabyte-scale BSS/OSS data, in-network analysis. CDP/HDInsight plus their own OpenStack/K8s.
  3. Government and public: Isolated data centers with air gaps. In-house lakehouses. Iceberg + Spark + Trino on K8s.
  4. Manufacturing: OT (operational technology) plus IT integration. Time-series databases (InfluxDB, TimescaleDB) plus Spark.
  5. Healthcare and insurance: Patient data protection. In-house HDFS or Ceph plus Spark.

The recent shift in on-prem big data is gradual migration to Kubernetes. K8s instead of YARN, MinIO/Ceph instead of HDFS, ArgoCD plus Spark Operator plus Flink Operator instead of Cloudera Manager. In 2026 "you can run a lakehouse architecture on-prem" feels normal.

A Spark Operator manifest looks like this.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: nightly-iceberg-compaction
  namespace: data
spec:
  type: Scala
  mode: cluster
  image: registry.internal/spark:4.0.0
  mainApplicationFile: s3a://jobs/compaction.jar
  sparkVersion: "4.0.0"
  driver:
    cores: 2
    memory: 4g
    serviceAccount: spark
  executor:
    cores: 4
    instances: 8
    memory: 8g
  deps:
    jars:
      - "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.13/1.6.0/iceberg-spark-runtime-3.5_2.13-1.6.0.jar"
  hadoopConf:
    "fs.s3a.endpoint": "https://minio.internal:9000"
    "fs.s3a.path.style.access": "true"

Building and submitting:

# Install the Spark Operator
helm repo add spark-operator https://kubeflow.github.io/spark-operator
helm install spark-operator spark-operator/spark-operator \
  --namespace spark-operator --create-namespace \
  --set sparkJobNamespaces='{data}'

# Quick-start the Iceberg REST catalog
docker run -d --name iceberg-rest \
  -p 8181:8181 \
  -e CATALOG_WAREHOUSE=s3://lakehouse/ \
  -e CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO \
  apache/iceberg-rest-fixture:latest

# Submit the job
kubectl apply -f nightly-iceberg-compaction.yaml
kubectl get sparkapplications -n data

Data engineering hiring market — May 2026 snapshot

Keywords in the data engineering hiring market shifted fast.

The tool frequency in LinkedIn searches for "data engineer" (rough observation, May 2026):

ToolFrequencyNote
SparkVery highEffectively required
AirflowVery highEffectively required
dbtVery highListed in most new postings
Snowflake/BigQuery/DatabricksVery highOne or two required
Iceberg/DeltaHighExploded since 2024
Trino/PrestoMediumRequired in specific domains
FlinkMediumStreaming roles
KafkaVery highEffectively required
KubernetesMedium-highPlatform roles
Hadoop/HiveLow (legacy)Maintenance roles only

The "Hadoop operations" requirement on new postings has fallen sharply. Instead, "experience operating Spark/Trino on top of S3/Iceberg" has become the standard.

Learning path — recommended entry route for 2026 data engineers

If you are starting in data engineering today, this is the recommended May 2026 order.

  1. Master SQL deeply. PostgreSQL or MySQL deeply.
  2. Python + pandas/Polars/DuckDB for local data processing. DuckDB also doubles as lakehouse learning material.
  3. Apache Spark: Start with PySpark. Not via RDDs but via DataFrame/SQL.
  4. Apache Airflow + dbt: Orchestration plus transformation. The core of the modern data stack.
  5. Apache Kafka plus Flink or Spark Streaming for streaming basics.
  6. One cloud: AWS (S3 + EMR + Athena + Glue) or GCP (GCS + Dataproc + BigQuery).
  7. Table format: Iceberg deeply. Delta you absorb naturally inside Databricks environments.
  8. Trino: Federated query plus ad-hoc analysis.
  9. Kubernetes basics: Up to the Spark/Flink Operator level.
  10. Data modeling/Kimball: Dimensional modeling remains essential.

Books: Designing Data-Intensive Applications (Martin Kleppmann), Fundamentals of Data Engineering (Joe Reis & Matt Housley), The Data Warehouse Toolkit (Kimball).

Videos and blogs: Databricks Data + AI Summit, Snowflake Summit, Airflow Summit, Iceberg / Trino Summit keynotes, plus Korean data-engineering conferences (DEvFest, Pycon Korea Data Track).

References

Read the original: https://www.youngju.dev/blog/culture/2026-05-16-hadoop-spark-flink-trino-iceberg-delta-lake-hudi-airflow-dbt-data-engineering-2026-deep-dive.en

Comments

No comments yet.

Sign in to leave a comment