Tag: #spark
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 15 posts
Iceberg v3 Row Lineage: Row IDs Aren't Stored in the File
Apache Iceberg format v3 writes row lineage — a stable identifier for every row — into the spec, and it isn't optional: any v3 table has it on unconditionally. But that value isn't stored in the data file — it's computed
2026-07-16 · 18 min read #iceberg#lakehouse#table-format#data-engineering#sparkPySpark 4.2 Puts Python UDFs on Arrow by Default — the Story of 132 out of 225 Type-Coercion Cells Changing
Apache Spark 4.2.0, released on July 14, 2026, flipped the default of spark.sql.execution.pythonUDF.arrow.enabled from false to true under SPARK-54555. Vendor blogs describe this as a faster columnar path without rewriti
2026-07-16 · 16 min read #spark#python#arrow#data-engineeringModern Hadoop & Big Data Ecosystem 2026 Deep Dive - Hadoop 3.4 · Spark 4 · Hive 4 · Kafka 4 · Flink 2 · Iceberg · Trino
In 2026 Hadoop and the broader big-data ecosystem are not dead — they have been relocated. Hadoop 3.4 Erasure Coding and Ozone, Spark 4 ANSI mode and Spark Connect, Hive 4 Iceberg integration, Kafka 4 with KRaft GA and t
2026-05-16 · 26 min read #english#hadoop#spark#hive#kafkaModern Scala 2026 — Scala 3.6 / 3.7 / ZIO 2 / Cats Effect 3 / Pekko / Tapir / Mill / Scala CLI Deep Dive
Everything about the 2026 Scala ecosystem — new features in Scala 3.6/3.7, Scala 4 preview, sbt 2.x vs Mill vs Scala CLI, ZIO 2 vs Cats Effect 3, the Pekko/Akka BSL drama, Play/http4s/Tapir, Scala.js/Native, Spark, Liche
2026-05-16 · 26 min read #scala#scala-3#sbt#mill#scala-cliHadoop Ecosystem & Data Engineering 2026 Deep Dive - Hadoop, Spark, Flink, Trino, Iceberg, Delta Lake, Hudi, Airflow, dbt
A complete look at the data engineering stack in May 2026. The truth behind the "Hadoop is dead" headline (HDFS is shrinking, but YARN and the lakehouse pattern survive), the conclusion of the Iceberg vs Delta Lake vs Hu
2026-05-16 · 25 min read #hadoop#spark#flink#trino#prestoData Lakehouse & Modern Data Engineering 2026 — Iceberg / Delta / Hudi / Paimon / Tabular (Databricks acquisition) / Trino / Spark 4 / Flink 2 / DataFusion Deep Dive
Data engineering in 2026 is no longer the era of "data warehouse vs data lake." Apache Iceberg emerged as the winner of the 2024-25 table format war — Netflix, Apple, LinkedIn, Stripe, Airbnb all converged on it. Databri
2026-05-16 · 30 min read #data-engineering#data-lakehouse#apache-iceberg#delta-lake#apache-hudiIs Hadoop Dead? — The Evolution of the Big Data Stack, From Hadoop to Lakehouse (Spark, Iceberg, Delta, and Where Things Actually Stand in 2026)
Is Hadoop dead? More precisely — it has been demoted from the default. The HDFS plus YARN plus MapReduce trio is no longer how anyone starts a new analytics platform. Spark replaced MapReduce, object storage replaced HDF
2026-05-14 · 21 min read #hadoop#big-data#spark#iceberg#delta-lakeColumnar Storage Deep Dive 2025: Parquet, ORC, Apache Arrow, Dremel — Why Analytics DBs Are 10,000x Faster
How Snowflake, BigQuery, and Spark query terabytes in seconds. A complete tour of Parquet Dremel structure, ORC, Apache Arrow in-memory format, RLE/Dictionary compression, and vectorized execution.
2026-04-15 · 13 min read #columnar-storage#parquet#orc#apache-arrow#dremelApache Spark Internals Deep Dive 2025: RDD, Catalyst Optimizer, Tungsten, Whole-Stage Codegen, Shuffle
Why Spark runs the same query 100x faster than MapReduce. RDD to DataFrame/Dataset, Catalyst optimizer, Tungsten project, whole-stage code generation, and shuffle optimization — a complete internal analysis.
2026-04-15 · 13 min read #spark#catalyst#tungsten#rdd#whole-stage-codegenData Engineering Fundamentals — ETL, Data Warehouses, Streaming, and Data Lakes
How do you build data pipelines? From ETL/ELT and data warehouses to streaming, batch processing, Spark, Kafka, and Airflow.
2026-04-13 · 17 min read #data-engineering-aws#data-engineering#etl#data-warehouse#sparkData Engineering Pipeline Complete Guide 2025: ETL/ELT, Spark, Airflow, Real-Time Streaming
Everything about data engineering! ETL vs ELT, Apache Spark (PySpark), Apache Airflow (DAG/Operator/Sensor), real-time streaming (Kafka+Flink), dbt (data transformation), data warehouses (BigQuery/Snowflake/Redshift), da
2026-03-25 · 18 min read #data-engineering#etl#elt#spark#airflowDatabricks AI Engineer (FDE) Complete Guide: Spark, Unity Catalog, RAG to Customer Deployment
A complete analysis of the Databricks AI Engineer (FDE) JD. From Spark/Delta Lake/Unity Catalog tech stack, Lakehouse architecture, RAG pipeline construction, to customer deployment skills — 25 interview questions and an
2026-03-23 · 34 min read #databricks#fde#spark#delta-lake#unity-catalogApache Spark in Production: Performance Tuning, Shuffle, Skew, AQE, and Streaming Operations
A practical production guide to Apache Spark covering the execution model that matters, shuffle and skew, AQE, cache strategy, and Structured Streaming operational checklists.
2026-03-17 · 6 min read #spark#spark-tuning#data-engineering#shuffle#adaptive-query-executionApache Iceberg Data Lakehouse Table Format Guide
A comprehensive guide to Apache Iceberg table format for data lakehouse architecture, covering schema evolution, partitioning, time travel, and query optimization.
2026-03-14 · 27 min read #apache-iceberg#data-lakehouse#big-data#hadoop#sparkHow to Install Spark on YARN
Learn how to install Spark on YARN.
2022-11-26 · 7 min read #hadoop#spark#yarn