Tag: #big-data
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 10 posts
Revisiting Hadoop Architecture — HDFS, YARN, and Beyond
A diagram-driven walkthrough of Hadoop core internals — HDFS, YARN, and how MapReduce works. We look at how Hadoop role has shifted in the era of object storage, the cloud, and the lakehouse, and how we should understand
2026-06-27 · 30 min read #hadoop#hdfs#yarn#mapreduce#lakehouseModern Hadoop & Big Data Ecosystem 2026 Deep Dive - Hadoop 3.4 · Spark 4 · Hive 4 · Kafka 4 · Flink 2 · Iceberg · Trino
In 2026 Hadoop and the broader big-data ecosystem are not dead — they have been relocated. Hadoop 3.4 Erasure Coding and Ozone, Spark 4 ANSI mode and Spark Connect, Hive 4 Iceberg integration, Kafka 4 with KRaft GA and t
2026-05-16 · 26 min read #english#hadoop#spark#hive#kafkaHadoop Ecosystem & Data Engineering 2026 Deep Dive - Hadoop, Spark, Flink, Trino, Iceberg, Delta Lake, Hudi, Airflow, dbt
A complete look at the data engineering stack in May 2026. The truth behind the "Hadoop is dead" headline (HDFS is shrinking, but YARN and the lakehouse pattern survive), the conclusion of the Iceberg vs Delta Lake vs Hu
2026-05-16 · 25 min read #hadoop#spark#flink#trino#prestoIs Hadoop Dead? — The Evolution of the Big Data Stack, From Hadoop to Lakehouse (Spark, Iceberg, Delta, and Where Things Actually Stand in 2026)
Is Hadoop dead? More precisely — it has been demoted from the default. The HDFS plus YARN plus MapReduce trio is no longer how anyone starts a new analytics platform. Spark replaced MapReduce, object storage replaced HDF
2026-05-14 · 21 min read #hadoop#big-data#spark#iceberg#delta-lakeColumnar Storage Deep Dive 2025: Parquet, ORC, Apache Arrow, Dremel — Why Analytics DBs Are 10,000x Faster
How Snowflake, BigQuery, and Spark query terabytes in seconds. A complete tour of Parquet Dremel structure, ORC, Apache Arrow in-memory format, RLE/Dictionary compression, and vectorized execution.
2026-04-15 · 13 min read #columnar-storage#parquet#orc#apache-arrow#dremelThe Complete Guide to Probabilistic Data Structures 2025: Bloom Filter, HyperLogLog, Count-Min Sketch, and Real-World Usage
Everything about probabilistic data structures! Bloom Filter, Counting Bloom Filter, HyperLogLog, Count-Min Sketch, MinHash, Cuckoo Filter — the memory-accuracy tradeoff, and real-world usage in RocksDB/Redis/Cassandra/S
2026-04-15 · 14 min read #bloom-filter#hyperloglog#count-min-sketch#probabilistic#data-structuresApache Spark Internals Deep Dive 2025: RDD, Catalyst Optimizer, Tungsten, Whole-Stage Codegen, Shuffle
Why Spark runs the same query 100x faster than MapReduce. RDD to DataFrame/Dataset, Catalyst optimizer, Tungsten project, whole-stage code generation, and shuffle optimization — a complete internal analysis.
2026-04-15 · 13 min read #spark#catalyst#tungsten#rdd#whole-stage-codegenApache Iceberg Data Lakehouse Table Format Guide
A comprehensive guide to Apache Iceberg table format for data lakehouse architecture, covering schema evolution, partitioning, time travel, and query optimization.
2026-03-14 · 27 min read #apache-iceberg#data-lakehouse#big-data#hadoop#sparkHBase Practical Guide: From Large-Scale NoSQL Data Store Design to Operations
A comprehensive guide covering everything you need for real-world HBase operations, from understanding the data model and architecture to table design, RowKey strategies, read/write performance optimization, Region manag
2026-03-08 · 16 min read #hadoop#hbase#nosql#big-data#databaseHadoop Ecosystem Practical Guide: HDFS, MapReduce, and YARN Core Concepts
A comprehensive guide covering the architecture and internals of Hadoop's core components -- HDFS, MapReduce, and YARN -- along with practical operational know-how from cluster setup to performance tuning and monitoring.
2026-03-08 · 14 min read #hadoop#hdfs#mapreduce#yarn#big-data