Tag: #shuffle
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 2 posts
Apache Spark Internals Deep Dive 2025: RDD, Catalyst Optimizer, Tungsten, Whole-Stage Codegen, Shuffle
Why Spark runs the same query 100x faster than MapReduce. RDD to DataFrame/Dataset, Catalyst optimizer, Tungsten project, whole-stage code generation, and shuffle optimization — a complete internal analysis.
2026-04-15 · 13 min read #spark#catalyst#tungsten#rdd#whole-stage-codegenApache Spark in Production: Performance Tuning, Shuffle, Skew, AQE, and Streaming Operations
A practical production guide to Apache Spark covering the execution model that matters, shuffle and skew, AQE, cache strategy, and Structured Streaming operational checklists.
2026-03-17 · 6 min read #spark#spark-tuning#data-engineering#shuffle#adaptive-query-execution