Tag: #apache-arrow
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 4 posts
Parquet 2.13.0 Adds nan_count and IEEE 754 Total Order — How Float Statistics Got Fixed After 3 Years and 3 Months
Parquet's min/max statistics are the mechanism that lets query engines skip data. But on floating-point columns, this mechanism has been subtly broken for over 10 years — NaN returns false in every comparison, so it gets
2026-07-16 · 17 min read #data-engineering#parquet#columnar-storage#apache-arrowData Lakehouse & Modern Data Engineering 2026 — Iceberg / Delta / Hudi / Paimon / Tabular (Databricks acquisition) / Trino / Spark 4 / Flink 2 / DataFusion Deep Dive
Data engineering in 2026 is no longer the era of "data warehouse vs data lake." Apache Iceberg emerged as the winner of the 2024-25 table format war — Netflix, Apple, LinkedIn, Stripe, Airbnb all converged on it. Databri
2026-05-16 · 30 min read #data-engineering#data-lakehouse#apache-iceberg#delta-lake#apache-hudiPolars 1.x vs Pandas — End of an Era? A Modern DataFrame Deep Dive for 2026
Polars hit 1.x with a stable-API promise, Pandas 2.2 finally adopted PyArrow as a real backend, and the gravity of the dataframe world shifted. Is Polars actually replacing Pandas? This deep dive walks through the Rust p
2026-05-14 · 20 min read #polars#pandas#dataframe#apache-arrow#duckdbColumnar Storage Deep Dive 2025: Parquet, ORC, Apache Arrow, Dremel — Why Analytics DBs Are 10,000x Faster
How Snowflake, BigQuery, and Spark query terabytes in seconds. A complete tour of Parquet Dremel structure, ORC, Apache Arrow in-memory format, RLE/Dictionary compression, and vectorized execution.
2026-04-15 · 13 min read #columnar-storage#parquet#orc#apache-arrow#dremel