Tag: #data-engineering
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 26 posts
Following 657,607 Links, and the Lifespan of a URL — Why Links Die of Connection Failure, Not 404
A study published its results after following, in 2026, every one of roughly 655,000 shortened links created between 2009 and 2014. More important than the headline number is the composition of the failures: most dead li
2026-08-14 · 6 min read #web#data-engineering#archival#reliability#url-designOpen Source Worth Watching Right Now (5) Data and ML Pipelines
A data pipeline is not something a single scheduler solves. Ingestion, transformation, orchestration, execution engines, the model lifecycle, and search stores have each become the territory of a different tool. This pos
2026-08-12 · 5 min read #open-source#data-engineering#mlops#python#rustHow to Read the 2026 Data Tooling Landscape — Layer-by-Layer Decisions, Not a Tool List
On July 28, 2026, "A Data Tooling Landscape Guide for Developers" landed on GeekNews and picked up 51 points. It is a good map, but a map does not choose your destination for you. This post redraws the same landscape acr
2026-07-31 · 16 min read #data-engineering#lakehouse#iceberg#duckdb#data-platformAfter Airflow 2 EOL — The Real Work List for Going 2 to 3, and Where 3.x Stands at 3.3
Apache Airflow 2 passed EOL on April 22, 2026, and it no longer receives security patches. But the road from 2 to 3 is not a pip upgrade — it is an architecture change. Direct metadata-DB access from workers disappears,
2026-07-17 · 15 min read #data-engineering#airflow#workflow-engine#migrationParquet 2.13.0 Adds nan_count and IEEE 754 Total Order — How Float Statistics Got Fixed After 3 Years and 3 Months
Parquet's min/max statistics are the mechanism that lets query engines skip data. But on floating-point columns, this mechanism has been subtly broken for over 10 years — NaN returns false in every comparison, so it gets
2026-07-16 · 17 min read #data-engineering#parquet#columnar-storage#apache-arrowIceberg v3 Row Lineage: Row IDs Aren't Stored in the File
Apache Iceberg format v3 writes row lineage — a stable identifier for every row — into the spec, and it isn't optional: any v3 table has it on unconditionally. But that value isn't stored in the data file — it's computed
2026-07-16 · 18 min read #iceberg#lakehouse#table-format#data-engineering#sparkDuckDB Gets a Client-Server Protocol — What Quack Changes and What It Doesn't
On May 12, 2026, the DuckDB team announced Quack — a client-server protocol built on top of HTTP that lets two DuckDB instances become client and server to each other, so the same database can be read and written concurr
2026-07-16 · 18 min read #database#duckdb#olap#analytics#data-engineeringPySpark 4.2 Puts Python UDFs on Arrow by Default — the Story of 132 out of 225 Type-Coercion Cells Changing
Apache Spark 4.2.0, released on July 14, 2026, flipped the default of spark.sql.execution.pythonUDF.arrow.enabled from false to true under SPARK-54555. Vendor blogs describe this as a faster columnar path without rewriti
2026-07-16 · 16 min read #spark#python#arrow#data-engineeringFrom Documents to a Knowledge Graph: An Honest Pipeline
'Extract a knowledge graph from your documents' looks like a single LLM call in a demo. But turning a customer's documents into a graph you can actually query is a six-stage pipeline, and most of the cost and pain lives
2026-07-15 · 10 min read #knowledge-graph#ai#llm#data-engineeringThe Complete Guide to LLM Training Data Preprocessing — From Web Crawls to Token Packing, with the Latest Papers
Good models come from good data, and good data comes from a preprocessing pipeline. This post walks through the entire pretraining data process step by step — web crawl collection → text extraction → language identificat
2026-07-09 · 7 min read #ai#llm#data-engineering#preprocessing#trainingDuckDB in Practice — The Era of Embedded Analytics That Ends on Your Laptop
Why is DuckDB, often called the SQLite of OLAP, so loved? We start from the architecture and cover direct Parquet querying, hands-on examples of window functions, PIVOT, and ASOF JOIN, zero-copy integration with pandas,
2026-06-12 · 14 min read #duckdb#database#olap#analytics#parquetHadoop Ecosystem & Data Engineering 2026 Deep Dive - Hadoop, Spark, Flink, Trino, Iceberg, Delta Lake, Hudi, Airflow, dbt
A complete look at the data engineering stack in May 2026. The truth behind the "Hadoop is dead" headline (HDFS is shrinking, but YARN and the lakehouse pattern survive), the conclusion of the Iceberg vs Delta Lake vs Hu
2026-05-16 · 25 min read #hadoop#spark#flink#trino#prestoData Lakehouse & Modern Data Engineering 2026 — Iceberg / Delta / Hudi / Paimon / Tabular (Databricks acquisition) / Trino / Spark 4 / Flink 2 / DataFusion Deep Dive
Data engineering in 2026 is no longer the era of "data warehouse vs data lake." Apache Iceberg emerged as the winner of the 2024-25 table format war — Netflix, Apple, LinkedIn, Stripe, Airbnb all converged on it. Databri
2026-05-16 · 30 min read #data-engineering#data-lakehouse#apache-iceberg#delta-lake#apache-hudiIs Hadoop Dead? — The Evolution of the Big Data Stack, From Hadoop to Lakehouse (Spark, Iceberg, Delta, and Where Things Actually Stand in 2026)
Is Hadoop dead? More precisely — it has been demoted from the default. The HDFS plus YARN plus MapReduce trio is no longer how anyone starts a new analytics platform. Spark replaced MapReduce, object storage replaced HDF
2026-05-14 · 21 min read #hadoop#big-data#spark#iceberg#delta-lakePolars 1.x vs Pandas — End of an Era? A Modern DataFrame Deep Dive for 2026
Polars hit 1.x with a stable-API promise, Pandas 2.2 finally adopted PyArrow as a real backend, and the gravity of the dataframe world shifted. Is Polars actually replacing Pandas? This deep dive walks through the Rust p
2026-05-14 · 20 min read #polars#pandas#dataframe#apache-arrow#duckdbData Engineering Complete Guide — Lakehouse, Streaming, dbt, Orchestration, Data Mesh (Season 2 Ep 8, 2025)
Data engineering is no longer "running ETL scripts." In 2025 a data engineer designs Lakehouse architectures (Iceberg, Delta, Hudi), combines Streaming (Flink, Kafka) with Batch (Spark), standardizes modeling with dbt, r
2026-04-15 · 10 min read #data-engineering#lakehouse#iceberg#delta-lake#hudiData Engineering Fundamentals — ETL, Data Warehouses, Streaming, and Data Lakes
How do you build data pipelines? From ETL/ELT and data warehouses to streaming, batch processing, Spark, Kafka, and Airflow.
2026-04-13 · 17 min read #data-engineering-aws#data-engineering#etl#data-warehouse#sparkData Engineering Pipeline Complete Guide 2025: ETL/ELT, Spark, Airflow, Real-Time Streaming
Everything about data engineering! ETL vs ELT, Apache Spark (PySpark), Apache Airflow (DAG/Operator/Sensor), real-time streaming (Kafka+Flink), dbt (data transformation), data warehouses (BigQuery/Snowflake/Redshift), da
2026-03-25 · 18 min read #data-engineering#etl#elt#spark#airflowSnowflake Data Engineer Career Guide: Mastering the Cloud Data Warehouse King
A complete analysis of the Snowflake Data Engineer role. Snowflake architecture (compute-storage separation), Snowpark, Cortex AI, Iceberg Tables, dbt integration, streaming (Snowpipe), cost optimization — 20 interview q
2026-03-23 · 24 min read #snowflake#data-engineering#data-warehouse#sql#dbtDatabricks AI Engineer (FDE) Complete Guide: Spark, Unity Catalog, RAG to Customer Deployment
A complete analysis of the Databricks AI Engineer (FDE) JD. From Spark/Delta Lake/Unity Catalog tech stack, Lakehouse architecture, RAG pipeline construction, to customer deployment skills — 25 interview questions and an
2026-03-23 · 34 min read #databricks#fde#spark#delta-lake#unity-catalog