Where data pipelines meet AI
Hands-on notes from a working data engineer on data pipelines, LLMs, RAG, and the tools that connect them.
- Data Engineering
- LLMs & RAG
- AI Tools
- Spark & Lakehouse
- Career & Learning
Latest posts
Kubernetes Scale-Down Was Eating Our Messages: Finding the Missing 0.01%
An email ETL pipeline lost 10-odd messages per 100,000: pods killed mid-message on scale-down. The fix: a graceful exit plus a fallback queue.
Migrating Production Spark Pipelines off Databricks: What Broke, and How We Proved the Data Matched
Moving dozens of Spark pipelines off Databricks: silent SQL translation bugs, hidden state, stale inputs, hash-based parity checks, and how AI helped.
How MapReduce Works: From Hadoop Word Count to GROUP BY in Spark SQL
How Hadoop MapReduce works, with diagrams, a runnable word count, and how a GROUP BY on Databricks or WeData turns into map, shuffle and reduce.
How Spark Works: Jobs, Stages and Tasks, Explained with an Email Header Analysis
How Apache Spark runs your code: driver, executors, jobs, stages, tasks and the shuffle, shown with a Zeppelin notebook that analyzes email headers on S3.
Building an Automated Data Quality Framework: Rules, Trend Detection, and LLM False-Alarm Filtering
How to build automated data quality checks: rule-based checkers, regression trend detection, ID pattern analysis, and LLM false-alarm filtering.
Graceful Shutdown for ETL Workers on Kubernetes: Stop Losing Data on Scale-Down
Handle SIGTERM in Go and Python ETL workers, size terminationGracePeriodSeconds, and avoid the pitfalls that still lose data when EKS scales down.