Notes on data engineering, cloud & integrations.
A walkthrough of turning a Confluence space into a queryable, cited knowledge base using Cloud Composer, BigQuery's native AI functions, and Gemini — including the silent failures that almost made it into production.
A practical walkthrough of building RAG on Google Cloud — RAG Engine, Vector Search, and BigQuery vector search — including the failure modes that actually show up once a prototype becomes a production pipeline.
When a data platform grows to hundreds of pipelines, "does this table still exist?" stops being a question you can answer by memory. Here's how to turn INFORMATION_SCHEMA into an actual audit system instead of a one-off query you run when something breaks.
A single Cloud Composer environment works fine until it doesn't. Here's what actually breaks as DAG count and team count grow, and what a real production cutover to a multi-instance architecture looks like in practice.
Why the same Beam pipeline behaves completely differently depending on whether it's reading a bounded batch or an unbounded stream — and how that difference shapes the way data actually lands in your BigQuery tables.