Which customers are about to stop buying?
A medallion lakehouse over a year of real transactions from a UK online retailer: daily files land in bronze, are cleaned and merged into silver with quality gates, become customer features in gold, and feed a churn model evaluated on a month it never saw.
Pipeline
Every table is a Delta table: ACID writes, schema enforcement, and a version history that the gold layer pins, so any snapshot can be rebuilt exactly.
Daily incremental runs
Incremental vs full rebuild
One new day processed with MERGE, compared with rebuilding silver from all of bronze.
Data quality
Every batch passes through the same gate. Critical checks stop the run before anything downstream is built; warnings are tracked over time. Rows that can't be trusted go to quarantine with a reason rather than disappearing.
Checks across all batches
What the gate caught
Rows kept out of silver, and why.
Table layout: Z-ordering
Churn model
Out-of-time test
Cumulative gains
Share of next quarter's churners found by contacting the riskiest X% of customers.
What drives the prediction
Churn rate by snapshot
Customers active before each date who didn't buy in the following 90 days.
Highest-risk customers today
How it works
Bronze: land as received
Daily CSV drops are appended to Delta as strings with source file and ingest time. Files already loaded are skipped, so re-runs never duplicate a day.
Silver: MERGE, don't append
Rows are typed with try_cast (a bad value quarantines one row instead of failing the batch), hashed into a line_id, and MERGEd, pruned to the affected month partitions.
Gold: point-in-time features
Monthly snapshots of 19 behavioural features per customer, computed only from data before the snapshot, with a 90-day churn label. Built from a pinned silver version (time travel).
Quality gates
Row reconciliation (every input row inserted, de-duplicated, already present or quarantined), unique keys, required columns, and drift warnings, stored as a Delta table.
Out-of-time evaluation
Trained on March–July snapshots, model chosen on August, tested once on September. Spark ML pipelines logged to MLflow with metrics and the fitted model.
Reproducible
CI downloads the data, runs the whole pipeline on a fresh Spark + Delta, runs unit tests on every layer, and fails if row counts or model metrics differ from the published ones.
Data: Online Retail (Chen, Sain & Guo, 2012), 1 Dec 2010 – 9 Dec 2011, CC0 via the onlineretail CRAN package. Customer IDs are the dataset's anonymised numbers.