Lakewright solutions / brief
Stream Postgres change data capture into Iceberg for Databricks MERGE
You can tail a Postgres database's changes into an append-only Apache Iceberg changelog — no Kafka, no Debezium cluster, no JVM — and materialize current state in Databricks with one MERGE.
How is this different from Debezium?
Debezium requires Kafka + Kafka Connect — an operational platform of its own. Lakewright is one process speaking Postgres's built-in logical decoding (pgoutput/test_decoding; PG10+, nothing to install server-side). Positions checkpoint two-phase: the LSN is confirmed only after the Iceberg commit lands.
What lands in the changelog?
One row per change: _op (I/U/D/R), _position (LSN), capture time, key columns, and the row image. Deletes are appended _op=D rows — the changelog is immutable and point-in-time reconstructable, which auditors prefer to a mutated mirror.
How do I get current state?
Downstream, where MERGE engines excel:
MERGE INTO orders t USING (
SELECT * FROM orders_changes QUALIFY
ROW_NUMBER() OVER (PARTITION BY id ORDER BY _position DESC) = 1
) s ON t.id = s.id
WHEN MATCHED AND s._op = 'D' THEN DELETE
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED AND s._op != 'D' THEN INSERT *;
Snowflake tasks/streams and dbt incremental models follow the same shape. MySQL binlog sources work identically (--source mysql).
FAQ
The question that matters
"Which orders were amended more than twice before they shipped, and what did the amendments change?"
A question that is simply unanswerable against a mirrored table, because the intermediate states have been overwritten. Against the changelog it is a count of update events per key.
The answer comes back as a ranked list of orders with three or more updates before their fulfilment event, and — because each row carries the full after-image — the assistant can show what actually changed between versions: in the sample data the majority are address corrections clustered in the first hour after order placement, which is a checkout-form problem rather than an operations problem. A smaller group churns on line-item quantity over several days, which is a different conversation entirely, with a different owner.
That distinction is the point. Current-state tables tell you what is true now; the changelog tells you what happened, which is usually where the money is.
Want this run against your data?
We start with a read-only scan and an inventory report — no installation on your systems, nothing leaves your environment. Most engagements produce findings the team didn't know about in the first afternoon.