Lakewright solutions / brief
Stream Postgres change data capture into Iceberg for Databricks MERGE
You can tail a Postgres database's changes into an append-only Apache Iceberg changelog — no Kafka, no Debezium cluster, no JVM — and materialize current state in Databricks with one MERGE.
How is this different from Debezium?
Debezium requires Kafka + Kafka Connect — an operational platform of its own. Lakewright is one process speaking Postgres's built-in logical decoding (pgoutput/test_decoding; PG10+, nothing to install server-side). Positions checkpoint two-phase: the LSN is confirmed only after the Iceberg commit lands.
What lands in the changelog?
One row per change: _op (I/U/D/R), _position (LSN), capture time, key columns, and the row image. Deletes are appended _op=D rows — the changelog is immutable and point-in-time reconstructable, which auditors prefer to a mutated mirror.
How do I get current state?
Downstream, where MERGE engines excel:
MERGE INTO orders t USING (
SELECT * FROM orders_changes QUALIFY
ROW_NUMBER() OVER (PARTITION BY id ORDER BY _position DESC) = 1
) s ON t.id = s.id
WHEN MATCHED AND s._op = 'D' THEN DELETE
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED AND s._op != 'D' THEN INSERT *;
Snowflake tasks/streams and dbt incremental models follow the same shape. MySQL binlog sources work identically (--source mysql).