Lakewright solutions / brief
Automate SFTP CSV drops into Apache Iceberg tables
The vendor CSV drop — daily files, shifting schemas, no contract — becomes a governed feed: detected, deduplicated, loaded to Iceberg, and watched for new arrivals under a human-approved plan.
What makes vendor drops hard?
Encodings (UTF-16, Windows-1252), delimiter surprises, renamed columns, re-sent files, and gzip/zip wrappers. Lakewright detects encoding and dialect by evidence, expands archives transparently, journals every file by content digest (re-sends are no-ops), and surfaces schema drift instead of loading it silently.
The steps
lakewright scan sftp-drop/ lakewright approve plans/vendor_returns.plan.yaml lakewright serve --plans plans --rest $CATALOG # the ever-running watcher
FAQ
The question that matters
"Which vendors are actually delivering on schedule, and did anyone send us the same file twice?"
Punctuality becomes a metric rather than an impression.
The assistant answered with vendors ranked by median lateness against their own established cadence — not against a single SLA, because each vendor has a different rhythm and the watcher learned each one. Two vendors drift consistently by a day or more; one is erratic, which is worse operationally than consistently late because nothing downstream can be scheduled around it.
On the duplicate question the journal answers directly: several arrivals during the quarter were byte-identical re-sends that landed nothing, and the assistant can list them by vendor and date. That is the conversation to have with a supplier — not "you are unreliable," but "these seven files on these dates were resent, and this one arrived four days late three months running."
Want this run against your data?
We start with a read-only scan and an inventory report — no installation on your systems, nothing leaves your environment. Most engagements produce findings the team didn't know about in the first afternoon.