Ducklake is a new lake storage format by MotherDuck. It aims to solve some of the issues with Iceberg by centralizing metadata in Postgres, instead of directly on blob storage.SQLFlow is a stream processing engine that ingests data from kafka, runs sql against that stream and sinks the output.
SQLFlow has a duckdb context available during stream processing. This make it trivial to stream data from Kafka to Ducklake!
https://sql-flow.com/docs/tutorials/ducklake-sink/
https://github.com/turbolytics/sql-flow