Show HN: SwiftLake – Lightweight SQL Engine for Iceberg Tables

6 points

9 months ago

At Arcesium, we needed a way to run SQL-driven CRUD operations, manage slowly-changing dimensions, and handle schema evolution on our cloud data lakes—without the overhead of spinning up distributed clusters or relying on vendor-managed query services.

That’s why we built SwiftLake, a lightweight open-source library that brings Apache Iceberg and DuckDB together:

  - Lightweight & cluster-free: Runs as a single Java service with embedded DuckDB—no  distributed infra required.
  - Full CRUD and SCD1/SCD2 support: Reads, inserts/appends, updates, deletes, and merges with ACID guarantees.
  - Iceberg advantages: Schema evolution, partition evolution, and time travel.
  - Developer-friendly APIs: Simple interfaces to embed directly into apps or data workflows.
  - Cloud storage support: Works with S3 today, with GCS and ADLS coming soon.

SwiftLake is a good fit for ETL/ELT pipelines, data quality checks, and analytical workloads where spinning up heavy distributed systems would be overkill.

SwiftLake is open source under Apache 2.0:

  - GitHub: https://github.com/arcesium/swiftlake
  - Blog: https://medium.com/arcesium-engineering-blog/your-data-just-got-swifter-meet-swiftlake-7568cc85ee57

We’re just getting started—would love to hear your feedback and ideas from the HN community.