Show HN: Serverless OLAP with Seafowl and GCP

54 points

3 years ago

Hello HN! I'm an engineer at Splitgraph and recently started learning Rust so I could make my first contribution to Seafowl [0], an early stage analytical database. Along the way I figured out a database hosting hack on GCP and wanted to share it with HN. It's a way to achieve "true" scale to zero database hosting that could be useful for certain side projects or spiky traffic situations.

A recurring problem I've faced with side projects is the need for Postgres, but no desire to deploy or maintain new instances. So when I learned GCP's "always free" tier includes serverless [1] I got curious to see if I could run a database.

While a lot of classic databases aren't usually a great fit for serverless, Seafowl separates compute, storage and catalog (catalog == a SQLite file of metadata). [2] Last month I was able to introduce GCS bucket compat to Seafowl, which enabled me to mount the catalog via gcsfuse (i.e. an adapter that allows attaching GCS buckets to local filesystems). Upshot: while FUSE does add HTTP requests to container startup, init time remains comparatively quick, even cold starts, because fetching is limited to the single catalog SQLite file only.

With this approach you get a URL you can query directly from your FE if you want, e.g. fetch() can send SELECT * ... queries straight from your users' browser. You could plot a graph from a static React frontend, or observablehq.com editor, with no persistent backend needed. So at times when nobody's using your app, 100% of your stack can scale to zero with obvious cloud spend advantages. And even if you exceed free tier limits, being PAYG offers a good chance you'll come out ahead on hosting costs anyway.

NB: Seafowl is an early stage project, so it's not really suitable if you need transactions or fast single-row writes. Otherwise, this could be a nice way to get free database hosting at a big 3 cloud provider, especially for e.g. read-only analytical reporting queries.

Feedback and suggestions are appreciated. Hope it helps you! More available if you want [3].

[0] https://seafowl.io/docs/getting-started/introduction

[1] https://cloud.google.com/run/pricing#cpu-requests

[2] Neon is another interesting project that separates compute and storage. https://neon.tech/blog/architecture-decisions-in-neon

One issue I observed was a noticeably longer startup time vs this FUSE approach, which I believe may be related to Postgres connection setup time/roundtrips. Looking forward to trying Neon again in future.

[3] https://www.splitgraph.com/blog/deploying-serverless-seafowl

2 comments