Show HN: Lightweight Durable Workflows Built on Postgres

5 points

a year ago

Hi HN! This is Qian here with Peter (KraftyOne) and Jeremy (jedberg). We’re building DBOS, an open-source, lightweight durable workflows library that you can add to Python apps in just a few lines of code. It’s comparable to popular open-source workflow and queue libraries like Airflow and Celery, but more lightweight with a greater focus on reliability and automatically recovering from failures.

Our goal in building DBOS is to make workflows lightweight and flexible so you can add them to your existing apps with minimal work. Everything you need to run durable workflows and queues is contained in this Python library. You don’t need to manage a separate workflow server: just install the library, connect it to a Postgres database (to store workflow/queue state) and you’re good to go.

DBOS workflows make your program durable by checkpointing its state in Postgres. If your program ever fails, when it restarts all your workflows will automatically resume from the last completed step. You add durable workflows to your existing program by annotating ordinary functions as workflows and steps:

  from dbos import DBOS

  @DBOS.step()
  def step_one():
      ...

  @DBOS.step()
  def step_two():
      ...

  @DBOS.workflow()
  def workflow():
    step_one()
    step_two()

The workflow is just an ordinary Python function. You can call it any way you like–from a FastAPI handler, in response to events, wherever you’d normally call a function.

We’ve just released DBOS Python 1.0. This enhances workflows with many powerful features we’ve been building over the last few months, including:

- Durable queues. Postgres-backed queues with all the queuing features of BullMQ/Celery (concurrency limits, rate limits, timeouts, priority, deduplication, etc.). Plus, they integrate with durable workflows, so you can write a workflow that enqueues 1K tasks, waits for and processes their results, and automatically recovers from any interruption.

- Programmatic workflow management. Your workflows are stored as rows in a Postgres table, so you have full programmatic control over them. Write scripts to query workflow executions, batch pause or resume workflows, or even restart failed workflows from a specific step. This makes it much easier to diagnose and recover from bugs and failures that affect thousands of workflows.

- Full support for both sync and async Python–write your workflows and steps as code either synchronously or asynchronously, it all works out of the box.

- Improved tooling, including dashboards, workflow graph visualization, workflow management via web UI, and more.

We’d love to hear your feedback and hope you can try DBOS out!