Show HN: Pq – Simple, durable background tasks in Python using Postgres

5 points

3 months ago

At work we were using python-rq for background tasks. It does the job for simple things, but we kept bumping into limitations. We needed to schedule tasks hours / days out and trust they'd survive a restart. We wanted periodic tasks with proper overlap control. So we built a scheduling / enqueuing system around Postgres to bring these durability capabilities to python-rq. This worked fine for a while but was trickier to reason about due to its more complicated architecture (we'd run two separate services just for getting jobs from Postgres into the rq Redis queue, plus N actual task workers).

pq is a simpler approach: It's a Postgres-backed background-task library for Python, using SELECT ... FOR UPDATE SKIP LOCKED for concurrency safety. You just run your N task workers, that's it. The stuff I think is worth knowing about: - Transactional enqueueing -- you can enqueue a task inside the same DB transaction as your writes. If the transaction rolls back, the task never exists. This is the thing Redis literally can't give you. Fork isolation -- every task runs in a forked child process. If it OOMs, segfaults, or leaks memory, the worker just keeps going. The parent monitors via os.wait4() and handles timeouts, OOM kills, and crashes cleanly. - Periodic tasks -- intervals or cron expressions, with overlap control (skip the tick if the previous run is still going), pause/resume without deleting the schedule. Priority queues -- five levels from BATCH to CRITICAL. You can dedicate workers to only process tasks with specific priorities. - Three tables in your main database schema: pq_tasks for one-off tasks, pq_periodic for periodic tasks and pq_alembic_version to track its own schema version (pq manages its own migrations).

There's also client IDs for idempotency and correlation with other tables in your application DB, upsert for debouncing (only the latest version of a task runs), lifecycle hooks that execute in the forked child (useful for fork-unsafe stuff like OpenTelemetry), and async task support.

What it won't do: high throughput (you're polling Postgres). If you need 10k+ tasks/sec or complex DAGs, use something else. For the kind of workload most apps actually have, it's probably sufficient.

pip install python-pq // uv add python-rq repo at https://github.com/ricwo/pq, docs at https://ricwo.github.io/pq