Ask HN: What do you use to manage chains of long-running jobs?

4 points

9 years ago

For running chains of multi-hour jobs with dependencies I've always used Jenkins but always been unsatisfied with it. I've also heard of Luigi but haven't given it a shot yet. Is there anything else similar I should look at? I've had trouble finding a single name for programs like this to search with.

Luigi: https://github.com/spotify/luigi

Example jobs:

  - download a bunch of raw data from different sources
  - clean and transform raw data into a single format
  - push a lot of data into a DB
  - routine ML model updates
  - generate reports

These would have dependency relations, some would be short and some would be long, sometimes they fail (which may be important or not), etc...

5 comments