Situation: I have lot of large file conversions from many customers which needs to be executed. Currently, I have a dashboard created which POSTs to a remote server. The endpoint on the remote server will shell execute a python script to launch file conversion. When it finishes, the python script will POST back to the server running dashboard.I've been looking at solutions like https://github.com/irmen/Pyro4 but I can't help but feel like I'm reinventing the wheel here. What I would need is a way to launch long running tasks on a cluster of remote worker servers that would be aware of load balancing.
What task management framework is suitable for my case?