Process launchers
October 9, 2013 7:00 PM   Subscribe

What are the go-to systems these days for launching and managing a whole bunch of processes?

We're using a home-grown script to launch several thousand shell scripts in a regression test suite, and would like to move to something more robust. At the moment, the needs are pretty simple (keep a fixed number of processes running at any given time, on one CPU.) Maybe someday we'll think about things like farming off to clusters, but for right now what's most important is finding something robust and easy to use.

I've run into one candidate (celery), but I've been surprised to not find a whole slew, so I think I've got the wrong google keywords in mind. (wasn't even sure what to put in the keywords here.) I imagine there's a range of such systems, emphasizing different features. Can you help me understand the playing field?
posted by spbmp to Technology (6 answers total) 2 users marked this as a favorite
If you have access to gnu xargs, it will have a -P option which lets you run processes in parallel with a set limit.
posted by gryftir at 9:56 PM on October 9, 2013

I don't have a specific suggestion, other than using the term "job" rather than "process" in your searches.
posted by Good Brain at 10:54 PM on October 9, 2013

Also "batch" and "queuing."
posted by Good Brain at 10:56 PM on October 9, 2013

What about an automation tool like Testing Anywhere?
posted by judith at 3:42 AM on October 10, 2013

Thanks for the keywords.

Using xargs is a fascinatingly simple idea (and I didn't know it could do the parallelization management like that.) There's a funny thing about it, which is that if you leave off the first argument, it assumes you want "echo", so you can't just pass in a list of lines you could type at the shell.
posted by spbmp at 6:44 AM on October 10, 2013

xargs led me to gnu parallel, which looks like it might wind up working out great for us (and has a lot of the features we may want to grow into. This tutorial illustrates a lot of the capabilities.)
posted by spbmp at 8:40 AM on October 10, 2013

« Older Help me buy a new computer   |   I want to learn about electronics, not plumbing. Newer »
This thread is closed to new comments.