Process launchers
October 9, 2013 7:00 PM Subscribe
What are the go-to systems these days for launching and managing a whole bunch of processes?
We're using a home-grown script to launch several thousand shell scripts in a regression test suite, and would like to move to something more robust. At the moment, the needs are pretty simple (keep a fixed number of processes running at any given time, on one CPU.) Maybe someday we'll think about things like farming off to clusters, but for right now what's most important is finding something robust and easy to use.
I've run into one candidate (celery), but I've been surprised to not find a whole slew, so I think I've got the wrong google keywords in mind. (wasn't even sure what to put in the keywords here.) I imagine there's a range of such systems, emphasizing different features. Can you help me understand the playing field?
We're using a home-grown script to launch several thousand shell scripts in a regression test suite, and would like to move to something more robust. At the moment, the needs are pretty simple (keep a fixed number of processes running at any given time, on one CPU.) Maybe someday we'll think about things like farming off to clusters, but for right now what's most important is finding something robust and easy to use.
I've run into one candidate (celery), but I've been surprised to not find a whole slew, so I think I've got the wrong google keywords in mind. (wasn't even sure what to put in the keywords here.) I imagine there's a range of such systems, emphasizing different features. Can you help me understand the playing field?
I don't have a specific suggestion, other than using the term "job" rather than "process" in your searches.
posted by Good Brain at 10:54 PM on October 9, 2013
posted by Good Brain at 10:54 PM on October 9, 2013
Also "batch" and "queuing."
posted by Good Brain at 10:56 PM on October 9, 2013
posted by Good Brain at 10:56 PM on October 9, 2013
Response by poster: Thanks for the keywords.
Using xargs is a fascinatingly simple idea (and I didn't know it could do the parallelization management like that.) There's a funny thing about it, which is that if you leave off the first argument, it assumes you want "echo", so you can't just pass in a list of lines you could type at the shell.
posted by spbmp at 6:44 AM on October 10, 2013
Using xargs is a fascinatingly simple idea (and I didn't know it could do the parallelization management like that.) There's a funny thing about it, which is that if you leave off the first argument, it assumes you want "echo", so you can't just pass in a list of lines you could type at the shell.
posted by spbmp at 6:44 AM on October 10, 2013
Best answer: xargs led me to gnu parallel, which looks like it might wind up working out great for us (and has a lot of the features we may want to grow into. This tutorial illustrates a lot of the capabilities.)
posted by spbmp at 8:40 AM on October 10, 2013
posted by spbmp at 8:40 AM on October 10, 2013
This thread is closed to new comments.
posted by gryftir at 9:56 PM on October 9, 2013