On 07/27/2018 02:47 PM, Lux, Jim (337K) wrote:
I’ve just started using Jupyter to organize my Pythonic ramblings..
What would be kind of cool is to have a high level way to do some
embarrassingly parallel python stuff, and I’m sure it’s been done, but
my google skills appear to be lacking (for all I know there’s someone
at JPL who is doing this, among the 6000 people doing stuff here).
What I’m thinking is this:
I have a high level python script that iterates through a set of data
values for some model parameter, and farms out running the model to
nodes on a cluster, but then gathers the results back.
So, I’d have N copies of the python model script on the nodes.
Almost like a pythonic version of pdsh.
Yeah, I’m sure I could use lots of subprocess() and execute() stuff
(heck, I could shell pdsh), but like with all things python, someone
has probably already done it before and has all the nice hooks into
the Ipython kernel.
I didn't do this with ipython or python ... but this was effectively the
way I parallelized NCBI BLAST in 1998-1999 or so. Wrote a perl script
to parse args, construct jobs, move data, submit/manage jobs, recover
results, reassemble output. SGI turned that into a product.
--
Joe Landman
e: joe.land...@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
