-----Original Message----- From: Beowulf [mailto:beowulf-boun...@beowulf.org] On Behalf Of Joe Landman Sent: Friday, July 27, 2018 11:54 AM To: beowulf@beowulf.org Subject: Re: [Beowulf] Jupyter and EP HPC
On 07/27/2018 02:47 PM, Lux, Jim (337K) wrote: > > I’ve just started using Jupyter to organize my Pythonic ramblings.. > > What would be kind of cool is to have a high level way to do some > embarrassingly parallel python stuff, and I’m sure it’s been done, but > my google skills appear to be lacking (for all I know there’s someone > at JPL who is doing this, among the 6000 people doing stuff here). > > What I’m thinking is this: > > I have a high level python script that iterates through a set of data > values for some model parameter, and farms out running the model to > nodes on a cluster, but then gathers the results back. > > So, I’d have N copies of the python model script on the nodes. > > Almost like a pythonic version of pdsh. > > Yeah, I’m sure I could use lots of subprocess() and execute() stuff > (heck, I could shell pdsh), but like with all things python, someone > has probably already done it before and has all the nice hooks into > the Ipython kernel. > I didn't do this with ipython or python ... but this was effectively the way I parallelized NCBI BLAST in 1998-1999 or so. Wrote a perl script to parse args, construct jobs, move data, submit/manage jobs, recover results, reassemble output. SGI turned that into a product. -- yes.. but I was hoping someone had done that for Jupyter.. >>> for parametervalue in parametervaluelist: .... result = simulation(parametervalue) Results.append(result) _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf