You might find some enlightening tidbits at around the 26:55 mark here re Ipyparallel and Kubernetes:
http://www.rce-cast.com/Podcast/rce-116-jupyter.html On Sat, Jul 28, 2018 at 1:01 PM, Andrew Holway <andrew.hol...@gmail.com> wrote: > Hi Jim, > > There is a group at JPL doing Kubernetes. It might be interesting to ask > them if you can execute Jobs on their clusters. > > Cheers, > > Andrew > > On 27 July 2018 at 20:47, Lux, Jim (337K) <james.p....@jpl.nasa.gov> > wrote: > >> I’ve just started using Jupyter to organize my Pythonic ramblings.. >> >> >> >> What would be kind of cool is to have a high level way to do some >> embarrassingly parallel python stuff, and I’m sure it’s been done, but my >> google skills appear to be lacking (for all I know there’s someone at JPL >> who is doing this, among the 6000 people doing stuff here). >> >> >> >> What I’m thinking is this: >> >> I have a high level python script that iterates through a set of data >> values for some model parameter, and farms out running the model to nodes >> on a cluster, but then gathers the results back. >> >> >> >> So, I’d have N copies of the python model script on the nodes. >> >> Almost like a pythonic version of pdsh. >> >> >> >> Yeah, I’m sure I could use lots of subprocess() and execute() stuff >> (heck, I could shell pdsh), but like with all things python, someone has >> probably already done it before and has all the nice hooks into the Ipython >> kernel. >> >> >> >> >> >> >> >> James Lux >> >> Task Manager, DARPA High Frequency Research (DHFR) Space Testbed >> >> Jet Propulsion Laboratory (Mail Stop 161-213) >> >> 4800 Oak Grove Drive >> >> Pasadena CA 91109 >> >> (818)354-2075 (office) >> >> (818)395-2714 (cell) >> >> >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > >
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
