On Tue 04/15/14 12:38PM +0100, James Harrison wrote: > On 14/04/2014 14:13, Gavin W. Burris wrote: > > Hi, Lawrence. > > > > Maybe rolling something with a ZeroMQ / 0MQ would be a good idea. > > This could be a very fast way to pass many small messages that hand > > off those connections and tasks. > > http://zguide.zeromq.org/page:all > > > > I am very intrigued by this kind of queuing and think it is the > > path forward for many workflows, traditional cluster and cloud. It > > allows for asynchronous communication and fault tolerance in large > > distributed systems. I've used this to wrap a grid engine queue, > > passing python objects that execute in parallel on many nodes. > > It's cool stuff. > > FWIW my $dayjob currently involves building a media analysis system > designed to run on cloud (public, private) infrastructure. ZeroMQ and > friends are not all of the answer by a long shot - there's a lot more > complexity involved in making a workflow and processing system robust > to all the failure conditions you can experience when dealing with > hundreds or thousands of nodes than simple queueing. It is a nice tool > to implement such systems with but don't underestimate the complexity > involved! >
Yes! No doubt! The "simple" queues presuppose a massive distributed system to take advantage of. Bonus points if that system can interchangeably be an in-house cluster or major cloud provider. I would be very interested to hear what your preferred tools and APIs are for your analysis system. I can easily default to the job script and qsub workflow, but restful cloud APIs and simple queues seem to be next-level for some workflows. Cheers, -- Gavin W. Burris Senior IT Project Leader Research and Innovation Team The Wharton School University of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
