-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 14/04/2014 14:13, Gavin W. Burris wrote: > Hi, Lawrence. > > Maybe rolling something with a ZeroMQ / 0MQ would be a good idea. > This could be a very fast way to pass many small messages that hand > off those connections and tasks. > http://zguide.zeromq.org/page:all > > I am very intrigued by this kind of queuing and think it is the > path forward for many workflows, traditional cluster and cloud. It > allows for asynchronous communication and fault tolerance in large > distributed systems. I've used this to wrap a grid engine queue, > passing python objects that execute in parallel on many nodes. > It's cool stuff.
FWIW my $dayjob currently involves building a media analysis system designed to run on cloud (public, private) infrastructure. ZeroMQ and friends are not all of the answer by a long shot - there's a lot more complexity involved in making a workflow and processing system robust to all the failure conditions you can experience when dealing with hundreds or thousands of nodes than simple queueing. It is a nice tool to implement such systems with but don't underestimate the complexity involved! - -- Cheers, James Harrison -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) iQIcBAEBAgAGBQJTTRpCAAoJENTyYHL8dmp9P9IQAJc6zSv0Soo43oOnnlAEm1Oe +DmpLxsh5a/TkM6JqoEAhzf9MtqAxeznxpSJ0CcJ13K5HidyTqqGmB4nEQV0f3Sd BkTR8E+0KSHIPeHQBikcxMJd5eRm++JvZ9tNaIbqwpDbnJ688S6hoin7dcpAu5J5 Qq+psvazgrAaOaffFid4uBpI18GJh3+kbdEQ6Muda8LQre2EzISA+Q7zTRMnpuN3 bm9cOMoq5ricpbC9MImhJII9M1LpnSUlaqP/yCYqM/Fi29CKNikDev1hWTHAzGn4 8qc7JEhHVpyf2cCAxjOsI7XTDnk+fLoBu9reW+3zn7TjeiTUiAmRTDZ8hMKV/D/7 GGfVvgBTDuIK9RgfpHz0LfaRMi4rhGcUfxqfbSZRQb/XJQuqRJRh98oWY1oY4+do SaoAZwaS/KnhVvj5iYh9c/JEkc23sBhfHcQccpcbaXeFfKn1hVw7nvjvK+Fw2Z8r Ed/xZK9GSS0rw7rMGkL3o6AvNvpgrwojPfQ9yxbd5K1M1+kpZOllh1XNYxY/2Nyz 95Wu715w7BuRDXeykduz7CRRzzdHeJUhvPxzAXL1cXxETFOezS/oD8SaO5HH8uAZ ZKi2/BaoX18ctwxNBp4FE4SbVbs9112VdA6CmYtroRCR3ZCJK1DaJG6LyWmaVqSQ 1zzcVX91ozPOYNd2ld38 =n39x -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
