Hi, Lawrence. Maybe rolling something with a ZeroMQ / 0MQ would be a good idea. This could be a very fast way to pass many small messages that hand off those connections and tasks. http://zguide.zeromq.org/page:all
I am very intrigued by this kind of queuing and think it is the path forward for many workflows, traditional cluster and cloud. It allows for asynchronous communication and fault tolerance in large distributed systems. I've used this to wrap a grid engine queue, passing python objects that execute in parallel on many nodes. It's cool stuff. Cheers. On Mon 04/14/14 08:59AM -0400, Lawrence Stewart wrote: > This inquiry brings up a subject that has long puzzled me. In my (limited) > experience, > resource managers like SLURM (which I've used) are pretty good at handling > flows > of independent MPI jobs that set up, run, and exit. > > I don't know what to use to manage a pool of servers such as a collection of, > say, video > transcoders. A particular instance of the transcoder might be able to handle > up to > N streams, but because the streams start and stop at different times, the > server is always > doing something. > > Are there standard packages to manage keeping a collection of daemon servers > running on > a cluster, and then managing the connections handled by those servers? > > -Larry > > On 2014, Apr 14, at 2:32 AM, Yew, Tze Hui <tze.hui....@intel.com> wrote: > > > Hi Michael Will, > > > > I’m TH and am interested with this > > http://www.beowulf.org/pipermail/beowulf/2005-January/011626.html. I’m > > currently looking at a solution to launch an object detection app on Host, > > with the GUI running on Host and the compute nodes doing all the video > > processing and analytics part. I see this solution is similar to my > > approach but I couldn’t find any information on how to setup. May I know > > where to more info about this? As I googled, Cinelerra is a video > > processing editor, which has nothing to do with clustering. Can you guide > > me on this? Appreciate your help. > > > > Thanks. > > > > FYI, I have tried to send jobs(with graphics) using MPI previously but > > failed, seems like MPI only suitable for data processing but not video > > processing. > > > > > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > > To change your subscription (digest mode or unsubscribe) > > visithttp://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Gavin W. Burris Senior IT Project Leader Research and Innovation Team The Wharton School University of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
