Hi,
Am 16.10.2007 um 16:08 schrieb Sean Ward:
I've started work on a web service which contains several
potentially long running processing steps (molecular dynamics),
which are perfect to farm out to the fairly large (90 node) Beowulf
I have access to. The primary issue is translating requests from
the event driven web service, to job queues, and back again upon
completion. Specifically, the major queuing systems I have
immediate access to (Sun Grid Engine and Condor) only support e-
mail based notification of job completion. Starting jobs isn't an
issue, as my service can simply ssh over and execute shell scripts
as needed to start things up, the problem is reliably being
informed when the jobs fail or complete, via any programmatic
method (such as executing a shell script, calling a web service via
SOAP/etc, or an asynchronous message library). My other problem,
ensuring that these web service requests don't starve in house jobs
on the Beowulf is easily handled via the priority levels built into
all the various job managers, although being able to checkpoint a
long running job would be a plus (such as is supported by Condor).
if it's possible to compile your program with Condor-Checkpointing,
then you can even compile it as a standalone application which
includes checkpointing. This you can use with any other
queuingsystem. It's of course best to choose one which supports
checkpointing like SGE. It has no built-in checkpointing (like
Condor), but has extra setup options to make its handling easy.
-- Reuti
I am currently investigating modifications to either Condor (more
complex to update, but checkpoint is useful) or Ruby Queue (very
easy to update for reliable notification) to solve this issue, but
wanted to be sure I wasn't overlooking any existing solutions to
programmatic based queuing and receiving notifications on jobs in a
Beowulf environment...
-Sean
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
