what's wrong with 500k job submissions? to me, the existence of "array
jobs"
is an admission that the job/queueing system is inefficient.
When I compare this e.g. to C:
- a loop like:
for (i=1;i<=100000;i++)
printf("Hello from run %d.\n", i);
- and you can guess: 100,000 times:
printf("Hello from run %d.\n", i++);
While the execution time is nearly the same, compilation of the first one is
faster by far.
this implies that the compiler's lexer sucks, which is as close to my
point this strained analogy can get.
that it can't compile sequential code very well and generates a big
executable?
big executables are fine (this one would be a whole 2 MB!).
The compiler must read the source, and SGE has to read the job
requirements for every job again and again and store it.
if SGE has to keep re-reading user files, this suggests to me that its
design is poor. it's obvious to me that a scheduler should be based on
a production-quality DB, for instance, and should clearly give some
thought to performance when there are many runnable jobs.
(PS: not to mention, that a for-loop/array-job is easier to handle for the
user)
is it? an array job, in my experience at least, is just syntactic sugar
for submission. all the internals of job scheduling still have to operate,
though as you point out, the scheduler may take advantage of the fact that
each sub-job has identical resource requirements. it still needs to dispatch
each sub-job separately, each SJ may fail in unique ways, etc. in the end,
the submission and resource-matching code of the scheduler has gotten a
break, but everything else works just as hard. consider, for instance, that
flat naming schemes are conceptually simpler, but array jobs break that.
remember that the user still has to write some sort of script to evaluate
whether each SJ worked, and recover from the failures.
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
