Considering how widely WRF is used, I'm surprised it has this problem.
Unfortunately, I can't offer you any advice. I've installed it once or
twice for others to use, but that's the limit of my experience with it.
It sounds like your users are being penny-wise and pound-foolish. If a
non-exclusive job starts sooner but eventually fails and has to be run
again, is it really a faster time to solution than the job that has to
wait a little longer in the queue, but succeeds the first time? I assume
you're using a backfill scheduler and require wallclock times with
submissions. If so, do you know how accurate your users wallclock times
are? That could delay job starts in exclusive mode, too.
Sounds like to might need to 'educate' your users a bit.
Prentice
On 07/30/2015 02:51 PM, Tom Harvill wrote:
Hi Prentice,
Thank you for your reply. Yes, it's 'bad' code. It's WRF mostly. If
you have suggestions for that app I'm
all ears. We don't control the code-base. We're also not allowed to
update it except between projects
which is very infrequent.
It would be ideal if we could discretely control memory allocations to
individual processes within
a job but I don't expect it's possible. I wanted to reach out to this
list of experts in case we might be
missing something.
The resistance comes from increased wait times as a result of
staggered serial jobs that prevent
allocations within a node exclusively. Yes, the users would probably
get better aggregate turnaround
time if they waited for node exclusivity...
...Tom
On 7/30/2015 1:37 PM, Prentice Bisbal wrote:
Tom,
I don't want to be 'that guy', but it sounds like the root-cause of
this problem is the programs themselves. A well-written parallel
program should balance the workload and data pretty evenly across the
nodes. Is this software written by your own researchers, open-source,
or a commercial program? In my opinion, your efforts would be better
spent fixing the program(s), if possible, than finding a scheduler
with the feature you request, which I don't think exists.
If you can't fix the software, I think you're out of luck.
I was going to suggest requesting exclusive use of nodes (whole-node
assignment) the easiest solution. What is the basis for the resistance?
Prentice
On 07/30/2015 11:34 AM, Tom Harvill wrote:
Hi,
We run SLURM with cgroups for memory containment of jobs. When users
request
resources on our cluster many times they will specify the number of
(MPI) tasks and
memory per task. The reality of much of the software that runs is
that most of the
memory is used by MPI rank 0 and much less on slave processes. This
is wasteful
and sometimes causes bad outcomes (OOMs and worse) during job runs.
AFAIK SLURM is not able to allow users to request a different amount
of memory
for different processes in their MPI pool. We used to run
Maui/Torque and I'm fairly
certain that feature is not present in that scheduler either.
Does anyone know if any scheduler allows the user to request
different amounts of
memory per process? We know we can move to whole-node assignment to
remedy
this problem but there is resistance to that...
Thank you!
Tom
Tom Harvill
Holland Computing Center
hcc.unl.edu
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin
Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
