As Joe mention, the way we handle this is by using cluster schedulers
sitting on robust hardware platforms that are capable of handling
large numbers of job submissions without problems. Grid Engine and
Platform LSF are two capable products that come to mind and scale well.
The fact that your users are using "qsub" is a good thing that you
certainly want to encourage. It puts their job under the control of a
scheduler and allows you to do policy based allocation of your
computing resources.
The alternative is your users bypassing the scheduler altogether by
SSH'ing to a node and just manually starting programs. Attempts to
bypass the scheduler are common in some environments so consider
yourself lucky that your users are using the scheduler at all!
The problem with specific users or perl loops bringing the system
down with a giant load of rapid qsub submissions is usually best
handled on a per-user or per-use case level.
Its more of a matter of education and making sure your users have a
resource who can help them with their job scripts and the general
tasks of cluster application integration. Your users are not
intentionally trying to cause problems on the system (most likely)
but it appears clear that they may need some assistance on how to
better use the existing cluster.
Not giving users sufficient application integration and cluster
scripting support resources is a problem I see all the time. Too many
cluster operators think that training users on a few scheduler
submission and status commands is all the integration help that they
need to provide. The end result is someone writing a shell or perl
script that tries to submit a few million short running tasks all at
once ...
Ways you can deal with the situation:
- Examine the user scripts, see if their script can be altered to put
"more work" into each individual qsub job submission. This will
reduce the number of qsub commands required
- Tell your users that the use of rapid loops for job submission is
causing system problems. Work with them to introduce a small delay
into their submissions. It is to everyone's best interest not to
bring down the master scheduler
- Look into a feature that some scheduling systems call "array jobs"
or "job arrays" -- For schedulers that support this feature it is a
very very powerful way to use a single qsub/bsub command to launch
hundreds of thousands of jobs. I know that a SGE design goal is to
support the submission of a single job array with up to 500,000
individual sub tasks. Both SGE and LSF do job arrays very well.
This feature only works well if the workflow includes similar
commands that vary only slightly (like the input file or a command
line argument for instance).
So in summary:
- Be happy users are issuing qsub commands at all !
- Treat the looping problem as a sign that your users may need some
application integration assistance/education
- Work with the users that are causing problems, see if they can
introduce a delay
- Look into "array job" functionality
Regarding the problem of people bypassing the scheduler and logging
into nodes directly via SSH to run tasks -- I've posted on this exact
topic on this list before, you may be able to find it in an archive
somewhere. In short, my belief is that you'll never win the
technological "arms race" with the users when you try to block users
who are bypassing the scheduler.
Depending on your organizational environment, it is better to treat
the problem of users bypassing the scheduler as a Management/HR/
Policy problem rather than a technological problem. Set up a good
scheduler with resource allocation policies that have been accepted
by the users. Then make a policy that everyone who wants to use
shared resources must operate under the scheduler. After that, make
sure that people are informed that scheduler/cluster abuse is a
policy matter that will be referred up the management chain and
eventually to the human resources department. It's a matter of
policy and acceptable use, not technology.
My $.02
-Chris
On Aug 1, 2006, at 5:36 PM, Xu, Jerry wrote:
Hi, Thanks, Joe.
I am not meaning to "ban" anything immediately, I am just curious
how often
this happen to the HPC community.
Perl/shell is really strong tool, one example is to use loop to
submit huge
mount of jobs and puts burden on scheduler server, the other
example is to have
one job sit idle and frequently to use system call to detect the
job status and
resubmit jobs again and again; the other example is that use system
call and ssh
to each node and run stuff and bypass the scheduler... It just
drives me crazy
sometime.
How do you guys handle issue like this?
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
