On Tue, Oct 29, 2019 at 4:49 PM Lux, Jim (US 337K) <james.p....@jpl.nasa.gov> wrote:
> True, there’s tons of info in qstat -f, however, doesn’t qstat stop > showing my job after it completes, though? > Unless the system administrator sets a grace period for completed jobs to stay on the queue. You could ask your sysadmin to set that up: qmgr -c 'set server keep_completed=7200' will keep the completed jobs information available for two hours (time in seconds). I think the default is zero seconds, maybe a bit more. I think it increases the pbs_server memory use, hence there may be some reluctance to do this, but for a modest amount of time in a not extremely busy server it may not be too bad. However, most (maybe all) of this information is stored in the accounting logs, after the job completes. I think read access to those logs is by default blocked to regular users, but maybe you can try to mollify the sysadmin policy. :) There are three records there for each job, one for when the job is submitted, another for when the job starts, and one for when it ends, tagged by Q,S,E, respectively. Maybe there’s a switch that retrieves “last data”? > > I don't know, but a longer "keep_completed" (above) may produce a similar effect. You can also ask the system administrator to write a PBS epilogue script to print out the job requested and used resources to the job STDOUT log. I did this here. It is useful information for every user. Sample output: job_id=95649.master job_user=gus job_group=gus job_name=FJXndg job_session_id=10623 job_requested_resources=neednodes=3:ppn=32,nodes=3:ppn=32,walltime=24:00:00 job_used_resources=cput=875:53:15,mem=105991292kb,vmem=152856944kb,walltime=09:36:08 job_queue=production job_account= job_exit_code=0 This is just printing variables that are available from PBS at the job's end. Prologue/epilogue scripts are documented here: http://docs.adaptivecomputing.com/torque/3-0-5/a.gprologueepilogue.php I suppose I could put a qstat -f in as the last thing in the sub job file. > > > > resources_used.cpupercent = 99 > > resources_used.cput = 00:34:26 > > resources_used.mem = 11764kb > > resources_used.ncpus = 4 > > resources_used.vmem = 1135264kb > > resources_used.walltime = 00:34:28 > qstat -f $PBS_JOBID (I think this is the environment variable.) It may work. Wouldn't some statistics extracted with qstat -f from previous jobs provide a reasonable guess for the information you need to set up future jobs? Gus Correa > > and then there’s /usr/bin/time which might also help. > > > > > > > *From: *Beowulf <beowulf-boun...@beowulf.org> on behalf of Gus Correa < > g...@ldeo.columbia.edu> > *Date: *Tuesday, October 29, 2019 at 11:34 AM > *To: *"beowulf@beowulf.org" <beowulf@beowulf.org> > *Subject: *[EXTERNAL] Re: [Beowulf] PBS question > > > > > > > > On Tue, Oct 29, 2019 at 2:00 PM Lux, Jim (US 337K) via Beowulf < > beowulf@beowulf.org> wrote: > > I’m doing some EP job arrays using PBS, and I was wondering if there’s a > way to find out how much resources the job actually consumed. > > For instance, if I do a select 1:ncpus=4 does it actually use 4 CPUs? > > Likewise, is there a “memory high water mark”. > > > > The idea here is that since my job competes with all the other jobs, the > better I can describe my resource needs (i.e. not over-require) the faster > my job gets through the system. > > > > Or are these things I need to instrument in some other way. > > > > -- > > > > ncpus=4 requests 4 cpus/cores, which PBS/Torque will allocate to your job > alone (unless the nodes are tagged as work shared). > > It is up to you use these cpus (e.g. MPI: mpirun np=4; OpenMP: > OMP_THREAD_NUM=4) or not (a serial program). > > "qstat -f" produces a fair amount of information about the job resources. > > The accounting logs, in server_priv/accounting/YYYMMDD also do, if you > have access to them. > > > > You can request a certain amount of memory with the -l (lowercase L) > switch of qsub. > > See 'man qsub' for syntax, and 'man pbs_resources' for which resources can > be requested (particularly pmem, pvmem, vmem). > > > > I hope this helps, > > Gus Correa > > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf > >
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
