Hi Dan, If you use PBS/torque or some other batch system that could use it, the epilogue script in http://bellatrix.pcl.ox.ac.uk/~ben/pbs/ may help you:
" When running parallel jobs on Linux clusters with MPICH and PBS, "slave" MPICH processes are often left behind on one or nodes at job abortion. PBS makes no attempt to clean up processes on any node except the master node, and so these processes can linger for some time. The approach used at the WGR/PDL lab is to kill these processes by means of a second MPI-enabled program, which is run on the same set of nodes that the main job was run on, by the PBS epilogue facility. This program kills all of the user's processes that have the relevant PBS job ID in their environment, so should leave other jobs on the same machine untouched. To set up this system, this C program should be compiled with mpicc and installed as /usr/local/bin/mpicleanup on every MPI node. This epilogue script should then be used by PBS on every node (usually it needs to be installed as /usr/spool/PBS/mom_priv/epilogue) to call the MPICH cleanup program properly at job termination." I have just started using it so I cannot say if it works in all the cases. Hope it helps, -- Diego. On Friday 08 September 2006 12:56, [EMAIL PROTECTED] wrote: > Hello All > > Any one have a method/script that they would be willing to pass along that > could be used in the process of terminating user processes out on the > compute nodes that we never properly cleaned up? Thanks! > Dan > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
