On 06/25/2014 07:14 PM, Ellis H. Wilson III wrote:
Catching up:
1. My comment on loving Gentoo but not suggesting it for HPC
deployment assumed old-style installation across the entire cluster as
the sole OS. Obviously netboot/VM/Docker beats the hell out of such a
static implementation, I just wasn't interpreting the OP's question
that way, which was my bad. Joe nailed it in this regard.
2. My new place of work (Panasas) supports RHEL 6 (maybe 7, I'm green
here), so it's there's one datapoint where RHEL is still supported.
We also support a bunch of other distros though, so it's certainly not
exclusive. I /hope/ to someday push for Gentoo weak-module support.
We'll see about that.
3. I cannot count on my hands and toes the number of times restrictive
distros like RHEL bit me in the ass when I tried to run any number of
HPC applications to push my new/shiny/academic file system. Holy
smokes what I would have given for a flexible environment then.
3.a continues below:
On 06/25/2014 04:50 PM, Prentice Bisbal wrote:
On 06/25/2014 03:08 PM, Kilian Cavalotti wrote:
On Wed, Jun 25, 2014 at 10:29 AM, Andrew M.A. Cater
<amaca...@galactic.demon.co.uk> wrote:
RHEL doesn't cut it for these people: they know that they want later
GCC / different commercial compilers / hand written assembly - a
later
kernel with a smarter scheduler ...
SCL really doesn't work - it's stil not up to it.
One way to deal with this is to separate user applications from the
OS, as much as possible. And compilers could be considered as user
applications.
You can just use a very minimal OS on your compute nodes, then compile
and install all the user facing bits in a shared location. You hand an
environment modules system to the users and off they go. Systems such
as EasyBuild (https://hpcugent.github.io/easybuild/) aim to facilitate
this by allowing easy compilation and installation of scientific
software (based on descriptive specification files, à la Gentoo
ebuilds), including dependencies, and by automatically generating
environment modules.
This way, you don't really care what the underlying OS is. You can
have as many versions of GCC, Python, R, Perl, Ruby or anything
installed alongside each other with no side effect, as long as you
load the right module before running your job. It's like a
distro-agnostic ebuild system.
You can keep the distro the hardware vendor recommends to retain
support (for interconnect drivers, parallel filesystems and such)
while making your users happy with the newest versions of the software
they need^Wwant.
I agree with this approach. I've been doing this for years, and it's
really not has hard as people make it out to be. There's the occasional
'dependency Hell' situation, but that's not usually that bad unless you
are building a GUI application. Fortunately, GUI users aren't too common
in HPC. Overall, I find compiling Perl, R, etc. from source and
installing each version in it's own installation directory much easier
then learning how to get package managers to allow you to install
different versions of the same packages in a sane way.
3.a: Agree with some of the above from Kilian and Prentice. The
absolute best scenario (IMHO) for users and administrators is to
provide an environment where users who want full control and want to
take the responsibility for support onto themselves (e.g., me and all
my fellow systems researchers in my PhD) can do so, while users who
want all of that "nonsense" to get out of the way so they can run
their application X to do "real" science without having to recode it
for bleeding-edge (or super-old) version of GCC/Perl/Python/etc can
also do so. The ONLY place this is possible is in a flexible
(VM/Docker/etc) environment. Administrators just need to tell the
latter group "choose from ImageA, ImageB, and ImageC to run your
application." Counter-argument will be "I have to manage those
images." Counter-counter argument is "Is that really harder than
rolling special environments to cope with those users (which you'll
have to do anyhow)?"
I disagree with needed separate images in a VM/Docker/etc. environment,
maybe if your doing system-level research, but most HPC users work
exclusively in the user-space,and just want to run their
MATLAB/NAMD/LAMMPS, whatever job. In this case, just installing
user-space applications and libraries in a different path from the
distro-supplied versions is adequate. Modifying PATH and a few other
environment variables that most users can handle is all you need
modules, lmod, softenv and other utilities make that even easier for users.
If you're doing lower-level research involving kernel modules or kernel
tuning, yes, you will need VMs or something. But this is usually
'research', not 'production' HPC.
I ended up doing very crazy root-stealing, chroot-establishing things
to get my science done in my PhD. If you prevent intelligent people
from doing their work, they are going to be your worst nightmare.
Don't kid yourselves if you think you are doing anyone favors by
providing super-static OS environments like RHEL for your users. You
are just being lazy (and not the good kind of programmer lazy).
This is so true. If you are a roadblock to your users, they will find a
way around you.
Best,
ellis
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
