----- Forwarded message from Kragen Javier Sitaker <[EMAIL PROTECTED]> -----
From: Kragen Javier Sitaker <[EMAIL PROTECTED]>
Date: Tue, 20 Nov 2007 14:12:04 -0500
To: [EMAIL PROTECTED]
Subject: Joe Blaylock's notes on running a MacOS cluster, Nov. 2007
User-Agent: Mutt/1.5.9i
Disordered thoughts on using MacOS X for HPC.
By Joe Blaylock, 2007-11.
Recollections:
* we were the first people to ever try that particular combination:
Tiger on Xeons with Intel's ICC 10 compiler suite and MKL linear
algebra libraries. Blazing new territory is never easy.
* We didn't use XGrid or Apple's cluster management stuff, only
Server Admin and ARD.
* Pov-Ray was easy; OpenMPI was easy; using Myrinet over 10Gig
Ethernet was easy
* GAMESS was more challenging, but we got it working somewhat. We
still don't know how to run jobs of type ccsd(t), which require
System V shared memory.
* We never got POP to work.
* Apparently, ICC 10 has some bugs. There were several times when we
were trying to use it to build, IIRC, GAMESS or POP, and it would
give illegal instruction errors during compile. Or it would build
a binary that we would run, and then it would do something
horrible (like hang the machine (probably a bug interaction
between icc and MacOSX).
* OpenDirectory doesn't seem ready for prime time. It's pretty easy
to set up, but it's unreliable and mysterious. In MacOS X, there
seems to be a fundamental disconnect between things in the CLI
world and things in the GUI world. Setting something up in one
place won't necessarily be reflected in the other place. I'm sure
that this is all trivial, if you're a serious Darwin user. But
none of us were. So for example, you set up your NFS exports in
the Server Admin tool, rather than by editing /etc/exports. The
Admin tool won't put anything into /etc/exports. So if you're on
the command line, how do you check what you're exporting? With the
complexity of LDAP, this becomes a real problem. You set up
accounts on your head node, and say to export that information.
But perhaps you create an account, but can't log into it on a
node. If you're ssh'd in from the outside, where do you check to
see (from the command-line) what the authentication system is
doing? Our local Mac guru couldn't tell us. And then you'd create
another account, and the first one would start working again. WTF?
* This may be the most frustrating thing about working with OS X
Server. The CLI is the redheaded stepchild, and lots of HPC is
mucking about on the command-line. You can use VNC to connect to
ARD (but only if a user is logged in on the desktop and running
ARD!), but it's slow, and only provides desktop control, not
cluster management. ARD can then be run on the desktop, to provide
desktop control of the nodes in the cluster, and some cluster
management: run unix command everywhere, shut nodes down, etc.
There were a handful of tasks which seemed important, but which I
couldn't figure out how to do on the command-line at all. The most
heinous of these is adding and removing users to/from LDAP.
* Most of the time, I found it more convenient to use a 'for' loop
that would ssh to nodes to run some command for me.
* MacOS X lacks a way to do cpu frequency scaling. This killed us in
the competition. We couldn't scale cores to save on our power
budget, we could only leave them idle.
* Being a Linux dude, I found having to have license keys for my
operating systems, and (separately) my administration and
management tools, to be odious in the extreme. Having to
separately license ICC and IFORT and MKL just added frustration
and annoyance.
* We didn't make detailed performance comparisons between stuff
built with the intel suite and things built with, e.g., the GNU
suite and GotoBLAS. We were too busy just trying to get everything
to work. I'm sure that Intel produces better code under normal
circumstances, but we had lots of cases where version 10 couldn't
even produce viable binaries. So, make of that what you will.
What I would recommend (if you were going to use MacOS X):
* Learn Darwin, in detail. Figure out the CLI way to do everything,
and do it. In fact, forget Mac OS X; just use Darwin. Learn the
system's error codes, figure out how to manipulate fat binaries
(and how to strip them to make skinny ones), be able to manipulate
users, debug the executing binaries, etc. Consider looking into
the Apple disk imaging widget so you can boot the nodes diskless.
What I would do differently (whether I stick with MacOS X or not):
* diskless clients
* Flash drive for head node
* no GPUs
* Get Serial Console set up and available, even if you don't use it
routinely
* CPU Frequency Scaling!!
* many more, smaller cores. we had 36 at 3GHz. this was crazy. We
were way power hungry.
* Go to Intel 45nm dies.
----- End forwarded message -----
--
Eugen* Leitl <a href="http://leitl.org";>leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
