It's nice to see some Apple centric notes here. Excellent writeup.
I've been involved in building and supporting Apple clusters running
Sun Grid Engine for years now, dating back to the time when the first
Xserves were released and we got some notoriety for using Apple 2nd
generation iPods as bootable firewire drives for auto-imaging our
cluster nodes.
Our apple cluster experience is more focused on batch-style compute
farming rather than "true HPC" but I can toss some comments into the
mix here - I've commented down below on some of the points that were
raised in the write-up.
It's sad to hear from people attending SuperComputing that Apple did
not have a booth. The consensus reported back to me was that "Apple
has nothing to show in the HPC space ..." and that sort of goes along
with what we've been seeing with Apple turning back from enterprise
sales and focusing almost exclusively on the consumer market. Sad to
hear -- we've been waiting on refreshed Xserves for way too long now.
At this point I'd guess that the product may be dead or discontinued.
On Nov 20, 2007, at 2:19 PM, Kragen Javier Sitaker wrote:
This is not strictly about Beowulfs, but it is probably of interest to
their users.
My friend Joe's team from Indiana University just fielded a MacOS
cluster for the Supercomputing '07 Cluster Challenge. His experiences
weren't that great; I encouraged him to jot something quick down so
that
other people could benefit from his hard-won lessons.
There's more information about the challenge at
http://sc07.supercomp.org/?pg=clusterchlng.html&pp=conference.html.
----- Forwarded message from Kragen Javier Sitaker
<[EMAIL PROTECTED]> -----
From: Kragen Javier Sitaker <[EMAIL PROTECTED]>
To: kragen-fw
Subject: Joe Blaylock's notes on running a MacOS cluster, Nov. 2007
Disordered thoughts on using MacOS X for HPC.
By Joe Blaylock, 2007-11.
Recollections:
* we were the first people to ever try that particular combination:
Tiger on Xeons with Intel's ICC 10 compiler suite and MKL linear
algebra libraries. Blazing new territory is never easy.
* We didn't use XGrid or Apple's cluster management stuff, only
Server Admin and ARD.
* Pov-Ray was easy; OpenMPI was easy; using Myrinet over 10Gig
Ethernet was easy
* GAMESS was more challenging, but we got it working somewhat. We
still don't know how to run jobs of type ccsd(t), which require
System V shared memory.
* We never got POP to work.
* Apparently, ICC 10 has some bugs. There were several times when
we
were trying to use it to build, IIRC, GAMESS or POP, and it would
give illegal instruction errors during compile. Or it would build
a binary that we would run, and then it would do something
horrible (like hang the machine (probably a bug interaction
between icc and MacOSX).
* OpenDirectory doesn't seem ready for prime time. It's pretty easy
to set up, but it's unreliable and mysterious. In MacOS X, there
seems to be a fundamental disconnect between things in the CLI
world and things in the GUI world. Setting something up in one
place won't necessarily be reflected in the other place. I'm sure
that this is all trivial, if you're a serious Darwin user. But
none of us were. So for example, you set up your NFS exports in
the Server Admin tool, rather than by editing /etc/exports. The
Admin tool won't put anything into /etc/exports. So if you're on
the command line, how do you check what you're exporting? With
the
complexity of LDAP, this becomes a real problem. You set up
accounts on your head node, and say to export that information.
But perhaps you create an account, but can't log into it on a
node. If you're ssh'd in from the outside, where do you check to
see (from the command-line) what the authentication system is
doing? Our local Mac guru couldn't tell us. And then you'd create
another account, and the first one would start working again.
WTF?
* This may be the most frustrating thing about working with OS X
Server. The CLI is the redheaded stepchild, and lots of HPC is
mucking about on the command-line. You can use VNC to connect to
ARD (but only if a user is logged in on the desktop and running
ARD!), but it's slow, and only provides desktop control, not
cluster management. ARD can then be run on the desktop, to
provide
desktop control of the nodes in the cluster, and some cluster
management: run unix command everywhere, shut nodes down, etc.
There were a handful of tasks which seemed important, but which I
couldn't figure out how to do on the command-line at all. The
most
heinous of these is adding and removing users to/from LDAP.
Prior to Apple releasing the Xserve, there were many things that
required a GUI to accomplish. Right around the time they released the
Xserves though the OS improved to the point where you could just about
everything via the command line and/or a serial console.
ServerAdmin even has a CLI variant that can do everything that the
GUI. I think 'serveradmin' along with 'networksetup' were the two main
CLI tools we used over and over again.
OpenDirectory is more of a pain - there is a CLI tool but you also end
up working with standard openldap commands and binaries.
The best reference we found (in addition to lots of blood, sweat and
tears) was a PDF that Apple publishes called "Mac OS X Server Command
Line Administration" which can be found here: http://www.apple.com/server/pdfs/Command_Line.pdf
All things considered we've found that working on headless Apple boxes
using a serial console is certainly possible, it's a bit slower than
Linux due to the need to look up cryptic CLI commands in the PDF but
it works for about 99% of the things we ever really needed to do. I
think one GUI only thing that bit us once was the fact that during
initial OS install if you want to software RAID your disks you can
only do this via a GUI prompt.
* Most of the time, I found it more convenient to use a 'for' loop
that would ssh to nodes to run some command for me.
We use passwordless SSH and 'dsh' or 'pdsh' utilities on just about
every Apple system we work on. There are many times when the "do
something on N nodes" automated process is necessary. heh
* MacOS X lacks a way to do cpu frequency scaling. This killed us
in
the competition. We couldn't scale cores to save on our power
budget, we could only leave them idle.
* Being a Linux dude, I found having to have license keys for my
operating systems, and (separately) my administration and
management tools, to be odious in the extreme. Having to
separately license ICC and IFORT and MKL just added frustration
and annoyance.
This is not really a "secret" but Apple does not publish it easily and
it took some internal contacts to discover ...
Apple is capable of producing what is called a "Watermarked Serial
Number" that will license all servers in a cluster or a subnet. You
still have to enter a serial number in the OS but having the single
Watermark serial number allows you to use the same values which makes
it scriptable or something you can bake into a netboot'ed image.
Watermarked serial numbers can not be requested by mere mortals. You
have to request this from your sales rep and apparently there is some
internal system that the sales rep can use to generate the watermarked
serial number. This is the first thing we tell our Apple using friends
and colleagues as it is a significantly nice thing to have.
* We didn't make detailed performance comparisons between stuff
built with the intel suite and things built with, e.g., the GNU
suite and GotoBLAS. We were too busy just trying to get
everything
to work. I'm sure that Intel produces better code under normal
circumstances, but we had lots of cases where version 10 couldn't
even produce viable binaries. So, make of that what you will.
What I would recommend (if you were going to use MacOS X):
* Learn Darwin, in detail. Figure out the CLI way to do everything,
and do it. In fact, forget Mac OS X; just use Darwin. Learn the
system's error codes, figure out how to manipulate fat binaries
(and how to strip them to make skinny ones), be able to
manipulate
users, debug the executing binaries, etc. Consider looking into
the Apple disk imaging widget so you can boot the nodes diskless.
What I would do differently (whether I stick with MacOS X or not):
* diskless clients
* Flash drive for head node
* no GPUs
* Get Serial Console set up and available, even if you don't use it
routinely
* CPU Frequency Scaling!!
* many more, smaller cores. we had 36 at 3GHz. this was crazy. We
were way power hungry.
* Go to Intel 45nm dies.
Regards,
Chris
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
