Hi Mark,
It's my experience that when building large clusters, issues of space,
power and cooling are often harder and more time-consuming to resolve
than actually getting the cluster itself purchased, commissioned, and
operating. For
that is somewhat perplexing, since the space/power/cooling issues aren't
really _that_ complicated. I think it's one of those areas where too much
choice leads to harder decisions.
Over the past three years, I've been closely involved in the construction
of two large cluster rooms. (400kW and 500kW). Once is conventional
raised floor air cooling and the second is water cooled racks. In both
cases, they were institutional construction projects, and took
substantially more than a year.
In the past, I have done remodelling efforts at the 40kW level. This, I
agree, is something that can be done in a matter of a couple of months.
But when scaled up by a factor of ten it's substantially more difficult.
perhaps it also reflects the fact that we're still not really
comfortable with the state of affairs - for instance, vendors still
advocate blade servers, which if fully populated are basically
uncoolable (~24 KW/rack!).
With good design (water cooled racks) you can cool a 24kW system.
example I've recently taken up a new position in Hannover Germany where as
part of my start-up package the MPG is building a cluster room (450 square
meters floor space, 500kW cooling, 800kW UPS, with the option to double
cooling/power in four years).
those numbers seem strange to me - unless I've botched conversion, the
cluster I sit next to is about 4.7 KW/sq-m.
In my case, it turns out that the most cost-effective systems are LOW
density ones. So the room has been designed to accomodate these, with a
lower kW/m^2 value. And the physical space itself was 'free' so it's nice
to have some elbow room.
(such a large UPS seems strange too - did they choose it based on poor
quality line power? we have none of our compute hardware on UPS, and
don't have problems,
I have had power-related problems in the past, and have found that the
lower maintenance needs and higher reliability of UPS backed systems are
worth it.
The room is being designed with a 20-year lifetime. When amortized over
that time (at least 6 or 7 clusters) the one-time cost of the UPS is
negligable.
since modern PDUs seem to ride out the typical 1-second glitch without
much trouble...)
That's interesting. Where does the PDU store 1 second of power?
end of this year. So total design and construction time is 2.3 years. In
that's a bit extreme, I think. our room was a bare-slab reno and took a
bit over a year.
That's fast!
another one of our sites was built from scratch and took about 1.5.
Fair enough. From my experience this 1.5 years from scratch is probably
typical.
construction time will be about 0.5 years. The cost of the cluster room is
about equal to the cost of the initial cluster that will go into it. But
the
strange. I'm pretty sure the cost ratio we see is more like 4:1 for the
from-scratch site (and closer to 10:1 for renos.)
When I built a 40kW system (rennovation) the ratio was 10:1. In other
words the room rennovation cost was 10% of the cluster cost. In that case
the construction was done by a University 'Physical Plant' as a
remodelling effort. No management or controls.
When I built a 400kW system (bare room) the cost ratio was about 2:1. In
that case the construction was done by a state agency, and there were a
lot of management and process requirement costs. This did result in a
higher-quality room but I think that it probably doubled the construction
costs.
The room that I am building now would *also* be about 2:1, except for the
fact that we are using liquid cooled racks (we have a very low ceiling).
This effectively doubles the cost, hence the 1:1 ratio.
Fortunately, over the 20 year lifetime that the room will be in use, I
expect that at least six generations of cluster will go into it. So in
the end the ratio should be at least 6:1.
Cheers,
Bruce
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
