Hi Greg,
We have some switches made by Fortinet (model FS500) which we like. They were
previously designed and marketed by Woven Systems, which was then acquired by
Fortinet.
Cheers,
Bruce
On Sep 2, 2010, at 3:17 AM, Greg Lindahl wrote:
> I'm in the market for 48-port 10gig switches (pr
controllers and firmware
or was this two or three years ago? It's my impression that (when used
with compatible drives and drive backplanes) the latest generation of
Areca hardware is quite stable.
Cheers,
Bruce
Bruce Allen wrote:
What was needed to fix the systems? Reboot?
l over and stopped accessing disks.
gerry
Bruce Allen wrote:
Hi Gerry,
So far the only problem we have had is with one Areca card that had a bad
2GB memory module. This generated lots of (correctable) single bit errors
but eventually caused real problems. Could you say something about the
r
he systems we've built that way.
gerry
Bruce Allen wrote:
Hi Xavier,
PPS: We've also been doing some experiments with putting
OpenSolaris+ZFS on some of our generic (Supermicro + Areca) 16-disk
RAID systems, which were originally intended to run Linux.
I think that DESY proved some
Hi Xavier,
PPS: We've also been doing some experiments with putting
OpenSolaris+ZFS on some of our generic (Supermicro + Areca) 16-disk
RAID systems, which were originally intended to run Linux.
I think that DESY proved some data corruption with such
configuration, so they switched to OpenS
Hi Xavier,
Would it be possible to get a (private) copy of your Jumpstart config file
and the custom install scripts? Reading these and modifying them will
probably be quite a bit quicker than developing our own from scratch.
Sure. I'll send them in private after review.
Thanks very much.
uce
PS: Loic: thanks for passing this on to your colleauge!
PPS: We've also been doing some experiments with putting OpenSolaris+ZFS
on some of our generic (Supermicro + Areca) 16-disk RAID systems, which
were originally intended to run Linux.
On Mon, 14 Apr 2008, Xavier Canehan wrote:
Hi Loic,
(I'm picking up a 6-month old thread.)
On Wed, 5 Sep 2007, Loic Tortay wrote:
As of today we have 112 X4500, 112U are almost 3 racks which is quite a
lot due to our floor space constraints.
We're now doing the comissioning of our new cluster which includes about
30 Linux storage se
Hi Carsten,
Here is the database system used by ROCKS:
http://www.rocksclusters.org/rocks-documentation/reference-guide/4.1/database.html
Rocks is an open-source Linux cluster distribution that enables end users
to easily build computational clusters, grid endpoints and visualization
tiled-di
The myrinet connection was working right, but sometimes a user program
just got stuck - one of the processes was sleeping, and all others were
running. Then, the program hangs.
Any suggestions? I can provide any log necessary.
Ivan, you probably already know this, but if not it can be very us
Jeremy,
You might be better off posting to the Linux Kernel Mailing List (LKML)
about this issue. There are a few experts here (Don, Joe, Mike, ) who
might know, but LKML is more likely to give you correct guidance quickly.
cheers,
Bruce
On Mon, 24 Sep 2007, Jeremy Fleming wrote
I've heard of people making clusters out of Alphas running Linux before -
can't remember where, but I did see someone had done it.
Back in 1998 I built a 48-node DEC AXP cluster. I think it was the
largest AXP cluster in operation for a few months, then a larger one was
built at one of the na
What do they use in high dollar gear that really needs this? Fluorinert,
such as FC77. Totally inert, very low viscosity, doesn't tend to cling to
the electronics, etc. available in a variety of boiling points so you can do
"ebullient cooling" (aka 2 phase cooling)which doesn't need a pump, an
In any case by the end of the year I should have at least ten X4500s, and
can do some testing myself. But your collection is an order of magnitude
larger, so you can collect much more useful statistics. If those
statistics show no data corruption, then someone like myself with many
fewer systems
UGU isn't what it used to be (but neither am I). I'm having trouble
finding a man page for fsprobe; can you specifiy a flavor of unix?
http://fuji.web.cern.ch/fuji/fsprobe/
___
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (dige
Hi Loic,
This thread has been evolving, but I'd like to push it back a bit.
Earlier in the thread you pointed out the CERN study on silent data
corruption:
http://fuji.web.cern.ch/fuji/talk/2007/kelemen-2007-C5-Silent_Corruptions.pdf
Actually, I was not the one who pointed out this study bu
Hi Leif,
I still think it would be interesting to see how often one gets data
corruption from other sources than disk errors (presuming ZFS is
perfect). Data corruption is data corruption even if its from bad cache
memory.
I will try to get fsprobe deployed on as much of the Nordic LHC stora
Hi Loic,
This thread has been evolving, but I'd like to push it back a bit.
Earlier in the thread you pointed out the CERN study on silent data
corruption:
http://fuji.web.cern.ch/fuji/talk/2007/kelemen-2007-C5-Silent_Corruptions.pdf
If you are not already doing this, would it be possible fo
you wrote...
Cheers,
Bruce
On Wed, 5 Sep 2007, Loic Tortay wrote:
According to Bruce Allen:
[...]
This is (in my opinion) probably the only real issue with the X4500.
The system disk(s) must be with the data disks (since there are "only"
48 disks slots) and the two bootable
Hi Loic,
[...]
In a system with 24 x 500 GB disks, I would like to have usable storage of
20 x 500 GB and use the remaining disks for redundancy. What do you
recommend? If I understand correctly I can't boot from ZFS so one or more
of the remaining 4 disks might be needed for the OS.
This i
Hi Loic,
I'm planning to buy a handful (8 to 16) x4500's a bit later this year, so
I am quite interested in the details.
All RAID operations are done in software by ZFS (which acts as both a
filesystem and a volume manager).
ZFS has a "scrub" command that does a background scan of a pool (a
Hi Leif,
I think you should add a parity verification to this - not just a
sector scan.
Good idea. In other words, don't just check that all disk surfaces
are readable by the disk device. Also check that the data is
consistent from the RAID perspective. Did I understand you correctly?
Exa
When I buy RAID controllers, I put this requirement directly in my bid
specifications. I say something like the following:
--
During a READ operation, if the RAID controller finds an unreadable
(uncorrectable) disk sector, then it
Hi Jakob,
We are now very busy working out how we're going to accomplish this; I
would like to follow up to this e-mail in a few months, with a
description of what we ended up doing.
A big thank you to all of you!
Over a number of years this mailing list has consistently offered a high
SNR
s who are interested, please drop by my office for a chat!
Cheers,
Bruce
On Thu, 30 Aug 2007, Andrew Walrond wrote:
Bruce Allen wrote:
Yes, precisely.
When I buy RAID controllers, I put this requirement directly in my bid
s
D-6) what is the probability of two failed disks within the
rebuild time window, and how likely is it that uncorrectable sectors have
appeared during that time?
Cheers,
Bruce
On Fri, 24 Aug 2007, Jeffrey B. Layton wrote:
Bruce Allen wrote:
Hi Jeff,
OK, I see the point. You are not wor
Hi Jakob,
2) You read it. In which case the disk can see that the checksum has
failed. There's nothing it can do about this except give you a read
error on that block.
A good RAID controller *may* help you with scenario (2) above; it can read the
block from a mirror or compute it from parity
n the rebuild time window, and how likely is it that uncorrectable
sectors have appeared during that time?
Cheers,
Bruce
On Fri, 24 Aug 2007, Jeffrey B. Layton wrote:
Bruce Allen wrote:
Hi Jeff,
OK, I see the point. You are not worried about multiple unreadable sectors
maki
Hi Jeff,
OK, I see the point. You are not worried about multiple unreadable
sectors making it impossible to reconstruct lost data. You are worried
about 'whole disk' failure.
I definitely agree that this is a possible problem. In fact we operate
all of our UWM data archives (about 300 TB)
While not a typical beowulf, I thought the list would have some input
for this "storage cluster" or whatever we should call it :)
Some years ago, a Caltech researcher Roy Williams
http://www.cacr.caltech.edu/~roy/ started calling such things 'datawulfs'.
I don't know if he coined the term, but
Glen,
Are your NFS clients and servers networked with gigabit ethernet? If not,
just hit . If so, keep reading...
If your network can handle jumbo frames (MTU of 6000 or 9000, rather than
the standard MTU of 1500) then shifting to jumbo frames might improve your
NFS performance.
I have a
Hi Jeff,
For this reason, in a RAID system with a lot of disks it is important to
scan the disks looking for unreadable (UNC = uncorrectable) data blocks on
a regular basis. If these are found, then the missing data at that
Logical Block Address (LBA) has to be reconstructed from the *other*
Hi David,
A couple of years ago we had a lot of CMOS battery deaths on systems that
were about 3 years old (and virtually always powered on). These were
Intel consumer-grade motherboards.
In the past year, a large number of Supermicro Opteron motherboards had
their batteries dying within a
That's interesting. Where does the PDU store 1 second of power?
I don't know about "PDU" per se -- but units that do significant power
conditioning (up to the extreme of dual isolation transformers) usually
have big capacitors to buffer surges and load variations. I wouldn't
have expected t
Hi Mark,
It's my experience that when building large clusters, issues of space,
power and cooling are often harder and more time-consuming to resolve
than actually getting the cluster itself purchased, commissioned, and
operating. For
that is somewhat perplexing, since the space/power/coolin
Thanks Mike!
To undertand power use in the past I have found it very helpful to look at
OEM motherboard manuals. For example the Intel OEM motherboard manuals
typically include a table listing expected power consumption for a system
built around the motherboard. The table includes standby, no
I have spun up Sun x4100 dual core, dual processor to 100% processor
usage and normal HD writes and measured the actually powerusage at 267
watts. Obviously higher than normal HD usage (such as swapping) would
drive the number up,
Even heavy disk thrashing would only drive the number up by a f
On Wed, 18 Apr 2007, Joshua Baker-LePain wrote:
If anyone is looking (or knows someone looking) for a HPC/cluster sysadmin in
or around San Francisco, I would greatly appreciate hearing from you
off-list. I'll be moving out there at the end of July, and I'd really like
to be able to pay the r
Amrik Singh wrote:
Recently we started noticing very high (70-90%) wait states on the file
servers when compute nodes. We have tried to optimize the NFS through
increasing the number of daemons and the rsize and wsize but to no avail.
PS: All the nodes are running SuSE 10.0 and servers are r
Hey Jon,
Fun to see you here!! I was just looking through some old Goleta pictures
last week.
Just for kicks have a look at these figures:
http://www.lsc-group.phys.uwm.edu/beowulf/nemo/design/SMC_8508T_Performance.html
This was part of a study that we did to select edge switches for the N
Hmmm is the April 1 date purely a coincidence??
On Sun, 1 Apr 2007, Douglas Eadline wrote:
I just posted some interesting news on Cluster Monkey.
http://www.clustermonkey.net//content/view/192/1/
--
Doug
___
Beowulf mailing list, Beowulf@beowulf.
Hi Stu,
Actually I run it (Lustre) in production and I'm not a kernel hacker.
Thank you for this snapshot of 'real world' Lustre use. At the risk of
hijacking this thread (or borrowing it...) could I ask you a question
about Lustre? I've always been interested in Lustre but never used it.
Hi Thomas,
If I understand correctly, your NFS server is connected via a 10Gb/s link
to the switch, and you've spent some effort to tune it. What sort of
aggregate NFS performance are you seeing? Is it at the level of 300 or
400 MG/sec? Could you please provide a few details about the hardw
smartctl version 5.32 Copyright (C) 2002-4 Bruce Allen
the machine I checked has 5.33, and 5.36 is the date on the sources I grabbed
in early dec. the machine in question is running HP XC 3.0, based on RHEL4's
2.6.9 - obviously with some backports.
Device: ATA HDT722525DLA380 Ve
On Tue, 6 Feb 2007, Joshua Baker-LePain wrote:
On Tue, 6 Feb 2007 at 2:17pm, Bruce Allen wrote
Here is an important firmware update for WDC WDYS series drives on RAID
controllers. Without this update you will see period drive dropouts and
rebuilds on the RAID sets. We've been s
Here is an important firmware update for WDC WDYS series drives on
RAID controllers. Without this update you will see period drive dropouts
and rebuilds on the RAID sets. We've been seeing this a lot with some
Areca controllers; I am hoping that this firmware update will fix the
problem.
Hi Mark,
For my next cluster room, I am hoping to use 'liquid cooled' racks make by
Knurr (CoolTherm, http://www.thermalmanagement.de/). The scale is 67 racks
x 7.5kW heat removal per rack (36 U usable per rack).
7.5 KW/rack isn't much; are you designing low-power nodes?
Yup -- I guess so.
Dear Beowulf list,
For my next cluster room, I am hoping to use 'liquid cooled' racks make by
Knurr (CoolTherm, http://www.thermalmanagement.de/). The scale is 67
racks x 7.5kW heat removal per rack (36 U usable per rack).
Does anyone on the list have experiences with these or similar racks
erformance impact from enabling scrubbing
is on your systems? did you do any before/after benchmarking?
Thanks,
-stephen
Bruce Allen wrote:
On Sun, 3 Sep 2006, Mark Hahn wrote:
ECC Features
ECCEnabled
ECC Scrub RedirectionEnabled
Dram ECC Scrub CTLDisabled
on, 4 Sep 2006, stephen mulcahy wrote:
Hi Bruce,
Do you have any idea what the performance impact from enabling scrubbing
is on your systems? did you do any before/after benchmarking?
Thanks,
-stephen
Bruce Allen wrote:
On Sun, 3 Sep 2006, Mark Hahn wrote:
ECC Features
ECCEn
On Sun, 3 Sep 2006, Mark Hahn wrote:
ECC Features
ECC Enabled
ECC Scrub Redirection Enabled
Dram ECC Scrub CTL Disabled
Chip-Kill Disabled
DCACHE ECC Scrub CTLDisabled
L2
Stephen,
Search the archives of this list to find some advice from me and others
about setting the ECC features.
Cheers,
Bruce
On Thu, 31 Aug 2006, stephen mulcahy wrote:
Hi,
I'm maintaining a 20-node cluster of Tyan K8SREs (4GB RAM, dual Opteron
270s) which are being used primari
Just to throw my 2 cents worth into these discussions, I've recently taken
delivery of 34 of the following machines, so around 200 TB total usable
storage. Cost is $6.7k each, so about $1200 per Terabyte.
The local disks exceed 200 MB/sec read/write speeds. As NFS servers we
can get about 10
Date: Fri, 05 May 2006 10:08:47 +0100
From: John Hearns <[EMAIL PROTECTED]>
Subject: Re: [Beowulf] 512 nodes Myrinet cluster Challanges
On Fri, 2006-05-05 at 10:23 +0200, Alan Louis Scheinine wrote:
Since you'all are talking about IPMI, I have a question.
The newer Tyan boards have a plug-in I
ve some problem size it's RAM
usage increases for other reasons?
What about the "PxQ" parameters? For 676 cores we are using square P=Q
but change this to use 1280 cores. Does anyone know of problems with
running xhpl when P != Q
I can (somewhat) answer this question in the case of Supermicro H8SSL-i
motherboards with the Supermicro plug in IPMI2 card. The situation is
identical to what you describe: the motherboard has two copper gigE ports,
and the IPMI card 'piggybacks' on one of those ports.
The anwer is 'it depen
rsh typically uses ports in the 513-1023 range. With a 640 node cluster
we are running out of ports. This leads to messages such as:
"rcmd: socket: All ports in use"
Are there any standard solutions to this other than 'use ssh'? We already
have net.ipv4.tcp_tw_recycle = 1 in /etc/sysctl.conf
What breakers are not tested for is the trip time when operated at under its
current rating. It is not uncommon especially as the breakers age or have
been tripped repeatedly, to trip at less than 100% of its rating. I've
replaced breakers many times that would trip when under 80-95% of its rate
In our experience it's a very good idea to observe the "don't operate a
circuit at more than 80% of rated capacity". We just had to replace a
150A 3-phase breaker which died after being operated a bit too close
(within 10A) of its rated capacity for a number of months.
Cheers,
Bruce
Bill,
I have similar issues in purchasing. U. of Wisconsin requires that we use
a sealed bid system, so writing good specs is important.
You can get some protection by requiring that in their submisssion a
bidder provide you with MTBF (Mean Time Between Failure) statistical data
for ALL fans
For what it's worth, the Opteron 175 (2 x 2.2 GHz cores) is identical
silicon to the AMD64 x2 4400+. The qualification/testing process is
apparently NOT the same for the two types of chip, and the register that
returns 'chip ID' is programmed differently. Other than that, I have been
told by A
On Tue, Mar 28, 2006 at 10:01:03PM -0600, Bruce Allen wrote:
After discussions with some of their technical people, the conclusion was
that the switch had enough buffer memory to run wire speed with MTU=1500
but not with MTU=9000.
Our testing results are here:
http://www.lsc-group.phys.uwm.edu
I suggest that you do some careful netperf testing of the 2724 before you
purchase it. We recently tested some 2716 switches with jumbo frames
(MTU=9000) and found the performance to be well below the advertised 'wire
speed'. The response from Dell was 'when we say it's wire speed, we don't
r
http://www.supermicro.com/Aplus/motherboard/Opteron/HT1000/H8SSL-i.cfm
http://www.supermicro.com/products/chassis/1U/?chs=813
On Tue, 21 Mar 2006, John Hearns wrote:
On Tue, 2006-03-21 at 08:41 +0100, Eugen Leitl wrote:
* 1U chassis
* 1 dual-core amd64
* 4 SATA drives
* 1 32-bit PCI slot (pre
Hi Mark,
according to the chart in the AMD doc, if you already have two 2-rank dimms
installed, further dimms need to be single-rank dimms to stay at ddr400.
the wording of the doc makes it seem like this is actually up to the board
vendor, though - how they configure the memory controller in bi
perhaps this is TMI, but the answer depends on which cpu rev you're
using (and whether the bios optimizes the memory controller.) the AMD
bios-writers doc cited earlier shows that 4 dimms (2xdouble, 2xsingle)
can still be used at ddr400 on rev e. no config uses >6 CS's at DDR400,
and some drop
More details at the GAMMA web page, http://www.disi.unige.it
Did you mean:
http://www.disi.unige.it/project/gamma/
___
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.or
I have seen 200MB/s over infiniband. Needless to say I was not overly
impressed, expecting more something in the 800-900MB/s
range. (pci-x bottlenecked infiniband)
PCI-X should be 133 MHz x 8 bytes = 1034 MB/s. Any idea why it
bottlenecked at 200?
Cheers,
Bruce
_
spec.org has a benchmark that is nfs-specific but it does not give you MB/s
for a special access pattern.
Instead, it gives you operations per second as well as response time at
that load for a mix of different access patterns, filesizes between 1K
and 1M.
http://www.spec.org/benchmarks.html
Hi Peter, others,
I'd like to know the fastest that anyone has seen an NFS server run, over
either a 10Gb/s ethernet link or a handful of link aggregated
(channel-bonded) Gb/s ethernet lines.
First, don't hijack threads, can't you guys have any mercy on us with thread
capable e-mail clients?
Hi Joe,
don't understand or believe this. If the server's local disks can
read/write at 300MB/s and the networking can run substantially faster than
100 MB/s, I don't see any constraint to faster operation. But perhaps
someone on this list can provide real-world data (or say why it can't
wo
I'd like to know the fastest that anyone has seen an NFS server run, over
either a 10Gb/s ethernet link or a handful of link aggregated
(channel-bonded) Gb/s ethernet lines.
This would be with a small number of clients making large file sequential
reads from the same NFS host/server. Please a
L2 Cache BG Scrub
not DRAM related
Data Cache BG Scrub
not DRAM related
[Note from Bruce: can anyone on the list make recommendations about this
last two, non-DRAM-related SCRUB settings??]
Sounds like we must not Scrub the memory too frequently because i
Salut Velu!
Hey bruce, World is small isn't it ;))
Yup. (Actually if you look back through the archives you'll find the
first annoucment of smartmontools was on this list.)
[...]
I would appreciate advice about:
-- how to configure these settings
-- pointers to relevant AMD/Serverwork
On Thu, 26 Jan 2006, Mark Hahn wrote:
(in the absence of windows, there's no reason to use builtin fraid over
MD.)
I'm not sure about this. I had the impression that the 'builtin fraid'
on this system has Linux support from Broadcom, including both some OS
drivers and user-side tools (CLI a
http://www.supermicro.com/Aplus/motherboard/Opteron/HT1000/H8SSL-R10.cfm
We too are waiting for SMicro to release the Serverworks boards with
**functioning** RAID on them. For example we are using their 1020P-T
server, but for now have to use a 3rd party card, such as the 3Ware
9550 to get this
Dear Beowulf list,
Our new cluster nodes (Supermicro H8SSL-i motherboard) have Opteron 175
CPUs, (unregistered) ECC memory dimms, and a serverworks HT1000 chipset.
The BIOS offers a number of ECC configuration options. I would like
advice about how to set these. We're running a recent Linux
On Wed, 11 Jan 2006, Maurice Hilarius wrote:
http://www.supermicro.com/Aplus/motherboard/Opteron/HT1000/H8SSL-R10.cfm
Has anyone on the list seen or used a Supermicro H8SSL-R10 motherboard?
We too are waiting for SMicro to release the Serverworks boards with
**functioning** RAID on them. For
78 matches
Mail list logo
