Re: Dataimport destroys our harddisks

2010-12-02 Thread Sven Almgren
That's the same series we use... we hade problems when running other disk-heavy operations like rsync and backup on them too.. But in our case we mostly had hangs or load > 180 :P... Can you simulate very heavy random disk i/o? if so then you could check if you still have the same problems... Tha

Re: Dataimport destroys our harddisks

2010-12-02 Thread Robert Gründler
On Dec 2, 2010, at 15:43 , Sven Almgren wrote: > What Raid controller do you use, and what kernel version? (Assuming > Linux). We hade problems during high load with a 3Ware raid controller > and the current kernel for Ubuntu 10.04, we hade to downgrade the > kernel... > > The problem was a bug i

Re: Dataimport destroys our harddisks

2010-12-02 Thread Sven Almgren
What Raid controller do you use, and what kernel version? (Assuming Linux). We hade problems during high load with a 3Ware raid controller and the current kernel for Ubuntu 10.04, we hade to downgrade the kernel... The problem was a bug in the driver that only showed up with very high disk load (a

Re: Dataimport destroys our harddisks

2010-12-02 Thread Robert Gründler
> The very first thing I'd ask is "how much free space is on your disk > when this occurs?" Is it possible that you're simply filling up your > disk? no, i've checked that already. all disks have plenty of space (they have a capacity of 2TB, and are currently filled up to 20%. > > do note that a

Re: Dataimport destroys our harddisks

2010-12-02 Thread Erick Erickson
The very first thing I'd ask is "how much free space is on your disk when this occurs?" Is it possible that you're simply filling up your disk? do note that an optimize may require up to 2X the size of your index if/when it occurs. Are you sure you aren't optimizing as you add items to your index?

Dataimport destroys our harddisks

2010-12-02 Thread Robert Gründler
Hi, we have a serious harddisk problem, and it's definitely related to a full-import from a relational database into a solr index. The first time it happened on our development server, where the raidcontroller crashed during a full-import of ~ 8 Million documents. This happened 2 weeks ago, and