That's a tiny little index there ;) Circa 100GB? What do you see if you run vmstat 2 while the optimization is happening? Non-idle CPU? A pile of IO? Is there a reason for such a small heap on a machine with 32GB of RAM?
Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: F Knudson <[EMAIL PROTECTED]> > To: solr-user@lucene.apache.org > Sent: Thursday, February 28, 2008 9:54:50 AM > Subject: Optimization taking days/weeks > > > Optimization time on solr index has turned into days/weeks. > We are using solr 1.2. > We use one box to build/optimize indexes. This index is copied to another > box for searching purposes. > We welcome suggestions/comments, etc. We are a bit stumped on this. > Details are below. > > Box details > Proc: 8 Dual Core 2.6GHz > Mem: 32 GB > OS: Red Hat Linux Enterprise 4 > Kernel: 2.6.9-55.0.12.ELlargesmp > > These are details from the index currently in use. Search response time is > very acceptable (searchers are very happy) > Optimization time - 10433 (12/11/07) > index size - 229486464 > # of records - 84960570 > index directory > flasher# ls -l > total 229486464 > -rw-r--r-- 1 flknud staff 22197926593 Dec 12 08:07 _2bl6.fdt > -rw-r--r-- 1 flknud staff 679684560 Dec 12 08:20 _2bl6.fdx > -rw-r--r-- 1 flknud staff 208 Dec 12 08:23 _2bl6.fnm > -rw-r--r-- 1 flknud staff 40176405625 Dec 12 09:28 _2bl6.frq > -rw-r--r-- 1 flknud staff 594723994 Dec 12 09:41 _2bl6.nrm > -rw-r--r-- 1 flknud staff 47616340310 Dec 12 12:07 _2bl6.prx > -rw-r--r-- 1 flknud staff 76708079 Dec 12 12:25 _2bl6.tii > -rw-r--r-- 1 flknud staff 6154384415 Dec 12 12:42 _2bl6.tis > -rw-r--r-- 1 flknud staff 20 Dec 12 12:48 segments.gen > -rw-r--r-- 1 flknud staff 44 Dec 12 12:48 segments_2c64 > -------------------------- > > current directory listing > indexed new records - Jan 22 and Jan 27 > # of records - 85032470 > optimization time - 558188 > > There were no out of memory errors. There was 800961792KB left in the > directory. The files were not > collapsed as expected. There are still files dated Jan 22 and Jan 27. > > A new optimization was started Feb. 11 and continues. > This is a snapshot of the index directory. > > We have at least another million records to add. Plus weekly updates of > approximately 103K records. > We are using the direct indexing method. > java settings used - java -Xmx1024M -Xms1024M > > The files continue to grow so work is progressing. > snapshot 2/21/08 > -bash-3.00$ ls -ltr > total 205396680 > -rw-r--r-- 1 flknud users 208 Jan 10 07:15 _2bm7.fnm > -rw-r--r-- 1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt > -rw-r--r-- 1 flknud users 679819760 Jan 10 08:09 _2bm7.fdx > -rw-r--r-- 1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq > -rw-r--r-- 1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx > -rw-r--r-- 1 flknud users 6155230704 Jan 16 18:16 _2bm7.tis > -rw-r--r-- 1 flknud users 76704158 Jan 16 18:16 _2bm7.tii > -rw-r--r-- 1 flknud users 594842294 Jan 16 18:18 _2bm7.nrm > -rw-r--r-- 1 flknud users 208 Jan 22 08:57 _2bpa.fnm > -rw-r--r-- 1 flknud users 10806426 Jan 22 08:57 _2bpa.fdt > -rw-r--r-- 1 flknud users 371200 Jan 22 08:57 _2bpa.fdx > -rw-r--r-- 1 flknud users 21114330 Jan 22 08:57 _2bpa.frq > -rw-r--r-- 1 flknud users 25683573 Jan 22 08:57 _2bpa.prx > -rw-r--r-- 1 flknud users 9225592 Jan 22 08:57 _2bpa.tis > -rw-r--r-- 1 flknud users 118660 Jan 22 08:57 _2bpa.tii > -rw-r--r-- 1 flknud users 324804 Jan 22 08:57 _2bpa.nrm > -rw-r--r-- 1 flknud users 198 Jan 22 09:00 _2bpl.fnm > -rw-r--r-- 1 flknud users 1335931 Jan 22 09:00 _2bpl.fdt > -rw-r--r-- 1 flknud users 36800 Jan 22 09:00 _2bpl.fdx > -rw-r--r-- 1 flknud users 2646708 Jan 22 09:00 _2bpl.frq > -rw-r--r-- 1 flknud users 3781824 Jan 22 09:00 _2bpl.prx > -rw-r--r-- 1 flknud users 1429176 Jan 22 09:00 _2bpl.tis > -rw-r--r-- 1 flknud users 18582 Jan 22 09:00 _2bpl.tii > -rw-r--r-- 1 flknud users 32204 Jan 22 09:00 _2bpl.nrm > -rw-r--r-- 1 flknud users 198 Jan 22 09:01 _2bpm.fnm > -rw-r--r-- 1 flknud users 121716 Jan 22 09:01 _2bpm.fdt > -rw-r--r-- 1 flknud users 3200 Jan 22 09:01 _2bpm.fdx > -rw-r--r-- 1 flknud users 205961 Jan 22 09:01 _2bpm.frq > -rw-r--r-- 1 flknud users 302114 Jan 22 09:01 _2bpm.prx > -rw-r--r-- 1 flknud users 233641 Jan 22 09:01 _2bpm.tis > -rw-r--r-- 1 flknud users 3036 Jan 22 09:01 _2bpm.tii > -rw-r--r-- 1 flknud users 2804 Jan 22 09:01 _2bpm.nrm > -rw-r--r-- 1 flknud users 198 Jan 27 14:00 _2bpn.fnm > -rw-r--r-- 1 flknud users 227962 Jan 27 14:00 _2bpn.fdt > -rw-r--r-- 1 flknud users 7200 Jan 27 14:00 _2bpn.fdx > -rw-r--r-- 1 flknud users 437798 Jan 27 14:00 _2bpn.frq > -rw-r--r-- 1 flknud users 593858 Jan 27 14:00 _2bpn.prx > -rw-r--r-- 1 flknud users 516031 Jan 27 14:00 _2bpn.tis > -rw-r--r-- 1 flknud users 6814 Jan 27 14:00 _2bpn.tii > -rw-r--r-- 1 flknud users 6304 Jan 27 14:00 _2bpn.nrm > -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpo.fnm > -rw-r--r-- 1 flknud users 231456 Jan 27 14:01 _2bpo.fdt > -rw-r--r-- 1 flknud users 7200 Jan 27 14:01 _2bpo.fdx > -rw-r--r-- 1 flknud users 448401 Jan 27 14:01 _2bpo.frq > -rw-r--r-- 1 flknud users 616557 Jan 27 14:01 _2bpo.prx > -rw-r--r-- 1 flknud users 587697 Jan 27 14:01 _2bpo.tis > -rw-r--r-- 1 flknud users 7801 Jan 27 14:01 _2bpo.tii > -rw-r--r-- 1 flknud users 6304 Jan 27 14:01 _2bpo.nrm > -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpp.fnm > -rw-r--r-- 1 flknud users 229944 Jan 27 14:01 _2bpp.fdt > -rw-r--r-- 1 flknud users 6400 Jan 27 14:01 _2bpp.fdx > -rw-r--r-- 1 flknud users 462865 Jan 27 14:01 _2bpp.frq > -rw-r--r-- 1 flknud users 652003 Jan 27 14:01 _2bpp.prx > -rw-r--r-- 1 flknud users 574096 Jan 27 14:01 _2bpp.tis > -rw-r--r-- 1 flknud users 7452 Jan 27 14:01 _2bpp.tii > -rw-r--r-- 1 flknud users 5604 Jan 27 14:01 _2bpp.nrm > -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpq.fnm > -rw-r--r-- 1 flknud users 53438 Jan 27 14:01 _2bpq.fdt > -rw-r--r-- 1 flknud users 1600 Jan 27 14:01 _2bpq.fdx > -rw-r--r-- 1 flknud users 103059 Jan 27 14:01 _2bpq.frq > -rw-r--r-- 1 flknud users 157149 Jan 27 14:01 _2bpq.prx > -rw-r--r-- 1 flknud users 179711 Jan 27 14:01 _2bpq.tis > -rw-r--r-- 1 flknud users 2489 Jan 27 14:01 _2bpq.tii > -rw-r--r-- 1 flknud users 1404 Jan 27 14:01 _2bpq.nrm > -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpr.fnm > -rw-r--r-- 1 flknud users 200777 Jan 27 14:01 _2bpr.fdt > -rw-r--r-- 1 flknud users 6400 Jan 27 14:01 _2bpr.fdx > -rw-r--r-- 1 flknud users 416658 Jan 27 14:01 _2bpr.frq > -rw-r--r-- 1 flknud users 570959 Jan 27 14:01 _2bpr.prx > -rw-r--r-- 1 flknud users 489593 Jan 27 14:01 _2bpr.tis > -rw-r--r-- 1 flknud users 6451 Jan 27 14:01 _2bpr.tii > -rw-r--r-- 1 flknud users 5604 Jan 27 14:01 _2bpr.nrm > -rw-r--r-- 1 flknud users 236 Jan 27 14:01 segments_2cap > -rw-r--r-- 1 flknud users 20 Jan 27 14:01 segments.gen > -rw-r--r-- 1 flknud users 0 Feb 11 09:56 write.lock > -rw-r--r-- 1 flknud users 208 Feb 11 09:56 _2bps.fnm > -rw-r--r-- 1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt > -rw-r--r-- 1 flknud users 680259760 Feb 11 12:26 _2bps.fdx > -rw-r--r-- 1 flknud users 24416256 Feb 21 06:42 _2bps.tii > -rw-r--r-- 1 flknud users 1685057536 Feb 21 06:56 _2bps.tis > -rw-r--r-- 1 flknud users 29885433856 Feb 21 07:32 _2bps.frq > -rw-r--r-- 1 flknud users 38229336064 Feb 21 07:32 _2bps.prx > -- > View this message in context: > http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
