We are a bit concerned regarding the index size. At least no response (so far) as indicated that the size is unmanagable. We killed the process - will move to Java6 - and will use vmstat to monitor the new optimization process. At what index size would you begin to worry? Or is it a combination of index size, optimization time, and response time? We are data rich here! Thanks Frances
Otis Gospodnetic wrote: > > That's a tiny little index there ;) Circa 100GB? > > What do you see if you run vmstat 2 while the optimization is happening? > Non-idle CPU? A pile of IO? Is there a reason for such a small heap on a > machine with 32GB of RAM? > > Otis > > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > ----- Original Message ---- >> From: F Knudson <[EMAIL PROTECTED]> >> To: solr-user@lucene.apache.org >> Sent: Thursday, February 28, 2008 9:54:50 AM >> Subject: Optimization taking days/weeks >> >> >> Optimization time on solr index has turned into days/weeks. >> We are using solr 1.2. >> We use one box to build/optimize indexes. This index is copied to another >> box for searching purposes. >> We welcome suggestions/comments, etc. We are a bit stumped on this. >> Details are below. >> >> Box details >> Proc: 8 Dual Core 2.6GHz >> Mem: 32 GB >> OS: Red Hat Linux Enterprise 4 >> Kernel: 2.6.9-55.0.12.ELlargesmp >> >> These are details from the index currently in use. Search response time >> is >> very acceptable (searchers are very happy) >> Optimization time - 10433 (12/11/07) >> index size - 229486464 >> # of records - 84960570 >> index directory >> flasher# ls -l >> total 229486464 >> -rw-r--r-- 1 flknud staff 22197926593 Dec 12 08:07 _2bl6.fdt >> -rw-r--r-- 1 flknud staff 679684560 Dec 12 08:20 _2bl6.fdx >> -rw-r--r-- 1 flknud staff 208 Dec 12 08:23 _2bl6.fnm >> -rw-r--r-- 1 flknud staff 40176405625 Dec 12 09:28 _2bl6.frq >> -rw-r--r-- 1 flknud staff 594723994 Dec 12 09:41 _2bl6.nrm >> -rw-r--r-- 1 flknud staff 47616340310 Dec 12 12:07 _2bl6.prx >> -rw-r--r-- 1 flknud staff 76708079 Dec 12 12:25 _2bl6.tii >> -rw-r--r-- 1 flknud staff 6154384415 Dec 12 12:42 _2bl6.tis >> -rw-r--r-- 1 flknud staff 20 Dec 12 12:48 segments.gen >> -rw-r--r-- 1 flknud staff 44 Dec 12 12:48 segments_2c64 >> -------------------------- >> >> current directory listing >> indexed new records - Jan 22 and Jan 27 >> # of records - 85032470 >> optimization time - 558188 >> >> There were no out of memory errors. There was 800961792KB left in the >> directory. The files were not >> collapsed as expected. There are still files dated Jan 22 and Jan 27. >> >> A new optimization was started Feb. 11 and continues. >> This is a snapshot of the index directory. >> >> We have at least another million records to add. Plus weekly updates of >> approximately 103K records. >> We are using the direct indexing method. >> java settings used - java -Xmx1024M -Xms1024M >> >> The files continue to grow so work is progressing. >> snapshot 2/21/08 >> -bash-3.00$ ls -ltr >> total 205396680 >> -rw-r--r-- 1 flknud users 208 Jan 10 07:15 _2bm7.fnm >> -rw-r--r-- 1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt >> -rw-r--r-- 1 flknud users 679819760 Jan 10 08:09 _2bm7.fdx >> -rw-r--r-- 1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq >> -rw-r--r-- 1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx >> -rw-r--r-- 1 flknud users 6155230704 Jan 16 18:16 _2bm7.tis >> -rw-r--r-- 1 flknud users 76704158 Jan 16 18:16 _2bm7.tii >> -rw-r--r-- 1 flknud users 594842294 Jan 16 18:18 _2bm7.nrm >> -rw-r--r-- 1 flknud users 208 Jan 22 08:57 _2bpa.fnm >> -rw-r--r-- 1 flknud users 10806426 Jan 22 08:57 _2bpa.fdt >> -rw-r--r-- 1 flknud users 371200 Jan 22 08:57 _2bpa.fdx >> -rw-r--r-- 1 flknud users 21114330 Jan 22 08:57 _2bpa.frq >> -rw-r--r-- 1 flknud users 25683573 Jan 22 08:57 _2bpa.prx >> -rw-r--r-- 1 flknud users 9225592 Jan 22 08:57 _2bpa.tis >> -rw-r--r-- 1 flknud users 118660 Jan 22 08:57 _2bpa.tii >> -rw-r--r-- 1 flknud users 324804 Jan 22 08:57 _2bpa.nrm >> -rw-r--r-- 1 flknud users 198 Jan 22 09:00 _2bpl.fnm >> -rw-r--r-- 1 flknud users 1335931 Jan 22 09:00 _2bpl.fdt >> -rw-r--r-- 1 flknud users 36800 Jan 22 09:00 _2bpl.fdx >> -rw-r--r-- 1 flknud users 2646708 Jan 22 09:00 _2bpl.frq >> -rw-r--r-- 1 flknud users 3781824 Jan 22 09:00 _2bpl.prx >> -rw-r--r-- 1 flknud users 1429176 Jan 22 09:00 _2bpl.tis >> -rw-r--r-- 1 flknud users 18582 Jan 22 09:00 _2bpl.tii >> -rw-r--r-- 1 flknud users 32204 Jan 22 09:00 _2bpl.nrm >> -rw-r--r-- 1 flknud users 198 Jan 22 09:01 _2bpm.fnm >> -rw-r--r-- 1 flknud users 121716 Jan 22 09:01 _2bpm.fdt >> -rw-r--r-- 1 flknud users 3200 Jan 22 09:01 _2bpm.fdx >> -rw-r--r-- 1 flknud users 205961 Jan 22 09:01 _2bpm.frq >> -rw-r--r-- 1 flknud users 302114 Jan 22 09:01 _2bpm.prx >> -rw-r--r-- 1 flknud users 233641 Jan 22 09:01 _2bpm.tis >> -rw-r--r-- 1 flknud users 3036 Jan 22 09:01 _2bpm.tii >> -rw-r--r-- 1 flknud users 2804 Jan 22 09:01 _2bpm.nrm >> -rw-r--r-- 1 flknud users 198 Jan 27 14:00 _2bpn.fnm >> -rw-r--r-- 1 flknud users 227962 Jan 27 14:00 _2bpn.fdt >> -rw-r--r-- 1 flknud users 7200 Jan 27 14:00 _2bpn.fdx >> -rw-r--r-- 1 flknud users 437798 Jan 27 14:00 _2bpn.frq >> -rw-r--r-- 1 flknud users 593858 Jan 27 14:00 _2bpn.prx >> -rw-r--r-- 1 flknud users 516031 Jan 27 14:00 _2bpn.tis >> -rw-r--r-- 1 flknud users 6814 Jan 27 14:00 _2bpn.tii >> -rw-r--r-- 1 flknud users 6304 Jan 27 14:00 _2bpn.nrm >> -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpo.fnm >> -rw-r--r-- 1 flknud users 231456 Jan 27 14:01 _2bpo.fdt >> -rw-r--r-- 1 flknud users 7200 Jan 27 14:01 _2bpo.fdx >> -rw-r--r-- 1 flknud users 448401 Jan 27 14:01 _2bpo.frq >> -rw-r--r-- 1 flknud users 616557 Jan 27 14:01 _2bpo.prx >> -rw-r--r-- 1 flknud users 587697 Jan 27 14:01 _2bpo.tis >> -rw-r--r-- 1 flknud users 7801 Jan 27 14:01 _2bpo.tii >> -rw-r--r-- 1 flknud users 6304 Jan 27 14:01 _2bpo.nrm >> -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpp.fnm >> -rw-r--r-- 1 flknud users 229944 Jan 27 14:01 _2bpp.fdt >> -rw-r--r-- 1 flknud users 6400 Jan 27 14:01 _2bpp.fdx >> -rw-r--r-- 1 flknud users 462865 Jan 27 14:01 _2bpp.frq >> -rw-r--r-- 1 flknud users 652003 Jan 27 14:01 _2bpp.prx >> -rw-r--r-- 1 flknud users 574096 Jan 27 14:01 _2bpp.tis >> -rw-r--r-- 1 flknud users 7452 Jan 27 14:01 _2bpp.tii >> -rw-r--r-- 1 flknud users 5604 Jan 27 14:01 _2bpp.nrm >> -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpq.fnm >> -rw-r--r-- 1 flknud users 53438 Jan 27 14:01 _2bpq.fdt >> -rw-r--r-- 1 flknud users 1600 Jan 27 14:01 _2bpq.fdx >> -rw-r--r-- 1 flknud users 103059 Jan 27 14:01 _2bpq.frq >> -rw-r--r-- 1 flknud users 157149 Jan 27 14:01 _2bpq.prx >> -rw-r--r-- 1 flknud users 179711 Jan 27 14:01 _2bpq.tis >> -rw-r--r-- 1 flknud users 2489 Jan 27 14:01 _2bpq.tii >> -rw-r--r-- 1 flknud users 1404 Jan 27 14:01 _2bpq.nrm >> -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpr.fnm >> -rw-r--r-- 1 flknud users 200777 Jan 27 14:01 _2bpr.fdt >> -rw-r--r-- 1 flknud users 6400 Jan 27 14:01 _2bpr.fdx >> -rw-r--r-- 1 flknud users 416658 Jan 27 14:01 _2bpr.frq >> -rw-r--r-- 1 flknud users 570959 Jan 27 14:01 _2bpr.prx >> -rw-r--r-- 1 flknud users 489593 Jan 27 14:01 _2bpr.tis >> -rw-r--r-- 1 flknud users 6451 Jan 27 14:01 _2bpr.tii >> -rw-r--r-- 1 flknud users 5604 Jan 27 14:01 _2bpr.nrm >> -rw-r--r-- 1 flknud users 236 Jan 27 14:01 segments_2cap >> -rw-r--r-- 1 flknud users 20 Jan 27 14:01 segments.gen >> -rw-r--r-- 1 flknud users 0 Feb 11 09:56 write.lock >> -rw-r--r-- 1 flknud users 208 Feb 11 09:56 _2bps.fnm >> -rw-r--r-- 1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt >> -rw-r--r-- 1 flknud users 680259760 Feb 11 12:26 _2bps.fdx >> -rw-r--r-- 1 flknud users 24416256 Feb 21 06:42 _2bps.tii >> -rw-r--r-- 1 flknud users 1685057536 Feb 21 06:56 _2bps.tis >> -rw-r--r-- 1 flknud users 29885433856 Feb 21 07:32 _2bps.frq >> -rw-r--r-- 1 flknud users 38229336064 Feb 21 07:32 _2bps.prx >> -- >> View this message in context: >> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- View this message in context: http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15762378.html Sent from the Solr - User mailing list archive at Nabble.com.
