We had one issue with our Solr production deployment couple of weeks back. Following is more info about it.
Server Setup =============== Platform: Sun Solaris Ultrasparc JDK: 1.5 Solr: 1.2 Index Size: ~15GB Topology: One master and two slaves Problem Statement =============== Every day we index different contents into the Solr master and run the optimize at the end of it. Snapshooter is triggered at the end of Optimize and it creates a snapshot of the index. After couple of hours, Slaves pull the latest snapshot and install it to serve the searches. Couple of weeks back, Slaves didn't pull the snapshot and when researched we found that one file (.nrm) was missing in the snapshot created by master, as shown below. # ls -lrth total 29877000 -rw-r--r-- 1 jboss staff 1.9K Apr 14 11:00 _429o.fnm -rw-r--r-- 1 jboss staff 28M Apr 14 12:18 _429o.fdx -rw-r--r-- 1 jboss staff 9.8G Apr 14 12:18 _429o.fdt -rw-r--r-- 1 jboss staff 501M Apr 14 12:43 _429o.tis -rw-r--r-- 1 jboss staff 6.5M Apr 14 12:43 _429o.tii -rw-r--r-- 1 jboss staff 2.4G Apr 14 12:43 _429o.prx -rw-r--r-- 1 jboss staff 1.2G Apr 14 12:43 _429o.frq -rw-r--r-- 1 jboss staff 44 Apr 14 12:44 segments_53gt -rw-r--r-- 1 jboss staff 20 Apr 14 12:44 segments.gen -rw-r--r-- 1 jboss staff 351M Apr 14 12:44 _429o.nrm <--- This is the missing file in the snapshot! # cd snapshot.20090414124449/ # ls -lrth total 29157784 -rw-r--r-- 1 jboss staff 20 Apr 14 12:44 segments.gen -rw-r--r-- 1 jboss staff 1.9K Apr 14 12:44 _429o.fnm -rw-r--r-- 1 jboss staff 9.8G Apr 14 13:03 _429o.fdt -rw-r--r-- 1 jboss staff 28M Apr 14 13:03 _429o.fdx -rw-r--r-- 1 jboss staff 1.2G Apr 14 13:06 _429o.frq -rw-r--r-- 1 jboss staff 2.4G Apr 14 13:12 _429o.prx -rw-r--r-- 1 jboss staff 501M Apr 14 13:13 _429o.tis -rw-r--r-- 1 jboss staff 6.5M Apr 14 13:13 _429o.tii -rw-r--r-- 1 jboss staff 44 Apr 14 13:13 segments_53gt ***This snapshot is missing _429o.nrm file!*** Has anybody faced this issue (missing a file, or may be missing .nrm file) might happen? Any insight is greatly appreciated. Couple of other questions. 1. For index of size 15GB, how much of breathing space is required (both memory and diskspace) in master and slave? 2. Would Slaves pull the snapshots if any of the files are missing in index? (I guess that snapshot-puller will pull but may not be able to install it?) 3. We are using Solr 1.2 and contemplating to upgrade to 1.3. What is your experience in this path of upgrade? Is it strongly recommended (based on any critical bugs which were fixed?) Thanks, Santhosh.
