Dear fellow Solr users, I've been using Solr for a very short time now and I'm stuck. I'm trying to index a drupal website consisting of 1.2 million smaller nodes and 300k larger nodes (~400kb avg)..
I'm using Solr 3.5 on a dedicated Ubuntu 10.04 box with 3TB of diskspace and 16GB of memory. I've tried using the sun JRE and OpenJDK, both resulting in the same problem. Indexing works great until my .fdt file reaches the size of 4.9GB/ 5217987319b. At this point when Solr starts merging it just keeps on merging, starting over and over.. Java is using all the available memory even though Xmx is set at 8G. When I restart Solr everything looks fine until merging is triggered. Whenever it hangs the server load averages 3, searching is possible but slow, the solr admin interface is reachable but sending new documents leads to a time-out. I've tried using several different settings for MergePolicy and started reindexing a couple of times but the behavior stays the same. My current solrconf.xml can be found here: http://pastebin.com/NXDT0B8f. I'm unable to find errors in the log which makes it really difficult to debug.. Could anyone point me in the right direction? I've already asked my question on stackoverflow without receiving a solution: http://stackoverflow.com/questions/9993633/apache-solr-3-5-hangs-when-indexing. Maybe it can provide you with some more information. Kind regards! Bram Rongen
