Just to clarify, I'm not worried about the virtual memory getting bigger, the issue is that after doing a lot of adds without a commit the performance dramatically decreases until we do the commit. This didn't use to be a problem with 1.4
Willem On Fri, Sep 30, 2011 at 10:20 AM, Willem Basson <willem.bas...@gmail.com>wrote > Hi there > > We are currently upgrading from solr 1.4 to 3.4 and have seen some issues > with our specific use-case. > As background we drop the whole index and then add all our documents in one > big build before we commit and then optimise. > This way we can revert the build if there are any issues and this won't be > replicated out to our slave instances. > > We use the following flags: -Xmx2g -Xms2g > With 1.4 it works pretty well, and the process doesn't use much more than > 2GB of memory as the index is being built. Garbage collector kicks in quite > a bit but performance are pretty decent throughout the build. > For 3.4 though the process builds up to use a lot more memory, about 20 to > 30 GB and it starts swapping and grinding to a bit of a halt. This means it > takes about 10 times longer to complete the build, but it does complete. > We have about a 100k documents and they aren't massive, but they aren't > small either. They do have a lot of fields though, some have 6000+ fields. > > Monitoring the heap size I can see that it stays under 2GB and garbage > collector seems to kick in just like with 1.4 > > While we could increase the memory and do a few other things to make this > less of an issue I would really like to know if anyone has any idea what the > problem could be and what we could do to try and change the behaviour in > config. Our solrconfig.xml file is the same for 1.4 and 3.4, we haven't made > any changes. > > Thanks > > Willem Basson >
