: solr, 100 documents at a time. I was doing a commit after each of those : but after what Yonik says I will remove it and commit only after each : batch of 25k.
do the commit only when you think it's neccessary to expose those docs to your search clients, one of which may be "you" checking on the progress of your index build. : Q1: I've got autocommit set to 1000 now.. in solrconfig.xml, should i : disable it in this scenario? i'm guessing you don't want that if you are doing full builds on a regular basis. it's intent is for indexes that are being continuously updated and you just want to know that eventually a commit will happen 9wihtout needing to ever call it explicilty) : Q2: To decide which of those 25k are going to be indexed, we need to do : a query for each (this is the main reason to optimize before a new DB : batch is indexed), each of these 25k queries take around 30ms which is : good enough for us, but i've observed every ~30 queries the time of one : search goes up to 150ms or even 1200ms. Then it does another ~30, etc. I : guess there is something happening inside the server regularly that : causes it. Any clues what it can be and how can i minimize that time? are these queries happening simultenously with the updates? the autocommiting will be causing a newSearcher to be opened, and the first search on it will have to pay some added cost. besdies autocommit, there is nothing that happens automaticly on a recuring basis in Solr .. there may be something else running on your box that is using ram, which is taking away from the disk page cache, which causes some searches to need to rerad pages (pure speculation) : Q3: The 25k searches are done without any cumulative effect on : performance (avg/search is ~30ms from start to end). But if inmmediately : after start posting documents to the index tomcat peaks CPU. But if i : stop tomcat, and then post the 25k documents without doing those : searches they're very quick. Is there any reason why the searches would : affect tomcat to justify this? Just to clarify, searches are NOT done at : the same time as indexing. i'm having trouble understanding your question ... how can you post documenst after stopping tomcat? -Hoss
