I index in 10K batches and commit after 5 index cyles (after 50K). Is there any limitation that I can't search during commit or auto-warming? I got 8 CPU cores and only 2 were showing busy (using top) - so it's unlikely that the CPU was pegged.
2009/4/12 Noble Paul നോബിള് नोब्ळ् <noble.p...@gmail.com>: > If you use StreamingUpdateSolrServer it POSTs all the docs in a single > request. 10 million docs may be a bit too much for a single request. I > guess you should batch it in multiple requests of smaller chunks, > > It is likely that the CPU is really hot when the autowarming is hapening. > > getting a decent search perf w/o autowarming is not easy . > > autowarmCount is an attribute of a cache .see here > http://wiki.apache.org/solr/SolrCaching > > On Mon, Apr 13, 2009 at 3:32 AM, vivek sar <vivex...@gmail.com> wrote: >> Thanks Shalin. >> >> I noticed couple more things. As I index around 100 million records a >> day, my Indexer is running pretty much at all times throughout the >> day. Whenever I run a search query I usually get "connection reset" >> when the commit is happening and get "blank page" when the >> auto-warming of searchers is happening. Here are my questions, >> >> 1) Is this coincidence or a known issue? Can't we search while commit >> or auto-warming is happening? >> 2) How do I stop auto-warming? My search traffic is very low so I'm >> trying to turn off auto-warming after commit has happened - is there >> anything in the solrconfig.xml to do that? >> 3) What would be the best strategy for searching in my scenario where >> commits may be happening all the time (I commit every 50K records - so >> every 30-60 sec there is a commit happening followed by auto-warming >> that takes 40 sec)? >> >> Search frequency is pretty low for us, but we want to make sure that >> whenever it happens it is fast enough and returns result (instead of >> exception or a blank screen). >> >> Thanks for all the help. >> >> -vivek >> >> >> >> On Sat, Apr 11, 2009 at 1:48 PM, Shalin Shekhar Mangar >> <shalinman...@gmail.com> wrote: >>> On Sun, Apr 12, 2009 at 2:15 AM, vivek sar <vivex...@gmail.com> wrote: >>> >>>> >>>> The problem is I don't see any error message in the catalina.out. I >>>> don't even see the request coming in - I simply get blank page on >>>> browser. If I keep trying the request goes through and I get respond >>>> from Solr, but then it become unresponsive again or sometimes throws >>>> "connection reset" error. I'm not sure why would it work sometimes and >>>> not the other times for the same query. As soon as I stop the Indexer >>>> process things start working fine. Any way I can debug this problem? >>>> >>> >>> I'm not sure. I've never seen this issue myself. >>> >>> Could you try using the bundled jetty instead of Tomcat or on a different >>> box just to make sure this is not an environment specific issue? >>> >>> -- >>> Regards, >>> Shalin Shekhar Mangar. >>> >> > > > > -- > --Noble Paul >
