Your continuous deletes won't affect performance noticeably, that's true. But you're really doing bad things with the commit after every add or delete. You haven't said whether you have a master/ slave setup or not, but assuming you're searching on the same machine you're indexing to, each time you commit, you're forcing the underlying searcher to close and re-open and any attendant autowarming to occur. All to get a single document searchable. 20 times a second. If you have a master/ slave setup, you're forcing the slave to fetch the changed parts of the index every time it polls, which is better than what's happening on the master, but still rather often.
400K documents isn't very big by Solr standards, so unless you can show performance problems, I wouldn't be concerned about index size, as Otis says, your per-document commit is probably hurting you far more than any index size savings. I'd actually think carefully about whether you need even 10 second commits. If you can stretch that out to minutes, so much the better. But it all depends upon your problem space. Best Erick On Mon, Feb 6, 2012 at 2:59 AM, prasenjit mukherjee <prasen....@gmail.com> wrote: > Thanks Otis. commitWithin will definitely work for me ( as I > currently am using 3.4 version, which doesnt have NRT yet ). > > Assuming that I use commitWithin=10secs, are you saying that the > continuous deletes ( without commit ) wont have any affect on > performance ? > I was under the impression that deletes just mark the doc-ids ( > essentially means that the index size will remain the same ) , but > wont actually do the compaction till someone calls optimize/commit, is > my assumption not true ? > > -Thanks, > Prasenjit > > On Mon, Feb 6, 2012 at 1:13 PM, Otis Gospodnetic > <otis_gospodne...@yahoo.com> wrote: >> Hi Prasenjit, >> >> It sounds like at this point your main enemy might be those per-doc-add >> commits. Don't commit until you need to see your new docs in results. And >> if you need NRT then use softCommit option with Solr trunk >> (http://search-lucene.com/?q=softcommit&fc_project=Solr) or use commitWithin >> to limit commit's "performance damage". >> >> >> Otis >> >> ---- >> Performance Monitoring SaaS for Solr - >> http://sematext.com/spm/solr-performance-monitoring/index.html >> >> >> >>>________________________________ >>> From: prasenjit mukherjee <prasen....@gmail.com> >>>To: solr-user <solr-user@lucene.apache.org> >>>Sent: Monday, February 6, 2012 1:17 AM >>>Subject: effect of continuous deletes on index's read performance >>> >>>I have a use case where documents are continuously added @ 20 docs/sec >>>( each doc add is also doing a commit ) and docs continuously getting >>>deleted at the same rate. So the searchable index size remains the >>>same : ~ 400K docs ( docs for last 6 hours ~ 20*3600*6). >>> >>>Will it have pauses when deletes triggers compaction. Or with every >>>commits ( while adds ) ? How bad they will effect on search response >>>time. >>> >>>-Thanks, >>>Prasenjit >>> >>> >>>
