Soft commits and NRT don't exist in Solr 3.x so I'm really confused. In 3.6 you have to do hard commits and they're expensive 1/sec is far too often...
Erick On Wed, Jul 24, 2013 at 5:10 AM, Mikhail Khludnev <mkhlud...@griddynamics.com> wrote: > Hello, > > First of all, I don't think it can commit (even soft) every second, afaik > it's too frequent for typical deployment. Hence, if you really need such > (N)RT I suggest you experiment with it right now, to face the bummer > sooner. > Also, one second durability sounds like over-expectation for Solr, it > sounds like OLTP requirements. > Then, now Solr has some sort of pre-indexing record storage called > UpdateLog. try to experiment with syncLevel = FSYNC vs FLUSH . > That's how it works, when document arrives for indexing it's written into > update log, which is plain binary file. Indexing works as-is relying on > RAMbuffer. When node dies, RAMbuffer dies, but updateLog is persistent, > during startup Solr recovers uncommitted updates from updateLog. Caveat! > UpdateLog has HashMap internally which easily hits OOM on rare commits. > > > > On Wed, Jul 24, 2013 at 2:56 AM, SolrLover <bbar...@gmail.com> wrote: > >> Currently I am using SOLR 3.5.X and I push updates to SOLR via queue >> (Active >> MQ) and perform hard commit every 30 minutes (since my index is relatively >> big around 30 million documents). I am thinking of using soft commit to >> implement NRT search but I am worried about the reliability. >> >> For ex: If I have the hard autocommit set to 10 minutes and a softcommit >> every second, new documents will show up every second but in case of JVM >> crash or power goes out I will lose all the documents after the last hard >> commit. >> >> I was thinking of using a backup database or another SOLR index that I can >> use as a backup and write the document from queue in both places (one with >> soft commit, another index with just the push updates with normal hard >> commits (or) write simultaneously to a db and delete the rows once the hard >> commit is successful after making sure that we didn't lose any records). >> >> Does someone have any other idea to improve the reliability of the push >> updates when using soft commit? >> >> >> >> >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/How-to-make-soft-commit-more-reliable-tp4079892.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > > > -- > Sincerely yours > Mikhail Khludnev > Principal Engineer, > Grid Dynamics > > <http://www.griddynamics.com> > <mkhlud...@griddynamics.com>
