Hello, First of all, I don't think it can commit (even soft) every second, afaik it's too frequent for typical deployment. Hence, if you really need such (N)RT I suggest you experiment with it right now, to face the bummer sooner. Also, one second durability sounds like over-expectation for Solr, it sounds like OLTP requirements. Then, now Solr has some sort of pre-indexing record storage called UpdateLog. try to experiment with syncLevel = FSYNC vs FLUSH . That's how it works, when document arrives for indexing it's written into update log, which is plain binary file. Indexing works as-is relying on RAMbuffer. When node dies, RAMbuffer dies, but updateLog is persistent, during startup Solr recovers uncommitted updates from updateLog. Caveat! UpdateLog has HashMap internally which easily hits OOM on rare commits.
On Wed, Jul 24, 2013 at 2:56 AM, SolrLover <bbar...@gmail.com> wrote: > Currently I am using SOLR 3.5.X and I push updates to SOLR via queue > (Active > MQ) and perform hard commit every 30 minutes (since my index is relatively > big around 30 million documents). I am thinking of using soft commit to > implement NRT search but I am worried about the reliability. > > For ex: If I have the hard autocommit set to 10 minutes and a softcommit > every second, new documents will show up every second but in case of JVM > crash or power goes out I will lose all the documents after the last hard > commit. > > I was thinking of using a backup database or another SOLR index that I can > use as a backup and write the document from queue in both places (one with > soft commit, another index with just the push updates with normal hard > commits (or) write simultaneously to a db and delete the rows once the hard > commit is successful after making sure that we didn't lose any records). > > Does someone have any other idea to improve the reliability of the push > updates when using soft commit? > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/How-to-make-soft-commit-more-reliable-tp4079892.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
