Re: Commit after every document - alternate approach

2016-03-04 Thread Shawn Heisey
On 3/3/2016 11:36 PM, sangs8788 wrote: > When a commit fails, the document doesnt get cleared out from MQ and there is > a task which runs in a background to republish the files to SOLR. If we do a > batch commit we will not know we will end up redoing the same batch commit > again. We currenlty ha

Re: Commit after every document - alternate approach

2016-03-04 Thread Emir Arnautovic
.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261575.html Sent from the Solr - User mailing list archive at Nabble.com. -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/

Re: Commit after every document - alternate approach

2016-03-03 Thread sangs8788
this message in context: http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261587.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-03 Thread Walter Underwood
xt: > http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261575.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-03 Thread Walter Underwood
om MQ. > > Thanks > Sangeetha > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261575.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-03 Thread sangs8788
context: http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261576.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-03 Thread sangs8788
failure the document can be resent from MQ. Thanks Sangeetha -- View this message in context: http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261575.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-02 Thread Varun Thacker
Hi Sangeetha, Well I don't think you need to commit after every document add. You can rely on Solr's transaction log feature . If you are using SolrCloud it's mandatory to have a transaction log . So every documents get written to the tlog . Now say a node crashes even if documents were not commi

Re: Commit after every document - alternate approach

2016-03-02 Thread Emir Arnautovic
Hi Sangeetha, What is sure is that it is not going to work - with 200-300K doc/hour, there will be >50 commits/second, meaning there are <20ms time for doc+commit. You can do is let Solr handle commits and maybe use real time get to verify doc is in Solr or do some periodic sanity checks. Are y

Commit after every document - alternate approach

2016-03-02 Thread sangeetha.subraman...@gtnexus.com
Hi All, I am trying to understand on how we can have commit issued to solr while indexing documents. Around 200K to 300K document/per hour with an avg size of 10 KB size each will be getting into SOLR . JAVA code fetches the document from MQ and streamlines it to SOLR. The problem is the client