Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-06 Thread Chris Hostetter
: We tried to post the same manually from SOLR ADMIN / documents UI. It got : indexed successfully. We are sure that it's not duplicate issue. We are : using default update handler and doesn't configure for custom one. We fire : the request to index using direct HTTP request using XML : format.

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-06 Thread Ganesh M
Hi Shawn, Good to know about this configuration in shardHandler. We will try this settings and keep you posted on status. Hope setting changes will resolve the issue. Regards, Ganesh On 06-Sep-2016 10:32 pm, "Ganesh M" mailto:ganesh.sudhakar@gmail.com>> wrote: Hi Shawn, Good to know abou

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-06 Thread Ganesh M
Hi Shawn, Good to know about this configuration in shardHandler. We will try this settings and keep you posted on status. Hopefully it resolved the issue. Regards, Ganesh On 05-Sep-2016 10:02 pm, "Shawn Heisey" wrote: > On 9/4/2016 10:02 PM, Ganesh M wrote: > > We have captured all traffic of

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-05 Thread Shawn Heisey
On 9/4/2016 10:02 PM, Ganesh M wrote: > We have captured all traffic of HTTP POST request going out from app I'm the one you've interacted with on IRC for this issue. If this index has multiple shards, one thing that might be a problem here is the ShardHandler that's internal to Solr. This is t

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-05 Thread Alexandre Rafalovitch
On 5 September 2016 at 11:02, Ganesh M wrote: > My big question is why is that SOLR can't throw the error when it's not > able to handle the request due to concurrency or for other reason. Solr SHOULD throw an error if there is an issue. The problem is that the concurrency is a HARD problem with

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-05 Thread Ganesh M
Hi Alex, We have captured all traffic of HTTP POST request going out from app server to SOLR request. Only once that particular document with that id ( in our case it's rowkey ) is going out to SOLR. Also in the SOLR side, we have enabled localhost_access logs and we could see only once that docu

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-05 Thread Ganesh M
Hi Dheerendra, This doesn't always happens. When we add single document, no issue on that. It get's added. But when add in parallel with 50 threads concurrently, out of 2000 documents 10 documents are getting missed ( not getting indexed ). When this is happening, we also tried to do hard commit m

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-04 Thread Alexandre Rafalovitch
I can't tell anything from the document provided. So, here would be my thoughts: If what you see is some sort of concurrency issues, the documents missed/dropped would unlikely be exactly the same ones. So, if you see the same documents dropped, it is much more likely to be something to do with do

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-04 Thread Dheerendra Kulkarni
Can you try this: 1. Add the document 2. Follow up by optimize in the core admin ui, If above works then you may need to check your commit. Regards, Dheerendra On Sun, Sep 4, 2016 at 10:47 PM, Ganesh M wrote: > Hi Alex, > We tried to post the same manually from SOLR ADMIN / documents UI. It g

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-04 Thread Ganesh M
Hi Alex, We tried to post the same manually from SOLR ADMIN / documents UI. It got indexed successfully. We are sure that it's not duplicate issue. We are using default update handler and doesn't configure for custom one. We fire the request to index using direct HTTP request using XML format. W

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-04 Thread Alexandre Rafalovitch
Can you identify the specific documents that 'fail'? What happens if you post them manually? Try posting them manually but with one field super-distinct to see whether it made it in. What happens if you post it to an empty index (copy definition and try). Also, what's your request handler's parame

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-04 Thread Ganesh M
Some more information on this... Most of documents get indexed properly. Few documents are not getting indexed. All documents POST are seen in the localhost_access and 200 OK response is seen in local host access file. But in catalina, there are some difference in the logs for which are indexin

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-04 Thread Ganesh M
Nitin, Thanks for reply. Our each document has unique id and its hbase rowkey id. So it will be unique only. So there is no chance of duplicates id being send. On Sun 4 Sep, 2016 12:41 pm Nitin Kumar, mailto:nitinkumar.i...@gmail.com>> wrote: Please check doc's unique key(Id). All keys shd be

Re: Solr document missing or not getting indexed though we get 200 ok status from server

2016-09-04 Thread Nitin Kumar
Please check doc's unique key(Id). All keys shd be unique. Else docs having same id will be replaced. On 04-Sep-2016 12:13 PM, "Ganesh M" wrote: > Hi, > we are keep sending documents to Solr from our app server. Single document > per request, but in parallel of 10 request hits solr cloud in a se