Re: SOLR-11504: Provide a config to restrict number of indexing threads

2017-11-02 Thread Michael McCandless
Actually, it's one lucene segment per *concurrent* indexing thread. So if you have 10 indexing threads in Lucene at once, then 10 in-memory segments will be created and will have to be written on refresh/commit. Elasticsearch uses a bounded thread pool to service all indexing requests, which I th

Re: SOLR-11504: Provide a config to restrict number of indexing threads

2017-11-02 Thread Emir Arnautović
Hi Nawab, > One indexing thread in lucene corresponds to one segment being written. I > need a fine control on the number of segments. I didn’t check the code, but I would be surprised that it is how things work. It can appear that it is working like that if each client thread is doing commit

Re: SOLR-11504: Provide a config to restrict number of indexing threads

2017-11-01 Thread Nawab Zada Asad Iqbal
Well, the reason i want to control number of indexing threads is to restrict number of "segments" being created at one time in the RAM. One indexing thread in lucene corresponds to one segment being written. I need a fine control on the number of segments. Less than that, and I will not be fully u

Re: SOLR-11504: Provide a config to restrict number of indexing threads

2017-11-01 Thread Shawn Heisey
On 10/31/2017 4:57 PM, Nawab Zada Asad Iqbal wrote: I hit this issue https://issues.apache.org/jira/browse/SOLR-11504 while migrating to solr6 and locally working around it in Lucene code. I am thinking to fix it properly and hopefully patch back to Solr. Since, Lucene code does not want to keep