Re: Slower indexing speed in Solr 8.0.0

2019-04-03 Thread Zheng Lin Edwin Yeo
Hi David, Yes, I do have this field "_root_" in the schema. However, I don't think I have use the field, and there is no difference in the indexing speed after I remove the field. Regards, Edwin On Wed, 3 Apr 2019 at 22:57, David Smiley wrote: > Hi Edwin, > > I'd like to rule something o

Re: Slower indexing speed in Solr 8.0.0

2019-04-03 Thread David Smiley
Hi Edwin, I'd like to rule something out. Does your schema define a field "_root_"? If you don't have nested documents then remove it. It's presence adds indexing weight in 8.0 that was not there previously. I'm not sure how much though; I've hoped small but who knows. ~ David Smiley Apache Lu

Re: Slower indexing speed in Solr 8.0.0

2019-04-03 Thread David Smiley
What/where is this benchmark? I recall once Ishan was working with a volunteer to set up something like Lucene has but sadly it was not successful On Wed, Apr 3, 2019 at 6:04 AM Đạt Cao Mạnh wrote: > Hi guys, > > I'm seeing the same problems with Shalin nightly indexing benchmark. This > happen

Re: Slower indexing speed in Solr 8.0.0

2019-04-03 Thread Toke Eskildsen
On Wed, 2019-04-03 at 18:04 +0800, Zheng Lin Edwin Yeo wrote: > I have tried to set all the docValues in my schema.xml to false and > do the indexing again. > There isn't any difference with the indexing speed as compared to > when we have enabled the docValues. Thank you for sparing me the work.

Re: Slower indexing speed in Solr 8.0.0

2019-04-03 Thread Zheng Lin Edwin Yeo
Hi Toke, I have tried to set all the docValues in my schema.xml to false and do the indexing again. There isn't any difference with the indexing speed as compared to when we have enabled the docValues. Seems like the cause of the regression might be somewhere else? Regards, Edwin On Wed, 3 Apr

Re: Slower indexing speed in Solr 8.0.0

2019-04-03 Thread Đạt Cao Mạnh
Hi guys, I'm seeing the same problems with Shalin nightly indexing benchmark. This happen around this period git log --before=2018-12-07 --after=2018-11-21 On Wed, Apr 3, 2019 at 8:45 AM Toke Eskildsen wrote: > On Wed, 2019-04-03 at 15:24 +0800, Zheng Lin Edwin Yeo wrote: > > Yes, I am using Do

Re: Slower indexing speed in Solr 8.0.0

2019-04-03 Thread Toke Eskildsen
On Wed, 2019-04-03 at 15:24 +0800, Zheng Lin Edwin Yeo wrote: > Yes, I am using DocValues for most of my fields. So that's a culprit. Thank you. > Currently we can't share the test data yet as some of the records are > sensitive. Do you have any data from CSV file that you can test? Not really.

Re: Slower indexing speed in Solr 8.0.0

2019-04-03 Thread Zheng Lin Edwin Yeo
Yes, I am using DocValues for most of my fields. I am using dynamicField, in which I have appended the field name with things like _s, _i, etc in the CSV file. Currently we can't share the test data yet as some of the recor

Re: Slower indexing speed in Solr 8.0.0

2019-04-02 Thread Toke Eskildsen
On Wed, 2019-04-03 at 10:17 +0800, Zheng Lin Edwin Yeo wrote: > What could be the reason that causes the indexing to be slower in > Solr 8.0.0? As Aroop states there can be multiple explanations. One of them is the change to how DocValues are handled in 8.0.0. The indexing impact should be tiny, b

Re: Slower indexing speed in Solr 8.0.0

2019-04-02 Thread Zheng Lin Edwin Yeo
I'm using external zookeeper, running on Solr Cloud with one shards and two replicas. This is a testing setup, so there is only one machine. The input data is coming from CSV file. I am indexing one CSV file at a time, and each CSV file contains 3 million records. I'm indexing using the code from t

Re: Slower indexing speed in Solr 8.0.0

2019-04-02 Thread Aroop Ganguly
Indexing speeds are function of a lot of variables in my experience. What is your setup like? What kind of cluster you have, the number of shards you created, the number of machines etc? Where is your input data coming from? What technology do you use to indexing (simple java threads or somethi

Re: Slower indexing speed in Solr 8.0.0

2019-04-02 Thread Zheng Lin Edwin Yeo
For additional info, I am still using the same version of the major components like ZooKeeper, Tika, Carrot2 and Jetty. Regards, Edwin On Wed, 3 Apr 2019 at 10:17, Zheng Lin Edwin Yeo wrote: > Hi, > > I am setting up the latest Solr 8.0.0, and I am re-indexing the data from > scratch in Solr 8.

Slower indexing speed in Solr 8.0.0

2019-04-02 Thread Zheng Lin Edwin Yeo
Hi, I am setting up the latest Solr 8.0.0, and I am re-indexing the data from scratch in Solr 8.0.0 However, I found that the indexing speed is slower in Solr 8.0.0, as compared to the earlier version like Solr 7.7.1. I have not changed the schema.xml and solrconfig.xml yet, just did a change of