Increasing the number of rows should not have this kind of impact in either version of Solr, so I think there’s something fundamentally strange in your setup.
Whether returning 10 or 300 documents, every document has to be scored. There are two differences between 10 and 300 rows: 1> when returning 10 rows, Solr keeps a sorted list of 10 doc, just IDs and score (assuming you’re sorting by relevance), when returning 300 the list is 300 long. I find it hard to believe that keeping a list 300 items long is making that much of a difference. 2> Solr needs to fetch/decompress/assemble 300 documents .vs. 10 documents for the response. Regardless of the fields returned, the entire document will be decompresses if you return any fields that are not docValues=true. So it’s possible that what you’re seeing is related. Try adding, as Alexandre suggests, &debug to the query. Pay particular attention to the “timings” section too, that’ll show you the time each component took _exclusive_ of step <2> above and should give a clue. All that said, fq clauses don’t score, so scoring is certainly involved in why the query takes so long to return even 10 rows but gets faster when you move the clause to a filter query, but my intuition is that there’s something else going on as well to account for the difference when you return 300 rows. Best, Erick > On Sep 29, 2020, at 8:52 PM, Alexandre Rafalovitch <arafa...@gmail.com> wrote: > > What do the debug versions of the query show between two versions? > > One thing that changed is sow (split on whitespace) parameter among > many. It is unlikely to be the cause, but I am mentioning just in > case. > https://lucene.apache.org/solr/guide/8_6/the-standard-query-parser.html#standard-query-parser-parameters > > Regards, > Alex > > On Tue, 29 Sep 2020 at 20:47, Permakoff, Vadim > <vadim.permak...@verisk.com> wrote: >> >> Hi Solr Experts! >> We are moving from Solr 6.5.1 to Solr 8.5.0 and having a problem with long >> query, which has a search text plus many OR and AND conditions (all in one >> place, the query is about 20KB long). >> For the same set of data (about 500K docs) and the same schema the query in >> Solr 6 return results in less than 2 sec, Solr 8 takes more than 10 sec to >> get 10 results. If I increase the number of rows to 300, in Solr 6 it takes >> about 10 sec, in Solr 8 it takes more than 1 min. The results are small, >> just IDs. It looks like the relevancy scoring plays role, because if I move >> this query to filter query - both Solr versions work pretty fast. >> The right way should be to change the query, but unfortunately it is >> difficult to modify the application which creates these queries, so I want >> to find some temporary workaround. >> >> What was changed from Solr 6 to Solr 8 in terms of scoring with many >> conditions, which affects the search speed negatively? >> Is there anything to configure in Solr 8 to get the same performance for >> such query like it was in Solr 6? >> >> Thank you, >> Vadim >> >> ________________________________ >> >> This email is intended solely for the recipient. It may contain privileged, >> proprietary or confidential information or material. If you are not the >> intended recipient, please delete this email and any attachments and notify >> the sender of the error.
