[ 
https://issues.apache.org/jira/browse/SOLR-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris M. Hostetter updated SOLR-14137:
--------------------------------------
    Attachment: hoss_experiement1.tgz
        Status: Open  (was: Open)

So.... i actually started trying to reproduce Erick's observations yesterday 
based on what little info i could find in the ticket/thread about his 
methodology – before he posted his github repo.

I'm attaching a tgz containing all the scripts, timing logs and graphs produced 
by my experiments (just not the raw 'docs.csv' generated by the scripts, or the 
resulting indexes)

My testing was based on:
 * comparing 8.4.1 to 7.7.2
 * using _default configsets for each
 * single core indexes of 1 million docs
 ** (because i didn't want to sit around waiting to build a 20 million doc 
index on my laptop)
 * text field built using a the "word list" of 2000 words
 ** and each doc had 50 random words.
 * date field with unique values for each doc, starting with "NOW" and going 
backword 0-120seconds using random milliscond deltas

My initial queries were "date boosted" disjunction queries of 10 words (from 
the word list) using rows=0.

I compared the timing _from stopwatches run by the query client_ of running 
1000 queries in a single thread to running 1000 queries each in 16 concurrent 
threads, and didn't see any obvious and/or massive impacts of concurrent 
queries in how the query times compared between 7x and 8x – What I did see was 
that the mean time of the 7x queries was ~80-85% of the mean time of the 8x 
queries. (FWIW: erick's "second-run.png" seems to show 77% relative mean 
response time comaring 7.7.1 to 8.3.1)

Based on those observations, I stopped worrying about doing heavy concurrent 
threaded query testing, and focused on single thread query testing since it let 
me iterate faster.

As far as i can tell, date boosting isn't a significant factor in the relateive 
speed of 7x vs 8x queries – date boosted queries slower are then w/o a date 
boost, but that relative difference in query time doesn't seem to depend on the 
solr version?

*It seems like in general disjunctions may have gotten slower?*

1000 date boosted "match all docs" queries in 7x has a mean request time of 96% 
of the same queries in 8x – which seems close enough to suggest it may just be 
noise, or something related to jetty level HTTP changes between version...

meanwhile 1000 single term queries _w/o any date boosting_ on 7x clocks in at 
80% the mean request time of those same queries in 8x.

1000 date boosted queries containing 100 disjunction clauses (where ~33% aren't 
in the word list and won't match any docs) results in 7x having a mean request 
time of 82% of the same queries in 8x .... that changes to only 84% when we 
remove the date boost – suggesting that date boosting isn't a significant 
factor here in relative speed of queries between 7x and 8x.
----
Obviously index size (both in terms of number of docs and number of terms in 
the corpus) is going to be a big factor, but I think the most important 
question to start with is...

[~erickerickson]: did you do any testing of just basic 1, 10, or 100 clause 
disjunction queries _w/o any date boosting_ to see how basic query performance 
compared between these versions of solr?

> Boosting by date (and perhaps others) shows a steady decline 6.6->8.3
> ---------------------------------------------------------------------
>
>                 Key: SOLR-14137
>                 URL: https://issues.apache.org/jira/browse/SOLR-14137
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Erick Erickson
>            Priority: Major
>         Attachments: Screen Shot 2019-12-19 at 2.35.41 PM.png, Screen Shot 
> 2019-12-19 at 3.09.37 PM.png, Screen Shot 2019-12-19 at 3.31.16 PM.png, 
> hoss_experiement1.tgz, second_run.png
>
>
> Moving a user's list discussion over here.
> {color:#000000}The very short form is that from Solr 6.6.1 to Solr 8.3.1, the 
> throughput for date boosting in my tests dropped by 40+%{color}
> {color:#000000}I’ve been hearing about slowdowns in successive Solr releases 
> with boost functions, so I dug into it a bit. The test setup is just a 
> boost-by-date with an additional big OR clause of 100 random words so I’d be 
> sure to hit a bunch of docs. I figured that if there were few hits, the 
> signal would be lost in the noise, but I didn’t look at the actual hit 
> counts.{color}
> {color:#000000}I saw several Solr JIRAs about this subject, but they were 
> slightly different, although quite possibly the same underlying issue. So I 
> tried to get this down to a very specific form of a query.{color}
> {color:#000000}I’ve also seen some cases in the wild where the response was 
> proportional to the number of segments, thus my optimize experiments.{color}
> {color:#000000}Here are the results, explanation below. O stands for 
> optimized to one segment. I spot checked pdate against 6.6, 7.1 and 8.3 and 
> they weren’t significantly different performance wise from tdate. All have 
> docValues enabled. I ran these against a multiValued=“false” field. All the 
> tests pegged all my CPUs. Jmeter is being run on a different machine than 
> Solr. Only one Solr was running for any test.{color}
> {color:#000000}Solr version   queries/min   {color}
> {color:#000000}6.6.1              3,400          {color}
> {color:#000000}6.6.1 O           4,800          {color}
> {color:#000000}7.1                 2,800           {color}
> {color:#000000}7.1 O             4,200           {color}
> {color:#000000}7.7.1              2,400           {color}
> {color:#000000}7.7.1 O          3,500            {color}
> {color:#000000}8.3.1             2,000            {color}
> {color:#000000}8.3.1 O          2,600            {color}
> {color:#000000}The tests I’ve been running just index 20M docs into a single 
> core, then run the exact same 10,000 queries against them from jmeter with 24 
> threads. Spot checks showed no hits on the queryResultCache.{color}
> {color:#000000}A query looks like this: {color}
> {color:#000000}rows=0&\{!boost b=recip(ms(NOW, 
> INSERT_FIELD_HERE),3.16e-11,1,1)}text_txt:(campaigners OR adjourned OR 
> anyplace…97 more random words){color}
> {color:#000000}There is no faceting. No grouping. No sorting.{color}
> {color:#000000}I fill in INSERT_FIELD_HERE through jmeter magic. I’m running 
> the exact same queries for every test.{color}
> {color:#000000}One wildcard is that I did regenerate the index for each major 
> revision, and the chose random words from the same list of words, as well as 
> random times (bounded in the same range though) so the docs are not 
> completely identical. The index was in the native format for that major 
> version even if slightly different between versions. I ran the test once, 
> then ran it again after optimizing the index.{color}
> {color:#000000}I haven’t dug any farther, if anyone’s interested I can throw 
> a profiler at, say, 8.3 and see what I can see, although I’m not going to 
> have time to dive into this any time soon. I’d be glad to run some tests 
> though. I saved the queries and the indexes so running a test would  only 
> take a few minutes.{color}
> {color:#000000}While I concentrated on date fields, the docs have date, int, 
> and long fields, both docValues=true and docValues=false, each variant with 
> multiValued=true and multiValued=false and both Trie and Point (where 
> possible) variants as well as a pretty simple text field.{color}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to