I forgot to mention, the fields being used in the function query are indexed
fields. They are mostly text fields that cannot have DocValues
-----Original Message-----
From: Webster Homer <webster.ho...@milliporesigma.com>
Sent: Thursday, July 23, 2020 2:07 PM
To: solr-user@lucene.apache.org
Subject: RE: How to measure search performance
Hi Erick,
This is an example of a pseudo field: wdim_pno_:if(gt(query({!edismax
qf=searchmv_pno v=$q}),0),1,0) I get your point that it would only be applied
to the results returned and not to all the results. The intent is to be able to
identify which of the fields matched the search. Our business people are keen
to know, for internal reasons.
I have not done a lot of function queries like this, does using edismax make
this less performant? My tests have a lot of variability but I do see an effect
on the QTime for adding these, but it is hard to quantify. It could be as much
as 10%
Thank you for your quick response.
Webster
-----Original Message-----
From: Erick Erickson <erickerick...@gmail.com>
Sent: Thursday, July 23, 2020 12:52 PM
To: solr-user@lucene.apache.org
Subject: Re: How to measure search performance
This isn’t usually a cause for concern. Clearing the caches doesn’t necessarily
clear the OS caches for instance. I think you’re already aware that Lucene uses
MMapDirectory, meaning the index pages are mapped to OS memory space. Whether
those pages are actually _in_ the OS physical memory or not is anyone’s guess
so depending on when they’re needed they might have to be read from disk. This
is entirely independent of Solr’s caches, and could come into play even if you
restarted Solr.
Then there’s your function queries for the pseudo fields. This is read from the
docValues sections of the index. Once again the relevant parts of the index may
or may not be in the OS memory.
So comparing individual queries is “fraught” with uncertainties. I suppose you
could reboot the machines each time ;) I’ve only ever had luck averaging a
bunch of unique queries when trying to measure perf differences.
Do note that function queries for pseudo fields is not something I’d expect to
add much overhead at all. The reason is that they’re only called for the top N
docs that you’re returning, not part of the search at all. Consider a function
query involved in scoring. That one must be called for every document that
matches. But a function query for a pseudo field is only called for the docs
returned in the packet, i.e. the “rows” parameter.
Best,
Erick
> On Jul 23, 2020, at 11:49 AM, Webster Homer
> <webster.ho...@milliporesigma.com> wrote:
>
> I'm trying to determine the overhead of adding some pseudo fields to one of
> our standard searches. The pseudo fields are simply function queries to
> report if certain fields matched the query or not. I had thought that I could
> run the search without the change and then re-run the searches with the
> fields added.
> I had assumed that the QTime in the query response would be a good metric to
> use when comparing the performance of the two search queries. However I see
> that the QTime for a query can vary by more than 10%. When testing I cleared
> the query cache between tests. Usually the QTime would be within a few
> milliseconds of each other, however in some cases there was a 10X or more
> difference between them.
> Even cached queries vary in their QTime, though much less.
>
> I am running Solr 7.7.2 in a solrcloud configuration with 2 shards and 2
> replicas/shard. Our nodes have 32Gb memory and 16GB of heap allocated to solr.
>
> I am concerned that these discrepancies indicate that our system is not tuned
> well enough.
> Should I expect that a query's QTime really is a measure of the query's
> inherent performance? Is there a better way to measure query performance?
>
>
>
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> you must not copy this message or attachment or disclose the contents to any
> other person. If you have received this transmission in error, please notify
> the sender immediately and delete the message and any attachment from your
> system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not
> accept liability for any omissions or errors in this message which may arise
> as a result of E-Mail-transmission or for damages resulting from any
> unauthorized changes of the content of this message and any attachment
> thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not
> guarantee that this message is free of viruses and does not accept liability
> for any damages caused by any virus transmitted therewith.
>
>
>
> Click http://www.merckgroup.com/disclaimer to access the German, French,
> Spanish and Portuguese versions of this disclaimer.
This message and any attachment are confidential and may be privileged or
otherwise protected from disclosure. If you are not the intended recipient, you
must not copy this message or attachment or disclose the contents to any other
person. If you have received this transmission in error, please notify the
sender immediately and delete the message and any attachment from your system.
Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept
liability for any omissions or errors in this message which may arise as a
result of E-Mail-transmission or for damages resulting from any unauthorized
changes of the content of this message and any attachment thereto. Merck KGaA,
Darmstadt, Germany and any of its subsidiaries do not guarantee that this
message is free of viruses and does not accept liability for any damages caused
by any virus transmitted therewith.
Click http://www.merckgroup.com/disclaimer to access the German, French,
Spanish and Portuguese versions of this disclaimer.
This message and any attachment are confidential and may be privileged or
otherwise protected from disclosure. If you are not the intended recipient, you
must not copy this message or attachment or disclose the contents to any other
person. If you have received this transmission in error, please notify the
sender immediately and delete the message and any attachment from your system.
Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept
liability for any omissions or errors in this message which may arise as a
result of E-Mail-transmission or for damages resulting from any unauthorized
changes of the content of this message and any attachment thereto. Merck KGaA,
Darmstadt, Germany and any of its subsidiaries do not guarantee that this
message is free of viruses and does not accept liability for any damages caused
by any virus transmitted therewith.
Click http://www.merckgroup.com/disclaimer to access the German, French,
Spanish and Portuguese versions of this disclaimer.
