I went through other queries for which we are getting `The request took too
long to iterate over doc values` warning. As pointed by Erick I have cross
check all the fields that are being used in query and there is no such field
against which we are searching and it as index=false and docValues=true.
Few observations I would like to share here:
- We are performing a load test on our system and the above timeout warning
is occurring for only those queries which are fetching a large number of
documents.
- I had stopped all the load on the system and fired same queries (for which
we were getting timeout warning). Here is solr response:
Solr Response:
response: {
numFound: 6082251,
start: 0,
maxScore: 4709.594,
docs: [ ]
}
The response was quite weird (header is saying there are `6082251` docs
found but `docs` array is empty) also there was no timeout warning in logs.
Then I increased `timeAllowed` to 5000ms (default is 1000ms). This time
`docs` array was not empty and in fact there was an increase in numFound
count. This clearly points that query was not able to complete in 1000ms
(default timeAllowed).
I have following question:
1. Is doc value is as effiecient as ExternalFileField for functional query?
2. Why I got warning message when system was under load but no when there
was no laod?
When we were performing load test (load scale is same) with
ExternalFileField type were not getting any warning messages in our logs.
raj.yadav wrote
> Hey Erick,
>
> In cases for which we are getting this warning, I'm not able to extract
> the
> `exact solr query`. Instead logger is logging `parsedquery ` for such
> cases.
> Here is one example:
>
> ````
> 2020-09-29 13:09:41.279 WARN (qtp926837661-82461) [c:mycollection
> s:shard1_0 r:core_node5 x:mycollection_shard1_0_replica_n3]
> o.a.s.s.SolrIndexSearcher Query: [+FunctionScoreQuery(+*:*, scored by
> boost(product(if(max(const(0),
> sub(float(my_doc_value_field1),const(500))),const(0.01),
>
> if(max(const(0),sub(float(my_doc_value_field2),const(290))),const(0.2),const(1))),
>
> sqrt(product(sum(const(1),float(my_doc_value_field3),float(my_doc_value_field4)),
> sqrt(sum(const(1),float(my_doc_value_field5))))))))
> #BitSetDocTopFilter]; The request took too long to iterate over doc
> values.
> Timeout: timeoutAt: 1635297585120522 (System.nanoTime():
> 1635297690311384),
> DocValues=org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$8@7df12bf1
>
> ````
>
> As per my understanding query in the above case is `q=*:*`. And then there
> is boost function which uses functional query on my_doc_value_field*
> (fieldtype doc_value_field i.e having index=false and docValue = true) to
> reorder matched docs. If docValue works efficiently for _function queries_
> then why this warning are coming?
>
>
> Also, we do use frange queries on doc_value_field (having index=false and
> docValue = true).
> example:
> {!frange l=1.0}my_doc_value_field1
>
>
> Erick Erickson wrote
>> Let’s see the query. My bet is that you are _searching_ against the field
>> and have indexed=false.
>>
>> Searching against a docValues=true indexed=false field results in the
>> equivalent of a “table scan” in the RDBMS world. You may use
>> the docValues efficiently for _function queries_ to mimic some
>> search behavior.
>>
>> Best,
>> Erick
>
>
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
