On 5/11/2018 9:26 AM, Andy C wrote:
> Why are range searches more efficient than wildcard searches? I guess I
> would have expected that they just provide different mechanism for defining
> the range of unique terms that are of interest, and that the merge
> processing would be identical.
I hope I can explain the reason that wildcard queries tend to be slow.
I will use an example field from one of my own indexes.
Choosing one of the shards of my main index, and focusing on the
"keywords" field for that Solr core: Here's the histogram data that the
Luke handler gives for this field:
"histogram":[
"1",14095268,
"2",767777,
"4",425610,
"8",312156,
"16",236743,
"32",177718,
"64",122603,
"128",80513,
"256",52746,
"512",34925,
"1024",24770,
"2048",17516,
"4096",11467,
"8192",7748,
"16384",5210,
"32768",3433,
"65536",2164,
"131072",1280,
"262144",688,
"524288",355,
"1048576",163,
"2097152",53,
"4194304",12]}},
The first entry means that there are 14 million terms that only appear
once in the keywords field across the whole index. The last entry means
that there are twelve terms that appear 4 million times in the keywords
field across the whole index.
Adding this all up, I can see that there are a little more than 16
million unique terms in this field.
This means that when I do a "keywords:*" query, that Solr/Lucene will
expand this query such that the query literally contains 16 million
individual terms. It's going to take time just to make the query. And
then that query will have to be executed. No matter how quickly each
term in the query executes, doing 16 million of them is going to be slow.
Just for giggles, I used my dev server to execute that "keywords:*"
query on this single shard. The reported QTime in the response was
18017 milliseconds. Then I ran the full range query. The reported
QTime for that was 14569 milliseconds. Which is honestly slower than I
thought it would be, but faster than the wildcard. The number of unique
terms in the field affects both kinds of queries, but the effect of a
large number of terms on the wildcard is usually greater than the effect
on the range.
> Would a search such as:
>
> field:c*
>
> be more efficient if rewritten as:
>
> field:[c TO d}
On most indexes, probably. That would depend on the number of terms in
the field, I think. But there's something to consider: Not every
wildcard query can be easily rewritten as a range. I think this one is
impossible to rewrite as a range: field:abc*xyz
I tried your c* example as well on my keywords field. The wildcard had
a QTime of 1702 milliseconds. The range query had a QTime of 1434
milliseconds. The numFound on both queries was identical, at 16399711.
Thanks,
Shawn
