Radu Gheorghe created SOLR-15008:
------------------------------------
Summary: Avoid building OrdinalMap for each facet
Key: SOLR-15008
URL: https://issues.apache.org/jira/browse/SOLR-15008
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: Facet Module
Affects Versions: 8.7
Reporter: Radu Gheorghe
Attachments: Screenshot 2020-11-19 at 12.01.55.png
I'm running against the following scenario:
* [JSON] faceting on a high cardinality field
* few matching documents => few unique values
Yet the query almost always takes a long time. Here's an example taking almost
4s for ~300 documents and unique values (edited a bit):
{code:java}
"QTime":3869,
"params":{
"json":"{\"query\": \"*:*\",
\"filter\": [\"type:test_type\", \"date:[1603670360 TO 1604361599]\",
\"unique_id:49866\"]
\"facet\":
{\"keywords\":{\"type\":\"terms\",\"field\":\"keywords\",\"limit\":20,\"mincount\":20}}}",
"rows":"0"}},
"response":{"numFound":333,"start":0,"maxScore":1.0,"numFoundExact":true,"docs":[]
},
"facets":{
"count":333,
"keywords":{
"buckets":[{
"val":"value1",
"count":124},
...
{code}
I did some [profiling with our Sematext
Monitoring|https://sematext.com/docs/monitoring/on-demand-profiling/] and it
points me to OrdinalMap building. If I read the code right, an OrdinalMap is
built with every facet. And it's expensive since there are many unique values
in the shard (previously, there we more smaller shards, making latency better,
but this approach doesn't scale for this particular use-case).
If I'm right up to this point, I see a couple of potential improvements,
[inspired from
Elasticsearch|[https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-execution-hint]:]
# Keep the OrdinalMap cached until the next softCommit, so that only the first
query takes the penalty
# Allow faceting on actual values (a Map) rather than ordinals, for situations
like the one above where we have few matching documents. We could potentially
auto-detect this scenario (e.g. by configuring a threshold) and use a Map when
there are few documents
I'm curious about what you're thinking:
* would a PR/patch be welcome for any of the two ideas above?
* do you see better options? am I missing something?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
