[
https://issues.apache.org/jira/browse/SOLR-13807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057947#comment-17057947
]
Michael Gibney commented on SOLR-13807:
---------------------------------------
Still working at [PR #751|https://github.com/apache/lucene-solr/pull/751], I
separated the earlier monolithic commit (and recent incremental adjustments)
into two logical commits: first introducing single-sweep collection of facet
term counts across different DocSet domains, second introducing a term facet
count cache. (N.b. tests are passing for each commit, but validation is failing
based on intentional nocommits).
[~hossman], I have yet to expand on the test stub you introduced, but if you
don't think it's premature to take a second look at this now that the two
features have been separated into logical commits, I'd appreciate any feedback
you have to offer. I was reluctant to force-push, and wasn't sure whether to
open new PRs or work with the existing one; but I left the old (monolithic +
test-stub-patch + iterative-adjustments) available
[here|https://github.com/magibney/lucene-solr/tree/SOLR-13132-mingled-sweep-and-cache],
and figured the new 2-commit push would clarify things and be a good jumping
off point for however we want to proceed (whether new PRs, etc...).
I know I had said I would make single-sweep collection dependent on facet
cache, but (as you can see) I went the opposite way. Functionality-wise,
facet-cache would have made sense first, but code/structure-wise, sweep-first
was much cleaner and clearer.
> Caching for term facet counts
> -----------------------------
>
> Key: SOLR-13807
> URL: https://issues.apache.org/jira/browse/SOLR-13807
> Project: Solr
> Issue Type: New Feature
> Components: Facet Module
> Affects Versions: master (9.0), 8.2
> Reporter: Michael Gibney
> Priority: Minor
> Attachments: SOLR-13807__SOLR-13132_test_stub.patch
>
>
> Solr does not have a facet count cache; so for _every_ request, term facets
> are recalculated for _every_ (facet) field, by iterating over _every_ field
> value for _every_ doc in the result domain, and incrementing the associated
> count.
> As a result, subsequent requests end up redoing a lot of the same work,
> including all associated object allocation, GC, etc. This situation could
> benefit from integrated caching.
> Because of the domain-based, serial/iterative nature of term facet
> calculation, latency is proportional to the size of the result domain.
> Consequently, one common/clear manifestation of this issue is high latency
> for faceting over an unrestricted domain (e.g., {{\*:\*}}), as might be
> observed on a top-level landing page that exposes facets. This type of
> "static" case is often mitigated by external (to Solr) caching, either with a
> caching layer between Solr and a front-end application, or within a front-end
> application, or even with a caching layer between the end user and a
> front-end application.
> But in addition to the overhead of handling this caching elsewhere in the
> stack (or, for a new user, even being aware of this as a potential issue to
> mitigate), any external caching mitigation is really only appropriate for
> relatively static cases like the "landing page" example described above. A
> Solr-internal facet count cache (analogous to the {{filterCache}}) would
> provide the following additional benefits:
> # ease of use/out-of-the-box configuration to address a common performance
> concern
> # compact (specifically caching count arrays, without the extra baggage that
> accompanies a naive external caching approach)
> # NRT-friendly (could be implemented to be segment-aware)
> # modular, capable of reusing the same cached values in conjunction with
> variant requests over the same result domain (this would support common use
> cases like paging, but also potentially more interesting direct uses of
> facets).
> # could be used for distributed refinement (i.e., if facet counts over a
> given domain are cached, a refinement request could simply look up the
> ordinal value for each enumerated term and directly grab the count out of the
> count array that was cached during the first phase of facet calculation)
> # composable (e.g., in aggregate functions that calculate values based on
> facet counts across different domains, like SKG/relatedness – see SOLR-13132)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]