On Mon, Aug 19, 2024 at 2:32 PM Michael Gibney <mich...@michaelgibney.net> wrote: > For a more robust solution than fussing with G1HeapRegionSize, I'm > wondering if it might be appropriate to change the implementation of > BitDocSet so that larger instances will be backed by an array of > multiple smaller FixedBitSet instances. This would introduce some > extra complexity for DocSets over large indexes, but it shouldn't be > terrible; it could be cleanly implemented and would ensure that we > never allocate humongous objects in the service of BitDocSets ...
IMO I don't think this is quite the right direction to go in. Instead, we should prioritize a FixedBitSet per segment -- basically a segment level filterCache. This exists at the Lucene level (IndexSearcher QueryCache); we could stop disabling it. It'd go with the new multiThreaded query stuff wonderfully. But would then want to do changes in BitDocSet so that it becomes an aggregate of FixedBitSets sourced from that cache. At least this is a strawman proposal; needs more thought! Note that Yonik resisted ending Solr's top level FixedBitSet on the grounds that benchmarks are needed to show that a segment approach is better. But that's challenging -- we will find the status quo is faster for some situations and a segment level is faster for others. Shrug. ~ David --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
