msokolov commented on PR #13542: URL: https://github.com/apache/lucene/pull/13542#issuecomment-2214811101
I wonder if we should tackle the issue with caching / cloning scorers? We have scorers/scorerSuppliers that do a lot of up-front work when created and we don't want to duplicate that work when two threads search the same segment. OTOH two threads cannot share the same Scorer instance since it has internal state namely the iterator it advances, and maybe more. This seems like the biggest problem to tackle - maybe it would make sense to start there? OTOH we need some kind of execution framework (this PR) that we can use to exercise such changes - I guess it's a kind of leapfrogging problem, gotta start somewhere. Given that, maybe we create a long-running branch we can use to iterate on this in multiple steps before merging to main? As for the Scorer-cloning I poked at it and it seems tricky. I think that Weight can cache Scorer and/or ScorerSupplier, and I guess BulkScorer. Then when asked for a scorer-type object that it already has it can create a shallowCopy(). Shallow-copying should be implemented so as to do the least amount of work to create a new instance that can be iterated independently. This will be up to each Scorer/ScorerSupplier/BulkScorer to figure out -- maybe some of them just return null and let Weight create a new one as it does today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
