jimczi commented on PR #16847: URL: https://github.com/apache/lucene/pull/16847#issuecomment-6290546941
I did another pass on merges and merge policies. A segment still uses an int for its doc ids. So a single merge can never go over `MAX_DOCS`, even if the whole index now holds more. Tiered already capped its merges for that. I made `LogMergePolicy` do the same, so it leaves a few more segments instead of overflowing. Soft deletes count here too, and I missed at first that the count can be an approximation. The policy estimates how many deletes a merge will reclaim, but a merge can keep more than that. So the guard looks at the docs the merge really writes, not the estimate. When the retention only moves forward this stays on the safe side, since the retained set only shrinks by merge time. I also added a guard in `IndexWriter`. It fails the merge early with a clear message if a policy still picks one that is too big. And `BaseMergePolicyTestCase` now checks this, so a custom merge policy catches it in its own tests. @iverase thanks for testing this, it pushed me to go through a few more cases while I was here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
