msfroh commented on issue #15412: URL: https://github.com/apache/lucene/issues/15412#issuecomment-3514338121
Hmm... that is a neat idea! It's almost like a "`LogTimeRangeMergePolicy`". I could definitely see some value in a timestamp-aware merge policy that merges older stuff into bigger segments while grouping newer stuff into smaller segments. Another possible approach could involve rolling over and combining indexes. This approach is (I think) more common on OpenSearch/Elasticsearch. Basically, every day you roll over to a new index and the old index becomes read-only. While you might keep the daily indexes for the past seven days, maybe you combine the 7 days before that into a weekly index (and the 7 days before that, and the 7 days before that). Maybe combine some of your weekly indexes into monthly indexes. In other words, solve the problem at the index level instead of at the segment level. (Again, I also like the idea of solving it at the segment level -- I'm just pitching alternatives in case they're helpful.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
