walterddr opened a new pull request #8176: URL: https://github.com/apache/pinot/pull/8176
this is a follow up with several previous attempt to fix segment file deletion. Goal === Add support to control segment retention period on a per-table basis. Challenge === in previous Approaches - we previously attempt to add field in TableConfig that controls the retention period. this causes many corner case problems 1. when table is deleted, so is the tableConfig, thus we need to store the retention period override somewhere. 2. when retention period changed for a table config (e.g. say from 3 days to 7 days) this means we need to rescanned all the previously deleted segments based on the new retention, however files that are older than 3 days but newer than 7 days would've already been deleted. Therefore it is semantically hard to do a post-deletion retention override. New Approach === In this PR we avoid these issues entirely by employing one practice: segments retentions are set when they are being deleted based on the tableConfig override or the default cluster setting AT THE TIME OF DELETION. no retro editing of the override will be supported for any deleted segments. This means if there's any new changes to the cluster level or table level override, they will only apply to newly deleted segments and will not affect any segments that are previously deleted. See: #8078 and #8069 for more details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
