walterddr edited a comment on issue #8092: URL: https://github.com/apache/pinot/issues/8092#issuecomment-1036477050
some caveat for our current implementation 1. table deletion requires a "sync" deletion on all segments on deep store before table config were deleted. this means if there's any failure before the table is completely deleted, error might've been returned and it is up to the client to handle and reissue deletion. 2. segment deletion however requires us to first delete the property stores and asynchronously delete the segment files on deep store. since the return is suppose to be quick. Ideally speaking what we needed here decouple the deep store deletion from the table/segment deletion. -> for table/segment deletion, as long as their ideal state is being removed from ZK. we consider them to be deleted. -> for deep store data, utilizing RetentionManager to do the clean up. however, this causes problems if table / segments with the exact same name is re-created (for example if one were to replace a corrupted segment with a newly ingested one). So challenge here is resolve operations similar to this one that are ultimately requires a "sync" deletion across zk and deep store. solution 1: have some sort of tombstone mechanism to indicate that the table/segment is marked as deleted, but do not recreate the same identifier again until the clean up has been completely done. solution 2: make versioning on table/segment that always increment version number when a new table / segment is created. solution 3: ??? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
