aokolnychyi commented on PR #5760: URL: https://github.com/apache/iceberg/pull/5760#issuecomment-1536714175
@chenjunjiedada, one more clarification to make sure I understand. Flink upsert produces position files to dedup records within the same batch and those position deletes prevent from compaction, right? The issue is not related to concurrency, every rewrite data files would fail afterwards? Can we solve this by adapting `validateNoNewDeletesForDataFiles` in `RewriteFiles` to not conflict when we detect a conflicting position delete file sequence number (different from data sequence number) is the same as the rewritten data file sequence number? This would leverage file sequence numbers added recently, not data sequence numbers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
