amogh-jahagirdar commented on PR #13555: URL: https://github.com/apache/iceberg/pull/13555#issuecomment-3079340267
I was talking to @aokolnychyi earlier and he brought up a promising approach of using a special identifier that's used internally in the spark compaction implementation, when loading a table for a compaction (i.e. something like `rewrite_<task_group_uuid>". Then we could have some handling similar to what we do for [time travel ](https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java#L884). The really nice thing about this approach is we don't need extensions when running compaction on v3+ tables. I'll give this a go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
