aokolnychyi commented on code in PR #9447:
URL: https://github.com/apache/iceberg/pull/9447#discussion_r1468199919
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##########
@@ -118,4 +126,15 @@ private InternalRow[] toOutputRows(RewriteManifests.Result
result) {
public String description() {
return "RewriteManifestsProcedure";
}
+
+ private RewriteManifestsSparkAction checkAndApplyFilter(
+ RewriteManifestsSparkAction action, String where, Identifier ident) {
+ if (where != null) {
+ Expression expression = filterExpression(ident, where);
Review Comment:
I agree that there are some cases where it could be beneficial, but like you
said, it would only work in some special cases (e.g. time-based partitioning
and writing data only to new partitions and never to old ones). However, I am
not sure how generic this is. If we add this new argument, it may not work in a
lot of cases, causing confusion.
What do you think about using the underlying action directly for your needs?
There, you can pass a closure and have sophisticated filtering logic even
beyond partition bounds (e.g. you can check snapshot IDs).
> By the way, In V1 table with snapshot-id inherit=true, it still need write
all manifests in driver when execute copyManifests function, I find that this
propertity is no longer affects V2 table, Is any plan to remove or default true
this propertity in the V1 table, or is there still any unsafe?
I highly recommend enabling that property in all tables if you know that it
will not brake some old writers/readers. We can't make that guarantee on behalf
of the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
