Re: [PR] Core, Spark: Scan only live entries in RewriteTablePathUtil [iceberg]

2025-01-24 Thread via GitHub
flyrain commented on PR #12006: URL: https://github.com/apache/iceberg/pull/12006#issuecomment-2613545458 > > Yes, thanks for fixing the issue (found by our internal usage). > > I wonder, because the deleted entry may be important for CDC (to mark that this file at some point existed), is

Re: [PR] Core, Spark: Scan only live entries in RewriteTablePathUtil [iceberg]

2025-01-24 Thread via GitHub
dramaticlly commented on code in PR #12006: URL: https://github.com/apache/iceberg/pull/12006#discussion_r1928946351 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -710,7 +710,7 @@ private boolean fileExist(String path)

Re: [PR] Core, Spark: Scan only live entries in RewriteTablePathUtil [iceberg]

2025-01-24 Thread via GitHub
dramaticlly commented on code in PR #12006: URL: https://github.com/apache/iceberg/pull/12006#discussion_r1928932547 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -923,16 +1005,20 @@ protected void checkFileNum(

Re: [PR] Core, Spark: Scan only live entries in RewriteTablePathUtil [iceberg]

2025-01-23 Thread via GitHub
szehon-ho commented on code in PR #12006: URL: https://github.com/apache/iceberg/pull/12006#discussion_r1928243065 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -923,16 +1005,20 @@ protected void checkFileNum(

Re: [PR] Core, Spark: Scan only live entries in RewriteTablePathUtil [iceberg]

2025-01-23 Thread via GitHub
dramaticlly commented on PR #12006: URL: https://github.com/apache/iceberg/pull/12006#issuecomment-2611724920 > Yes, thanks for fixing the issue (found by our internal usage). > > I wonder, because the deleted entry may be important for CDC (to mark that this file at some point existe

Re: [PR] Core, Spark: Scan only live entries in RewriteTablePathUtil [iceberg]

2025-01-20 Thread via GitHub
szehon-ho commented on PR #12006: URL: https://github.com/apache/iceberg/pull/12006#issuecomment-2603148129 Yes, thanks for fixing the issue (found by our internal usage). I wonder, because the deleted entry may be important for CDC (to mark that this file at some point existed), is a