Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-11-20 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1848650926 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -971,4 +979,109 @@ public int hashCode() { return Objects.hashCode

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-11-20 Thread via GitHub
RussellSpitzer merged PR #11157: URL: https://github.com/apache/iceberg/pull/11157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-11-19 Thread via GitHub
manuzhang commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1849635195 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -273,4 +273,22 @@ public void testMigrateWith

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-11-19 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1848654925 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -971,4 +979,109 @@ public int hashCode() { return Objects.hashCode

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-11-19 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1848675267 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -273,4 +273,22 @@ public void testMigrat

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-11-19 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1848647496 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -971,4 +979,109 @@ public int hashCode() { return Objects.hashCode

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-11-18 Thread via GitHub
manuzhang commented on PR #11157: URL: https://github.com/apache/iceberg/pull/11157#issuecomment-2484823743 @RussellSpitzer Can we move this PR forward? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-30 Thread via GitHub
manuzhang commented on PR #11157: URL: https://github.com/apache/iceberg/pull/11157#issuecomment-2447432767 @RussellSpitzer I've reverted to lazy executor service. Please check again. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on PR #11157: URL: https://github.com/apache/iceberg/pull/11157#issuecomment-2444918297 Main main reason was that the API that is specified in the API Module allows withExecutorService(executor service) so unless we want to break that api (which we could) we need to

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-28 Thread via GitHub
manuzhang commented on PR #11157: URL: https://github.com/apache/iceberg/pull/11157#issuecomment-2443260806 @RussellSpitzer I can revert to previous commit and this is Spark specific, but can you elaborate on why `LazyExecutorService` is better than`SerializableSupplier`? I agree with you t

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-28 Thread via GitHub
RussellSpitzer commented on PR #11157: URL: https://github.com/apache/iceberg/pull/11157#issuecomment-2442221029 After thinking about this for a while, I think you are probably right that we need to build a specific LazyExecutorService like you did originally. I'm sorry I lend you on a goos

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-28 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1819436344 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -711,7 +719,7 @@ public static void importSparkPartitions( spec,

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-28 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1819435386 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -711,7 +719,7 @@ public static void importSparkPartitions( spec,

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-28 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1819368101 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -711,7 +719,7 @@ public static void importSparkPartitions( spec,

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-26 Thread via GitHub
manuzhang commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1817893959 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -711,7 +719,7 @@ public static void importSparkPartitions( spec,

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-26 Thread via GitHub
manuzhang commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1817894453 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -711,7 +719,7 @@ public static void importSparkPartitions( spec,

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-26 Thread via GitHub
manuzhang commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1817893959 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -711,7 +719,7 @@ public static void importSparkPartitions( spec,

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-26 Thread via GitHub
manuzhang commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1817876302 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -92,6 +98,8 @@ import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-25 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1817300773 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -711,7 +719,7 @@ public static void importSparkPartitions( spec,

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-25 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1817289632 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -92,6 +98,8 @@ import org.apache.spark.sql.catalyst.plans.logical.Logica

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-10-25 Thread via GitHub
RussellSpitzer commented on code in PR #11157: URL: https://github.com/apache/iceberg/pull/11157#discussion_r1817288548 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -92,6 +98,8 @@ import org.apache.spark.sql.catalyst.plans.logical.Logica