Re: [PR] Spark 3.5: Parallelize reading files in snapshot and migrate procedures [iceberg]

[via GitHub]

RussellSpitzer commented on code in PR #10037:
URL: https://github.com/apache/iceberg/pull/10037#discussion_r1559706583


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java:
##########
@@ -232,4 +237,42 @@ public void testMigrateEmptyTable() throws Exception {
     Object result = scalarSql("CALL %s.system.migrate('%s')", catalogName, 
tableName);
     assertThat(result).isEqualTo(0L);
   }
+
+  @TestTemplate
+  public void testMigrateWithParallelism() throws IOException {
+    assumeThat(catalogName).isEqualToIgnoringCase("spark_catalog");
+
+    testWithParallelism(-1);
+    testWithParallelism(0);
+    testWithParallelism(1);
+    testWithParallelism(5);
+  }
+
+  private void testWithParallelism(int parallelism) throws IOException {
+    String location = Files.createTempDirectory(temp, 
"junit").toFile().toString();
+    sql(
+        "CREATE TABLE %s (id bigint NOT NULL, data string) USING parquet 
LOCATION '%s'",
+        tableName, location);
+    sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
+    sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
+
+    try (MockedStatic<MoreExecutors> executors = 
mockStatic(MoreExecutors.class)) {
+      ProcedureUtil.TestExecutorService testService = new 
ProcedureUtil.TestExecutorService();
+      executors
+          .when(() -> 
MoreExecutors.getExitingExecutorService(any(ThreadPoolExecutor.class)))
+          .thenReturn(testService);
+
+      Object result =
+          scalarSql(
+              "CALL %s.system.migrate(table => '%s', parallelism => %d)",
+              catalogName, tableName, parallelism);
+      assertThat(result).as("Should have added two files").isEqualTo(2L);
+
+      assertThat(testService.getExecutedTasks()).isEqualTo(parallelism > 1 ? 2 
: 0);

Review Comment:
   This is a very complicated way to check whether the class was used or not.
   
   This is also generally not what we want to do, we should essentially have
   ```
   usesSingleThreaded() {
     // Asserts that the test is not using a custom thread pool
   }
   
   usesMultiThreaded() {
      Asserts using thread pool
      Asserts thread pool size
   }
   
   invalidParallelism() {
     assertThrows()
   }
   
   Then tests are
   
     usesSingleThreaded(1)
     usesMultiThreaded(2)
     usesMultiThreaded(10)
     invalidParallelism(-3)
     }



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org