huaxingao commented on code in PR #18726:
URL: https://github.com/apache/iceberg/pull/18726#discussion_r4528253392


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadMetrics.java:
##########
@@ -341,4 +344,58 @@ public void testDeleteMetrics() throws 
NoSuchTableException {
         .hasEntrySatisfying(
             "skippedDeleteFiles", sqlMetric -> 
assertThat(sqlMetric.value()).isEqualTo(0));
   }
+
+  @TestTemplate
+  public void numSplitsWhenTaskReadsSeveralPartitions() {
+    sql(
+        "CREATE TABLE %s (id BIGINT, dep STRING, region STRING) USING iceberg "
+            + "PARTITIONED BY (dep, region)",
+        tableName);
+    sql(
+        "INSERT INTO %s VALUES (1, 'hr', 'us'), (2, 'hr', 'eu'), (3, 'hr', 
'apac'), "
+            + "(4, 'ops', 'us'), (5, 'ops', 'eu')",
+        tableName);
+
+    // joining on a subset of the partition keys makes Spark group the 3 hr 
and the 2 ops
+    // partitions into one task each, so every task opens several readers
+    Map<String, String> spjConf =
+        ImmutableMap.of(
+            SQLConf.V2_BUCKETING_ENABLED().key(),
+            "true",
+            
SQLConf.V2_BUCKETING_ALLOW_JOIN_KEYS_SUBSET_OF_PARTITION_KEYS().key(),
+            "true",
+            SQLConf.REQUIRE_ALL_CLUSTER_KEYS_FOR_CO_PARTITION().key(),
+            "false",
+            SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(),
+            "false",
+            SQLConf.AUTO_BROADCASTJOIN_THRESHOLD().key(),
+            "-1",
+            SparkSQLProperties.PRESERVE_DATA_GROUPING,
+            "true");
+
+    for (String vectorized : new String[] {"true", "false"}) {
+      sql(
+          "ALTER TABLE %s SET TBLPROPERTIES ('%s'='%s')",
+          tableName, TableProperties.PARQUET_VECTORIZATION_ENABLED, 
vectorized);
+
+      withSQLConf(
+          spjConf,
+          () -> {
+            Dataset<Row> df =
+                spark.sql(
+                    String.format(
+                        "SELECT * FROM %s a JOIN %s b ON a.dep = b.dep", 
tableName, tableName));
+            df.collect();
+
+            List<SparkPlan> scans =
+                
seqAsJavaListConverter(df.queryExecution().executedPlan().collectLeaves()).asJava();
+            assertThat(scans).hasSize(2);
+            for (SparkPlan scan : scans) {
+              assertThat(scan.metrics().get("numSplits").get().value())
+                  .as("numSplits with vectorization %s", vectorized)
+                  .isEqualTo(5L);

Review Comment:
   This only checks `numSplits`. The fix also changes `numDeletes`. Could this 
test also delete a few rows with merge-on-read, like `testDeleteMetrics` does, 
and check that `numDeletes` counts deletes from all the partitions in the task?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to