RussellSpitzer commented on code in PR #13907:
URL: https://github.com/apache/iceberg/pull/13907#discussion_r2317167636
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java:
##########
@@ -133,7 +133,47 @@ public void removeTables() {
sql("DROP TABLE IF EXISTS %s", tableName(OTHER_TABLE_NAME));
}
- // TODO: add tests for truncate transforms once SPARK-40295 is released
+ // TODO: SPJ does not currently leverage truncate(...) partition transforms
for partition
+ // alignment.
+ // SPARK-40295 improved related areas, but full truncate support is tracked
in SPARK-50593.
+ // This test documents current behavior; update/extend once SPARK-50593
lands.
+ @TestTemplate
+ public void testJoinWithTruncatePartitioning() {
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING) "
+ + "USING iceberg "
+ + "PARTITIONED BY (truncate(4, dep)) "
+ + "TBLPROPERTIES (%s)",
+ tableName, tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName);
+ sql("INSERT INTO %s VALUES (2L, 200, 'software')", tableName);
+ sql("INSERT INTO %s VALUES (3L, 300, 'software')", tableName);
+
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING) "
+ + "USING iceberg "
+ + "PARTITIONED BY (truncate(4, dep)) "
+ + "TBLPROPERTIES (%s)",
+ tableName(OTHER_TABLE_NAME), tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')",
tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (2L, 200, 'software')",
tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (3L, 300, 'software')",
tableName(OTHER_TABLE_NAME));
+
+ // TODO(SPARK-50593): Once truncate transforms are leveraged by SPJ,
expected shuffles with SPJ
Review Comment:
I'm not sure what we gain from the tests prior to 50593 going in. I would
probably hold off on this unless we think we are getting some other coverage
here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
