jbewing commented on code in PR #13534:
URL: https://github.com/apache/iceberg/pull/13534#discussion_r2205067882
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java:
##########
@@ -370,6 +370,54 @@ public void testJoinsWithCompatibleSpecEvolution() {
tableName(OTHER_TABLE_NAME));
}
+ @TestTemplate
+ public void testJoinsWhichReorderPartitionsViaSpecEvolution() {
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (bucket(8, id), int_col, dep) "
+ + "TBLPROPERTIES (%s)",
+ tableName, tablePropsAsString(TABLE_PROPERTIES));
+
+ Table table = validationCatalog.loadTable(tableIdent);
+
+ // evolve the spec in the first table by reordering the final two
partition columns by first
+ // removing them and re-adding them
+ table.updateSpec().removeField("int_col").commit();
+ table.updateSpec().removeField("dep").commit();
+
+ table.updateSpec().addField("dep").commit();
+ table.updateSpec().addField("int_col").commit();
Review Comment:
Then SPJ would be disabled as the grouping key for table1 would be reported
as `(bucket(8, id))` instead of `(bucket(8, id), int_col, dep)` given that only
`bucket(8, id)` is an active partition spec. I can solidify this behavior with
a unit test in this class if you'd like, but generally the behavior for
[TestPartitioning](https://github.com/apache/iceberg/blob/bcf9c69c098b54d31cbd803d62a2609d3814c3df/core/src/test/java/org/apache/iceberg/TestPartitioning.java#L230-L241)
applies to how the grouping spec is calculated. The only real change
introduced in this PR being that the ordering of keys within the grouping spec
can to follow the ordering of the most recent active partition spec of the
table first rather than ascending fieldId order (e.g. following the order at
which partitions were originally added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
