shivadarshan-devadiga opened a new pull request, #18995: URL: https://github.com/apache/iceberg/pull/18995
Closes #18948. This keeps the `TestStoragePartitionedJoins` tests that @manuzhang wrote in the Spark verification drafts #18340 (Spark 4.0.5), #18339 (Spark 4.1.4) and #18294 (Spark 4.2.1) as permanent tests. The tests are taken from the drafts unchanged. In `spark/v4.2` the new tests are placed before `testJoinsHourToDays` rather than after it, to avoid conflicting with #18990. #18383 also adds a test in that area, so whichever of #18383 and this PR merges second needs a rebase; I'm happy to do it if this one is second. Coverage added: | Coverage | 4.0 | 4.1 | 4.2 | |---|---|---|---| | Broadcast joins ordered by a computed join-key expression, ascending and descending ([SPARK-59905](https://issues.apache.org/jira/browse/SPARK-59905)) | ✓ | ✓ | ✓ | | Mixed-type partition keys with a non-leading subset join key, with partial clustering off and on ([SPARK-60203](https://issues.apache.org/jira/browse/SPARK-60203)); compatibility coverage on 4.2 | ✓ | ✓ | ✓ | | Runtime filtering with a trailing partition key | ✓ | ✓ | | | One-side shuffle with binary partition keys | ✓ | ✓ | | | Chained or successive joins whose intermediate output has unknown partition keys | ✓ | ✓ | | | Duplicate partition-key aliases, and different projected key subsets | | ✓ | | | Columnar `UNION ALL` preserving key grouping | | ✓ | | | Joins over checkpointed Iceberg scans, one or both sides checkpointed | ✓ | | | The aggregate over a partially clustered join is left to #18383, as the issue asks. Testing: - The full `TestStoragePartitionedJoins` suite passes on the versions main pins: 80 tests on Spark 4.0.5, 84 on 4.1.4 and 72 on 4.2.1, with 0 failures. - On the releases before the fixes: - The broadcast-ordering tests fail on Spark 4.0.4, 4.1.3 and 4.2.0, with the rows in the wrong order (SPARK-59905). - `subsetJoinKeysWithMixedPartitionTypes` fails on 4.0.4 and 4.1.3 with `IllegalStateException: Unknown type for int field. Type name: java.lang.String` (SPARK-60203), and passes on 4.2.0, which was never affected. - `partiallyClusteredSubsetJoinKeysWithMixedPartitionTypes` passes on all three, so it is compatibility coverage. - `spotlessCheck` passes for all three modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
