shivadarshan-devadiga opened a new pull request, #18995:
URL: https://github.com/apache/iceberg/pull/18995

   Closes #18948.
   
   This keeps the `TestStoragePartitionedJoins` tests that @manuzhang wrote in 
the Spark verification drafts #18340 (Spark 4.0.5), #18339 (Spark 4.1.4) and 
#18294 (Spark 4.2.1) as permanent tests. The tests are taken from the drafts 
unchanged. In `spark/v4.2` the new tests are placed before 
`testJoinsHourToDays` rather than after it, to avoid conflicting with #18990. 
#18383 also adds a test in that area, so whichever of #18383 and this PR merges 
second needs a rebase; I'm happy to do it if this one is second.
   
   Coverage added:
   
   | Coverage | 4.0 | 4.1 | 4.2 |
   |---|---|---|---|
   | Broadcast joins ordered by a computed join-key expression, ascending and 
descending ([SPARK-59905](https://issues.apache.org/jira/browse/SPARK-59905)) | 
✓ | ✓ | ✓ |
   | Mixed-type partition keys with a non-leading subset join key, with partial 
clustering off and on 
([SPARK-60203](https://issues.apache.org/jira/browse/SPARK-60203)); 
compatibility coverage on 4.2 | ✓ | ✓ | ✓ |
   | Runtime filtering with a trailing partition key | ✓ | ✓ | |
   | One-side shuffle with binary partition keys | ✓ | ✓ | |
   | Chained or successive joins whose intermediate output has unknown 
partition keys | ✓ | ✓ | |
   | Duplicate partition-key aliases, and different projected key subsets | | ✓ 
| |
   | Columnar `UNION ALL` preserving key grouping | | ✓ | |
   | Joins over checkpointed Iceberg scans, one or both sides checkpointed | ✓ 
| | |
   
   The aggregate over a partially clustered join is left to #18383, as the 
issue asks.
   
   Testing:
   - The full `TestStoragePartitionedJoins` suite passes on the versions main 
pins: 80 tests on Spark 4.0.5, 84 on 4.1.4 and 72 on 4.2.1, with 0 failures.
   - On the releases before the fixes:
     - The broadcast-ordering tests fail on Spark 4.0.4, 4.1.3 and 4.2.0, with 
the rows in the wrong order (SPARK-59905).
     - `subsetJoinKeysWithMixedPartitionTypes` fails on 4.0.4 and 4.1.3 with 
`IllegalStateException: Unknown type for int field. Type name: 
java.lang.String` (SPARK-60203), and passes on 4.2.0, which was never affected.
     - `partiallyClusteredSubsetJoinKeysWithMixedPartitionTypes` passes on all 
three, so it is compatibility coverage.
   - `spotlessCheck` passes for all three modules.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to