adiranjan25 opened a new pull request, #18918: URL: https://github.com/apache/iceberg/pull/18918
## Summary Adds Spark 4.2 regression coverage for subset-key `MERGE INTO` operations involving Iceberg tables partitioned by `days(event_date)` and `bucket(2, id)`. Related to #18597. ## Changes - Adds a synthetic MERGE workload joining on `id`, a subset of the target's partition keys. - Exercises Spark Storage Partitioned Join behavior with partially clustered distribution enabled and disabled. - Checks physical-plan partition-group counts and validates the final MERGE results. ## Observations In the reproduced test environment: - **Partial clustering disabled:** 10 target input partitions were grouped into 2 execution groups. - **Partial clustering enabled:** The workload produced 10 execution groups. - Both configurations completed the MERGE correctly. These observations demonstrate partition-concentration behavior. They do not establish an out-of-memory failure or a performance improvement. ## Validation - Spark 4.2 targeted regression tests passed. - Full `TestStoragePartitionedJoinsInRowLevelOperations` test class passed. - Spotless formatting completed successfully. ## Scope This PR adds regression coverage; it does not implement node-local shuffle. The grouping-count assertion does not yet conclusively identify the target scan by physical-plan lineage. Maintainer feedback on strengthening this assertion and on the appropriate implementation approach for #18597 is welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
