shivadarshan-devadiga opened a new pull request, #18990:
URL: https://github.com/apache/iceberg/pull/18990

   Closes #18952.
   
   [SPARK-60044](https://github.com/apache/spark/pull/59509) freezes the 
ordering of a planned `GroupPartitionsExec` k-way merge. Before it, turning 
`spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` off after 
planning emptied the ordering the merge uses, so a parent that relied on the 
merge read the rows out of order. This adds an Iceberg-backed version of the 
upstream regression to `TestStoragePartitionedJoins` in `spark/v4.2`, as #18294 
left it untested.
   
   The new test, `testPlannedSortedMergeKeepsOrderingAfterConfigChange`:
   - creates a table partitioned by `(id, days(arrive_time), name)`, with one 
row per partition. The scan reports no ordering, so Spark derives `[id, name]` 
from the partition keys. The splits are sorted by the full key, so within `id = 
1` the `bb` split comes before the `aa` one.
   - plans `ROW_NUMBER() OVER (PARTITION BY id ORDER BY name)` with SPJ, 
`partitionKeyOrdering` and `preserveOrderingOnCoalesce` on, and checks that the 
window's ordering comes from a k-way merge (`SortedMerge`) and not a sort.
   - turns `partitionKeyOrdering` off, runs the already planned `Dataset`, and 
checks that the rows of `id = 1` are numbered `aa = 1`, `bb = 2`.
   - runs with AQE off and on, as the upstream test does.
   
   It is added to Spark 4.2 only: Spark 4.0 and 4.1 have no 
`GroupPartitionsExec` k-way merge, which SPARK-55715 added in 4.2.
   
   Testing:
   - The new test and the rest of `TestStoragePartitionedJoins` pass on Spark 
4.2.1, the version on main.
   - Against Spark 4.2.1 RC2 (staging repository `orgapachespark-1537`, the 
last build without SPARK-60044), Spark plans the k-way merge but the new test 
fails in both AQE modes: the rows of `id = 1` are numbered in split order 
(`expected: "aa" but was: "bb"`).
   - On Spark 4.2.0, Spark doesn't plan a k-way merge for this Iceberg scan at 
all and adds a `Sort` instead, with vectorized and row-based reads, so the bug 
can't be reached there and the test fails at its plan check.
   - `spotlessCheck` passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to