yashmayya opened a new pull request, #18997:
URL: https://github.com/apache/pinot/pull/18997
Follow-up to #18804. Adds test coverage verifying that **colocated
operations can be chained without a data shuffle**: when a chain of joins /
window functions / set operations is all keyed on the same partition column and
the colocation hints are applied, every inter-stage hash exchange in the plan
is pre-partitioned (direct, 1-to-1) and no full shuffle is inserted anywhere in
the chain.
This exercises the interaction of the three colocation hints:
- `joinOptions(is_colocated_by_join_keys='true')`
- `windowOptions(is_partitioned_by_window_keys='true')`
- `setOpOptions(is_colocated_by_set_op_keys='true')`
**Tests added (test-only, no production changes):**
- `QueryCompilationTest`: a planner-level invariant check
(`testColocatedOperationsChainWithoutShuffle`) that asserts every
hash-distributed exchange in the plan is pre-partitioned, across four chains:
- colocated join → window
- set op of two colocated joins
- set op of two colocated windows
- the triple: colocated join → window → set op
Plus a baseline (`testChainWithoutColocationHintsStillShuffles`)
confirming the same chain *does* shuffle without the hints, so the assertions
are not vacuous.
- `QueryHints.json`: two H2-validated runtime cases (a colocated join →
window, and a `UNION ALL` of two colocated joins) on tables physically
partitioned by the key, confirming the chained colocation also produces correct
results on both the V1 and V2 engines.
**Hint placement note:** because a set operation is an ancestor of its
branch `SELECT`s, chaining through a set op uses the outer-`SELECT`-wrap form
for the set-op hint with the join/window hints on each branch — this is what
lets colocation propagate through every level of the chain.
**Note:** this PR is stacked on #18804 (which introduces the `setOpOptions`
hint the set-op chains depend on). Its diff includes that commit until #18804
merges — the commit to review here is the top one (`Add tests for chaining
colocated joins, window functions and set operations without shuffle`); I'll
rebase once #18804 lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
