ankitsultana opened a new pull request, #15561:
URL: https://github.com/apache/pinot/pull/15561
**Note:** This does NOT touch any reachable code path so expected to be a
safe change. We are building the physical optimizer behind a flag to begin with
since the changes involved are significant.
### Summary
I am about done with the first version of the Physical Optimizer change.
While getting it running I found some bugs so this PR fixes some of them. These
are summarized below.
##### Support Mapping Source to Multiple Targets in PinotDistMapping
Calcite quite commonly generates projects, where the same input reference
exists multiple times. It looks something as follows. If we only allow a source
to map to at most 1 target, we were dropping distribution descriptors which was
leading to some unnecessary shuffles in some sample queries as shown below.
Hence, I have changed PinotDistMapping to instead be of the form
`Map<Integer, List<Integer>>`. Moreover, while computing new hash descriptors
for a node, I now compute all possible mappings. For instance, if a node is
partitioned on `$1` and we have a project such as the one show below, then the
Project node is partitioned by both `col2` and `col21`.
```
"\n PhysicalProject(col1=[$0], col2=[$1],
col21=[$1])",
```
Example query where this feature helps in getting rid of even more shuffles:
```
SET usePhysicalOptimizer=true; EXPLAIN PLAN FOR SELECT col1, col2 FROM a
WHERE col2 IN (SELECT col2 FROM a WHERE col3 = 'foo') AND col2 NOT IN (SELECT
col2 FROM a WHERE col3 = 'bar') AND col2 IN (SELECT col2 FROM a WHERE col3 =
'lorem')
```
##### Remove Semi-Join Dynamic Filtering Temporarily
This will be handled through a separate rule. To keep our scope low, I have
removed support for it for now.
##### Switch to Transformers instead of Rule Executors at Top Level
The `RelToPRelConverter` should execute transformers, which are the more
generic type, rather than rule executors.
##### Trait Assignment Update
We should first set the right distribution. If it is broadcast, then we
don't need to set any trait on the left and we can leave it Random (we'll
change this to Any later. see: F3 in the tracker)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
