Dandandan opened a new pull request, #24794:
URL: https://github.com/apache/datafusion/pull/24794
## Which issue does this PR close?
- Closes #.
## Rationale for this change
A `LeftSemi` hash join emits nothing while probing.
`adjust_indices_by_join_type`
returns empty index arrays for it, and every matched build row comes out at
the
end from the visited bitmap, the same way `LeftAnti` and `LeftMark` work.
`HashJoinExec::compute_properties` nevertheless reports
`EmissionType::Incremental` for `LeftSemi`, grouped with `Inner` and
`RightSemi` under the comment "If we only need to generate matched rows from
the probe side". That is not what `LeftSemi` does: its rows come from the
build
side. `LeftAnti` and `LeftMark` are correctly classified `Both`.
So the join is blocking, and a `LIMIT` above it buys nothing. On TPC-DS SF1:
```sql
select ws_order_number from web_sales ws
where exists (select 1 from web_returns wr
where wr.wr_order_number = ws.ws_order_number)
limit 10
```
| | |
|---|---|
| plans as `LeftSemi`, with `LIMIT 10` | 9.2 ms |
| the same query with no `LIMIT` at all | 8.6 ms |
| mirrored so it plans as `RightSemi`, with `LIMIT 10` | 2.8 ms |
## What changes are included in this PR?
A semi join only needs to know whether a build row has matched yet, and the
visited bitmap already carries that. This emits a build row when its bit
flips
from unset to set, which produces the same rows in the same order but while
probing rather than after it. The final stage then has nothing left to do.
The comment at that site already described the fix: "When visit the right
batch, we can output the matched left row and don't need to wait the end of
loop".
The query above now takes 2.7 ms, matching the `RightSemi` form, and
`EmissionType::Incremental` becomes true rather than aspirational.
Only the hash join changes. `NestedLoopJoinExec` still emits its `LeftSemi`
rows at the end, so `need_produce_result_in_final` is untouched.
## Are these changes tested?
Yes, by the existing coverage: the full sqllogictest suite (504 files) and
the
1117 join unit tests pass. Semi joins are heavily covered by `joins.slt`,
`subquery.slt` and the hash join's own tests, including across partition
modes
and batch sizes.
One expectation changed, in `push_down_filter_parquet.slt`. It is an `EXPLAIN
ANALYZE` whose join reports `input_batches=1, input_rows=2` where it used to
report `2` and `4`; `output_rows` is 2 either way. The join no longer holds
its
output back, so the consumer finishes and the probe scan stops earlier.
## Are there any user-facing changes?
`LeftSemi` hash joins are no longer blocking, so a `LIMIT` or any other
early-terminating consumer above one can now stop it early. Results are
unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
