github-actions[bot] commented on code in PR #70516:
URL: https://github.com/apache/doris/pull/70516#discussion_r4655992340
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPushDownVisitor.java:
##########
@@ -308,6 +318,25 @@ && canPushThroughJoinChild(join, false, ctx)) {
}
if (newTarget != null && newTarget.getInputSlots().size() == 1
&& !newTarget.equals(ctx.srcExpr)) {
+ if (expandByOuterJoin) {
Review Comment:
[P1] Stop expanded runtime filters at sensitive window expressions
`visitPhysicalWindow` pushes through whenever the probe slot is a common
partition key, without checking the window expressions or order keys. The new
outer-join branch can therefore filter the window input before a side-effecting
expression runs:
```text
InnerJoin(L.k = E.k)
LeftOuterJoin(L.k = R.k)
Scan L
Window(COUNT(assert_true(R.v > 10, 'bad')) OVER (PARTITION BY R.k))
Scan R
Scan E={2}
```
With `L={2}`, `E={2}`, and `R={(1,10),(2,20)}`, the expanded RF removes the
key-1 partition before the window evaluates `assert_true` and suppresses `bad`;
without expansion the window evaluates it and fails. Add a volatile/NoneMovable
barrier for window expressions (or conservatively stop this traversal) and a
regression for the non-preserved window path.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPushDownVisitor.java:
##########
@@ -308,6 +318,25 @@ && canPushThroughJoinChild(join, false, ctx)) {
}
if (newTarget != null && newTarget.getInputSlots().size() == 1
&& !newTarget.equals(ctx.srcExpr)) {
+ if (expandByOuterJoin) {
+ Plan preserved = join.getJoinType().isLeftOuterJoin()
? leftNode : rightNode;
+ Plan nonPreserved =
join.getJoinType().isLeftOuterJoin() ? rightNode : leftNode;
Review Comment:
[P1] Stop expanded runtime filters at sensitive sorts
The new outer-join branch can recurse through a retained
`PhysicalQuickSort`, but the visitor has no sort barrier:
```text
InnerJoin(L.k = E.k)
LeftOuterJoin(L.k = R.k)
Scan L
QuickSort(order by assert_true(R.v > 10, 'bad'))
Scan R
Scan E={2}
```
With `L={2}`, `E={2}`, and `R={(1,10),(2,20)}`, the expanded RF removes key
1 before the order expression runs and the query succeeds; without expansion
the sort raises `bad`. A derived-table sort can be retained when sort
elimination is disabled. Add a QuickSort barrier for volatile/NoneMovable order
keys (or conservatively fence sort traversal) and a regression with sort
elimination disabled.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPushDownVisitor.java:
##########
@@ -293,11 +297,17 @@ && canPushThroughJoinChild(join, false, ctx)) {
pushed |= rightNode.accept(this, ctx);
}
- // Expand through join conditions at non-builder inner/semi joins
+ // Outer joins only propagate a parent's ordinary join RF from the
preserved side.
+ // Decoupled RFs do not carry the parent equality here, so leave them
out of this expansion.
+ boolean expandByOuterJoin = !join.isMarkJoin()
Review Comment:
[P1] Stop expanded runtime filters at sensitive aggregates
The new outer-join branch can recurse through a `PhysicalHashAggregate` when
the probe is its direct group key, but the visitor has no aggregate barrier:
```text
InnerJoin(L.k = E.k)
LeftOuterJoin(L.k = R.k)
Scan L
Project(R.k, c)
HashAggregate(group by R.k,
COUNT(assert_true(R.v > 10, 'bad')) AS c)
Scan R
Scan E={2}
```
With `L={2}`, `E={2}`, and `R={(1,10),(2,20)}`, the expanded RF removes the
key-1 input before the aggregate argument runs and the query succeeds; without
expansion it raises `bad`. Add a volatile/NoneMovable aggregate barrier (or
conservatively stop traversal through such aggregates) and a regression for
this non-preserved aggregate path.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPushDownVisitor.java:
##########
@@ -293,11 +297,17 @@ && canPushThroughJoinChild(join, false, ctx)) {
pushed |= rightNode.accept(this, ctx);
}
- // Expand through join conditions at non-builder inner/semi joins
+ // Outer joins only propagate a parent's ordinary join RF from the
preserved side.
+ // Decoupled RFs do not carry the parent equality here, so leave them
out of this expansion.
+ boolean expandByOuterJoin = !join.isMarkJoin()
+ && ctx.exprOrder >= 0 && ctx.builderNode instanceof
PhysicalHashJoin
+ && (join.getJoinType().isLeftOuterJoin() ||
join.getJoinType().isRightOuterJoin());
+ // Expand through join conditions at non-builder joins.
if (!join.equals(ctx.builderNode)
&& ConnectContext.get() != null
&&
ConnectContext.get().getSessionVariable().expandRuntimeFilterByInnerJoin
- && (join.getJoinType() == JoinType.INNER_JOIN ||
join.getJoinType().isSemiJoin())) {
+ && (join.getJoinType() == JoinType.INNER_JOIN ||
join.getJoinType().isSemiJoin()
+ || expandByOuterJoin)) {
Review Comment:
[P1] Stop expanded runtime filters at volatile/non-movable filters
The new outer-join branch derives `R.k` and calls the generic visitor on the
non-preserved subtree, but `PhysicalFilter` has no barrier here
(`PlanVisitor.visitPhysicalFilter` recurses to its child). For example:
```text
InnerJoin(L.k = E.k)
LeftOuterJoin(L.k = R.k)
Scan L
Filter(assert_true(R.v > 10, 'bad'))
Scan R
Scan E={2}
```
With `L={2}`, `E={2}`, and `R={(1,10),(2,20)}`, expansion attaches the RF
for `E.k` to `R.k`, so the `k=1` row is removed before `assert_true` and the
query succeeds. Without expansion, the filter evaluates that row and raises
`bad`. Add a `containsVolatileOrNoneMovableExpression()` filter barrier and a
regression for this filtered non-preserved path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]