englefly commented on code in PR #66117:
URL: https://github.com/apache/doris/pull/66117#discussion_r3679755993
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/eageraggregation/EagerAggRewriter.java:
##########
@@ -83,13 +86,20 @@
* ->T2(D)
*/
public class EagerAggRewriter extends DefaultPlanRewriter<PushDownAggContext> {
- public static final int BIG_JOIN_BUILD_SIZE = 400_000;
+ public static final int BIG_JOIN_BUILD_SIZE = 1_000_000;
private static final double LOWER_AGGREGATE_EFFECT_COEFFICIENT = 10000;
private static final double LOW_AGGREGATE_EFFECT_COEFFICIENT = 1000;
private static final double MEDIUM_AGGREGATE_EFFECT_COEFFICIENT = 100;
+ private static final double HIGH_AGGREGATE_EFFECT_COEFFICIENT = 10;
+ private static final double SMALL_BROADCAST_REJECT_COEFFICIENT = 1000;
private static final String JOIN_CNT = "joinCnt";
private final StatsDerive derive = new StatsDerive(false);
+ @Override
+ public Plan visit(Plan plan, PushDownAggContext context) {
+ return plan;
Review Comment:
如果遇到非预期的node 类型,是不是可以考虑尝试在这个非预期的node上生成一个agg
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/eageraggregation/EagerAggRewriter.java:
##########
@@ -309,17 +345,25 @@ private Pair<Boolean, Boolean>
adjustPushSideForNullable(LogicalJoin<? extends P
return Pair.of(toLeft, toRight);
}
- private boolean isPassThroughBigJoin(LogicalJoin<? extends Plan, ? extends
Plan> join,
- PushDownAggContext context) {
- if (context.isPassThroughBigJoin()) {
- return true;
- } else {
- Statistics stats = join.right().getStats();
- if (stats == null) {
- stats = join.right().accept(derive, new
StatsDerive.DeriveContext());
- }
- return stats.getRowCount() > BIG_JOIN_BUILD_SIZE ||
SessionVariable.getEagerAggregationMode() > 0;
+ private boolean isSmallBroadcastJoin(LogicalJoin<? extends Plan, ? extends
Plan> join,
+ ConnectContext context) {
+ if (!JoinUtils.couldBroadcast(join)) {
+ return false;
}
+ SessionVariable sessionVariable = context.getSessionVariable();
+ Statistics stats = join.right().getStats();
+ if (stats == null) {
+ stats = join.right().accept(derive, new
StatsDerive.DeriveContext());
+ }
+ return stats.getRowCount() <=
sessionVariable.getBroadcastRowCountLimit()
+ && stats.getRowCount() <=
sessionVariable.eagerAggBroadcastRowCount;
+ }
+
+ private boolean isBottomJoin(LogicalJoin<? extends Plan, ? extends Plan>
join) {
Review Comment:
join.child(0) = filter(having clause)->agg->join2
这样的join 算bottomJoin吗
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/eageraggregation/EagerAggRewriter.java:
##########
@@ -105,6 +115,15 @@ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ?
extends Plan> join, P
return join;
}
}
+ ConnectContext connectContext =
context.getCascadesContext().getConnectContext();
+ boolean isSmallBroadcastBottomJoin = isSmallBroadcastJoin(join,
connectContext) && isBottomJoin(join);
+ if (context.isPassThroughJoinOrUnion() &&
connectContext.getSessionVariable().eagerAggregationOnBroadcastJoin
+ && isSmallBroadcastBottomJoin &&
!outputStringType(join.right())) {
Review Comment:
!outputStringType(join.right())
这是什么原因? 如果右侧有 struct/variant/array这些复杂类型呢
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/eageraggregation/PushDownAggContext.java:
##########
@@ -64,6 +64,8 @@ public class PushDownAggContext {
private final boolean passThroughBigJoin;
private final boolean needOutputCount;
+ private final boolean isPassThroughJoinOrUnion;
Review Comment:
为什么要关注 union?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/eageraggregation/EagerAggRewriter.java:
##########
@@ -179,6 +199,22 @@ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ?
extends Plan> join, P
return newJoin;
}
+ private boolean isPassThroughHeavyJoin(Plan joinChild, PushDownAggContext
context) {
+ if (context.isPassThroughBigJoin() ||
SessionVariable.getEagerAggregationMode() > 0) {
+ return true;
+ } else {
+ Statistics stats = joinChild.getStats();
+ if (stats == null) {
+ stats = joinChild.accept(derive, new
StatsDerive.DeriveContext());
+ }
+ return stats.getRowCount() > BIG_JOIN_BUILD_SIZE ||
outputStringType(joinChild);
Review Comment:
只检查 string 类型,没有检查其他复杂类型
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
