englefly opened a new pull request, #69760:
URL: https://github.com/apache/doris/pull/69760
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
The Nereids optimizer infers `is not null` predicates from join conditions,
e.g.
`t1 join t2 on t1.a = t2.b` infers `t1.a is not null` and `t2.b is not
null`. Those
predicates are marked as generated and `EliminateNotNull` always removes
them.
For a column with many nulls the inferred predicate is not redundant: when
it is
evaluated directly on a scan, the scan can skip the rows whose value is null.
Removing it forces the scan to emit all null rows, which the join then
discards.
This change keeps the inferred predicate instead of eliminating it when both:
- the filter which owns the predicate is evaluated directly on an OLAP scan,
and
- the number of nulls of the column (`colStats.numNulls`) is greater than
`keep_is_not_null_num_nulls_threshold` (new session variable, default
1000000,
a negative value disables the optimization).
The kept predicate is rebuilt as a normal `is not null` predicate, so the
second
`EliminateNotNull` run of the rewrite phase does not remove it again.
### Release note
Add the session variable `keep_is_not_null_num_nulls_threshold` (default
1000000).
The optimizer keeps an inferred `is not null` predicate above a scan when
the column
has more nulls than this threshold, so the scan can skip the null values. A
negative
value restores the previous behavior of always eliminating the inferred
predicate.
### Check List (For Author)
- Test: Unit Test / Regression test
- `EliminateNotNullTest` (new cases for keeping and eliminating the
inferred
predicate) plus the existing `InferJoinNotNullTest`,
`InferFilterNotNullTest`,
`EliminateOuterJoinTest` and `ConvertOuterJoinToAntiJoinTest` pass.
- New suite `nereids_rules_p0/not_null/keep_is_not_null_on_scan` passes
on a local
single-node cluster: with the threshold set to 1 the explain shows
`PREDICATES: val IS NOT NULL` on both scans, with the default or a
negative
threshold the predicate is eliminated as before.
- Behavior changed: Yes. The inferred `is not null` predicate is kept above
a scan when
the column has more nulls than the new session variable. Query results are
unchanged.
- Does this need documentation: Yes, the new session variable should be
documented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
