github-actions[bot] commented on code in PR #67354:
URL: https://github.com/apache/doris/pull/67354#discussion_r4230096789
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java:
##########
@@ -645,7 +645,12 @@ private Statistics
estimateColumnEqualToConstant(ComparisonPredicate cp, ColumnS
// stats-derive may be applied before type
coercion. so we need to try to catch
// comparison exception. for example: boolean
compare with int
if (((ComparableLiteral)
hot).compareTo((ComparableLiteral) constHand) == 0) {
- selectivity =
statsForLeft.getHotValues().get(hot);
+ float hotSel =
statsForLeft.getHotValues().get(hot);
Review Comment:
[P2] Preserve positive frequency for observed hot keys in IN and range
filters
The new guard covers equality only. With 1,000,000 non-null rows split
500,000/499,998/1/1 over four values, analyze stores
0.5000/0.5000/0.0000/0.0000. A two-value IN on the rare keys and an inclusive
range over them both sum the two zero ratios and estimate zero rows even though
two rows match. Those false zero stats still reach later join and cost
estimation. Preserve a positive bound for observed keys in these consumers too,
or retain their unrounded frequencies.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java:
##########
@@ -645,7 +645,12 @@ private Statistics
estimateColumnEqualToConstant(ComparisonPredicate cp, ColumnS
// stats-derive may be applied before type
coercion. so we need to try to catch
// comparison exception. for example: boolean
compare with int
if (((ComparableLiteral)
hot).compareTo((ComparableLiteral) constHand) == 0) {
- selectivity =
statsForLeft.getHotValues().get(hot);
+ float hotSel =
statsForLeft.getHotValues().get(hot);
+ // prevent precision-loss: the hot value's
ratio was rounded
+ // during analyze.
+ if (hotSel > 0) {
Review Comment:
[P1] Bound zero-ratio equality estimates by the collection precision
Four-place rounding still stores 0.0000 for an observed key below 0.005%.
With 12,509,034 rows, NDV 3, and 500 occurrences of a top-three key, this
fallback estimates about 4.17 million rows instead of 500. For `Join(Filter(a =
rare, Scan(A)), Scan(B: 1M rows))`, `InitJoinOrder` therefore misses the left <
100,000 swap; the actual 500-row left child would pass it, so the large right
build and MV refresh failure mode can remain. Use a small positive
precision/frequency bound for an observed rounded-zero key rather than the
uniform 1/NDV estimate.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java:
##########
@@ -112,7 +112,7 @@ public enum AnalyzeSampleAlgorithm {
+ "cte3 AS ("
+ "SELECT IFNULL(GROUP_CONCAT(CONCAT("
+ "REPLACE(REPLACE(t.`column_key`, \":\", \"\\\\:\"),
\";\", \"\\\\;\"), "
- + "\" :\", ROUND(t.`count` / ${rowCount2}, 2)), \" ;\"),
'') "
+ + "\" :\", ROUND(t.`count` / ${rowCount2}, 4)), \" ;\"),
'') "
Review Comment:
[P2] Handle all-hot IN maps before dividing residual frequency
For 100,000 non-null rows split 994/994/98,012 (NDV=3), this scale-4 format
stores .0099/.0099/.9801; parsing as floats makes their sum about .9999. In
`visitInPredicate`, IN on the two 994-row values takes the `nonHotRatio > 0`
branch, but both `newNdv - matchedHotCount` and `ndv - hotCount` are zero. The
resulting 0/0 becomes NaN, then `Statistics.withSel` treats it as selectivity
1: 100,000 estimated rows for 1,988 matches. The old scale-2 values took the
hot-only branch. Handle `ndv == hotCount` before the residual division.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
