englefly opened a new pull request, #69379:
URL: https://github.com/apache/doris/pull/69379
### What problem does this PR solve?
Backport of #69209 to `branch-4.1`.
`LINEAR_ANALYZE_TEMPLATE` collected hot values in a third CTE: it grouped
the sampled column by value, counted each group, and kept the top
`hot_value_collect_count` groups. That CTE re-reads the whole sample and is the
dominant cost of the template — the one sample analyze picks for a single
distribution column / single unique key column — and its cost is not bounded by
the sample size for the full-scan fallback.
This PR removes that CTE. `hot_value` keeps its position in the result set
and is reported as `NULL`, exactly like `FULL_ANALYZE_TEMPLATE` and
`MERGE_PARTITION_TEMPLATE` already do, and every reader already treats `NULL`
as "not collected" (`ColumnStatistic.hotValues == null`). Hot values are still
collected by `DUJ1_ANALYZE_TEMPLATE`, which derives them from the grouped
sample it already builds for the NDV estimate.
The params that only served the removed CTE (`subStringColName`, `rowCount2`
on the linear path, `hotValueCollectCount`) are now filled for the DUJ1
algorithm only.
### branch-4.1 adaptation
branch-4.1 still has a flat `org.apache.doris.statistics` package (no
`analysis/` / `model/` split), so the change is mapped onto:
-
`fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java`
-
`fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java`
- `fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java`
-
`fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java`
Skipped because the corresponding code does not exist in branch-4.1:
- The `AnalyzeCommand` rejection-message change and its test update.
branch-4.1 has no `WITH HOT VALUE` clause at all (no `HOT` token in the Nereids
lexer, no `PROPERTY_COLLECT_HOT_VALUE` in `AnalyzeProperties`), and
`AnalyzeCommand.validate` has no such check. For the same reason #69225 (accept
`WITH HOT VALUE` on a sample analyze) is a no-op for branch-4.1 and is not
included here.
- `PluginDrivenSampleAnalysisTask` (and its `subStringColName` fix) does not
exist in branch-4.1; the legacy `HMSAnalysisTask` is left unchanged.
- `test_full_analyze_hot_value.groovy` is master-only.
`physical_agg_regulator`: branch-4.1 cannot ask for hot values explicitly,
and its tables are distributed by a single column, so every sample analyze of
them now uses the linear estimator and collects no hot value. The expected plan
is regenerated accordingly — the skew-aware `hashAgg[LOCAL] +
PhysicalDistribute[DistributionSpecExecutionAny]` branch disappears from the
`!skew` case.
### Behaviour changes
- A sample analyze no longer collects hot values for columns estimated with
the linear estimator: a single distribution column / single unique key column,
and any sample analyze that falls back to a full table scan.
- The planner consequences (`ChildrenPropertiesRegulator` no longer picks
the skew-aware aggregation shape, `ShuffleKeyPruneUtils` skips the shuffle-key
prune, `SkewJoin` finds no hot value, `DecomposeRepeatWithPreAggregation`
treats the missing hot value as balanced) change plan quality only: every
consumer null-checks "not collected" and none of them produces a wrong result.
### Verification
- FE build: `BUILD SUCCESS`.
- Unit tests: `OlapAnalysisTaskTest` 17/17, `HMSAnalysisTaskTest` 8/8.
- Regression (local 1 FE + 1 BE cluster):
- `test_hot_value` ✅
- `hot_value_analyze_sync` ✅
- `physical_agg_regulator` ✅ (with the regenerated expected plan)
### Release note
Sample analyze no longer collects hot values for columns that use the linear
NDV estimator (single distribution / single key column, and the full-scan
fallback of a sample analyze), which removes a second aggregation over the
sampled data. Hot values are still collected by sample analyze for columns
estimated with DUJ1.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
