Doris-Breakwater commented on issue #69281: URL: https://github.com/apache/doris/issues/69281#issuecomment-6264500739
Breakwater-GitHub-Analysis-Slot: slot_b458a0c275e6 Initial triage: the reported baseline/count timing mismatch is supported by the code. It can inflate the planner's fallback cardinality when a load is included in the collected count but remains newer than the saved load-counter baseline. This concerns optimizer estimates; the inspected path does not establish incorrect query results. **Verified code path** Inspected upstream master at `57e8aa36ebac600037852c1c564bc0a5648cad60` and the currently unmerged #69028 at `dc2d63c6987ff9d63bc71dabc27497dad18d1bfd`: - Manual and automatic jobs capture `updateRows` during construction, before the analysis query executes. See [AnalysisManager](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/AnalysisManager.java#L419-L440) and [StatisticsAutoCollector](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/StatisticsAutoCollector.java#L257-L280). - `BaseAnalysisTask.runQuery` obtains the query result later and stores `colStatsData.count` through `addIndexRowCount`; that method overwrites the entry for the index. See [runQuery](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java#L675-L704). - On master, `TableStatsMeta.update` stores the job's earlier `updateRows` in column metadata. `getBaseIndexDeltaRowCount` subtracts the maximum base-index column baseline from the current counter. For an existing record with baseline 100, collected count 150, and current counter 150, the fallback therefore evaluates to 200. See [metadata update](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/TableStatsMeta.java#L164-L198), [delta calculation](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/TableStatsMeta.java#L257-L269), and [planner fallback](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java#L463-L480). - #69028 moves the baseline into `updatedRowsBase`, but still assigns `analyzedJob.updateRows` for ordinary analysis. Its synchronized publication prevents readers mixing two metadata updates; it does not align the job-build baseline with the query's data snapshot. See [PR metadata publication](https://github.com/apache/doris/blob/dc2d63c6987ff9d63bc71dabc27497dad18d1bfd/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/TableStatsMeta.java#L243-L292). **Reproduction distinction** The pre-existing mismatch and the exact post-TRUNCATE example need separate treatment. On inspected master, loads update an existing statistics record but do not create a missing one ([replayUpdateRowsRecord](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/AnalysisManager.java#L1353-L1416)). Consequently, the first post-TRUNCATE load does not itself establish `updatedRows = 100` there. Use an existing analyzed record to demonstrate the older race, or explicitly identify #69028 as the basis for the stated post-TRUNCATE sequence. The strict row-count fallback must also be active to observe the inflated scan estimate. **Suggested fix and validation** Carry the collected count and its collection-time baseline as one coherent result, preserving their association when concurrent column tasks overwrite an index entry and when metadata is published. Avoid simply repurposing the shared job-wide `updateRows`: column baselines also drive [automatic reanalysis decisions](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java#L840-L850). Sampling the counter immediately after the query addresses the illustrated earlier-load overlap, but remains an approximation. A load becoming visible after the query's snapshot and before that sample can be absorbed into the baseline without being present in the count, causing underestimation. Exact snapshot alignment requires coordinating the baseline with the visible versions read by the query. Also handle statistics-record removal/recreation so a query result cannot acquire a baseline from a different counter generation. Recommended deterministic coverage: 1. Pause after job construction, advance the counter from 100 to 150, then collect 150: the fallback should be 150. Advance the counter to 175 after capturing the count/baseline pair: it should become 175. 2. Exercise the actual query-result-to-job publication path, with concurrent column tasks completing out of order; a metadata-only arithmetic test would miss mismatched result/baseline publication. 3. Cover a load between query snapshot and counter sampling, plus DROP STATS/TRUNCATE record recreation. Retain unknown-count, user-injected-count, and rollup-only behavior. 4. Check synchronous/asynchronous manual and automatic analysis. Treat sampled and partition analysis separately: sample SQL supplies `${rowCount}` rather than obtaining the full table count through `COUNT(1)` ([templates](https://github.com/apache/doris/blob/57e8aa36ebac600037852c1c564bc0a5648cad60/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java#L153-L198)); an unconditional post-query baseline is not sufficient evidence of correctness for every collection mode. **Information still needed** No additional logs are needed to justify a focused regression test. To establish the cluster symptom and affected release scope, please provide the exact commit/build (including whether #69028 is applied), full versus sampled/partition analysis mode, job/task timestamps, load visibility ordering, collected count/current counter/baseline, and an EXPLAIN captured while strict tablet row counts remain unavailable. With the literal two-row inserts, the analogous inflated estimate is 6 for 4 rows; 200 requires the stated 100/50-row workload. Validation here is source inspection, not a running-cluster reproduction or an executed Java unit test. Released branches were not individually verified. Suggested next step: add the deterministic task-path regression first, then implement and review the paired count/baseline change against the collection modes above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
