englefly opened a new issue, #69281: URL: https://github.com/apache/doris/issues/69281
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master (the code is the same on the released branches; it predates the table-stats-record-on-load change) ### What's Wrong? `TableStatsMeta` pairs two values which are snapshotted at different moments: - `indexesRowCount[baseIndexId]`: the row count collected by an analyze task's `COUNT(1)`, which runs when the task executes (`BaseAnalysisTask.runQuery`, `jobInfo.addIndexRowCount(indexId, colStatsData.count)`). - `updatedRowsBase`: the load counter of the record at the time the analyze job is applied, currently `analyzedJob.updateRows` (`TableStatsMeta.update`). That value was captured when the **job was built** (`AnalysisManager.buildAnalysisInfoForTbl`, `infoBuilder.setUpdateRows(tableStatsStatus.updatedRows.get())`; `StatisticsAutoCollector.createAnalyzeJobForTbl` does the same). When a load becomes visible between those two moments, its rows are counted twice by the planner fallback: ``` load 100 rows updatedRows = 100 ANALYZE TABLE t (job is built) job.updateRows = 100 another 50 rows become visible updatedRows = 150 task executes COUNT(1) indexesRowCount[base] = 150 job is applied updatedRowsBase = 100 fallback estimate 150 + (150 - 100) = 200 (the table holds 150) ``` The fallback is `StatsCalculator.getOlapTableRowCount` -> `TableStatsMeta.getRowCountWithDeltaRows`, used while `OlapTable.getRowCountForIndex(indexId, strict = true)` returns -1, i.e. while some partition's tablets have not reported their row count yet (right after `TRUNCATE TABLE`, the first load of a partition, within `tablet_stat_update_interval_second`). In that window the double count becomes the scan cardinality. This is pre-existing: before the record-creation change the baseline was derived from `ColStatsMeta.updatedRows`, which is also `analyzedJob.updateRows` (`TableStatsMeta.update`), so master computes the same 200. The change only moved the baseline from the column statistics to a table level field (`updatedRowsBase`). ### What You Expected? The collected row count and its baseline should describe the same snapshot: the baseline should be the load counter at the visible version the analyze `COUNT(1)` reads, so that `collected + (current updatedRows - baseline)` never adds a row that is already part of `collected`. ### How to Reproduce? A cluster reproduction is timing dependent; the sequence is: ```sql CREATE TABLE t (k1 INT, k2 INT) DUPLICATE KEY(k1) PARTITION BY RANGE(k1) (PARTITION p1 VALUES LESS THAN (10)) DISTRIBUTED BY HASH(k1) BUCKETS 1; TRUNCATE TABLE t; -- removes the table stats record INSERT INTO t VALUES (1,1),(2,2); -- 100 rows in the real case ANALYZE TABLE t; -- async: the job is built now, updateRows = 100 INSERT INTO t VALUES (3,3),(4,4); -- 50 rows become visible before COUNT(1) runs -- the analyze task then executes COUNT(1) and collects 150 rows EXPLAIN SELECT * FROM t; -- while the tablets have not reported yet: -- cardinality = 150 + (150 - 100) = 200 ``` Deterministic model at the unit level: build an `AnalysisInfo` with `updateRows = 100`, apply it to a record whose `updatedRows` is 150 and whose collected count is 150, then `getRowCountWithDeltaRows` returns 200. ### Anything Else? Root cause pointers: - `AnalysisManager.buildAnalysisInfoForTbl` / `StatisticsAutoCollector.createAnalyzeJobForTbl`: `setUpdateRows(updatedRows.get())` when the job is built. - `BaseAnalysisTask.runQuery`: the row count is collected when the task executes. - `TableStatsMeta.update`: `updatedRowsBase.set(analyzedJob.userInject ? updatedRows.get() : analyzedJob.updateRows)`. - `TableStatsMeta.getBaseIndexDeltaRowCount` / `getRowCountWithDeltaRows`. Fix direction: capture the load counter when the analyze query collects the row count (after the query, so the remaining race biases to an under-count instead of a double count) and use it as the baseline; cover it with a deterministic "build job -> load -> count" test. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
