englefly opened a new pull request, #69028:
URL: https://github.com/apache/doris/pull/69028
An alternative implementation of the fix discussed in #68282, with a much
smaller change: instead of keeping
and resetting the statistics record across `TRUNCATE TABLE` (which also had
to order every record transition
with its journal entry), the record is recreated by the load itself, so a
truncation keeps removing it exactly
as it does today.
### What problem does this PR solve?
Issue Number: None (no upstream issue filed; the case is described below)
Problem Summary:
Reproduction on a single bucket duplicate key table:
CREATE TABLE t (k INT, v INT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k)
BUCKETS 1;
INSERT INTO t VALUES (1,1),(2,2),(3,3),(4,4),(5,5);
TRUNCATE TABLE t;
INSERT INTO t VALUES (1,1),(2,2),(3,3);
EXPLAIN SELECT * FROM t; -- reports cardinality=1, although the table
holds 3 rows
TRUNCATE TABLE replaces the partitions and tablets of the table with new
ones and removes its statistics
record. The backends report the row count of the new tablets only after the
delay of
`tablet_stat_update_interval_second` (60s by default), and a partition
reports no row count at all until its
first load becomes visible (`DatabaseTransactionMgr` sets `rowCountReported`
to false for the first
transaction of a partition), so during that interval
`OlapTable.getRowCountForIndex(..., strict = true)`
returns -1 for the whole table. The only other source of a row count is the
statistics record of the table,
and the truncation had just removed it, so the planner clamps the table to
one row: joins are ordered and
distributed for a table that looks empty instead of for the rows that were
just loaded.
The load now recreates the record itself. While the loading transaction
becomes visible,
`AnalysisManager.replayUpdateRowsRecord()` creates the record of a table
which has none in the state of a
table whose row count has never been collected, and accumulates the rows it
reports into it, so the planner
reads them instead of -1. The record is created with a baseline of 0
(`updatedRowsBase`) and a collected row
count of 0 for the base index, so every row loaded afterwards is a delta row
and the reported row count is
the number of rows the table holds.
The creation happens in `updateUpdatedRows()`, which is the path the master
walks when the loading
transaction becomes visible
(`DatabaseTransactionMgr.updateCatalogAfterVisible`) and the path a follower
walks while it replays the transaction state
(`replayUpsertTransactionState`), so every frontend ends up with
the record. It needs no edit log entry of its own and does not touch the
transaction state machine: the
record is a function of the transaction state the journal already carries.
The whole table `DROP STATS`, the
schema changes and the truncation keep removing the record as before, and
the next load recreates it. A
partition truncation is deliberately left out: the rows it reports are the
rows it removed, so it does not
create a record.
Two related fixes come with it, because the record is now the row count
source in more cases:
- the delta is only added to an index which keeps one row per base row
(`TableStatsMeta.keepsOneRowPerBaseRow`): an aggregating rollup, an index
which merges the rows of a unique
key table and a filtered materialized index have their own row count and
must not be reported with the
physical rows of the base index, and a row count which was never collected
is not combined with a delta;
- `StatisticsMetricCollector` decided "not analyzed" by the absence of a
record. A record now exists for a
loaded table which was never analyzed, so it decides by the analyze
timestamp instead.
### Known limitation
A record which is created for a table that already holds rows, for instance
right after a whole table
`DROP STATS` or after a schema change (the truncation case starts from an
empty table), only knows the rows
loaded after that point, so the table is estimated with them until the
backends report the tablets or an
analysis collects the row count. It is still closer than the unknown row
count it replaces: a merge key
table reports the physical row count in that window, which is an upper bound
of its logical row count, as it
does in the steady state today.
### Release note
None
### Check List (For Author)
- Test: new unit tests in `AnalysisManagerTest` (a load creates the record
and accumulates its rows, a load
which reports the rows by table id creates it as well, a replayed load
creates the same record, an existing
analyzed record is only accumulated, and an unresolvable table creates
none) and in `TableStatsMetaTest`
(the record of a table whose row count was never collected reports the
loaded rows; the baseline survives
dropping the column statistics; a legacy record derives it from the column
statistics; a stale analyze
snapshot and its baseline cancel out; a user injected row count is not
counted twice; the reader of the
snapshot never pairs two states; a row preserving rollup reports the
loaded rows while an aggregating one
and a filtered one do not). `AnalysisManagerTest` 21/21,
`TableStatsMetaTest` 12/12 (checkstyle not
skipped). Regression: the new suite `test_row_count_after_truncate` (the
truncate case, an aggregate rollup
and a projection rollup) and the existing `test_drop_stats_and_truncate`
both pass. Manual test on a local
FE and BE cluster with the reproduction above: the scan reports
cardinality=3 right after the load instead
of 1, a never analyzed table which receives 4 rows reports 4 instead of 1,
and `SHOW TABLE STATS` shows the
accumulated rows with an empty analyze time.
- Behavior changed: Yes. A table whose statistics record was removed reports
the rows its loads accumulate
instead of an unknown row count (planned as one row) while the backends
have not reported the new tablets
yet; the "not analyzed" metric counts a table by its analyze timestamp.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
