englefly commented on code in PR #68282:
URL: https://github.com/apache/doris/pull/68282#discussion_r4254643900
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/AnalysisManager.java:
##########
@@ -1487,6 +1487,75 @@ public void removeTableStats(long tableId) {
}
}
+ /**
+ * Remove the statistics record of a table and write the journal entry of
the removal while holding the
+ * monitor of the statistics records, so that this transition and a
concurrent truncation, which writes
+ * its own entry under the same monitor, are ordered the same way in
memory and in the journal. Replay
+ * paths, which must not journal, use {@link #removeTableStats(long)}.
+ */
+ public void removeTableStatsAndLog(long tableId) {
+ synchronized (idToTblStats) {
+ idToTblStats.remove(tableId);
+ Env.getCurrentEnv().getEditLog().logDeleteTableStats(new
TableStatsDeletionLog(tableId));
+ }
+ }
+
+ /**
+ * TRUNCATE TABLE removes all the data of the table, but the table itself
stays and can be loaded
+ * immediately. The stats record must be kept, otherwise the row count of
the newly loaded data can
+ * never be reported: the backends report the row count of the new empty
tablets with a delay of up to
+ * {@code tablet_stat_update_interval_second}, and without a record there
is nothing to accumulate the
+ * loaded rows into. So reset the record to the state of an empty table
instead of removing it.
+ *
+ * <p>The reset and the journal entry of the truncation are written while
holding the monitor of the
+ * statistics records, and so is the deletion of a record, so that the
order of two concurrent
+ * transitions between the states of a record is the same in memory and in
the journal: otherwise a
+ * truncation and a deletion which do not share a table lock could be
applied in one order and replayed
+ * in the other, leaving the followers with a record the master removed
(or the reverse).
+ *
+ * @return whether the record had to be created, i.e. the table had no
statistics record
+ */
+ public boolean resetTableStats(OlapTable table, TruncateTableInfo
truncateInfo) {
+ synchronized (idToTblStats) {
+ TableStatsMeta tableStats = idToTblStats.get(table.getId());
+ boolean recordCreated = false;
+ if (tableStats == null) {
+ tableStats = new TableStatsMeta(table);
+ idToTblStats.put(table.getId(), tableStats);
+ recordCreated = true;
+ }
+ // The entry records what this truncation did to the record, so
that the replay reproduces it.
+ truncateInfo.setTableStatsRecordCreated(recordCreated);
+ tableStats.reset(table);
+ Env.getCurrentEnv().getEditLog().logTruncateTable(truncateInfo);
+ return recordCreated;
+ }
+ }
+
+ /**
+ * Apply the transition which the given truncate entry recorded, instead
of the one this frontend would
+ * apply now.
+ *
+ * <p>A truncate creates the statistics record of a table which has none,
and that creation is not a
+ * journal entry of its own. The truncate entry therefore carries whether
the truncate created the record,
+ * so that the replay reproduces exactly what the master did and a
truncate entry never resurrects a
+ * record which another journaled transition, for instance a whole table
DROP STATS or the cleanup of an
+ * empty table by the analyzer, removed around it.
+ */
+ public void replayResetTableStats(OlapTable table, boolean recordCreated) {
Review Comment:
Fixed in `ae01164abcd` by removing the flag - your question is the reason it
can go.
Its purpose: the truncate entry recorded whether the truncation had created
the record, because a truncate creates the record of a table which has none and
that creation is not a journal entry of its own. The replay used it so that a
truncate entry could not resurrect a record which a concurrent journaled
deletion (a whole table `DROP STATS`, the analyzer's cleanup of an empty table)
had removed. Back then a record transition and its entry were not written
atomically, so the master could end up without the record while the entries
implied the opposite.
Since every transition and its entry are written under the same
`idToTblStats` monitor, the journal order is the mutation order and the entries
are replayed in that order. At a truncate entry the record state is therefore
the state the master had when it truncated the table, so the answer to
`tableStats == null` is simply "the master had no record either and created
one": `replayResetTableStats()` now creates the record when it is absent and
resets it otherwise, which is exactly the transition the truncation applied on
the master.
The combination you asked about (`tableStats == null && recordCreated ==
false`) now only occurs for entries written before the field existed; those get
the semantics of this change when they are replayed, i.e. the record is
created, which is what this PR wants: the record has to exist for the rows
loaded after the truncation to be reported.
`TruncateTableInfo` no longer carries the field (nor its accessors),
`resetTableStats()` no longer returns it, and
`AnalysisManagerTest#testReplayOfTruncateFollowsTheJournalOrder` covers the two
orders: an existing record is reset (its accumulated rows are dropped), an
absent one is created with `updatedRows = 0`, and a removal replayed afterwards
removes it again. Verified live as well: after a hard kill and a restart of the
frontend, the replay recreated the record of a truncated table with its 3
loaded rows, including a table which had never been analyzed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
