RoanHeNaN commented on issue #70206:
URL: https://github.com/apache/doris/issues/70206#issuecomment-6651384053
### Note on the current (coarse) grace implementation and a design question
The draft PR #70217 implements the grace watermark at **rowset
granularity**: in
`_init_delete_condition`, delete-sign pruning for the whole compaction is
deferred
until the newest input rowset is older than the grace window.
**Known limitation (coarse gate).** Because the decision is per-compaction
(not
per-row), under sustained ingestion a base compaction's input may always
contain a
recent rowset, so old tombstones that are already safe to prune get held
back too,
and delete-sign **rows** can accumulate in the base rowset. This is *delayed*
reclamation, not a permanent leak (an ingestion lull lets the next
compaction prune
them) and it is tunable via the grace value. It only grows the delete-sign
row
count, **not the delete bitmap** — compaction still runs and still reclaims
delete-bitmap entries for covered rows as usual.
**Why not row-level grace yet.** Keeping only the tombstones actually within
the
window (decided per winning row by its source-rowset age) would remove the
accumulation, but:
- it is a space optimization, not a correctness change — the coarse gate
already
fixes the resurrection/divergence bug in full;
- delete-sign filtering must run post-merge on the winning row, where per-row
source-rowset provenance isn't available; threading it through the merge
iterator
→ block reader → visibility filter is a cross-layer change on the hot
path, and
filtering pre-merge per rowset is incorrect (an older non-delete row in
another
input rowset would resurrect within the same compaction).
**Design question for maintainers.** Before investing in the heavier
version, which
direction is preferred:
1. keep the grace-watermark approach and add row-level grace as a follow-up,
or
2. instead persist a per-key "max deleted sequence" watermark that the
`KEY_NOT_FOUND` path can consult, so tombstones can be pruned immediately?
Happy to go either way based on your input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
