freemandealer opened a new pull request, #70142:
URL: https://github.com/apache/doris/pull/70142
### What problem does this PR solve?
Issue Number: None
Related PR: #67539
Problem Summary:
Export history expires relative to job creation, so a long-running export
can disappear immediately after it finishes. Online cleanup and image filtering
also use separate retention logic, and replaying cancelled attempts can replace
the label mapping owned by a non-cancelled attempt.
Measure retention from the finish time and preserve terminal jobs for at
least five minutes. Apply TTL cleanup first, then remove eligible terminal jobs
by finish time and job ID until the count limit is met. Finished and cancelled
jobs use the same ordering; active jobs and jobs with unknown or future finish
times remain protected.
Reuse this policy for online cleanup and image save/load. Only the ready
serving master backfills missing finish times through the existing state-update
journal, without holding the ExportMgr lock while waiting for the journal.
Preserve the existing missing-job replay guard and conditional label removal,
and keep label ownership consistent across image loading and state replay.
### Release note
Export history retention now starts when the job finishes, with a
five-minute minimum. Count-based cleanup removes the oldest eligible finished
or cancelled jobs first. Protected jobs can keep the history above the
configured count limit. A zero count limit removes all eligible terminal jobs;
a negative count limit disables only count-based cleanup.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test.
Validation on master `7866f5a36c02e15dc014aaef30e537a395f4059d`:
- Full FE build through `build.sh --fe -j 4`, including the UI built from
the committed lockfile.
- Checkstyle for `fe-common` and `fe-core`: zero violations.
- `run-fe-ut.sh --run
org.apache.doris.load.ExportHistoryPersistenceTest,org.apache.doris.load.ExportHistoryRetentionTest,org.apache.doris.load.loadv2.ExportMgrTest,org.apache.doris.load.ExportJobTest,org.apache.doris.load.ExportOutfileInfoTest`:
25 tests passed, zero failures, errors or skips.
- Coverage includes retention boundaries, a 10,000-job ordering case,
label registration races, real state/image serialization, replay, journal
blocking and failure propagation.
- Behavior changed:
- [ ] No.
- [x] Yes. Export history follows the finish-time retention policy
described above; configuration defaults are unchanged.
- Does this need documentation?
- [x] No. Existing configuration descriptions and the release note
explain the policy.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
