Yukang-Lian opened a new pull request, #67401:
URL: https://github.com/apache/doris/pull/67401
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
In cloud mode, a heavyweight schema change leaves NOT_READY shadow tablets
in the BE tablet cache. When the job is cancelled (or removed), those tablets
can never be converted or compacted, but they linger in the cache until the
recycler permanently deletes their meta, because meta-service keeps returning
their tablet meta before that.
Meanwhile the compaction scheduler records the max compaction score before
filtering, so such an abandoned tablet keeps feeding
`tablet_cumulative_max_compaction_score` with a large constant value, while
`/api/compaction_score` only enumerates TABLET_RUNNING tablets and never shows
it. The metric then reports a false backlog (observed: a flat 551 across many
BEs for days in production) until the BE restarts, and the metric and the HTTP
endpoint contradict each other, which makes the alarm impossible to diagnose.
Fix:
- MS reports whether a tablet still has an active schema change job via the
existing `get_rowset` RPC: one extra point read of the tablet job KV, only when
BE sets the new `need_alter_job_info` request field for its NOT_READY tablets.
The job KV is written on job start and cleared synchronously on commit/abort,
so it is an authoritative and timely liveness signal, independent of delayed
recycling.
- BE refreshes `CloudTablet::has_active_alter_job` on each rowset sync. A
NOT_READY tablet without an active alter job (a zombie tablet) is excluded from
the max compaction score stats, from compaction candidates, and from
`/api/compaction_score`, so the metric and the HTTP endpoint now describe the
same set of tablets: RUNNING plus in-progress schema change new tablets.
In-progress new tablets stay visible in both (they do get compacted via
`enable_new_tablet_do_compaction`).
- `/api/compaction_score` entries now carry `tablet_state` and
`not_scheduled_reason` (evaluated only for the returned top-n entries, so the
cost is bounded by `top_n`), and `/api/compaction/show` reports the full
schedulability breakdown including `is_zombie`, so a high score that nobody
compacts explains itself. The scheduler log line also reports `num_zombie`.
Rolling upgrade is safe in both directions: an old MS simply leaves the
response field unset and BE keeps today's behavior (the tablet is treated as
alter-in-progress).
Manual test on a single-node cloud cluster (schema change conversion blocked
via debug point, 60 double-written load txns, then CANCEL ALTER, recycler not
running):
- While the job is alive: gauge = 11 vs top1 = 12 and the shadow tablets
show up in top_n as `NOTREADY`, consistent for the first time.
- After cancel: within a few sync cycles the gauge falls back to the real
backlog level, top_n no longer lists the shadows, the scheduler log shows
`num_zombie=2`, and `/api/compaction/show` reports `is zombie tablet: true /
not scheduled reason: zombie`. Before this fix the gauge stayed pinned until BE
restart.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Abandoned (zombie) schema change shadow tablets no longer feed the
max compaction score metrics, are no longer picked as compaction candidates
(their prepare always fails against MS anyway), and are excluded from
`/api/compaction_score`; `/api/compaction_score` now also lists in-progress
schema change new tablets with a `tablet_state` field and a
`not_scheduled_reason` field.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_019Ee92HZPV9tu134TVB4EoT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
