Doris-Breakwater commented on issue #68537: URL: https://github.com/apache/doris/issues/68537#issuecomment-5862923444
Breakwater-GitHub-Analysis-Slot: slot_a64d0a38329a **Triage: credible cloud recycler / schema-change interaction; high impact when `force_immediate_recycle=true`.** The reported recycler `PREPARED` log for shadow index `1789639373134` precedes the `tablet_idx` lookup failure and FE cancellation. That sequence is consistent with the code path below. I have not independently inspected the original service logs or the exact deployed binary. **Code evidence.** In the [local `3.1.4-rc02` source](https://github.com/apache/doris/blob/7f5ba43de6c7f148e1dacdb7cb838394d2ec1537/cloud/src/recycler/recycler.cpp#L959-L977) and [current upstream master](https://github.com/apache/doris/blob/82f366e1f8017a5f404cc212be321ec7cfa2c29b/cloud/src/recycler/recycler.cpp#L1903-L1921), `calculate_index_expired_time()` returns `0` under `force_immediate_recycle` before checking `RecycleIndexPB.state`. [The index recycler](https://github.com/apache/doris/blob/82f366e1f8017a5f404cc212be321ec7cfa2c29b/cloud/src/recycler/recycler.cpp#L2970-L3038) therefore admits a `PREPARED` record, marks it `RECYCLING`, and calls `recycle_tablets()` for the whole index. [Tablet recycling removes `meta_tablet_idx_key`](https://github.com/apache/doris/blob/82f366e1f8017a5f404cc212be321ec7cfa2c29b/cloud/src/recycler/recycler.cpp#L3733-L3738), which [MetaService needs for the rowset commit](https://github.com/apache/doris/blob/82f366e1f8017a5f404cc212be321ec7cfa2 c29b/cloud/src/meta-service/meta_service.cpp#L203-L215). FE creates the `PREPARED` index with a schema-change deadline; successful `commit_index()` removes that recycle record. This supports the proposed cause, although confirmation for this specific incident still depends on matching the deletion of tablet `1789639379947` to the recycler run. **Operational next step.** Keep `force_immediate_recycle=false` on all relevant recycler instances and rerun the cancelled DDL only after confirming the setting and cleanup state. The setting is explicitly marked **“just for TEST”** and is declared [`CONF_Bool`](https://github.com/apache/doris/blob/82f366e1f8017a5f404cc212be321ec7cfa2c29b/cloud/src/common/config.h#L148-L150), which [defines it as non-mutable](https://github.com/apache/doris/blob/82f366e1f8017a5f404cc212be321ec7cfa2c29b/cloud/src/common/configbase.h#L97-L112); the issue’s claim that it is mutable should be corrected. **Suggested maintainer action.** Preserve the normal expiration policy for `PREPARED` indexes even when immediate recycling is enabled, while retaining eventual cleanup of abandoned preparations; evaluate a job-liveness guard and the commit/recycle race. Add a regression test with a live `PREPARED` shadow index under this flag, plus an expired/orphaned preparation and a `DROPPED` index. **Evidence requested to close the incident diagnosis:** effective recycler config at the time; the recycle record’s `state` and `expiration` plus the FE job timeout; recycler deletion/completion logs for index `1789639373134` and tablet `1789639379947`; the corresponding MetaService commit error and BE task log with timestamps/time zone; and the exact 3.1.4 build or commit. These distinguish this sequence from another cause of a missing `tablet_idx` key. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
