liutang123 opened a new issue, #52311: URL: https://github.com/apache/doris/issues/52311
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 2.1.9 ### What's Wrong? When useing workload group, if a query is queued in workload query queue, the replias planed in the query may be deleted by `TabletScheduler`. The log is as follows: Start a query and it will be appended to workload group's queue. ``` 2025-06-25 15:02:06,092 INFO (thrift-server-pool-110827|11493647) [OlapInsertExecutor.beforeExec():193] start insert [label_4f4131d425174394_a77e97f8867a71aa] with qury id 4f4131d425174394-a77e97f8867a71aa and txn id 58443425 ``` `TabletScheduler` remove a replia of tablet 5893068. ``` 2025-06-25 15:02:23,688 INFO (thrift-server-pool-87589|9004983) [TabletScheduler.addTablet():294] Add tablet to pending queue, tablet id: 5893068, status: REDUNDANT, state: PENDING, type: REPAIR, priority: VERY_HIGH, tablet size: 0, visible version: -1, committed version: -1 2025-06-25 15:02:24,673 INFO (tablet scheduler|31) [TabletScheduler.deleteReplicaInternal():1201] set decommission replica 61471402 on backend 60003594 of tablet 5893068 pre watermark txn id 58443855 2025-06-25 15:02:24,673 INFO (tablet scheduler|31) [TabletScheduler.deleteReplicaInternal():1219] set decommission replica 61471402 on backend 60003594 of tablet 5893068 post watermark txn id 58443856 2025-06-25 15:02:24,680 INFO (tablet scheduler|31) [TabletScheduler.deleteReplicaInternal():1258] delete replica. tablet id: 5893068, backend id: 60003594. reason: hih load backend, force: false 2025-06-25 15:02:24,680 INFO (tablet scheduler|31) [TabletScheduler.removeTabletCtx():1751] remove the tablet tablet id: 5893068, status: REDUNDANT, state: PENDING, tpe: REPAIR, priority: NORMAL, tablet size: 1414042232, visible version: 257060, committed version: 257060. err: redundant replica is deleted. because: redundant replca is deleted ``` BE report tablets to FE master: ``` 2025-06-25 15:02:45,228 INFO (report-thread|111) [ReportHandler.deleteFromBackend():1034] failed add to meta. tablet[5893068], backend[60003594] 2025-06-25 15:02:45,228 INFO (report-thread|111) [ReportHandler.deleteFromBackend():1050] delete tablet[5893068] from backend[60003594] because not found in meta ``` FE master send DELETE task to BE: ``` I20250625 15:02:45.228513 533173 task_worker_pool.cpp:337] successfully submit task|type=DROP|signature=5893068 I20250625 15:02:45.230566 9000 tablet_manager.cpp:541] begin drop tablet. tablet_id=5893068, replica_id=61471402, is_drop_table_or_partition=0 ``` When the query starts, BE will report `tablet not exist.` ``` 2025-06-25 15:03:22,130 INFO (thrift-server-pool-110827|11493647) [Coordinator.execInternal():783] dispatch load job: 4f4131d425174394-a77e97f8867a71aa to... 2025-06-25 15:03:22,287 WARN (thrift-server-pool-111112|11529066) [Coordinator.updateFragmentExecStatus():2777] one instance report fail, query_id=4f4131d425174394-a77e97f8867a71aa fragment_id=49 instance_id=0-0, be=60003594, error message: Status [errorCode=CANCELLED, errorMsg=(10.64.6.21)[CANCELLED]failed to get tablet: 5893068, reason: tablet does not exist. 10.64.6.21] ``` ### What You Expected? FE master doesn't delete the replica before query the query ends. OR Plan the query after get workload token. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
