xiacongling opened a new issue, #41747: URL: https://github.com/apache/doris/issues/41747
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 2.0.9 ### What's Wrong? when I alter a table to add a column, the progress changed to 'CANCELLED' state with the following erorr message: ``` errCode = 2, detailMessage = schema change tasks failed on same tablet reach threshold [[ALTER], signature: 9588886, backendId: 1237554, tablet id: 9588886, [ALTER], signature: 9588886, backendId: 1237552, tablet id: 9588886] ``` after searching the logs on BE, i found that the error is due to hard link creation cross file systems. the `storage_root_path` of the BE contains multiple directories, each for an SSD. Doris appears to have conducted cross-SSD hard link creation. ``` be.INFO.log.2024101210:I1012 10:23:53.739847 174192 task_worker_pool.cpp:264] successfully submit task|type=CREATE|signature=9588886|queue_size=1 be.INFO.log.2024101210:I1012 10:23:53.739900 152342 tablet_manager.cpp:270] begin to create tablet. tablet_id=9588886, table_id=519079, partition_id=7326298, replica_id=9588888 be.INFO.log.2024101210:I1012 10:23:53.740079 152342 tablet_meta.cpp:299] init tablet meta from pb: table_id: 519079 partition_id: 7326298 ............... be.INFO.log.2024101210:I1012 10:23:53.740762 152342 tablet_manager.cpp:344] success to create tablet. tablet_id=9588886 be.INFO.log.2024101210:I1012 10:23:53.740772 152342 task_worker_pool.cpp:1355] successfully create tablet|signature=9588886|tablet_id=9588886 be.INFO.log.2024101210:I1012 10:23:54.071437 174192 task_worker_pool.cpp:264] successfully submit task|type=ALTER|signature=9588886|queue_size=1 be.INFO.log.2024101210:I1012 10:23:54.071599 152367 task_worker_pool.cpp:1743] get alter table task, signature: 9588886 be.INFO.log.2024101210:I1012 10:23:54.071619 152367 schema_change.cpp:669] begin to do request alter tablet: base_tablet_id=7326299, new_tablet_id=9588886, alter_version=381 be.INFO.log.2024101210:I1012 10:23:54.071625 152367 schema_change.cpp:736] finish to validate alter tablet request. begin to convert data from base tablet to new tablet base_tablet=7326299.572003619.394099e52dc6d55e-7a7a0d78bf9a2490 new_tablet=9588886.708419933.484b4d433f831852-ac3a3492b722f1a1 be.INFO.log.2024101210:I1012 10:23:54.071753 152367 schema_change.cpp:808] begin to remove all data from new tablet to prevent rewrite. new_tablet=9588886.708419933.484b4d433f831852-ac3a3492b722f1a1 be.INFO.log.2024101210:I1012 10:23:54.071959 152367 schema_change.cpp:1007] begin to convert historical rowsets for new_tablet from base_tablet. base_tablet=7326299.572003619.394099e52dc6d55e-7a7a0d78bf9a2490, new_tablet=9588886.708419933.484b4d433f831852-ac3a3492b722f1a1 be.INFO.log.2024101210:I1012 10:23:54.072011 152367 schema_change.cpp:1028] schema change type, sc_sorting: 0, sc_directly: 0, base_tablet=7326299.572003619.394099e52dc6d55e-7a7a0d78bf9a2490, new_tablet=9588886.708419933.484b4d433f831852-ac3a3492b722f1a1 be.INFO.log.2024101210:W1012 10:23:54.072100 152367 local_file_system.cpp:236] [INTERNAL_ERROR]failed to create hard link from /data/be/ssd9/data/438/7326299/572003619/020000000084b6c3624d66f49b006ec639ef0262d9e42a94_0.dat to /data/be/ssd2/data/712/9588886/708419933/020000000084be52624d66f49b006ec639ef0262d9e42a94_0.dat: Invalid cross-device link be.INFO.log.2024101210:W1012 10:23:54.072145 152367 schema_change.cpp:455] fail to convert rowset., new_tablet=9588886.708419933.484b4d433f831852-ac3a3492b722f1a1, version=0-364, error status [E-100]fail to create hard link. from=/data/be/ssd9/data/438/7326299/572003619/020000000084b6c3624d66f49b006ec639ef0262d9e42a94_0.dat, to=/data/be/ssd2/data/712/9588886/708419933/020000000084be52624d66f49b006ec639ef0262d9e42a94_0.dat, errno=18 be.INFO.log.2024101210:W1012 10:23:54.072153 152367 schema_change.cpp:1096] failed to process the version. version=0-364, [E-100]fail to create hard link. from=/data/be/ssd9/data/438/7326299/572003619/020000000084b6c3624d66f49b006ec639ef0262d9e42a94_0.dat, to=/data/be/ssd2/data/712/9588886/708419933/020000000084be52624d66f49b006ec639ef0262d9e42a94_0.dat, errno=18 be.INFO.log.2024101210:I1012 10:23:54.072237 152367 schema_change.cpp:1045] finish converting rowsets for new_tablet from base_tablet. base_tablet=7326299.572003619.394099e52dc6d55e-7a7a0d78bf9a2490, new_tablet=9588886.708419933.484b4d433f831852-ac3a3492b722f1a1 be.INFO.log.2024101210:W1012 10:23:54.072257 152367 schema_change.cpp:974] failed to alter tablet. base_tablet=7326299.572003619.394099e52dc6d55e-7a7a0d78bf9a2490, drop new_tablet=9588886.708419933.484b4d433f831852-ac3a3492b722f1a1 be.INFO.log.2024101210:I1012 10:23:54.072273 152367 schema_change.cpp:688] finished alter tablet process, res=[E-100]fail to create hard link. from=/data/be/ssd9/data/438/7326299/572003619/020000000084b6c3624d66f49b006ec639ef0262d9e42a94_0.dat, to=/data/be/ssd2/data/712/9588886/708419933/020000000084be52624d66f49b006ec639ef0262d9e42a94_0.dat, errno=18 be.INFO.log.2024101210:W1012 10:23:54.072279 152367 task_worker_pool.cpp:1821] failed to alter tablet|signature=9588886|base_tablet_id=7326299|new_tablet_id=9588886|error=[E-100]fail to create hard link. from=/data/be/ssd9/data/438/7326299/572003619/020000000084b6c3624d66f49b006ec639ef0262d9e42a94_0.dat, to=/data/be/ssd2/data/712/9588886/708419933/020000000084be52624d66f49b006ec639ef0262d9e42a94_0.dat, errno=18 ``` ### What You Expected? any way to solve this problem? ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
