lyh20093867 opened a new issue, #39043: URL: https://github.com/apache/doris/issues/39043
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris-1.2.5-rc01(AVX2) RELEASE (build file://VM-10-6-centos@Unknown) ### What's Wrong? be crash frequently,almost one time per hour ------------------- start time: Wed Aug 7 15:13:39 CST 2024 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/be/lib/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] F0807 15:13:40.352217 102351 doris_main.cpp:408] fail to open StorageEngine, res=[INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage; *** Check failure stack trace: *** @ 0x55b0a223ccad google::LogMessage::Fail() @ 0x55b0a223f1e9 google::LogMessage::SendToLog() @ 0x55b0a223c816 google::LogMessage::Flush() @ 0x55b0a223f859 google::LogMessageFatal::~LogMessageFatal() @ 0x55b09c453a8b main @ 0x7f8760cfc555 __libc_start_main @ 0x55b09c45202a _start @ (nil) (unknown) *** Query id: 0-0 *** *** Aborted at 1723014820 (unix time) try "date -d @1723014820" if you are using GNU date *** *** Current BE git commitID: Unknown *** *** SIGABRT unkown detail explain (@0x3e800018fcf) received by PID 102351 (TID 0x7f87616dcb40) from PID 102351; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007F8760D10400 in /lib64/libc.so.6 2# gsignal in /lib64/libc.so.6 3# abort in /lib64/libc.so.6 4# 0x000055B0A2247699 in /opt/module/be/lib/doris_be 5# 0x000055B0A223CCAD in /opt/module/be/lib/doris_be 6# google::LogMessage::SendToLog() in /opt/module/be/lib/doris_be 7# google::LogMessage::Flush() in /opt/module/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /opt/module/be/lib/doris_be 9# main at /root/doris/be/src/service/doris_main.cpp:428 10# __libc_start_main in /lib64/libc.so.6 11# _start in /opt/module/be/lib/doris_be start time: Wed Aug 7 15:14:02 CST 2024 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/be/lib/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] *** Query id: fb1e59d3c354575-87ffc98a44c1f8b1 *** *** Aborted at 1723018291 (unix time) try "date -d @1723018291" if you are using GNU date *** *** Current BE git commitID: Unknown *** *** SIGSEGV address not mapped to object (@0x0) received by PID 104066 (TID 0x7efc645df700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# os::Linux::chained_handler(int, siginfo*, void*) in /opt/module//openjdk/jre/lib/amd64/server/[libjvm.so](http://libjvm.so/) 2# JVM_handle_linux_signal in /opt/module//openjdk/jre/lib/amd64/server/[libjvm.so](http://libjvm.so/) 3# signalHandler(int, siginfo*, void*) in /opt/module//openjdk/jre/lib/amd64/server/[libjvm.so](http://libjvm.so/) 4# 0x00007EFD7FF3E400 in /lib64/libc.so.6 5# je_arena_dalloc_promoted at ../src/arena.c:1604 6# __GI__dl_deallocate_tls in /lib64/ld-linux-x86-64.so.2 7# __free_stacks in /lib64/libpthread.so.0 8# __deallocate_stack in /lib64/libpthread.so.0 9# pthread_join in /lib64/libpthread.so.0 10# std::thread::join() at ../../../../../libstdc++-v3/src/c++11/[thread.cc:114](http://thread.cc:114/) 11# doris::ThreadGroup::join_all() in /opt/module/be/lib/doris_be 12# doris::KafkaDataConsumerGroup::start_all(doris::StreamLoadContext*) at /root/doris/be/src/runtime/routine_load/data_consumer_group.cpp:141 13# doris::RoutineLoadTaskExecutor::exec_task(doris::StreamLoadContext*, doris::DataConsumerPool*, std::function<void (doris::StreamLoadContext*)>) at /root/doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:306 14# std::_Function_handler<void (), std::_Bind_result<void, void (doris::RoutineLoadTaskExecutor::*(doris::RoutineLoadTaskExecutor*, doris::StreamLoadContext*, doris::DataConsumerPool*, doris::RoutineLoadTaskExecutor::submit_task(doris::TRoutineLoadTask const&)::$_0))(doris::StreamLoadContext*, doris::DataConsumerPool*, std::function<void (doris::StreamLoadContext*)>)> >::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 15# doris::PriorityThreadPool::work_thread(int) at /root/doris/be/src/util/priority_thread_pool.hpp:146 16# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/[thread.cc:84](http://thread.cc:84/) 17# start_thread in /lib64/libpthread.so.0 18# clone in /lib64/libc.so.6 start time: Wed Aug 7 16:11:32 CST 2024 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/be/lib/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] F0807 16:11:35.432806 12745 doris_main.cpp:408] fail to open StorageEngine, res=[INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage; *** Check failure stack trace: *** @ 0x555669752cad google::LogMessage::Fail() @ 0x5556697551e9 google::LogMessage::SendToLog() @ 0x555669752816 google::LogMessage::Flush() @ 0x555669755859 google::LogMessageFatal::~LogMessageFatal() @ 0x555663969a8b main @ 0x7fdaa7f5e555 __libc_start_main @ 0x55566396802a _start @ (nil) (unknown) *** Query id: 0-0 *** *** Aborted at 1723018295 (unix time) try "date -d @1723018295" if you are using GNU date *** *** Current BE git commitID: Unknown *** *** SIGABRT unkown detail explain (@0x3e8000031c9) received by PID 12745 (TID 0x7fdaa893eb40) from PID 12745; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007FDAA7F72400 in /lib64/libc.so.6 2# gsignal in /lib64/libc.so.6 3# abort in /lib64/libc.so.6 4# 0x000055566975D699 in /opt/module/be/lib/doris_be 5# 0x0000555669752CAD in /opt/module/be/lib/doris_be 6# google::LogMessage::SendToLog() in /opt/module/be/lib/doris_be 7# google::LogMessage::Flush() in /opt/module/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /opt/module/be/lib/doris_be 9# main at /root/doris/be/src/service/doris_main.cpp:428 10# __libc_start_main in /lib64/libc.so.6 11# _start in /opt/module/be/lib/doris_be --------------------------------------------- logs in be.INFO I0807 15:13:40.349098 103013 data_dir.cpp:745] path: /data1/doris/be/storage total capacity: 7999426224128, available capacity: 7905475198976 I0807 15:13:40.349145 103014 data_dir.cpp:745] path: /data2/doris/be/storage total capacity: 7999426224128, available capacity: 7907773960192 I0807 15:13:40.349195 103013 data_dir.cpp:192] path: /data1/doris/be/storage, hash: 8911238614356524183 I0807 15:13:40.349210 103014 data_dir.cpp:192] path: /data2/doris/be/storage, hash: 8825406747811533581 W0807 15:13:40.351935 103013 olap_meta.cpp:80] rocks db open failed, reason:IO error: While lock file: /data1/doris/be/storage/meta/LOCK: Resource temporarily unavailable W0807 15:13:40.351981 103014 olap_meta.cpp:80] rocks db open failed, reason:IO error: While lock file: /data2/doris/be/storage/meta/LOCK: Resource temporarily unavailable W0807 15:13:40.351994 103013 data_dir.cpp:204] init OlapMeta failed, error: [IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage W0807 15:13:40.352031 103013 data_dir.cpp:115] _init_meta failed, error: [IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage W0807 15:13:40.352030 103014 data_dir.cpp:204] init OlapMeta failed, error: [IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage W0807 15:13:40.352039 103013 storage_engine.cpp:208] Store load failed, status=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage, path=/data1/doris/be/storage W0807 15:13:40.352041 103014 data_dir.cpp:115] _init_meta failed, error: [IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage W0807 15:13:40.352072 103014 storage_engine.cpp:208] Store load failed, status=[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage, path=/data2/doris/be/storage W0807 15:13:40.352162 102351 storage_engine.cpp:172] _init_store_map failed, error: [INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage; W0807 15:13:40.352175 102351 storage_engine.cpp:99] open engine failed, error: [INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage; F0807 15:13:40.352217 102351 doris_main.cpp:408] fail to open StorageEngine, res=[INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage; ### What You Expected? don't crash ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
