ShummGen commented on issue #63609: URL: https://github.com/apache/doris/issues/63609#issuecomment-5450553533
[doris-core-backtrace.txt](https://github.com/user-attachments/files/31548105/doris-core-backtrace.txt) [doris-coredump-info.txt](https://github.com/user-attachments/files/31548104/doris-coredump-info.txt) Hi, Thank you for the detailed investigation and for narrowing down the specific state that needs to be inspected. A clarification: I have now encountered another BE crash on August 28, 2026 at 16:29:06 CST. The BE is still running the exact revision: b10073ad9ca17cd5685c4dd3b3ef650f256376d0 systemd recorded the crash as: Main process exited, code=dumped, status=11/SEGV Failed with result 'core-dump' The matching doris_be binary is still available. The core is also retained by systemd-coredump, although it is marked as truncated and reported as approximately 3.0 GB. Before this crash, the BE repeatedly reported storage read errors, including: - Bad page: checksum mismatch - ZSTD_decompressDCtx error: Restored data doesn't match checksum These errors occurred several times before the SIGSEGV. However, I agree that their temporal proximity alone does not establish that they directly caused the crash, so I will treat that relationship as unconfirmed. I will use the matching binary and core to collect the minimal GDB output you requested: set pagination off bt If the backtrace contains a ColumnDictI32::filter_by_selector frame, I will also provide: frame <N> info args info locals I will redact any sensitive information before posting the output. If the truncated core prevents GDB from recovering the relevant frame or variables, I would prefer to provide the core and matching binary through a private transfer channel recommended by a Doris maintainer rather than upload them publicly. I will post the GDB output in the next update. There is one additional issue on which I would appreciate your advice. Backend 192.168.1.203 repeatedly reports corrupted storage pages, while the other two backends, 192.168.1.201 and 192.168.1.202, have not shown the same problem. This does not appear to be limited to one query or one tablet. For example: 1. On August 28, tablet 1785483752796 reported: Bad page: checksum mismatch (actual=1158281431 vs expect=1638974542) The affected file was: /home/doris_test_local/be-storage/data/105/1785483752796/1825665516/0200000000004b520242299576392433df7d0ab6f284e5b6_0.dat 2. On August 27, a different tablet, 1785483752896, repeatedly failed during cumulative compaction with: ZSTD_decompressDCtx error: Restored data doesn't match checksum So far, these corruption errors have only been observed on backend 192.168.1.203. I am investigating the host hardware separately, including SMART data, kernel I/O errors, filesystem errors, and memory errors. Around the August 28 crash, the kernel log did not report OOM or an obvious disk I/O error. Could you please advise on the Doris-specific investigation and recovery procedure? In particular: - Is there a Doris-supported way to scan or verify all local tablets/rowsets on one backend for corrupted pages? - What is the safest way to identify and rebuild the corrupted replicas from healthy replicas on the other backends? - Should I mark individual replicas as bad, or would it be safer to decommission and rebuild the entire backend if corruption exists across multiple tablets? - Are there additional BE logs, HTTP APIs, tablet metadata, checksum tools, or configuration values that would help distinguish corruption introduced during writing from corruption caused later by disk, filesystem, memory, or page-cache problems? - Are there any known issues in revision b10073ad9ca17cd5685c4dd3b3ef650f256376d0 that could produce corrupted segment pages, especially during compaction? - Before rebuilding the replicas, is there any metadata or affected segment file that would be useful for maintainers to inspect? I will not delete or modify the affected .dat files manually. I would prefer to follow the supported recovery procedure and preserve any useful diagnostic evidence first. The systemd restart counter for this BE is also currently 2370. I am checking the historical crash records to determine how many of those restarts were SIGSEGV crashes and whether they correlate with storage corruption errors. My current workaround is as follows: 1. When a query encounters a checksum or ZSTD corruption error, I identify the affected tablet and replica on backend 192.168.1.203. 2. I use the Doris-supported replica status operation to mark that replica as bad. 3. I wait for Doris to repair the replica by cloning it from a healthy replica on backend 192.168.1.201 or 192.168.1.202. 4. After the repair completes, the same query can normally run successfully again. I do not manually delete or modify the segment files. This workaround confirms that the replicas on the other two backends are generally healthy and that rebuilding the replica on 192.168.1.203 temporarily resolves the query failure. However, the corruption is not an isolated event. It is extremely frequent on this backend. New data written every day, and sometimes data written within the same hour, can later produce bad-page checksum or ZSTD checksum errors on 192.168.1.203. I then have to repeat the same mark-bad and replica-repair procedure. Therefore, repairing individual replicas is only a temporary workaround. New corrupted replicas continue to appear on the same backend. Given this frequency, could you please advise whether I should immediately disable query/load on this backend and decommission/rebuild the entire backend instead of continuing to repair individual replicas? I would also like to know: - Which evidence should I preserve before repairing the next corrupted replica? - Is there a way to determine whether the page was already corrupted when Doris wrote it, or whether it became corrupted later? - Can Doris verify the segment immediately after writing or compaction? - Are there metrics or logs that can identify which write, flush, compaction, or clone operation created the affected segment? - Could a memory or page-cache problem on this host cause Doris to write corrupted segment data even when SMART and kernel logs report no disk I/O error? - Would you recommend replacing or fully testing the storage and memory before adding this backend back to the cluster? Because the problem affects newly written data and only occurs on 192.168.1.203, I currently suspect an ongoing host-local storage, filesystem, memory, or I/O-path problem rather than a few historical damaged files. However, I would appreciate guidance on how to confirm this from the Doris side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
