MoanasDaddyXu opened a new issue, #66368: URL: https://github.com/apache/doris/issues/66368
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues. ### Version Reproduced on `doris-0.0.0-1590e7e3228` in a non-cloud deployment with 1 FE and 3 BEs. Regression case: ```text regression-test/suites/stress/job_and_mtmv/ivm_ab/ivm_performance_scale_matrix_test.groovy ``` ### What's Wrong? A 107-column UNIQUE KEY Merge-on-Write table was configured with ROW binlog and historical values. A large Broker Load completed successfully in the first round, but a second load of the same source data with a different business-key prefix was cancelled after the BE hit an internal copy-on-write assertion: ```text [INTERNAL_ERROR][E6] COW::assert_mutable: use_count() > 1 ``` The first failure occurred while preparing ROW binlog source data. The same assertion was later observed in the normal vertical segment flush path. This is not a regression-test assertion failure: the load itself entered `CANCELLED` after the BE returned the internal error. ### What You Expected? A valid UNIQUE KEY MOW + ROW binlog Broker Load should complete successfully. If the workload exceeds a supported resource limit, Doris should return a clear resource-limit error instead of violating the copy-on-write ownership invariant in `IColumn::mutate()`. ### How to Reproduce? Create a wide UNIQUE KEY MOW table with the following relevant properties: ```text enable_unique_key_merge_on_write = true binlog.enable = true binlog.format = ROW binlog.need_historical_value = true inverted_index_storage_format = V3 AUTO PARTITION BY MONTH DISTRIBUTED BY HASH(trace_id) BUCKETS 128 ``` Run two large Broker Loads against the table. The reproduced workload used 1,000 source files and the following load settings: ```text load_parallelism = 3 exec_mem_limit = 8589934592 timeout = 43200 max_filter_ratio = 0.1 ``` Use a different business-key prefix in the second round so that it does not overwrite rows from the first round. Current reproduction scale: ```text round 1: ScannedRows = 2,500,000,000, succeeded round 2: ScannedRows = 1,052,674,279, cancelled round 2: LoadBytes = 906,586,501,986 round 2: FilteredRows = 0 ``` The issue has reproduced at approximately 1 TiB input scale. A smaller standalone reproduction has not been identified yet. ### Regression Test Result Primary stack: ```text doris::IColumn::mutate() const && at ../src/core/cow.h:325 doris::OlapBlockDataConvertor::OlapColumnDataConvertorSimple<...>::convert_to_olap() doris::OlapBlockDataConvertor::convert_column_data(unsigned long) doris::segment_v2::RowBinlogSourceDataWriter::prepare_by_source_block(...) doris::segment_v2::RowBinlogSegmentWriter::append_block(...) doris::SegmentFlusher::flush_single_block(...) doris::SegmentCreator::flush_single_block(...) doris::BaseBetaRowsetWriter::flush_memtable(...) doris::FlushToken::_flush_memtable_impl(...) doris::FlushToken::_flush_group_memtable(...) doris::PartOfGroupMemtableFlushTask::run() doris::ThreadPool::dispatch_thread() ``` The assertion was also observed through the regular segment writer path: ```text doris::IColumn::mutate() doris::OlapBlockDataConvertorSimple<...>::convert_to_olap() doris::segment_v2::VerticalSegmentWriter::write_batch() doris::SegmentFlusher::_add_rows() doris::SegmentFlusher::flush_single_block() ``` ### Initial Analysis The first load used the same schema, data source, field mapping, and load configuration and completed successfully. The failed load had `FilteredRows = 0`. All BEs remained alive, disk usage was about 50%, and the cluster still had available memory; there was no OOM, no-space error, node loss, or connection timeout. Under large concurrent memtable flushes, the ROW binlog source block and the regular segment writer may share a column object that is later mutated in place. This would cause `IColumn::mutate() const &&` to observe `use_count() > 1`. Please check the ownership and clone/mutate boundaries among: ```text RowBinlogSourceDataWriter::prepare_by_source_block shared/group memtable flush OlapBlockDataConvertorSimple VerticalSegmentWriter ``` ### Anything Else? Tracking issues: - #65265 - #65418 The remaining reproduction work is to reduce the data size and isolate the effect of bucket count, ROW binlog, and inverted indexes. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
