Formula24Code opened a new issue, #548: URL: https://github.com/apache/doris-flink-connector/issues/548
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version flink-doris-connector： ``` <dependency> <groupId>org.apache.doris</groupId> <artifactId>flink-doris-connector-1.14_2.11</artifactId> <version>1.1.1</version> </dependency> ``` doris版本是2.1.0-3.0.3（2.1.0以上每个版本都会出现这样的情况） ### What's Wrong? 当我使用flink-doris-connector-1.14连接doris2.1以上版本时，只要一启动flink，就算没有任何数据写入，BE的CPU使用率也很高 以下是flink没有任何数据写入时，doris2.1以上版本BE集群节点`top -H`命令的结果：  ### What You Expected? JAVA代码未做任何改动，这是我把doris降到2.0.15的BE集群节点`top -H`命令的结果：  ### How to Reproduce? 以下是我的DorisSink代码配置（sink并行度是10，checkpoint.interval=60000）： ```java DorisSink.Builder<String> builder = DorisSink.builder(); DorisOptions.Builder dorisBuilder = DorisOptions.builder(); dorisBuilder.setFenodes(properties.getRequired(ConfigProperty.DORIS_FE_HOST)) .setTableIdentifier(dorisSinkTable) .setUsername(properties.getRequired(ConfigProperty.DORIS_USER_NAME)) .setPassword(properties.getRequired(ConfigProperty.DORIS_PASSWORD)); Properties pro = new Properties(); pro.setProperty("format", "json"); pro.setProperty("read_json_by_line", "true"); DorisExecutionOptions executionOptions = DorisExecutionOptions.builder() .setLabelPrefix(properties.getRequired(ConfigProperty.DORIS_LABEL_PREFIX) + "_" + UUID.randomUUID()) .setStreamLoadProp(pro).build(); builder.setDorisReadOptions(DorisReadOptions.builder().build()) .setDorisExecutionOptions(executionOptions) .setSerializer(new SimpleStringSerializer()) .setDorisOptions(dorisBuilder.build()); return builder.build(); ``` ### Anything Else? 这是be.INFO截取一分钟的内容（doris版本2.1.7）： ``` I20250115 17:33:29.074241 124906 topic_subscriber.cpp:43] [topic_publish]begin handle topic info I20250115 17:33:29.074278 124906 topic_subscriber.cpp:46] [topic_publish]begin handle topic WORKLOAD_GROUP, size=1 I20250115 17:33:29.074291 124906 workload_group_listener.cpp:65] [topic_publish_wg]update workload group finish, wg info=TG[id = 1, name = normal, cpu_share = 1024, memory_limit = 3.88 GB, enable_memory_overcommit = true, version = 0, cpu_hard_limit = -1, scan_thread_num = 48, max_remote_scan_thread_num = 512, min_remote_scan_thread_num = 8, spill_low_watermark=50, spill_high_watermark=80, is_shutdown=false, query_num=1, read_bytes_per_second=-1, remote_read_bytes_per_second=-1], enable_cpu_hard_limit=false, cgroup cpu_shares=0, cgroup cpu_hard_limit=0, cgroup home path=, list size=1, thread info=[exec num:8, real_num:8, min_num:8, max_num:8],[l_scan num:48, real_num:48, min_num:48, max_num48],[r_scan num:8, real_num:8, min_num:8, max_num:512],[mem_tab_flush num:6, real_num:6, min_num:6, max_num:6] I20250115 17:33:29.074312 124906 workload_group_manager.cpp:133] [topic_publish_wg]finish clear unused workload group, time cost: 0 ms, deleted group size:0, before wg size=1, after wg size=1 I20250115 17:33:29.074316 124906 topic_subscriber.cpp:50] [topic_publish]finish handle topic WORKLOAD_GROUP I20250115 17:33:29.074317 124906 topic_subscriber.cpp:46] [topic_publish]begin handle topic WORKLOAD_SCHED_POLICY, size=0 I20250115 17:33:29.074321 124906 workload_sched_policy_listener.cpp:79] [workload_schedule]finish update workload schedule policy, size=0 I20250115 17:33:29.074323 124906 topic_subscriber.cpp:50] [topic_publish]finish handle topic WORKLOAD_SCHED_POLICY I20250115 17:33:29.113476 123123 compaction.cpp:364] start cumulative compaction. tablet=14603, output_version=[16-40], permits: 5 I20250115 17:33:29.113554 123123 merger.cpp:423] Start to do vertical compaction, tablet_id: 14603 I20250115 17:33:29.113560 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14603 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:29.118112 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14603 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:29.119540 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14603 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:29.122074 123123 tablet_meta_manager.cpp:293] remove old version delete bitmap, tablet_id: 14603 version: 40 removed keys size: 0 I20250115 17:33:29.122205 123123 compaction.cpp:751] succeed to do cumulative compaction is_vertical=1. tablet=14603, output_version=[16-40], current_max_version=40, disk=/data/doris/be_storage, segments=5, input_rowset_size=24427, output_rowset_size=6399, input_row_num=77, output_row_num=44, filtered_row_num=0, merged_row_num=33. elapsed time=0.008746s. cumulative_compaction_policy=size_based, compact_row_per_second=8800 I20250115 17:33:29.131130 123123 compaction.cpp:364] start cumulative compaction. tablet=14605, output_version=[16-40], permits: 5 I20250115 17:33:29.131183 123123 merger.cpp:423] Start to do vertical compaction, tablet_id: 14605 I20250115 17:33:29.131209 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14605 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:29.135079 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14605 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:29.136788 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14605 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:29.139050 123123 tablet_meta_manager.cpp:293] remove old version delete bitmap, tablet_id: 14605 version: 40 removed keys size: 0 I20250115 17:33:29.139092 123123 compaction.cpp:751] succeed to do cumulative compaction is_vertical=1. tablet=14605, output_version=[16-40], current_max_version=40, disk=/data/doris/be_storage, segments=5, input_rowset_size=21864, output_rowset_size=5767, input_row_num=60, output_row_num=39, filtered_row_num=0, merged_row_num=21. elapsed time=0.007971s. cumulative_compaction_policy=size_based, compact_row_per_second=7521 I20250115 17:33:31.168679 123123 compaction.cpp:364] start cumulative compaction. tablet=14607, output_version=[16-40], permits: 5 I20250115 17:33:31.168783 123123 merger.cpp:423] Start to do vertical compaction, tablet_id: 14607 I20250115 17:33:31.168792 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14607 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.173969 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14607 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.175699 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14607 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.177946 123123 tablet_meta_manager.cpp:293] remove old version delete bitmap, tablet_id: 14607 version: 40 removed keys size: 0 I20250115 17:33:31.178022 123123 compaction.cpp:751] succeed to do cumulative compaction is_vertical=1. tablet=14607, output_version=[16-40], current_max_version=40, disk=/data/doris/be_storage, segments=5, input_rowset_size=22746, output_rowset_size=5651, input_row_num=59, output_row_num=32, filtered_row_num=0, merged_row_num=27. elapsed time=0.009357s. cumulative_compaction_policy=size_based, compact_row_per_second=6300 I20250115 17:33:31.185534 123123 compaction.cpp:364] start cumulative compaction. tablet=14609, output_version=[16-40], permits: 5 I20250115 17:33:31.185592 123123 merger.cpp:423] Start to do vertical compaction, tablet_id: 14609 I20250115 17:33:31.185598 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14609 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.187340 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14609 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.188036 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14609 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.191551 123123 tablet_meta_manager.cpp:293] remove old version delete bitmap, tablet_id: 14609 version: 40 removed keys size: 0 I20250115 17:33:31.191594 123123 compaction.cpp:751] succeed to do cumulative compaction is_vertical=1. tablet=14609, output_version=[16-40], current_max_version=40, disk=/data/doris/be_storage, segments=5, input_rowset_size=23963, output_rowset_size=6549, input_row_num=80, output_row_num=44, filtered_row_num=0, merged_row_num=36. elapsed time=0.006071s. cumulative_compaction_policy=size_based, compact_row_per_second=13164 I20250115 17:33:31.202896 123123 compaction.cpp:364] start cumulative compaction. tablet=14611, output_version=[16-40], permits: 5 I20250115 17:33:31.202950 123123 merger.cpp:423] Start to do vertical compaction, tablet_id: 14611 I20250115 17:33:31.202956 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14611 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.205492 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14611 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.206312 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14611 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.209254 123123 tablet_meta_manager.cpp:293] remove old version delete bitmap, tablet_id: 14611 version: 40 removed keys size: 0 I20250115 17:33:31.209304 123123 compaction.cpp:751] succeed to do cumulative compaction is_vertical=1. tablet=14611, output_version=[16-40], current_max_version=40, disk=/data/doris/be_storage, segments=5, input_rowset_size=22423, output_rowset_size=5749, input_row_num=67, output_row_num=37, filtered_row_num=0, merged_row_num=30. elapsed time=0.006417s. cumulative_compaction_policy=size_based, compact_row_per_second=10432 I20250115 17:33:31.219981 123123 compaction.cpp:364] start cumulative compaction. tablet=14613, output_version=[16-40], permits: 5 I20250115 17:33:31.220053 123123 merger.cpp:423] Start to do vertical compaction, tablet_id: 14613 I20250115 17:33:31.220058 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14613 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.221055 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14613 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.223281 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14613 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.226034 123123 tablet_meta_manager.cpp:293] remove old version delete bitmap, tablet_id: 14613 version: 40 removed keys size: 0 I20250115 17:33:31.226079 123123 compaction.cpp:751] succeed to do cumulative compaction is_vertical=1. tablet=14613, output_version=[16-40], current_max_version=40, disk=/data/doris/be_storage, segments=5, input_rowset_size=24077, output_rowset_size=6570, input_row_num=93, output_row_num=45, filtered_row_num=0, merged_row_num=48. elapsed time=0.006104s. cumulative_compaction_policy=size_based, compact_row_per_second=15223 I20250115 17:33:31.236987 123123 compaction.cpp:364] start cumulative compaction. tablet=14615, output_version=[16-40], permits: 5 I20250115 17:33:31.237037 123123 merger.cpp:423] Start to do vertical compaction, tablet_id: 14615 I20250115 17:33:31.237043 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14615 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.239810 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14615 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.240382 123123 merger.cpp:389] estimate batch size for vertical compaction, tablet id: 14615 group data size: 0 row num: 0 consume bytes: 0 I20250115 17:33:31.244463 123123 tablet_meta_manager.cpp:293] remove old version delete bitmap, tablet_id: 14615 version: 40 removed keys size: 0 I20250115 17:33:31.244580 123123 compaction.cpp:751] succeed to do cumulative compaction is_vertical=1. tablet=14615, output_version=[16-40], current_max_version=40, disk=/data/doris/be_storage, segments=5, input_rowset_size=21429, output_rowset_size=5696, input_row_num=55, output_row_num=34, filtered_row_num=0, merged_row_num=21. elapsed time=0.007605s. cumulative_compaction_policy=size_based, compact_row_per_second=7229 I20250115 17:33:31.725417 123116 storage_engine.cpp:1127] start to delete unused rowset, size: 0 I20250115 17:33:31.725473 123116 storage_engine.cpp:1161] collected 0 unused rowsets to remove, skipped 0 rowsets due to use count > 1, skipped 0 rowsets due to don't need to delete file, skipped 0 rowsets due to delayed expired timestamp. I20250115 17:33:31.725476 123116 storage_engine.cpp:1177] removed all collected unused rowsets I20250115 17:33:32.994334 123008 wal_manager.cpp:486] Scheduled(every 10s) WAL info: [/data/doris/be_storage/wal: limit 25234618777 Bytes, used 0 Bytes, estimated wal bytes 0 Bytes, available 25234618777 Bytes.]; I20250115 17:33:35.183362 123933 workload_group_manager.cpp:192] Process Memory Summary: process memory used 2.29 GB(= 2.55 GB[vm/rss] - 264.12 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), sys available memory 7.84 GB(= 7.84 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), all workload groups memory usage: 1.02 MB, weighted_memory_limit_ratio: 0.8231815463325404 I20250115 17:33:43.039393 123008 wal_manager.cpp:486] Scheduled(every 10s) WAL info: [/data/doris/be_storage/wal: limit 25234618777 Bytes, used 0 Bytes, estimated wal bytes 0 Bytes, available 25234618777 Bytes.]; I20250115 17:33:43.052620 123153 olap_server.cpp:1165] cooldown producer get tablet num: 0 I20250115 17:33:45.228868 123933 workload_group_manager.cpp:192] Process Memory Summary: process memory used 2.29 GB(= 2.55 GB[vm/rss] - 260.80 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), sys available memory 7.84 GB(= 7.84 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), all workload groups memory usage: 1.02 MB, weighted_memory_limit_ratio: 0.8229555768688989 I20250115 17:33:53.090801 123008 wal_manager.cpp:486] Scheduled(every 10s) WAL info: [/data/doris/be_storage/wal: limit 25234618777 Bytes, used 0 Bytes, estimated wal bytes 0 Bytes, available 25234618777 Bytes.]; I20250115 17:33:55.230608 123933 workload_group_manager.cpp:192] Process Memory Summary: process memory used 2.29 GB(= 2.55 GB[vm/rss] - 260.41 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), sys available memory 7.84 GB(= 7.84 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), all workload groups memory usage: 1.02 MB, weighted_memory_limit_ratio: 0.8229302703886354 I20250115 17:33:59.077059 124906 topic_subscriber.cpp:43] [topic_publish]begin handle topic info I20250115 17:33:59.077091 124906 topic_subscriber.cpp:46] [topic_publish]begin handle topic WORKLOAD_GROUP, size=1 I20250115 17:33:59.077102 124906 workload_group_listener.cpp:65] [topic_publish_wg]update workload group finish, wg info=TG[id = 1, name = normal, cpu_share = 1024, memory_limit = 3.88 GB, enable_memory_overcommit = true, version = 0, cpu_hard_limit = -1, scan_thread_num = 48, max_remote_scan_thread_num = 512, min_remote_scan_thread_num = 8, spill_low_watermark=50, spill_high_watermark=80, is_shutdown=false, query_num=1, read_bytes_per_second=-1, remote_read_bytes_per_second=-1], enable_cpu_hard_limit=false, cgroup cpu_shares=0, cgroup cpu_hard_limit=0, cgroup home path=, list size=1, thread info=[exec num:8, real_num:8, min_num:8, max_num:8],[l_scan num:48, real_num:48, min_num:48, max_num48],[r_scan num:8, real_num:8, min_num:8, max_num:512],[mem_tab_flush num:6, real_num:6, min_num:6, max_num:6] I20250115 17:33:59.077122 124906 workload_group_manager.cpp:133] [topic_publish_wg]finish clear unused workload group, time cost: 0 ms, deleted group size:0, before wg size=1, after wg size=1 I20250115 17:33:59.077126 124906 topic_subscriber.cpp:50] [topic_publish]finish handle topic WORKLOAD_GROUP I20250115 17:33:59.077128 124906 topic_subscriber.cpp:46] [topic_publish]begin handle topic WORKLOAD_SCHED_POLICY, size=0 I20250115 17:33:59.077131 124906 workload_sched_policy_listener.cpp:79] [workload_schedule]finish update workload schedule policy, size=0 I20250115 17:33:59.077132 124906 topic_subscriber.cpp:50] [topic_publish]finish handle topic WORKLOAD_SCHED_POLICY I20250115 17:34:01.257515 123007 load_channel_mgr.cpp:216] cleaning timed out load channels I20250115 17:34:01.725680 123116 storage_engine.cpp:1127] start to delete unused rowset, size: 0 I20250115 17:34:01.725734 123116 storage_engine.cpp:1161] collected 0 unused rowsets to remove, skipped 0 rowsets due to use count > 1, skipped 0 rowsets due to don't need to delete file, skipped 0 rowsets due to delayed expired timestamp. I20250115 17:34:01.725739 123116 storage_engine.cpp:1177] removed all collected unused rowsets I20250115 17:34:03.065654 123153 olap_server.cpp:1165] cooldown producer get tablet num: 0 I20250115 17:34:03.140532 123008 wal_manager.cpp:486] Scheduled(every 10s) WAL info: [/data/doris/be_storage/wal: limit 25234618777 Bytes, used 0 Bytes, estimated wal bytes 0 Bytes, available 25234618777 Bytes.]; I20250115 17:34:05.274778 123933 workload_group_manager.cpp:192] Process Memory Summary: process memory used 2.29 GB(= 2.55 GB[vm/rss] - 264.64 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), sys available memory 7.84 GB(= 7.84 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), all workload groups memory usage: 1.02 MB, weighted_memory_limit_ratio: 0.8230149269193641 I20250115 17:34:06.325279 122786 vtablet_writer.cpp:973] VNodeChannel[149018-10151], load_id=347178cea5bb645-3e7cbbe88333649a, txn_id=74876, node=doris.node2:38060 mark closed, left pending batch size: 1 I20250115 17:34:06.325313 122786 vtablet_writer.cpp:973] VNodeChannel[149018-10150], load_id=347178cea5bb645-3e7cbbe88333649a, txn_id=74876, node=doris.node1:38060 mark closed, left pending batch size: 1 I20250115 17:34:06.325317 122786 vtablet_writer.cpp:973] VNodeChannel[149018-10152], load_id=347178cea5bb645-3e7cbbe88333649a, txn_id=74876, node=doris.node3:38060 mark closed, left pending batch size: 1 I20250115 17:34:06.325361 123422 tablets_channel.cpp:271] close tablets channel: (load_id=6545d160be25de91-ae5a950b5036c0b5, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.325382 123422 load_channel.cpp:219] txn 74881 closed tablets_channel 149018 I20250115 17:34:06.325431 123422 load_channel.cpp:71] load channel removed load_id=6545d160be25de91-ae5a950b5036c0b5, is high priority=0, sender_ip=doris.node2, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.326109 123425 tablets_channel.cpp:271] close tablets channel: (load_id=424549460a99c5b9-595a5bb5d2df4ebc, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.326133 123349 tablets_channel.cpp:271] close tablets channel: (load_id=d84becf81b38a43d-53d011fe261d65bd, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.326143 123425 load_channel.cpp:219] txn 74878 closed tablets_channel 149018 I20250115 17:34:06.326153 123349 load_channel.cpp:219] txn 74883 closed tablets_channel 149018 I20250115 17:34:06.326187 123349 load_channel.cpp:71] load channel removed load_id=d84becf81b38a43d-53d011fe261d65bd, is high priority=0, sender_ip=doris.node2, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.326195 123308 tablets_channel.cpp:271] close tablets channel: (load_id=114fd5aabe207e4d-61b7f07a7cdbf1a3, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.326210 123308 load_channel.cpp:219] txn 74880 closed tablets_channel 149018 I20250115 17:34:06.326246 123308 load_channel.cpp:71] load channel removed load_id=114fd5aabe207e4d-61b7f07a7cdbf1a3, is high priority=0, sender_ip=doris.node2, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.326272 123349 tablets_channel.cpp:271] close tablets channel: (load_id=8b4f1022d95abf3f-65cac47a96074b91, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.326295 123349 load_channel.cpp:219] txn 74879 closed tablets_channel 149018 I20250115 17:34:06.326326 123349 load_channel.cpp:71] load channel removed load_id=8b4f1022d95abf3f-65cac47a96074b91, is high priority=0, sender_ip=doris.node2, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.326332 123308 tablets_channel.cpp:271] close tablets channel: (load_id=0347178cea5bb645-3e7cbbe88333649a, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.326347 123308 load_channel.cpp:219] txn 74876 closed tablets_channel 149018 I20250115 17:34:06.326486 123308 load_channel.cpp:71] load channel removed load_id=0347178cea5bb645-3e7cbbe88333649a, is high priority=0, sender_ip=doris.node3, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.326376 123425 load_channel.cpp:71] load channel removed load_id=424549460a99c5b9-595a5bb5d2df4ebc, is high priority=0, sender_ip=doris.node2, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.326273 123559 vtablet_writer.cpp:1013] All node channels are stopped(maybe finished/offending/cancelled), sender thread exit. 347178cea5bb645-3e7cbbe88333649a I20250115 17:34:06.327868 123314 tablets_channel.cpp:271] close tablets channel: (load_id=8f4caf71e0458301-b9a3636a5fd01bb4, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.327890 123314 load_channel.cpp:219] txn 74877 closed tablets_channel 149018 I20250115 17:34:06.327930 123314 load_channel.cpp:71] load channel removed load_id=8f4caf71e0458301-b9a3636a5fd01bb4, is high priority=0, sender_ip=doris.node1, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.328035 123314 tablets_channel.cpp:271] close tablets channel: (load_id=0c40e926b8d42a65-7429714cb26e6fad, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.328047 123314 load_channel.cpp:219] txn 74882 closed tablets_channel 149018 I20250115 17:34:06.328071 123314 load_channel.cpp:71] load channel removed load_id=0c40e926b8d42a65-7429714cb26e6fad, is high priority=0, sender_ip=doris.node1, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.328542 122786 vtablet_writer.cpp:1584] total mem_exceeded_block_ns=0, total queue_push_lock_ns=0, total actual_consume_ns=93510, load id=347178cea5bb645-3e7cbbe88333649a I20250115 17:34:06.328560 122786 vtablet_writer.cpp:1631] finished to close olap table sink. load_id=347178cea5bb645-3e7cbbe88333649a, txn_id=74876, node add batch time(ms)/wait execution time(ms)/close time(ms)/num: {10152:(0)(0)(3)(1)} {10150:(0)(0)(3)(1)} {10151:(0)(0)(3)(1)} I20250115 17:34:06.329384 124030 fragment_mgr.cpp:730] Removing query 347178cea5bb645-3e7cbbe88333649a instance 347178cea5bb645-3e7cbbe88333649b, all done? true I20250115 17:34:06.329401 124030 fragment_mgr.cpp:736] Query 347178cea5bb645-3e7cbbe88333649a finished I20250115 17:34:06.329421 124030 query_context.cpp:156] Query 347178cea5bb645-3e7cbbe88333649a deconstructed, , deregister query/load memory tracker, queryId=347178cea5bb645-3e7cbbe88333649a, Limit=2.00 GB, CurrUsed=1.72 MB, PeakUsed=1.72 MB I20250115 17:34:06.329488 124030 query_context.cpp:188] Query 347178cea5bb645-3e7cbbe88333649a deconstructed, , deregister query/load memory tracker, queryId=347178cea5bb645-3e7cbbe88333649a, Limit=2.00 GB, CurrUsed=1.72 MB, PeakUsed=1.72 MB I20250115 17:34:06.330439 123404 tablets_channel.cpp:271] close tablets channel: (load_id=0d4aec47bff79f45-270e3ed0edb7f4b4, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.330466 123404 load_channel.cpp:219] txn 74885 closed tablets_channel 149018 I20250115 17:34:06.330513 123404 load_channel.cpp:71] load channel removed load_id=0d4aec47bff79f45-270e3ed0edb7f4b4, is high priority=0, sender_ip=doris.node1, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.338513 123921 stream_load.cpp:137] finished to execute stream load. label=test_check_48b9cc3d-e864-4b72-8d3b-d3bbbdaefaee_6_7, txn_id=74876, query_id=0347178cea5bb645-3e7cbbe88333649a, load_cost_ms=59992, receive_data_cost_ms=0, read_data_cost_ms=0, write_data_cost_ms=59945, commit_and_publish_txn_cost_ms=0, number_total_rows=0, number_loaded_rows=0, receive_bytes=0, loaded_bytes=0 I20250115 17:34:06.341405 123921 stream_load.cpp:208] new income streaming load request.id=3f49eaecf9a9ca2c-970e35d37d9413b6, job_id=-1, txn_id=-1, label=test_check_48b9cc3d-e864-4b72-8d3b-d3bbbdaefaee_6_8, elapse(s)=0, db=apply_check, tbl=test_check, group_commit=0 I20250115 17:34:06.370451 123299 tablets_channel.cpp:271] close tablets channel: (load_id=8b417aae5c405064-9122442e5e70f0ae, index_id=149018), sender id: 0, backend id: 10152 I20250115 17:34:06.370515 123299 load_channel.cpp:219] txn 74884 closed tablets_channel 149018 I20250115 17:34:06.370566 123299 load_channel.cpp:71] load channel removed load_id=8b417aae5c405064-9122442e5e70f0ae, is high priority=0, sender_ip=doris.node1, index id: 149018, total_received_rows: 0, num_rows_filtered: 0 I20250115 17:34:06.373654 123921 stream_load_executor.cpp:72] begin to execute stream load. label=test_check_48b9cc3d-e864-4b72-8d3b-d3bbbdaefaee_6_8, txn_id=74886, query_id=3f49eaecf9a9ca2c-970e35d37d9413b6 I20250115 17:34:06.373703 123921 fragment_mgr.cpp:778] query_id: 3f49eaecf9a9ca2c-970e35d37d9413b6, coord_addr: TNetworkAddress(hostname=doris.node3, port=39020), total fragment num on current host: 0, fe process uuid: 0, query type: LOAD, report audit fe:TNetworkAddress(hostname=doris.node3, port=39020) I20250115 17:34:06.373885 123921 fragment_mgr.cpp:819] Query/load id: 3f49eaecf9a9ca2c-970e35d37d9413b6, use workload group: TG[id = 1, name = normal, cpu_share = 1024, memory_limit = 3.88 GB, enable_memory_overcommit = true, version = 0, cpu_hard_limit = -1, scan_thread_num = 48, max_remote_scan_thread_num = 512, min_remote_scan_thread_num = 8, spill_low_watermark=50, spill_high_watermark=80, is_shutdown=false, query_num=1, read_bytes_per_second=-1, remote_read_bytes_per_second=-1], is pipeline: 1 I20250115 17:34:06.373907 123921 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: 3f49eaecf9a9ca2c-970e35d37d9413b6 limit: 0 I20250115 17:34:06.373937 123921 pipeline_fragment_context.cpp:253] Preparing instance 3f49eaecf9a9ca2c-970e35d37d9413b6|3f49eaecf9a9ca2c-970e35d37d9413b7, backend_num 0 I20250115 17:34:06.375121 123921 stream_load.cpp:214] finished to handle HTTP header, id=3f49eaecf9a9ca2c-970e35d37d9413b6, job_id=-1, txn_id=74886, label=test_check_48b9cc3d-e864-4b72-8d3b-d3bbbdaefaee_6_8, elapse(s)=0 I20250115 17:34:06.375916 123484 tablets_channel.cpp:136] open tablets channel (load_id=bb48363c7a07ec3d-b1e635a61649ccbf, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.376015 123484 tablets_channel.cpp:165] txn 74890: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.376796 122786 vtablet_writer.cpp:127] init new node for instance 0, incremantal:0 I20250115 17:34:06.376813 122786 vtablet_writer.cpp:127] init new node for instance 0, incremantal:0 I20250115 17:34:06.376819 122786 vtablet_writer.cpp:127] init new node for instance 0, incremantal:0 I20250115 17:34:06.377524 123495 tablets_channel.cpp:136] open tablets channel (load_id=d24f1c59a9695523-b1269f83ef863eb2, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.377616 123495 tablets_channel.cpp:165] txn 74888: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.377703 123495 tablets_channel.cpp:136] open tablets channel (load_id=47432bff74e8a688-ced51e5e70e7ffa7, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.377769 123495 tablets_channel.cpp:165] txn 74891: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.377863 123496 tablets_channel.cpp:136] open tablets channel (load_id=3f49eaecf9a9ca2c-970e35d37d9413b6, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.377990 123496 tablets_channel.cpp:165] txn 74886: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.379349 123434 tablets_channel.cpp:136] open tablets channel (load_id=fb47c1be20e351ec-9a41697f6dead197, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.379544 123434 tablets_channel.cpp:165] txn 74889: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.379670 123505 tablets_channel.cpp:136] open tablets channel (load_id=a54146c50789f267-cc09ad4ef4e263a0, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.379819 123505 tablets_channel.cpp:165] txn 74887: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.394963 123501 tablets_channel.cpp:136] open tablets channel (load_id=e441db732937aa08-706cedfb802bef93, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.395112 123501 tablets_channel.cpp:165] txn 74892: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.397425 123509 tablets_channel.cpp:136] open tablets channel (load_id=374f7d993f6ddc3b-d520fa461f262395, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.397567 123509 tablets_channel.cpp:165] txn 74893: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.401214 123429 tablets_channel.cpp:136] open tablets channel (load_id=a041e91ee1beda20-0c3ec04d64eb7a9b, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.401330 123429 tablets_channel.cpp:165] txn 74894: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.407635 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74876 I20250115 17:34:06.408002 123269 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74876, cost(us): 18, error_tablet_size=0, res=[OK] I20250115 17:34:06.408027 123269 task_worker_pool.cpp:1614] successfully publish version|signature=74876|transaction_id=74876|tablets_num=0|cost(s)=0 I20250115 17:34:06.420212 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74883 I20250115 17:34:06.420607 123269 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74883, cost(us): 22, error_tablet_size=0, res=[OK] I20250115 17:34:06.420632 123269 task_worker_pool.cpp:1614] successfully publish version|signature=74883|transaction_id=74883|tablets_num=0|cost(s)=0 I20250115 17:34:06.451469 123427 tablets_channel.cpp:136] open tablets channel (load_id=9d4e982a84f3a311-d7840f5e254125a1, index_id=149018), tablets num: 4 timeout(s): 259200 I20250115 17:34:06.451609 123427 tablets_channel.cpp:165] txn 74895: TabletsChannel of index 149018 init senders 1 with incremental off I20250115 17:34:06.492211 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74885 I20250115 17:34:06.492273 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74877 I20250115 17:34:06.492280 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74880 I20250115 17:34:06.492285 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74879 I20250115 17:34:06.492291 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74878 I20250115 17:34:06.492297 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74884 I20250115 17:34:06.492303 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74881 I20250115 17:34:06.492308 124906 task_worker_pool.cpp:337] successfully submit task|type=PUBLISH_VERSION|signature=74882 I20250115 17:34:06.492372 123269 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74885, cost(us): 9, error_tablet_size=0, res=[OK] I20250115 17:34:06.492383 123269 task_worker_pool.cpp:1614] successfully publish version|signature=74885|transaction_id=74885|tablets_num=0|cost(s)=0 I20250115 17:34:06.492416 123271 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74877, cost(us): 7, error_tablet_size=0, res=[OK] I20250115 17:34:06.492424 123271 task_worker_pool.cpp:1614] successfully publish version|signature=74877|transaction_id=74877|tablets_num=0|cost(s)=0 I20250115 17:34:06.492492 123272 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74880, cost(us): 7, error_tablet_size=0, res=[OK] I20250115 17:34:06.492499 123272 task_worker_pool.cpp:1614] successfully publish version|signature=74880|transaction_id=74880|tablets_num=0|cost(s)=0 I20250115 17:34:06.492538 123265 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74879, cost(us): 5, error_tablet_size=0, res=[OK] I20250115 17:34:06.492543 123265 task_worker_pool.cpp:1614] successfully publish version|signature=74879|transaction_id=74879|tablets_num=0|cost(s)=0 I20250115 17:34:06.492561 123266 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74878, cost(us): 4, error_tablet_size=0, res=[OK] I20250115 17:34:06.492568 123266 task_worker_pool.cpp:1614] successfully publish version|signature=74878|transaction_id=74878|tablets_num=0|cost(s)=0 I20250115 17:34:06.492657 123266 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74884, cost(us): 6, error_tablet_size=0, res=[OK] I20250115 17:34:06.492666 123266 task_worker_pool.cpp:1614] successfully publish version|signature=74884|transaction_id=74884|tablets_num=0|cost(s)=0 I20250115 17:34:06.492682 123269 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74881, cost(us): 3, error_tablet_size=0, res=[OK] I20250115 17:34:06.492686 123269 task_worker_pool.cpp:1614] successfully publish version|signature=74881|transaction_id=74881|tablets_num=0|cost(s)=0 I20250115 17:34:06.492717 123269 engine_publish_version_task.cpp:346] finish to publish version on transaction.transaction_id=74882, cost(us): 2, error_tablet_size=0, res=[OK] I20250115 17:34:06.492720 123269 task_worker_pool.cpp:1614] successfully publish version|signature=74882|transaction_id=74882|tablets_num=0|cost(s)=0 I20250115 17:34:07.248505 123294 tablet_manager.cpp:1025] find expired transactions for 0 tablets I20250115 17:34:07.271360 123294 tablet_manager.cpp:1058] success to build all report tablets info. tablet_count=15027 I20250115 17:34:08.907501 123293 data_dir.cpp:877] path: /data/doris/be_storage total capacity: 315574194176, available capacity: 252346187776, usage: 0.200359, in_use: 1 I20250115 17:34:08.915766 123293 storage_engine.cpp:373] get root path info cost: 8 ms. tablet counter: 15027 I20250115 17:34:13.185438 123008 wal_manager.cpp:486] Scheduled(every 10s) WAL info: [/data/doris/be_storage/wal: limit 25234618777 Bytes, used 0 Bytes, estimated wal bytes 0 Bytes, available 25234618777 Bytes.]; I20250115 17:34:15.313457 123933 workload_group_manager.cpp:192] Process Memory Summary: process memory used 2.30 GB(= 2.55 GB[vm/rss] - 262.40 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), sys available memory 7.84 GB(= 7.84 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), all workload groups memory usage: 1.02 MB, weighted_memory_limit_ratio: 0.8227695804550733 I20250115 17:34:23.076792 123153 olap_server.cpp:1165] cooldown producer get tablet num: 0 I20250115 17:34:23.229077 123008 wal_manager.cpp:486] Scheduled(every 10s) WAL info: [/data/doris/be_storage/wal: limit 25234618777 Bytes, used 0 Bytes, estimated wal bytes 0 Bytes, available 25234618777 Bytes.]; I20250115 17:34:25.351622 123933 workload_group_manager.cpp:192] Process Memory Summary: process memory used 2.29 GB(= 2.55 GB[vm/rss] - 260.79 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), sys available memory 7.84 GB(= 7.84 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), all workload groups memory usage: 1.02 MB, weighted_memory_limit_ratio: 0.8229612024497326 ``` ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
