wyxxxcat opened a new pull request, #49233: URL: https://github.com/apache/doris/pull/49233
### What problem does this PR solve? Issue Number: close #xxx ``` I20250324 17:49:24.592511 1013706 snapshot_loader.cpp:534] remote rowset 020000000000001f34486e234cc2d59301468c0cef156c81 was derived from local tablet 1742809606877 rowset 020000000000000234486e234cc2d59301468c0cef156c81, skip downloading these files I20250324 17:49:24.592561 1013706 snapshot_loader.cpp:534] remote rowset 020000000000002034486e234cc2d59301468c0cef156c81 was derived from local tablet 1742809606877 rowset 020000000000001934486e234cc2d59301468c0cef156c81, skip downloading these files I20250324 17:49:24.592617 1013706 snapshot_loader.cpp:554] link file from /mnt/disk2/wangyixuan/doris_sim/doris_master/be/0/storage/snapshot/20250324174918.6.86400/1742809606971/978928441/020000000000001934486e234cc2d59301468c0cef156c81_0.dat to /mnt/disk2/wangyixuan/doris_sim/doris_master/be/0/storage/snapshot/20250324174918.6.86400/1742809606971/978928441/020000000000002034486e234cc2d59301468c0cef156c81_0.dat I20250324 17:49:24.592677 1013706 snapshot_loader.cpp:579] skip download remote file 020000000000002034486e234cc2d59301468c0cef156c81_0.dat as its local version 020000000000001934486e234cc2d59301468c0cef156c81_0.dat already exists I20250324 17:49:24.592702 1013706 snapshot_loader.cpp:534] remote rowset 020000000000002134486e234cc2d59301468c0cef156c81 was derived from local tablet 1742809606877 rowset 020000000000001a34486e234cc2d59301468c0cef156c81, skip downloading these files I20250324 17:49:24.592752 1013706 snapshot_loader.cpp:554] link file from /mnt/disk2/wangyixuan/doris_sim/doris_master/be/0/storage/snapshot/20250324174918.6.86400/1742809606971/978928441/020000000000001a34486e234cc2d59301468c0cef156c81_0.dat to /mnt/disk2/wangyixuan/doris_sim/doris_master/be/0/storage/snapshot/20250324174918.6.86400/1742809606971/978928441/020000000000002134486e234cc2d59301468c0cef156c81_0.dat I20250324 17:49:24.592796 1013706 snapshot_loader.cpp:579] skip download remote file 020000000000002134486e234cc2d59301468c0cef156c81_0.dat as its local version 020000000000001a34486e234cc2d59301468c0cef156c81_0.dat already exists I20250324 17:49:24.592815 1013706 snapshot_loader.cpp:534] remote rowset 020000000000002234486e234cc2d59301468c0cef156c81 was derived from local tablet 1742809606877 rowset 020000000000001b34486e234cc2d59301468c0cef156c81, skip downloading these files I20250324 17:49:24.592852 1013706 snapshot_loader.cpp:554] link file from /mnt/disk2/wangyixuan/doris_sim/doris_master/be/0/storage/snapshot/20250324174918.6.86400/1742809606971/978928441/020000000000001b34486e234cc2d59301468c0cef156c81_0.dat to /mnt/disk2/wangyixuan/doris_sim/doris_master/be/0/storage/snapshot/20250324174918.6.86400/1742809606971/978928441/020000000000002234486e234cc2d59301468c0cef156c81_0.dat I20250324 17:49:24.592887 1013706 snapshot_loader.cpp:579] skip download remote file 020000000000002234486e234cc2d59301468c0cef156c81_0.dat as its local version 020000000000001b34486e234cc2d59301468c0cef156c81_0.dat already exists I20250324 17:49:24.592907 1013706 snapshot_loader.cpp:534] remote rowset 020000000000002334486e234cc2d59301468c0cef156c81 was derived from local tablet 1742809606877 rowset 020000000000001c34486e234cc2d59301468c0cef156c81, skip downloading these files I20250324 17:49:24.592942 1013706 snapshot_loader.cpp:554] link file from /mnt/disk2/wangyixuan/doris_sim/doris_master/be/0/storage/snapshot/20250324174918.6.86400/1742809606971/978928441/020000000000001c34486e234cc2d59301468c0cef156c81_0.dat to /mnt/disk2/wangyixuan/doris_sim/doris_master/be/0/storage/snapshot/20250324174918.6.86400/1742809606971/978928441/020000000000002334486e234cc2d59301468c0cef156c81_0.dat I20250324 17:49:24.592978 1013706 snapshot_loader.cpp:579] skip download remote file 020000000000002334486e234cc2d59301468c0cef156c81_0.dat as its local version 020000000000001c34486e234cc2d59301468c0cef156c81_0.dat already exists ``` ``` I20250324 17:49:30.599040 1013717 rowset.cpp:76] rowset is loaded. 020000000000002234486e234cc2d59301468c0cef156c81, rowset version:[4-4], state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:1742809606952 I20250324 17:49:30.599387 1013717 rowset.cpp:76] rowset is loaded. 020000000000002b34486e234cc2d59301468c0cef156c81, rowset version:[4-4], state from ROWSET_UNLOADED to ROWSET_LOADED. tabletid:1742809606952 I20250324 17:49:30.599442 1013717 beta_rowset.cpp:221] begin to remove files in rowset 020000000000002234486e234cc2d59301468c0cef156c81, version:4-4, tabletid:1742809606952 ``` ut_test ``` I20250328 15:22:21.599613 2738237 snapshot_loader.cpp:1347] finished to list files in local path: /mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/snapshot/20250328152221.18446744073709551615.172800/222/333, file num: 2 I20250328 15:22:21.599749 2738237 snapshot_loader.cpp:342] list remote files: http://127.0.0.1:1234/api/_tablet/_download?token=fake_token&channel=ingest_binlog&file=/mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/remote_snapshot/222/333, job: 3, task id: 222, remote be: TNetworkAddress(hostname=127.0.0.1, port=1234) I20250328 15:22:21.602023 2738237 snapshot_loader.cpp:247] clone begin to download file from: http://127.0.0.1:1234/api/_tablet/_download?token=fake_token&channel=ingest_binlog&file=/mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/remote_snapshot/222/333/333.hdr to: /mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/snapshot/20250328152221.18446744073709551615.172800/222/333/333.hdr.tmp. size(B): 1001, timeout(s): 300 I20250328 15:22:21.603295 2738237 snapshot_loader.cpp:418] link rowset files by compare 222.hdr and 333.hdr I20250328 15:22:21.603370 2738237 snapshot_loader.cpp:510] remote rowset 0200000000000003d94d5d45759f5448b85a280cc659c6b6 was derived from local tablet 222 rowset 0200000000000001d94d5d45759f5448b85a280cc659c6b6, skip downloading these files I20250328 15:22:21.603389 2738237 snapshot_loader.cpp:510] remote rowset 0200000000000004d94d5d45759f5448b85a280cc659c6b6 was derived from local tablet 222 rowset 0200000000000002d94d5d45759f5448b85a280cc659c6b6, skip downloading these files I20250328 15:22:21.603447 2738237 snapshot_loader.cpp:530] link file from /mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/snapshot/20250328152221.18446744073709551615.172800/222/333/0200000000000002d94d5d45759f5448b85a280cc659c6b6_0.dat to /mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/snapshot/20250328152221.18446744073709551615.172800/222/333/0200000000000004d94d5d45759f5448b85a280cc659c6b6_0.dat I20250328 15:22:21.604249 2738237 snapshot_loader.cpp:579] remote file: 0200000000000004d94d5d45759f5448b85a280cc659c6b6_0.dat, size: 232, md5: 3b61ab8303632eccddb9758e460e00a6 I20250328 15:22:21.604286 2738237 snapshot_loader.cpp:597] file 0200000000000004d94d5d45759f5448b85a280cc659c6b6_0.dat already exists, skip download url http://127.0.0.1:1234/api/_tablet/_download?token=fake_token&channel=ingest_binlog&file=/mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/remote_snapshot/222/333/0200000000000004d94d5d45759f5448b85a280cc659c6b6_0.dat I20250328 15:22:21.604517 2738237 snapshot_loader.cpp:686] begin to delete local snapshot file: /mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/snapshot/20250328152221.18446744073709551615.172800/222/333/0200000000000002d94d5d45759f5448b85a280cc659c6b6_0.dat, it does not exist in remote I20250328 15:22:21.604579 2738237 snapshot_loader.cpp:1099] finished to download snapshots. job: 3, task id: 222 ``` ``` slot_desc: col1 total column 1 before publish, tablet row nums:0 start to add rowset version:3-3 start to publish txn start to add inc rowset:0200000000000002d4402edb1b9ebb582b6731034765a49e, num rows:1, version:3-3 Original version: 3, 3 snapshot_path: /mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/snapshot/20250328175354.18446744073709551615.172800 Found linked file: /mnt/disk2/wangyixuan/doris_/doris/snapshot_data_test/snapshot/20250328175354.18446744073709551615.172800/222/333/0200000000000004d4402edb1b9ebb582b6731034765a49e_0.dat, size: 232 bytes ``` when switching between upstream and downstream, we hope to use the existing rowset to avoid downloading Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
