[GitHub] [doris] yuxuan-luo opened a new issue, #24573: [Bug] insert into ignore data lost

[via GitHub]

yuxuan-luo opened a new issue, #24573:
URL: https://github.com/apache/doris/issues/24573

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   
   ### Version
   
   master
   
   ### What's Wrong?
   
   使用 ignore 功能时数据丢失
   
   
   ### What You Expected?
   
   doris:
   CREATE TABLE `stream_load_csv` (
     `id` int(11) NULL,
     `name` varchar(10) NULL,
     `age` int(11) NULL DEFAULT "20",
     `city` varchar(10) NOT NULL DEFAULT "beijing",
     `balance` DECIMAL NULL
   ) ENGINE=OLAP
   UNIQUE KEY(`id`)
   COMMENT 'OLAP'
   DISTRIBUTED BY HASH(`id`) BUCKETS 1
   PROPERTIES (
   "replication_allocation" = "tag.location.default: 1",
   "is_being_synced" = "false",
   "storage_format" = "V2",
   "enable_unique_key_merge_on_write" = "true",
   "light_schema_change" = "true",
   "disable_auto_compaction" = "false",
   "enable_single_replica_compaction" = "false"
   );
   
   mysql:
   CREATE TABLE `stream` (
     `id` int NOT NULL,
     `name` varchar(255) DEFAULT '',
     `age` int DEFAULT NULL,
     `city` varchar(255) DEFAULT '',
     `balance` int DEFAULT NULL,
     PRIMARY KEY (`id`)
   ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3
   
   flink sql:
   CREATE TABLE doris_sink (
     `id` int ,
     `name` STRING ,
     `age` int ,
     `city` STRING,
     `balance` int 
   ) 
   WITH (
     'connector' = 'doris_test',
     'fenodes' = '10.0.3xxx:8030',
     'table.identifier' = 'hxxxx.xxxx',
     'username' = 'xxx',
     'password' = 'xxx',
     'sink.enable-delete' = 'true', 
     'sink.label-prefix' = 'doris_label_v2',
     'sink.properties.ignore_mode' = 'true',
     'sink.enable-2pc' = 'false'
   );
   
   CREATE TABLE cdc_mysql_source (
     `id` int ,
     `name` varchar ,
     `age` int ,
     `city` varchar,
     `balance` int ,
     PRIMARY KEY (`id`) NOT ENFORCED
   ) WITH (
    'connector' = 'mysql-cdc',
    'hostname' = '10xxxxx',
    'port' = '3306',
    'username' = 'xxxx',
    'password' = 'xxxxx',
    'database-name' = 'xxxx',
    'table-name' = 'xxxx'
   );
   
   insert into doris_sink select * from cdc_mysql_source;
   
   操作步骤：
   一、先在 stream_load_csv 表中插入数据 insert into stream_load_csv values         
(1,"kevin",18,"shenzhen",400),         (2,"bob",20,"beijing",500),         
(3,"alice",22,"shanghai",600),         (4,"jack",24,"hangzhou",700),         
(5,"tom",26,"guanzhou",800);
   二、在mysql表中插入数据并通过flink cdc 同步到 doris
   <img width="818" alt="image" 
src="https://github.com/apache/doris/assets/119841515/39940622-8209-4fa5-b3fa-142b50c32196";>
   三、用 stream load 导入数据
   <img width="263" alt="image" 
src="https://github.com/apache/doris/assets/119841515/88d84608-a3a9-4910-90d0-e0782e47dc9c";>
   四、在mysql再插入数据
   <img width="828" alt="image" 
src="https://github.com/apache/doris/assets/119841515/7ef464cd-f21f-4c45-bf99-46ff2915b647";>
   五、后续查询数据把数据丢失了：
   <img width="359" alt="image" 
src="https://github.com/apache/doris/assets/119841515/f1260566-9b21-47e0-85c4-d6322f31cc65";>
   
   
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org