gong commented on PR #2680: URL: https://github.com/apache/iceberg/pull/2680#issuecomment-1336717074
> > For example, we have a TiDB table that has six hundred million records. If we use flink streaming mode, it will cost too much time > > @coolderli Are you using the latest flink cdc connector and iceberg to export the stream ? I remember the latest flink cdc connector are refactored to use the netflix [DBLog algorithm](https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b) in parallel to export those existing RDBMS records in parallel. So in theory, if we don't have any performance blocker in the flink->iceberg path. There should not be any blocker that cost too much time. What's your bottleneck in your CDC exporting path ? @openinx Hello, Will not OOM be triggered if we use mysql-cdc2.0 to sync data to iceberg? Because mysql-cdc2.0 checkpoint in chunk level ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
