Guosmilesmile opened a new pull request, #12979: URL: https://github.com/apache/iceberg/pull/12979
Since we have support for RewriteDataFiles in Flink, we can now add it to Iceberg Sink v2 by using addPostCommitTopology to implement the related functionality. The relevant design document can be found at https://docs.google.com/document/d/16g3vR18mVBy8jbFaLjf2JwAANuYOmIwr15yDDxovdnA/edit?tab=t.0#heading=h.eyws2rxgmkhi In DataStream: ```java properties = ImmutableMap.of( FlinkWriteOptions.COMPACT_ENABLE.key(),"true", LockFactoryCreator.LOCK_TYPE_KEY,LockFactoryCreator.JDBC_LOCK, LockFactoryCreator.JDBC_URI,"jdbc:sqlite:file::memory:?ic" + UUID.randomUUID().toString().replace("-", ""), LockFactoryCreator.LOCK_ID,"test-lock-id", RewriteConfig.DATA_FILE_COUNT,"1", "flink-maintenance.lock.jdbc.init-lock-tables","true",RewriteConfig.REWRITE_ALL,"true" ); IcebergSink.forRowData(dataStream) .table(table) .tableLoader(tableLoader) .flinkConf(flinkConf) .append(); ``` In sql: ```sql SET table.exec.iceberg.use-v2-sink = true; SET compact-enabled = true; SET flink-maintenance.lock.type = 'jdbc'; SET flink-maintenance.lock.jdbc.uri = 'jdbc:sqlite:file::memory:?ic' ; SET flink-maintenance.lock.lock-id = 'test-lock-id'; SET flink-maintenance.rewrite.data-file-count = '1'; SET flink-maintenance.lock.jdbc.init-lock-tables= 'true'; SET flink-maintenance.rewrite.rewrite-alltrue = 'true'; insert into sink_table select * from source_table ; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
