pvary opened a new pull request, #8653: URL: https://github.com/apache/iceberg/pull/8653
### Summary The Flink community created a new Sink specification in [FLIP-143](https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API) with the explicit goal to guarantee the unified handling of the bounded and unbounded data streams. Later it was enhanced in [FLIP-191](https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction) so there is a well defined place to execute small files compaction. The deprecation of the old SinkFunction is postponed to somewhere around Flink 2.0 based on the discussion on the dev mailing list , so the migration is not extremely urgent, but having the possibility to use the PostCommitTopology to execute the compaction of the small files could provide immediate benefits for the users of the Iceberg-Flink integration. ### Previous work There is an existing Iceberg PR #4904 for the Sink migration by Liwei Li (https://github.com/hililiwei) and Kyle Bendickson (https://github.com/kbendick) with the [related documentation](https://docs.google.com/document/d/1G4O6JidAoKgbIdy8Ts73OfG_KBEMpsW-LkXIb89I5k8/edit#heading=h.qqlw5ghn3vp7) which is authored by the same team. The discussion there is stuck, and the PR has been out of date for almost a year now. **The current proposal builds heavily on their work and wants to keep them as the co-authors for the proposed change.** To start the discussion, I have created the following document. https://docs.google.com/document/d/1K1M4wb9r_Tr-SDsUvqLyBaI5F14eRcqe3-ud6io0Da0/edit?usp=sharing I propose the following timeline: 1. Review the design document 2. Update the PR 3. PR reviews 4. Merge the PR to the Iceberg source 5. Restart the discussion about the missing features in the Flink community by creating a FLIP 6. Discuss/review/merge the relevant Flink changes 7. Release the Flink changes 8. Create a PR in the Iceberg repo to start using the new Flink features 9. Merge the Iceberg PR 10. Be happy 😀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
