mxm commented on PR #17142: URL: https://github.com/apache/iceberg/pull/17142#issuecomment-5792580474
Thanks for clarifying! I was talking about V4 with IcebergSink, but you are right that we want to continue using DynamicSink in V4. There is nothing stopping us from integrating something like ConvertEqualityDeletes with DynamicSink. Even though lack of memory can can potential be an issue for a single table, multi-table scenarios are likely to amplify this issue. We might have to have to think about a scheduling strategy for the conversion process which can minimize the amount of data stored in memory and the amount of IO used. If we just run the current strategy as-is for many tables at once, it may become too resource intensive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
