swapna267 opened a new pull request, #18668: URL: https://github.com/apache/iceberg/pull/18668
In the dynamic sink, records can currently only be hash-distributed by the table's partition spec or by their equality fields. In certain scenarios, users would like to just distribute the rows by additional set of fields (different from partitioning spec). One use case , we saw was to achieve the reduced file sizes taking benefit of efficient compression in case of duplicate column values across rows. This use case is not specific to Dynamic Sink. This behavior can be easily achieved by sending the keyed stream to IcebergSink, which is not possible with Dynamic Sink. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
