ConeyLiu opened a new pull request, #7171: URL: https://github.com/apache/iceberg/pull/7171
We use the current PartitionSpec for IcebergStreamWriter, which is fixed and will not change after the job started. While the PartitionSpec for IcebergStreamWriter is refreshed with the table snapshot changing. This could fail the fink sink job when updating the partition spec. Because we use the wrong partition spec to write those DataFiles/DeleteFiles to ManifestFile. For example, we got the following error when updating the partition spec: <img width="940" alt="error" src="https://user-images.githubusercontent.com/12733256/226884127-cc8d6732-7cec-4a31-bfc1-0b2ccebc3780.png";> In this patch, we keep the same PartitionSpec between `IcebergStreamWriter` and `IcebergFilesCommitter` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/12733256/226884127-cc8d6732-7cec-4a31-bfc1-0b2ccebc3780.png"](https://user-images.githubusercontent.com/12733256/226884127-cc8d6732-7cec-4a31-bfc1-0b2ccebc3780.png"){kind=link}