vinjai commented on PR #871: URL: https://github.com/apache/iceberg-python/pull/871#issuecomment-2209659177
This PR solves for: 1. Writing sorted datasets to a partitioned or non-partitioned iceberg table. 2. Generating manifests with correct sort-order-id. 3. Integration tests to make sure sorted datasets are generated similar to spark sorting. Decisions taken: - If a sort transformation is not supported in PyIceberg, we will raise a warning related to the same and move ahead by writing the unsorted data with unsorted sort-order-id. What is not in the scope of this PR? - Performance improvement of the new sort function. (We will raise a separate issue for the same.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
