jbewing opened a new issue, #13634: URL: https://github.com/apache/iceberg/issues/13634
### Apache Iceberg version 1.9.0 ### Query engine Spark ### Please describe the bug š Currently, when writing files with SparkāI'm using 3.5.6āand Iceberg 1.9.0, when a table declares a sort orderāor honestly even when it doesn'tāI would expect that when writing files from the Spark compute engine in a manner that is ordered e.g. not using the fanout writer that the `sort_order_id` field be set for written data files in the manifests ([Data File Entry Manifest Spec](https://iceberg.apache.org/spec/#data-file-fields)). Currently, this field is never set when writing data files with Spark. [Per the Iceberg Table Spec on Sorting](https://iceberg.apache.org/spec/#sorting) > A data or delete file is associated with a sort order by the sort order's id within [a manifest](https://iceberg.apache.org/spec/#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. I realize that this is an optional field, so it's not required to be set, however, theoretically setting this field can unlock performance optimizations in the future. For example, I have a feature that I'd love to contribute after this one from an Iceberg fork which enables reporting file ordering to Spark during scans by implementing the `SupportsReportOrdering` interface to enable the query optimizer to eliminate redundant sorts. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
