CaptainAni187 opened a new pull request, #81: URL: https://github.com/apache/datafusion-iceberg/pull/81
Closes #39 With fanout disabled, `TaskWriter` passes the pieces from `RecordBatchPartitionSplitter::split` straight to `ClusteredWriter`, but the splitter groups rows in a `HashMap`, so they come out in arbitrary order. When a partition spans a batch boundary, `ClusteredWriter` can see it closed and then reopened, and fails with "Cannot write to partition that was previously closed" even though the input is sorted. This went with the `TaskWriter` side of the two options in the issue, so no iceberg-rust change is needed. On the clustered path the pieces are now sorted by the first row their partition value appears in the batch. The values come from the batch's `_partition` column through `arrow_struct_to_literal`, the same conversion the splitter uses, so the grouping matches. Fanout and unpartitioned writes are unchanged. This doesn't touch #38: the optimizer can still drop the sort before the writer, so fanout-disabled inserts through SQL need that one too. Test: `test_task_writer_clustered_partition_spans_batches` writes 50 sorted partitions of 3 rows in batches of 8, so most batches split a partition. Without the fix it fails every run with the error from the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
