andygrove commented on issue #5690: URL: https://github.com/apache/datafusion-comet/issues/5690#issuecomment-5590496985
The failure inventory on #5677 ([run 33995765073](https://github.com/apache/datafusion-comet/actions/runs/33995765073), commit `d86d3b5`) shows this also breaks `RewriteDataFiles`, not just direct writes. Recording it here so the fix covers both. `TestSparkDataWrite` (24 failures across all four Iceberg versions), writer produces half the expected files: ``` [Should have 8 DataFiles] Expected size: 8 but was: 4 [Should have 4 DataFiles] Expected size: 4 but was: 2 ``` from `testUnpartitionedCreateWithTargetFileSizeViaTableProperties` and `testPartitionedCreateWithTargetFileSizeViaOption`, so both the table property and the write option are affected. `TestRewriteDataFilesAction` (6 failures), the rewrite action's bin-packing lands on a different file count: ``` testBinPackCombineMediumFiles (formatVersion = 2) [Action should add 3 data files] expected: 3 but was: 4 testRemoveDangledPositionDeletesPartitionEvolution (formatVersion = 2) [Should rewrite 2 data files into 1 and remove 1 dangled position delete file] actual [2, 2, 0], expected [1, 2, 1] ``` The second one is worth a look on its own: the rewrite neither combined the two data files nor dropped the dangled position delete, which may be more than a file-sizing difference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
