NoahKusaba commented on PR #2522: URL: https://github.com/apache/datafusion-ballista/pull/2522#issuecomment-6024505361
Hey @villebro I'm going to close this PR. compact_view_columns works around an arrow-rs limitation: the IPC writer writes every data buffer of a Utf8View/BinaryView array, even when only a few rows reference them (apache/arrow-rs#7185). And since Ballista reads Parquet and SQL strings as plain Utf8 by default (schema_force_view_types=false), view columns rarely reach the sort-shuffle writer, so this code and this PR do little in a default setup. I think the right fix is in the arrow-rs IPC writer: write only the bytes the views reference. That would remove the blocker behind #1294 and make this workaround unnecessary. I'll look at fixing it upstream at a later date. After that we can cleanup this stuff from Ballista and see if we should re-enable schema_force_view_types=True again. I kinda opened up a can of worms I probably shouldn't have with this one :sweat: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
