adriangb commented on PR #21832: URL: https://github.com/apache/datafusion/pull/21832#issuecomment-4313969277
I have not read the code in detail (1.6k LOC) but I wonder how this will work in a world where the partition index of an opener is much less clear. Morselized scans (https://github.com/apache/datafusion/issues/20529) mean that filters will get applied at the morsel stage. Morsels won't belong to partitions per se / will be shared across partitions. So it seems to me that this entire feature is incompatible with the direction in which DataFusion is headed, unless I'm missing something. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
