gene-bordegaray commented on issue #21992: URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4366614993
I have added a draft PR #22002 which shows my vision for what partitioning can look like going forward and how this will apply to an optimization like dynamic filter routing. The API design was done by me but implementation details by AI and is not meant to be a mergeable PR. I provided a description of the API, would love any suggestions/ideas on this general approach. I separated it into 4 commits to show the logical progression of the implementation: 1. Adding Range partitioning variant that models what a custom partitioning trait would look like 2. Introduce the general partitioning trait using this model and have range partitioning implement it 3. Have file preserved partitioning use range partitioning rather than false advertising hash 4. Route dynamic filters using partitioning compatibility and create partitioning-local filters when range partitioned -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
