milenkovicm commented on issue #1241: URL: https://github.com/apache/datafusion-ballista/issues/1241#issuecomment-5450405688
> I put together the proposed two-step version of this work: > > 1. [feat: add data source codec extension seam #2388](https://github.com/apache/datafusion-ballista/pull/2388) adds a small, format-neutral codec seam for external table providers and physical plans. Ballista core keeps no format dependency, and existing Ballista nodes retain their current serialization path. > 2. [feat: add Hudi read integration #2389](https://github.com/apache/datafusion-ballista/pull/2389) is a stacked Hudi proof using that seam. It keeps Hudi in an independent `integrations/hudi` crate and executes a projected/filtered Parquet COW read through a two-worker standalone cluster. > > The Hudi proof currently pins [apache/hudi-rs#698](https://github.com/apache/hudi-rs/pull/698), which aligns its DataFusion adapter with DataFusion 55 / Arrow 59.2. The first slice intentionally excludes MOR custom execution, writes, dynamic loading, and broad fallback machinery so the interface remains small enough to review. @wirybeaver, i would be reluctant to merge this to main at the moment for multiple reasons 1. hudi does not really look maintained 2. this would increase workload on maintainers this would add to our dependencies and increase our release complexity 3. i would suggest for now to keep this as a standalone project out of ballista, if change are needed to ballista to accommodate that please let me know hope you understand, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
