kevinjqliu commented on issue #3554: URL: https://github.com/apache/iceberg-python/issues/3554#issuecomment-4819685614
I'd love to introduce datafusion to pyiceberg, but I do have some concerns with the approach outlined above. I think PyIceberg should continue to own the Iceberg semantics, and only use DataFusion as the execution engine for data-intensive operations. If we rely on iceberg-rust / pyiceberg-core for Iceberg semantics, then PyIceberg becomes limited by iceberg-rust’s feature set. That also effectively deprecate and bypass a lot of PyIceberg’s existing functionality. This was the original reason I stopped experimenting with the datafusion table provider I think the cleaner boundary is: PyIceberg handles everything and defers execution to dataFusion, much like how pyarrow is used today -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
