Tom-Newton commented on issue #19277: URL: https://github.com/apache/datafusion/issues/19277#issuecomment-5954098954
The specific problem is https://github.com/delta-io/delta-rs/issues/4623. However I have just found a workaround using `deltalake.query.QueryBuilder`, so that significantly reduces the urgency from our perspective. <details> <summary>How we are working around it in-case anyone is interested</summary> The limitation of this workaround is that we can only read delta tables. We were always aiming to use `ibis` but the normal `datafusion` + `ibis` connection depends on `datafusion-python`. To read delta tables we need to use the delta table provider from `delta-rs`, which requires some complex objects to be moved over the FFI. To avoid the FFI challenges we created `DeltaDataFusionBackend` which inherits from `ibis.backends.datafusion.Backend` and overrides a few methods. After compiling to SQL `DeltaDataFusionBackend` uses `deltalake.query.QueryBuilder` to execute it. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
