amogh-jahagirdar commented on issue #407: URL: https://github.com/apache/iceberg-python/issues/407#issuecomment-1938083316
@djouallah Thanks for reporting, could you provide some more specifics on the tests you are running which are exceeding memory? If you're able to share queries, the scale of the tables involved, and the hardware specs of what you're running on, that would be helpful context. Also sorry I'm not familiar with DuckDB views but at first glance they look like typical logical views. Iceberg has defined a spec for views which defines common metadata which can be used across engines (see [view spec](https://iceberg.apache.org/view-spec/)). There's a Java implementation of the spec, with spark engine integration slated for the 1.5 release, and Trino as well. I think if there was an Iceberg view representation which referenced the DuckDB view/queries you wanted then you could get what you wanted by first getting the representation for duckdb and then performing the computation on the view which could execute against DuckDB natively in your own code. That would require Python support for Iceberg views. Before all that though, I encourage if you can share all the test details you can so that way we can make sure we're doing things as intelligently as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
