pepijnve opened a new issue, #26992: URL: https://github.com/apache/datafusion/issues/26992
### Is your feature request related to a problem or challenge? `CatalogProvider` and `SchemaProvider` currently support addition of new child items using a registration mechanism takes an already created implementation instance (`dyn SchemaProvider` and `dyn TableProvider` resp.). The choice of the trait implementation types is handled externally to the provider implementation itself. This is an unfortunate design since it allows callers to pass in arbitrary schema and table implementations. The documentation does not really describe what each implementation is expected to do with these. Are they expected to accept and register arbitrary implementations as the in-memory catalog implementation does? Should they use the passed in implementation as a template to copy? If so, what should be done with the content of the table? Allowing arbitrary implementations for registration also does not match the query functions `CP::schema` and `SP::table` well. Both of those support returning an implementation of their choosing. If a catalog or schema implementation chooses to implement the 'copy template' strategy described in the previous paragraph then a read-after-write is likely to return a different value than what was registered. If the 'register as is' strategy is chosen, then every implementation needs to support listing and returning it's own implementations alongside arbitrary registered implementations. Neither option seems ideal. Finally, both registration functions are currently synchronous. This makes it more difficult than necessary to interact over the network with a remote catalog implementation. ### Describe the solution you'd like - The logical plan `CreateMemoryTable` is generalised to `CreateTable` - `SchemaProvider` is extended with a table creation function - `SchemaProvider` is extended with a view creation function - `CatalogProvider` is extended with a schema creation function - `SessionContext::create_catalog_schema` delegates this operation to the appropriate `CatalogProvider` - `SessionContext::create_memory_table` is renamed to `create_table` and delegates this operation to the appropriate `SchemaProvider` - `SessionContext::create_voew` delegates this operation to the appropriate `SchemaProvider` ### Describe alternatives you've considered Current workarounds: https://github.com/JanKaul/iceberg-rust/blob/9481bf10e60c743c7be722f30e3f3acd77649a21/datafusion_iceberg/src/planner.rs#L80 intercepts DDL statements and redirects them to its own implementations. This requires custom integration work for users of remote catalog extensions. Ideally a remote catalog implementation hooks into the existing extension points and doesn't require further ad hoc customisation. https://github.com/apache/datafusion-iceberg/blob/main/crates/datafusion/src/catalog/schema.rs#L147 treats the incoming table as a template to copy and asserts that it is empty. It works around the sync restriction by spawning a blocking task. It would be preferable for the library to handle this so that every remote catalog implementation doesn't have to implement this itself. ### Additional context _No response_ ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
