shyjsarah opened a new issue, #21778:
URL: https://github.com/apache/datafusion/issues/21778
### Is your feature request related to a problem or challenge?
Several methods on `CatalogProvider` and `SchemaProvider` perform
potentially fallible operations but have no way to report errors:
| Trait | Method | Current signature |
|-------|--------|-------------------|
| `CatalogProvider` | `schema_names()` | `-> Vec<String>` |
| `CatalogProvider` | `schema()` | `-> Option<Arc<dyn SchemaProvider>>` |
| `SchemaProvider` | `table_names()` | `-> Vec<String>` |
| `SchemaProvider` | `table_exist()` | `-> bool` |
For in-memory catalogs this is fine, but remote catalog implementations
(e.g. [paimon-rust](https://github.com/apache/paimon-rust), delta-rs Unity
Catalog) need to make network
calls in these methods. When a request fails (403, timeout, connection
refused, etc.), the implementation has no choice but to silently return an
empty vec / `None` / `false`,
which is indistinguishable from "no schemas exist" or "table not found".
This makes debugging very difficult — a network blip looks identical to an
empty catalog.
### Describe the solution you'd like
Change the return types to `Result<...>`:
```rust
// CatalogProvider
fn schema_names(&self) -> Result<Vec<String>>;
fn schema(&self, name: &str) -> Result<Option<Arc<dyn SchemaProvider>>>;
// SchemaProvider
fn table_names(&self) -> Result<Vec<String>>;
fn table_exist(&self, name: &str) -> Result<bool>;
This is a breaking API change. All existing implementations would need to
wrap their return values in Ok(...). The migration is mechanical but touches
every catalog
implementation.
Describe alternatives you've considered
1. Logging (current workaround): Add log::error! before returning the
fallback value. This is what we do in paimon-rust today — it surfaces the error
for debugging but callers
still can't react to it.
2. async versions of these methods: Would also solve the problem but is a
much larger change and has been intentionally avoided for planning performance
reasons (as documented in
CatalogProvider's doc comments).
3. Keep as-is: Accept that remote catalogs must silently swallow errors in
these methods.
Additional context
The other methods on these traits (table(), register_schema(),
deregister_schema(), register_table(), deregister_table()) already return
Result, so this would make the API
consistent.
Affected files (estimated): ~21 source files + FFI boundary in
datafusion-ffi.
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
