laskoviymishka commented on code in PR #2612:
URL: https://github.com/apache/iceberg-go/pull/2612#discussion_r4596849130
##########
catalog/hive/hive.go:
##########
@@ -784,7 +782,7 @@ func (c *Catalog) DropView(ctx context.Context, identifier
table.Identifier) err
return fmt.Errorf("failed to drop view %s.%s: %w", database,
viewName, err)
}
- fs, err := io.LoadFS(ctx, c.opts.props, metadataLocation)
+ fs, err := internal.LoadFSWithRegistry(ctx, c.opts.ioRegistry,
c.opts.props, metadataLocation)
Review Comment:
Should the registry tests cover
[`DropView`](https://github.com/apache/iceberg-go/blob/ea12c1758df95793746efce9d5414704498fa494/catalog/hive/hive.go#L785)
too? The Hive tests only run `LoadTable`, `CreateView` and `UpdateView` with a
non-nil registry. If a call site like this one lost its registry, IO would
quietly go to the global factory and the guard test wouldn't catch it. The same
gap exists for Hive `RegisterTable`/`CommitTable`/`LoadView`, SQL `CommitTable`
and Glue `RegisterTable`.
##########
catalog/glue/glue.go:
##########
@@ -201,11 +203,12 @@ func NewCatalog(opts ...Option) (*Catalog, error) {
}
cat := &Catalog{
- name: glueOps.name,
- glueSvc: glue.NewFromConfig(glueOps.awsConfig),
- catalogId: catalogId,
- awsCfg: &glueOps.awsConfig,
- props: iceberg.Properties(glueOps.awsProperties),
+ glueSvc: glue.NewFromConfig(glueOps.awsConfig),
+ catalogId: catalogId,
+ awsCfg: &glueOps.awsConfig,
+ props: iceberg.Properties(glueOps.awsProperties),
+ ioRegistry: glueOps.ioRegistry,
+ name: glueOps.name,
Review Comment:
Could the cosmetic hunks come out so the diff stays on the registry? Here
`name:` moved from first to last. Same for the removed blank line in hive
`NewCatalogWithClient`, the reflowed `CreateViewWithIOProperties` signature,
and the reworded `CreateStagedTableWithNamespaceKey` doc in
`catalog/internal/utils.go`.
##########
catalog/rest/vended_creds.go:
##########
@@ -545,17 +546,13 @@ func (p *prefixScopedIO) Close() error {
// loadRegistryFS preserves the global registry when none is configured.
func loadRegistryFS(ctx context.Context, registry *iceio.Registry, props
iceberg.Properties, location string) (iceio.IO, error) {
- if registry == nil {
- return iceio.LoadFS(ctx, props, location)
- }
-
- return registry.LoadFS(ctx, props, location)
+ return cataloginternal.LoadFSWithRegistry(ctx, registry, props,
location)
}
// loadRegistryFSFunc preserves the global registry for lazy loads when none
is configured.
func loadRegistryFSFunc(registry *iceio.Registry, props iceberg.Properties,
location string) func(context.Context) (iceio.IO, error) {
return func(ctx context.Context) (iceio.IO, error) {
- fs, err := loadRegistryFS(ctx, registry, props, location)
+ fs, err := cataloginternal.LoadFSWithRegistry(ctx, registry,
props, location)
Review Comment:
`loadRegistryFS` still has two callers (`:222`, `:416`), so this can keep
going through it as on `main`:
```suggestion
fs, err := loadRegistryFS(ctx, registry, props, location)
```
##########
view/view.go:
##########
@@ -147,45 +147,66 @@ func CreateView(
return CreateViewWithIOProperties(ctx, catalogName, viewIdent, schema,
representation, defaultNS, loc, props, props)
}
+// CreateViewWithIORegistry writes view metadata through the supplied
+// registry. A nil registry keeps the global FileIO behavior. The registry is
+// shared, not copied, so later Register and Unregister calls affect this
write.
+func CreateViewWithIORegistry(
Review Comment:
Could these four `...Registry` twins become one functional option on the
existing functions, e.g. `view.WithIORegistry(r)` passed variadically? Right
now
[view.go](https://github.com/apache/iceberg-go/blob/ea12c1758df95793746efce9d5414704498fa494/view/view.go#L150-L215)
and
[`WriteMetadataWithRegistry`](https://github.com/apache/iceberg-go/blob/ea12c1758df95793746efce9d5414704498fa494/view/write_metadata.go#L44-L60)
add four exported names with three suffix styles. After merge, removing any of
them is a break `apidiff` will flag, and the next knob would need a fifth twin.
##########
website/src/configuration.md:
##########
@@ -458,6 +458,10 @@ fs, err := registry.LoadFS(ctx, props, "myfs://bucket/key")
`io.NewRegistry()` copies schemes already registered globally (including
blank-imported cloud backends) but keeps later registrations independent.
`registry.Clone()` makes an isolated copy of another registry.
`registry.Names()` lists the schemes registered in that instance.
`registry.LoadFSFunc()` is the lazy factory equivalent. Global `io.LoadFS()`
and `io.LoadFSFunc()` continue using the default registry.
+To use the registry with a catalog, pass `WithIORegistry(registry)` to
`glue.NewCatalog` or `hive.NewCatalog`; call `hadoop.NewCatalogWithOptions(...,
hadoop.WithIORegistry(registry))` or `sql.NewCatalogWithOptions(...,
sql.WithIORegistry(registry))` for Hadoop and SQL. Direct view metadata writes
can use `view.CreateViewWithIORegistry` or
`view.CreateViewWithIOPropertiesAndRegistry`. The default remains global when
no registry is specified. An empty non-nil registry does **not** fall back to
global schemes. REST catalog support is tracked in
[#2366](https://github.com/apache/iceberg-go/pull/2366).
Review Comment:
Could this list REST instead of pointing at #2366? That PR is already merged
and
[`rest.WithIORegistry`](https://github.com/apache/iceberg-go/blob/dc09cf00fcdd59caca9d98421693423608d17111/catalog/rest/options.go#L31-L35)
is on `main`, so "tracked in" reads as pending. The [Hive and Glue option
lists](https://github.com/apache/iceberg-go/blob/ea12c1758df95793746efce9d5414704498fa494/website/src/configuration.md?plain=1#L227-L252)
also still omit `WithIORegistry`.
```suggestion
To use the registry with a catalog, pass `WithIORegistry(registry)` to
`glue.NewCatalog` or `hive.NewCatalog`, or `rest.WithIORegistry(registry)` for
REST; call `hadoop.NewCatalogWithOptions(..., hadoop.WithIORegistry(registry))`
or `sql.NewCatalogWithOptions(..., sql.WithIORegistry(registry))` for Hadoop
and SQL. Direct view metadata writes can use `view.CreateViewWithIORegistry` or
`view.CreateViewWithIOPropertiesAndRegistry`. The default remains global when
no registry is specified. An empty non-nil registry does **not** fall back to
global schemes.
```
##########
catalog/sql/sql.go:
##########
@@ -423,12 +425,21 @@ func (c *Catalog) isV0() bool { return c.schemaVersion ==
schemaV0 }
// SQLite callers that allow concurrent writers should configure WAL and a
busy timeout in the driver DSN so the
// settings apply to every pooled connection. NewCatalog does not mutate
connection settings on the caller-owned db.
func NewCatalog(name string, db *sql.DB, dialect SupportedDialect, props
iceberg.Properties) (*Catalog, error) {
+ return NewCatalogWithOptions(name, db, dialect, props)
+}
+
+// NewCatalogWithOptions builds a SQL catalog with optional isolated FileIO
wiring.
+func NewCatalogWithOptions(name string, db *sql.DB, dialect SupportedDialect,
props iceberg.Properties, opts ...Option) (*Catalog, error) {
Review Comment:
Why a separate `NewCatalogWithOptions` rather than `NewCatalog(name, db,
dialect, props, opts ...Option)`? Glue and Hive take `WithIORegistry` through
their existing variadic `NewCatalog`, while
[SQL](https://github.com/apache/iceberg-go/blob/ea12c1758df95793746efce9d5414704498fa494/catalog/sql/sql.go#L427-L432)
and
[Hadoop](https://github.com/apache/iceberg-go/blob/ea12c1758df95793746efce9d5414704498fa494/catalog/hadoop/hadoop.go#L206-L216)
end up with two constructors each. Appending the variadic keeps every direct
call compiling. It only breaks code that holds `sql.NewCatalog` as a typed func
value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]