tanmayrauth commented on issue #798:
URL: https://github.com/apache/iceberg-go/issues/798#issuecomment-4320784323
@laskoviymishka I am planning to pick this is up. What do you think about
the below plan?
Breakdown
1: Scaffold + path helpers
Add the Hadoop type constant to catalog/catalog.go and create the
catalog/hadoop/ package with the Catalog struct, NewCatalog constructor, and
init() self-registration. CatalogType() returns catalog.Hadoop and all other
interface methods stub out with a "not yet implemented" error, with a
compile-time var _ catalog.Catalog = (*Catalog)(nil) check. This PR also
includes all path helpers — namespaceToPath, tableToPath, metadataDir,
defaultTableLocation, metadataFilePath, and versionHintPath — along with
the version regex v([0-9]+)(?:\.gz)?\.metadata\.json and isTableDir(path) which
checks for matching metadata files. Tests cover registration via
catalog.Load, path construction, regex matching, and isTableDir true/false
cases.
2: Version-hint management
Implement readVersionHint (reads the integer from version-hint.text,
returns 0 on any error), writeVersionHint (writes to a temp file then renames,
best-effort with errors logged but not returned), and findVersion with the
full three-tier fallback: read hint and validate the file exists, scan
forward from the hinted version until no file is found, and if there's no hint
at all, list the metadata directory to find the max version then scan
forward. Tests cover round-trip read/write, missing hint, corrupt hint,
stale hint (v3 exists but hint says 2), scan-forward behavior, directory
listing fallback, and the no-metadata-files case.
3: Namespace operations
Implement all six namespace interface methods. CreateNamespace uses
os.MkdirAll and errors if the directory already exists or if non-empty
properties are passed. DropNamespace checks that there are no tables or child
namespaces before calling os.Remove. CheckNamespaceExists uses os.Stat.
ListNamespaces reads directory entries and filters out table dirs via
isTableDir. LoadNamespaceProperties returns a synthetic {"location": path} (not
persisted). UpdateNamespaceProperties returns an unsupported error. Tests
cover create, create duplicate, create nested (a.b.c), create with properties
(error), drop, drop non-existent, drop non-empty with tables and child
namespaces, list empty and mixed entries, check exists, load properties,
and update properties error.
4: CreateTable + LoadTable + CheckTableExists
Implement the three core table operations. CreateTable validates the
namespace exists, rejects custom locations, builds metadata via
table.NewMetadata, writes v1.metadata.json through a temp-file-plus-rename
pattern, and
does a best-effort version-hint write. LoadTable calls findVersion to get
the current version, builds the metadata path, and delegates to
table.NewFromLocation. CheckTableExists delegates to isTableDir. Tests cover
create-and-load round-trip, create with partition spec / sort order /
properties, reject custom location, create in non-existent namespace, create
duplicate, load non-existent, load with stale hint, and check exists
true/false.
5: ListTables + DropTable + RenameTable
Implement the remaining table catalog methods. ListTables verifies the
namespace exists then scans the namespace directory for table subdirs using
isTableDir. DropTable verifies the table exists then calls os.RemoveAll on
the table directory. RenameTable returns an unsupported error. Tests cover
list empty namespace, list with tables, list non-existent namespace, drop
existing table and verify directory is removed, drop non-existent table,
and rename returns error.
6: CommitTable
Implement the full atomic commit flow: validate requirements against
current metadata, apply updates via internal.UpdateTableMetadata, short-circuit
if metadata is unchanged, validate that the table location hasn't changed
and WRITE_METADATA_LOCATION is not set, write new metadata to a temp
{uuid}.metadata.json file, check that v{N+1}.metadata.json doesn't already
exist for conflict detection, atomically rename the temp file to the target
path, do a best-effort version-hint update, and clean up the temp file on
any error. This PR also handles create-via-commit where no existing table is
present. Tests cover single commit, sequential commits with version
increments, no-op commit, conflict detection by pre-creating the target
file, requirement failure, location change rejection, WRITE_METADATA_LOCATION
rejection, temp file cleanup on failure, and version-hint update
verification.
7: CLI integration
Add a case catalog.Hadoop: branch to the catalog switch in
cmd/iceberg/main.go, wire the cfg.Warehouse property, and add the blank import
for catalog/hadoop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
