fallintoplace opened a new issue, #1808:
URL: https://github.com/apache/iceberg-go/issues/1808
The orphan cleanup URI equivalence configuration is currently searched in
its raw, comma-separated form for every URI normalization.
For example, with `map[string]string{"s3,s3a,s3n": "s3"}`,
`applySchemeEquivalence` scans the configured groups and splits the key each
time. `applyAuthorityEquivalence` has the same pattern. On a large table, the
same equivalence rules are applied repeatedly to both referenced paths and
listed candidates.
This is a small cost per lookup, but orphan cleanup processes many paths.
The impact is speed x amount: any speed-up in equivalence lookup is multiplied
by the amount of paths compared, and by the number of configured equivalence
groups.
### Proposal
Flatten the configured equivalences once when building the cleanup config:
- turn each comma-separated group into direct `scheme -> canonical` and
`authority -> canonical` lookups
- preserve the current `WithEqualSchemes` and `WithEqualAuthorities` API
- define behavior for overlapping groups or conflicting canonical values
- use the flattened maps in per-path normalization and prefix-mismatch checks
### Acceptance criteria
- Equivalent schemes and authorities match exactly as they do today.
- Per-path lookup does not repeatedly split or scan the configured groups.
- Tests cover comma-separated groups, single entries, overlapping or
conflicting groups, and ADLS `container@host` authorities.
- Add a benchmark showing the effect as the number of paths and equivalence
groups grows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
