ashishkumar50 opened a new issue, #18921:
URL: https://github.com/apache/iceberg/issues/18921

   ### Feature Request / Improvement
   
   **Background**
   Apache Iceberg supports different storage systems through its FileIO 
abstraction. These storage systems may expose the same underlying data through 
multiple access protocols or filesystem interfaces.
   
   For example, an object storage system may provide access through native 
filesystem APIs, S3-compatible APIs, or other storage-specific protocols. 
Different compute engines may prefer different access mechanisms depending on 
their runtime environment, performance requirements, and available integrations.
   
   Currently, Iceberg file locations are persisted in table metadata and are 
used by compute engines to access metadata files, manifest lists, manifests, 
data files, and delete files.
   
   When multiple engines access the same Iceberg table through different 
storage protocols, differences in URI schemes and filesystem implementations 
can create interoperability challenges.
   
   One catalog supplies one base location (one scheme). Relative paths resolve 
against the catalog-provided base; if that base is storage://…, an engine 
wanting a different physical protocol still needs runtime, engine-side 
translation.
   Existing v1–v3 tables (and absolute paths in v4) keep fully-qualified URIs. 
Relative paths only help new writes; this FileIO serves the installed base with 
no metadata rewrite.
   
   **Problem Statement**
   There is a need to decouple the storage location represented in Iceberg 
metadata from the protocol or filesystem implementation used by a compute 
engine to access the underlying data.
   
   A consistent, canonical URI representation would allow multiple compute 
engines to access the same Iceberg table through their preferred storage 
protocols without requiring protocol-specific locations to be persisted in 
table metadata.
   
   Without such a mechanism, tables created through one storage access protocol 
may not be directly accessible to engines configured to use another protocol, 
even when both protocols address the same underlying storage.
   
   Additionally, Iceberg Format V4 introduces support for relative file paths, 
where file locations can be stored relative to the table location. This 
requires URI resolution mechanisms to account for both relative and absolute 
paths while maintaining consistent storage access across different protocols.
   
   **Proposed Solution**
   Introduce a CanonicalUriFileIO implementation that separates the canonical 
URI representation persisted in Iceberg metadata from the physical URI used for 
storage operations.
   
   The implementation should provide a configurable mechanism to resolve 
canonical URIs into protocol-specific URIs at runtime, based on the storage 
access configuration of the compute engine.
   
   The proposed implementation should:
   1. Maintain a consistent canonical URI representation in Iceberg table 
metadata, independent of the underlying storage access protocol.
   2. Resolve canonical URIs to the appropriate physical storage URI at runtime.
   3. Delegate filesystem operations to the configured underlying FileIO 
implementation.
   4. Apply URI resolution consistently to all file operations performed 
through this FileIO — manifest lists, manifests, data files, delete files, and 
metadata files when written/read via this FileIO.
   5. Ensure that newly written file locations preserve the appropriate logical 
representation (canonical absolute URI or relative path) according to the 
Iceberg table format version.
   6. Support configurable URI mappings without introducing dependencies on any 
specific storage system or filesystem implementation.
   
   **Example**
   Consider a storage system that exposes the same underlying data through 
multiple protocols:
   ```
   Compute engine             Canonical URI in Iceberg metadata      Physical 
access URI
   Engine A               storage://warehouse/db/table/...           
protocol-a://warehouse/db/table/...
   Engine B               storage://warehouse/db/table/...           
protocol-b://warehouse/db/table/...
   Engine C               storage://warehouse/db/table/...           
protocol-c://warehouse/db/table/...
   ```
   All engines operate on the same Iceberg table and share the same canonical 
metadata locations, while each engine resolves those locations to its 
configured storage access protocol.
   
   The URI mapping and underlying FileIO implementation should be configurable 
and extensible.
   
   **Expected Benefits**
   - Cross-engine interoperability: Allow different compute engines to access 
the same Iceberg tables using different storage protocols.
   - Protocol independence: Avoid coupling Iceberg metadata locations to a 
particular filesystem or storage access implementation.
   - Storage flexibility: Support storage systems exposing multiple access 
protocols or filesystem interfaces.
   - Consistent metadata: Maintain stable and consistent file locations across 
different compute engines.
   - Extensibility: Provide a reusable mechanism that can be adopted by 
different storage providers and integrations.
   - Reduced migration overhead: Avoid rewriting table metadata solely to 
accommodate a different storage access protocol, where the underlying locations 
can be resolved consistently.
   
   **Scope**
   - Introduce canonical URI resolution through the FileIO abstraction.
   - Support configurable URI scheme mappings.
   - Support delegation to different underlying FileIO implementations.
   - Ensure consistent URI resolution for read, write, delete, and other 
supported file operations.
   
   **Compatibility**
   The implementation should remain compatible with existing Iceberg table 
formats and catalog interfaces.
   
   No changes to the Iceberg table format specification should be required. 
Existing FileIO implementations and tables should continue to work without 
behavioral changes when canonical URI resolution is not configured.
   
   ### Query engine
   
   Spark
   
   ### Willingness to contribute
   
   - [x] I can contribute this improvement/feature independently
   - [ ] I would be willing to contribute this improvement/feature with 
guidance from the Iceberg community
   - [ ] I cannot contribute this improvement/feature at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to