vustef opened a new issue, #1912: URL: https://github.com/apache/iceberg-rust/issues/1912
### Is your feature request related to a problem or challenge? Currently we only support `s3://` style URLs for S3/AWS, and abfs(s)/wasb(s) style URLs for Azure data lake storage (ADLS) Gen2 and Azure blob storage. However, both services expose HTTP(S) endpoints. For S3, there are two ways (path-style and vhost), and also VPCs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html For Azure, http(s) URLs are always of form: - `http(s)://account.blob.core.windows.net/container/` for accounts without hierarchical namespace enabled (corresponds to wasb) - `http(s)://account.dfs.core.windows.net/container/` for accounts with hierarchical namespace enabled (ADLS Gen 2) For Azure, we might decide to simply parse these URLs and translate them to the currently supported wasb(s)/abfs(s) styles. However, these client drivers may have some limitations to what HTTP(s) protocol supports. For S3, translation might be more limiting - it's impossible for VPC URLs, and I'm not sure about behaviour if translating vhost/path-style to s3 paths - it's something to explore. I seem to remember that there are some configurations though that prevent access with path-style URLs. So perhaps the safe bet would be not to translate, but to always use the style as chosen in the metadata path and the metadata/manifest/manifest list files themselves. Note: this might be true for GCS, but I don't know much about its storage URLs. cc @Xuanwo ### Describe the solution you'd like _No response_ ### Willingness to contribute I would be willing to contribute to this feature with guidance from the Iceberg Rust community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
