krishnakaanchan-png opened a new pull request, #3884:
URL: https://github.com/apache/iceberg-python/pull/3884
# Rationale for this change
`parse_location` builds the path as `f"{uri.netloc}{uri.path}"`. That is
correct for S3, where the netloc is the bucket. For Azure the netloc is
`[email protected]`, so the account ends up inside the
path and PyArrow reads that whole first segment as the container name. Every
`PyArrowFileIO` call on a canonical ABFS location is therefore pointed at a
container that cannot exist, since container names allow only lowercase
letters, numbers and hyphens.
PyArrow itself handles these locations correctly.
`FileSystem.from_uri("abfss://[email protected]/wh/d.parquet")`
returns `myfs/wh/d.parquet`, so the problem is only on our side. After this
change `parse_location` returns the same path that PyArrow does.
The full trace and the Azure documentation references are in #2698.
This is the path half only. The other half is that `_initialize_azure_fs`
takes no netloc, so the account can come only from `adls.account-name` and
never from the location itself. I will send that separately, it needs a call on
precedence when the property and the location disagree.
Related to #2698. Not closing it here, since the account derivation is still
pending.
## Are these changes tested?
Yes. Two new parametrised tests over `abfs`, `abfss`, `wasb` and `wasbs`,
one for the account qualified form and one for the container only form.
I also added an S3 case to `test_parse_location`. That function has to keep
serving both shapes and there was nothing pinning the S3 behaviour that the new
branch has to preserve.
`make lint` is clean and `tests/io/test_pyarrow.py` passes.
## Are there any user-facing changes?
Yes. Locations of the form
`abfs[s]://<container>@<account>.dfs.core.windows.net/<path>` now resolve to
the correct container under `PyArrowFileIO`. Earlier they resolved to a
container named after the entire netloc. S3, HDFS and the container only Azure
form are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
