dwsmith1983 opened a new pull request, #7127:
URL: https://github.com/apache/datafusion-comet/pull/7127
## Which issue does this PR close?
Closes #6605.
Items 1 to 5 of that issue describe the native key lookup that #6059
replaced before it merged. The driver now resolves the mechanism through
hadoop-azure's own `AbfsConfiguration`, and the native store reads only the
global keys it is sent. So container-scoped keys, Fabric hosts, port-qualified
account keys, credential provider stores and the provider-class guard are all
Hadoop's decision. `AbfsAuthParitySuite` pins those cases per hadoop-azure
version: container-scoped OAuth keys, a port in the account, JCEKS stores and a
missing provider class. Fabric hosts have no parity case, but nothing in Comet
matches on the host suffix any more.
This PR covers item 6.
## Rationale for this change
Errors and debug lines in the native Azure store could print a whole URL,
including a password in its userinfo or a SAS token in its query:
- "Scheme of URL is not Azure: {url}" in `create_store`.
- object_store's own `Unable to recognise URL "{url}"`, from
`ObjectStoreScheme::parse`.
- object_store's `URL did not match any known pattern for scheme: {url}`,
which `MicrosoftAzureBuilder::build` returns for a host outside the public and
Fabric endpoints (a sovereign cloud, a private endpoint, Azurite) or a dotted
container or account name.
- The message for a mechanism the driver failed to resolve, and two debug
lines, which printed the userinfo with its password.
A URL from a Spark file listing doesn't normally carry a password or a SAS
query, since Hadoop's ABFS reads the part before `@` as the container. This is
defense in depth.
## What changes are included in this PR?
- `create_store` and `builder_for` in `azure.rs` hand object_store a copy of
the URL without the password, query or fragment. The container, which
object_store reads from the username, is kept.
- Messages name the URL by `scheme://[container@]host[:port]`, and the error
for a URL that isn't Azure by `scheme://host[:port]`, since the username of
another store's URL can be an access key ID.
## How are these changes tested?
New unit tests in `azure.rs`:
- A non-Azure URL and an unrecognised one, both with a username, password
and SAS query, fail without echoing any of them.
- An Azure URL on a host object_store doesn't recognise, with a password and
SAS query, fails without echoing either.
- An Azure URL with a SAS query still builds, and its container and path are
kept.
- The message for a driver-side resolution failure keeps the container and
drops the password.
Each leak test fails without the change, and handing object_store a URL
without the username fails the container test. `cargo test`, `cargo fmt` and
`cargo clippy` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
