morningman commented on PR #69031:
URL: https://github.com/apache/doris/pull/69031#issuecomment-6147988682
## 1. [Must fix] Give the storage connectivity probe the same HTTP client
default
**Why.** The default `http-client.type=urlconnection` is injected only in
`IcebergCatalogFactory.buildBaseCatalogProperties`.
`IcebergConnector.probeStorage` does not go through that method. It builds its
own map in `buildStorageProbeProperties` (`IcebergConnector.java:493`), with
only `s3.*` and `client.region`, and then loads an `S3FileIO` from it (`:467`).
Under iceberg 1.12.0 that `S3FileIO` falls back to the `apache` client type,
which is `Apache5HttpClient`, so building the client throws
`NoClassDefFoundError:
software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`. That is an
`Error`. It is not caught by `catch (Exception e)` (`:471`), so it never
becomes a "connectivity test failed" message. CREATE CATALOG fails with
`Unexpected exception: software/amazon/awssdk/http/apache5/...` whether the
credentials are right or wrong.
The failure was reproduced with iceberg 1.12.0 and AWS SDK 2.29.52, using
exactly the probe's properties against a closed local port. Without the key the
call throws `NoClassDefFoundError`. With `http-client.type=urlconnection` it
fails normally with `SdkClientException: Connection refused`.
`regression-test/suites/external_table_p0/test_connection/test_iceberg_rest_minio_connectivity.groovy`
covers this path and would fail in both of its probe cases.
**Fix.** Move the default into one helper and call it from both places. A
better version also copies the catalog's `http-client.*` keys into the probe
map, so the probe uses the same transport and proxy settings as the catalog.
```java
// IcebergCatalogFactory
/**
* iceberg-aws's default "apache" client is Apache5HttpClient since iceberg
1.12, which needs AWS SDK >= 2.32;
* Doris pins 2.29.x (#58566). Every map that ends up building an iceberg
AWS client must go through this.
*/
static void applyDefaultAwsHttpClient(Map<String, String> opts) {
opts.putIfAbsent(HttpClientProperties.CLIENT_TYPE,
HttpClientProperties.CLIENT_TYPE_URLCONNECTION);
}
```
```diff
// IcebergConnector.buildStorageProbeProperties
ioProps.put("s3.remote-signing-enabled", "false");
+ IcebergCatalogFactory.applyDefaultAwsHttpClient(ioProps);
```
Add a test written the same way as the PR's own factory test, so the probe
map is checked against the shipped SDK:
```java
@Test
public void storageProbePropertiesBuildAnS3ClientOnTheShippedSdk() {
Map<String, String> ioProps =
IcebergConnector.buildStorageProbeProperties(
IcebergCatalogProperties.TYPE_REST,
new
FakeS3CompatibleStorageProperties("S3").endpoint("http://127.0.0.1:1";)
.region("us-east-1").accessKey("ak").secretKey("sk"));
// MUTATION: drop the default -> "apache" -> NoClassDefFoundError -> red.
Assertions.assertDoesNotThrow(
() -> new
HttpClientProperties(ioProps).applyHttpClientConfigurations(S3Client.builder()));
}
```
After the fix, run the iceberg cases under
`external_table_p0/test_connection`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
