github-actions[bot] commented on code in PR #69031:
URL: https://github.com/apache/doris/pull/69031#discussion_r4261809851
##########
fe/pom.xml:
##########
@@ -246,8 +246,8 @@ under the License.
<module>fe-grpc</module>
</modules>
<properties>
- <!-- iceberg 1.9.1 depends avro on 1.12 -->
- <avro.version>1.12.1</avro.version>
+ <!-- iceberg 1.12.0 depends on avro 1.12.2 -->
+ <avro.version>1.12.2</avro.version>
Review Comment:
[P1] Adapt Hudi readers before raising the shared Avro version
This pin also upgrades `fe-connector-hudi` and `hadoop-hudi-scanner`. Avro
1.12.2 now rejects application classes by default, and this PR adds no
trusted-class/package configuration. For a Hudi 1.0 table with a completed
nonlegacy V2 commit, a COW incremental query runs `COWIncrementalRelation`'s
commit loop, which calls `HoodieTimeline.readCommitMetadata` ->
`CommitMetadataSerDeV2` -> `TimelineMetadataUtils.deserializeAvroMetadata` ->
`new SpecificDatumReader<>(HoodieCommitMetadata.class)`. Avro's
`SpecificData`/`ClassUtils` then reject
`org.apache.hudi.avro.model.HoodieCommitMetadata`, so the query fails even
though its data and timeline are valid. Scope the Avro bump to Iceberg or
register the required Hudi classes in both runtimes before initialization, and
cover a real V2 commit read. [Avro's release
notes](https://avro.apache.org/blog/2026/08/12/avro-1.12.2/) document this
incompatible default.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java:
##########
@@ -285,7 +285,7 @@ private static ManifestCacheValue
loadManifestCacheValue(ManifestFile manifest,
private static List<DataFile> loadDataFiles(ManifestFile manifest, Table
table) throws IOException {
List<DataFile> dataFiles = new ArrayList<>();
// .copy() is mandatory: the ManifestReader iterator reuses the same
object across iterations.
- try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest,
table.io())) {
+ try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest,
table.io(), table.specs())) {
Review Comment:
[P2] Preserve historical partition types in cached manifest reads
For a format-v1/v2 table identity-partitioned by `p INT`, save a snapshot,
promote `p` to BIGINT, then query the old snapshot with `WHERE p = 1`, manifest
caching enabled and batch streaming selected. `table.specs()` is rebound to the
current LONG schema, so this reader promotes the old Avro INT partition value
to `Long`. The pinned scan still binds its residual to historical INT;
`ManifestCacheFileScanTaskIterator.advance()` then calls
`PartitionData.get(Integer.class)` and throws `Wrong class, expected
java.lang.Integer, but was java.lang.Long`. With a cold/post-promotion cache
this fails the query because residual evaluation runs after
`streamingFileScanTasks`' eager fallback catch. The old reader used the
manifest's physical spec; the SDK historical path still uses the scan-bound
spec. Keep cached partition values consistent with historical scan typing,
including cache reuse across schemas, and cover a streamed historical scan
after promotion.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java:
##########
@@ -154,9 +156,22 @@ public static Map<String, String>
buildBaseCatalogProperties(Map<String, String>
opts.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
}
appendManifestCacheProperties(props, opts, metaCacheWeightLimited);
+ applyDefaultAwsHttpClient(opts);
return opts;
}
+ /**
+ * Defaults iceberg-aws's HTTP client to {@code urlconnection} unless
{@code opts} sets one. iceberg-aws's default
+ * "apache" client type is Apache5HttpClient since iceberg 1.12, which
needs an AWS SDK newer than the one Doris
+ * pins (2.29.x, see #58566), so every S3FileIO / Glue / STS client
iceberg builds from a map without this key fails
+ * with NoClassDefFoundError; url-connection-client ships with that SDK.
Every map that ends up building an
+ * iceberg AWS client must go through here. The key travels with
S3FileIO's properties, so the BE metadata
+ * scanner, which rebuilds the FileIO from them, uses the same client.
+ */
+ static void applyDefaultAwsHttpClient(Map<String, String> opts) {
+ opts.putIfAbsent(HttpClientProperties.CLIENT_TYPE,
HttpClientProperties.CLIENT_TYPE_URLCONNECTION);
Review Comment:
[P2] Apply the supported default to an empty HTTP client type
A catalog with `http-client.type=""` passes `checkCreateTimeOnlyRules()`,
and `putIfAbsent` preserves that empty value. [Iceberg 1.12's
`HttpClientProperties`](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/aws/src/main/java/org/apache/iceberg/aws/HttpClientProperties.java)
turns an empty type back into its default `apache`, which loads
`Apache5HttpClient` absent from Doris's pinned AWS SDK. An otherwise valid
S3/Glue catalog consequently fails on its first AWS call or CREATE storage
probe; ALTERing an existing type to an empty value also fails to restore the
documented default. The same empty value selected the working Apache4 default
before this upgrade. Normalize the empty value to `urlconnection` in the
catalog/probe maps, or reject it during statement validation, and cover this
case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
