Doris-Breakwater commented on issue #68431: URL: https://github.com/apache/doris/issues/68431#issuecomment-5790786847
Breakwater-GitHub-Analysis-Slot: slot_dd9f5ea13342 ### Initial assessment **High confidence: this is a valid connector packaging/classloader defect.** The reported `ArrayStoreException` is consistent with two `org.apache.http.HttpHost` classes loaded by different classloaders. The impact depends on deployment topology, but the supplied YARN application-mode layout is sufficient to trigger it. This issue currently has no labels. The affected build is maintained in `apache/doris-flink-connector`, so it should be routed to that connector's maintainers and labeled as a Flink-connector packaging bug if matching labels are available. ### Verified facts - I inspected the exact Maven Central artifact `org.apache.doris:flink-doris-connector-1.19:26.1.1` (SHA-256 `9db9096369f78901a601b1d14312fdcde2cea8fd8d809e998b4e857d74f7cd4d`). It contains `org/apache/http/HttpHost.class` and 766 entries under `org/apache/http/`, plus unrelocated Caffeine (696 entries), JSqlParser (516), Commons Lang/Logging, and related ancillary packages. - The release-tag shade configuration relocates Arrow, Netty, Jackson, Commons Codec, Guava, and Thrift, but not HttpComponents, Caffeine, JSqlParser, Commons Lang, or Commons Logging: [26.1.1 shade configuration](https://github.com/apache/doris-flink-connector/blob/de02e9d38898d8036117760e18702f619a85c74b/flink-doris-connector/flink-doris-connector-flink1/pom.xml#L222-L252). The relevant omissions are still present on current `master` at commit `8841db59b7ef8c66a3325188c8ce5100a6b44b95`. - One correction to the report: **Guava is already relocated in this exact artifact**. There are zero entries under `com/google/` and 3,768 entries under `org/apache/doris/shaded/com/google/`. Guava should therefore be removed from the list of confirmed unrelocated packages for 26.1.1. - Flink 1.19.3 defaults user code to child-first, but explicitly resolves `org.apache.flink.*`, `javax.annotation.*`, and logging packages parent-first. This supports the reported loader split: platform `NetworkConfig` and its HttpComponents classes can come from the parent, while job-side `HttpHost` comes from the child uber-jar. - A minimal two-classloader test using the exact 26.1.1 jar produces distinct `HttpHost` class identities and the same category of `ArrayStoreException` when a child-loaded instance is copied into a parent-loaded `HttpHost[]`. This validates the proposed Java-level failure mechanism independently of the full Flink job. - The jar also contains connector-owned `org/apache/flink/table/runtime/arrow/serializers/ArrowSerializer.class`. Its source explicitly says it was copied and modified from Flink: [ArrowSerializer source](https://github.com/apache/doris-flink-connector/blob/de02e9d38898d8036117760e18702f619a85c74b/flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/flink/table/runtime/arrow/serializers/ArrowSerializer.java#L19-L42). Flink 1.19.3's `flink-python` jar contains a class with the same FQCN, but different bytecode: the Doris copy calls `ArrowStreamWriter.end()` in `finishCurrentBatch()`, while the Flink copy does not. If `flink-python` is on the parent classpath, Flink's parent-first rule selects the platform class and can silently bypass the connector's intended modification. This independently confirms that emitting connector code in the `org.apache.flink.*` namespace is unsafe. ### Not yet verified / information still needed I did not reproduce the complete YARN application-mode job because the issue does not include the exact `flink-connector-elasticsearch8` version, full job POM/shade configuration, or complete TaskManager log. Those details are not needed to confirm the jar defect, but they are needed for a permanent integration regression test. Please attach: 1. The exact Elasticsearch connector jar filename/version and the job's effective dependency tree. 2. The complete job shade configuration (and dependency-reduced POM, if generated). 3. The full TaskManager exception and class-origin evidence for `NetworkConfig`, the list element's `HttpHost`, and the platform `HttpHost` (classloader plus `ProtectionDomain.getCodeSource()`, or JDK 11 `-Xlog:class+load=info` output). 4. A minimal reproducer project if it can be shared. ### Recommended maintainer actions 1. Add a connector integration test that puts the Doris connector in a child-loaded uber-jar and Elasticsearch 8/HttpComponents in the parent, then exercises the `List<HttpHost>` boundary. 2. Inventory the final release jar and isolate all connector-private dependencies. Relocate at least HttpComponents, Caffeine, and JSqlParser (and review Commons Lang/Logging and ancillary annotation/resource packages), or publish a thin artifact with normal dependency metadata. Because several public connector classes expose HttpComponents types (`HttpUtil#getHttpClient`, `HttpPutBuilder`, `HttpPostBuilder`, etc.), review and test the compatibility implications rather than adding relocations blindly. 3. Move the copied `ArrowSerializer` into a Doris-owned namespace and update connector references; a connector release should not define new classes under `org.apache.flink.*`. 4. Add an artifact-level CI assertion that rejects unapproved third-party prefixes and connector-owned `org/apache/flink/**` classes in every released Flink-version jar. 5. Ship the correction in a patch release and document affected versions. Treat the downstream filter as a temporary workaround only: stripping `org/apache/http/**` makes the connector depend on the platform's HttpComponents ABI and may replace this failure with linkage errors if versions differ. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
