cuidd2018 opened a new issue, #68431: URL: https://github.com/apache/doris/issues/68431
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris-flink-connector1.19:26.1.1 - Apache Flink 1.19.3, YARN **application mode** (child-first user code classloader) - `org.apache.doris:flink-doris-connector-1.19:26.1.1` - JDK 11, maven-shade-plugin 3.6.1 for the job uber-jar - `$FLINK_HOME/lib` contains `flink-connector-elasticsearch8` (platform-provided) ### What's Wrong? ## Evidence: embedded, unrelocated packages `jar tf flink-doris-connector-1.19-26.1.1.jar` shows, among others: org/apache/http/** <- Apache HttpComponents (httpcore + httpclient), unrelocated org/apache/commons/** com/github/benmanes/caffeine/** com/google/** <- Guava net/sf/jsqlparser/** javax/annotation/** android/annotation/** mozilla/** <- httpclient public-suffix dataset codegen/** org/apache/flink/table/runtime/arrow/serializers/** <- classes in Flink's OWN namespace Failure observed Our job uber-jar shades the Doris connector; the Elasticsearch connector comes from the platform lib. At runtime: 1. `org.apache.flink.connector.elasticsearch.sink.NetworkConfig` is loaded **parent-first** from `$FLINK_HOME/lib` (Flink's default parent-first patterns include `org.apache.flink.`), so it sees the platform's `org.apache.http.HttpHost`. 2. Job code is loaded **child-first** from the uber-jar, so it sees the `HttpHost` copy **embedded inside the Doris connector jar**. 3. Passing the job-side `List<HttpHost>` into `NetworkConfig` throws: java.lang.ArrayStoreException: arraycopy: type mismatch: can not copy org.apache.http.HttpHost[] into org.apache.http.HttpHost[] at java.base/java.util.Arrays.copyOf(Arrays.java:3722) at java.base/java.util.Arrays$ArrayList.toArray(Arrays.java:4341) at org.apache.flink.connector.elasticsearch.sink.NetworkConfig.getRestClient(NetworkConfig.java:84) at <our sink writer>... The same embedding also exposes every downstream job to `instanceof` / `ClassCastException` style splits for Guava, Caffeine and HttpComponents types whenever the platform or another connector provides the same classes. #Why this is a packaging defect 1. **Shade contract for library artifacts**: embedded dependencies must be relocated (e.g. `org.apache.doris.shaded.org.apache.http`) or not embedded at all. Shipping unrelocated copies of extremely common libraries guarantees collisions in every downstream uber-jar. 2. **Downstream cannot defend itself with normal dependency hygiene**: `maven-shade-plugin`'s `<artifactSet><excludes>` operates at artifact granularity, so excluding `org.apache.httpcomponents:httpcore` does NOT remove the copies embedded *inside* this connector's jar. The only workaround is a fragile, non-obvious per-artifact `<filter>` stripping `org/apache/http/**` from this artifact. 3. **Classes in Flink's own namespace are embedded** (`org.apache.flink.table.runtime.arrow.serializers.**`). Since Flink resolves `org.apache.flink.*` parent-first, whether these copies or the platform's are used depends on what happens to be in `$FLINK_HOME/lib` — nondeterministic behavior across clusters. ## Minimal reproduction 1. Build an uber-jar containing `flink-doris-connector-1.19:26.1.1` (compile scope) plus any code path that hands `org.apache.http.HttpHost` instances to a platform-side API consuming HttpComponents types (e.g. `NetworkConfig` from platform-provided `flink-connector-elasticsearch8`). 2. Submit via `flink run-application -t yarn-application` (child-first user classloader). 3. Observe the `ArrayStoreException` above during sink initialization. ### What You Expected? - Relocate all embedded third-party packages in the release jar (recommended), or - Publish a thin artifact declaring these dependencies normally instead of embedding them, or - At minimum, document prominently which packages are embedded unrelocated, so downstream projects can filter them consciously. ### How to Reproduce? Per-artifact shade filter in the job pom, plus defensive reconstruction of `HttpHost` instances at the classloader boundary: ```xml <filter> <artifact>org.apache.doris:flink-doris-connector-1.19</artifact> <excludes> <exclude>org/apache/http/**</exclude> </excludes> </filter> ``` Neither workaround should be necessary for a correctly packaged library artifact. ### Anything Else? Full `jar tf` listing, job pom and TaskManager logs available on request. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
