gongzexin commented on issue #4038: URL: https://github.com/apache/streampark/issues/4038#issuecomment-2817542490
> > mysql cdc uses the 3.2.0 package, then you should sink to paimon, and the error shows that paimon catalog does not support s3 for now > > 实际上0.8.2是支持的，这个任务成功跑起来了，日志显示jobGraph提交失败，restApi成功了 关于s3，flink在不同封装上表现差异比较大，原生支持较好，dinky提交也报类似错误失败，但streampark最终是跑起来了 另外，很奇怪动态参数没有生效，以下为重启streampark之后，运行custom任务的完整日志 > > ``` > 2024-09-11 14:52:37 | INFO | XNIO-1 task-2 | io.undertow.servlet:389] Initializing Spring DispatcherServlet 'dispatcherServlet' > 2024-09-11 14:52:37 | INFO | XNIO-1 task-2 | org.springframework.web.servlet.DispatcherServlet:525] Initializing Servlet 'dispatcherServlet' > 2024-09-11 14:52:37 | INFO | XNIO-1 task-2 | org.springframework.web.servlet.DispatcherServlet:547] Completed initialization in 3 ms > 2024-09-11 14:52:37 | INFO | XNIO-1 task-1 | org.apache.streampark.console.base.handler.GlobalExceptionHandler:61] Permission denied: Unauthorized > 2024-09-11 14:52:37 | INFO | XNIO-1 task-2 | org.apache.streampark.console.base.handler.GlobalExceptionHandler:54] Unauthenticated: This subject is anonymous - it does not have any identifying principals and authorization operations require an identity to check against. A Subject instance will acquire these identifying principals automatically after a successful login is performed be executing org.apache.shiro.subject.Subject.login(AuthenticationToken) or when 'Remember Me' functionality is enabled by the SecurityManager. This exception can also occur when a previously logged-in Subject has logged out which makes it anonymous again. Because an identity is currently not known due to any of these conditions, authorization is denied. > 2024-09-11 14:52:37 | INFO | XNIO-1 task-4 | org.apache.streampark.console.base.handler.GlobalExceptionHandler:54] Unauthenticated: This subject is anonymous - it does not have any identifying principals and authorization operations require an identity to check against. A Subject instance will acquire these identifying principals automatically after a successful login is performed be executing org.apache.shiro.subject.Subject.login(AuthenticationToken) or when 'Remember Me' functionality is enabled by the SecurityManager. This exception can also occur when a previously logged-in Subject has logged out which makes it anonymous again. Because an identity is currently not known due to any of these conditions, authorization is denied. > 14:54:07.624 [streampark-flink-app-bootstrap-0] DEBUG org.apache.streampark.common.util.CommandUtils - [StreamPark] Command execute: > java -classpath /opt/flink-1.18.1/lib/flink-dist-1.18.1.jar org.apache.flink.client.cli.CliFrontend --version > 14:54:08.065 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.common.conf.FlinkVersion - [StreamPark] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. > Version: 1.18.1, Commit ID: a8c8b1c > > 14:54:08.072 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] add flink shims urls classloader,flink version: > ----------------------------------------- flink version ----------------------------------- > flinkHome : /opt/flink-1.18.1 > distJarName : flink-dist-1.18.1.jar > flinkVersion : 1.18.1 > majorVersion : 1.18 > scalaVersion : 2.12 > shimsVersion : streampark-flink-shims_flink-1.18 > ------------------------------------------------------------------------------------------- > > 14:54:08.082 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-client-core_2.12-2.1.4.jar > 14:54:08.083 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include flink shims jar lib: streampark-flink-shims_flink-1.18_2.12-2.1.4.jar > 14:54:08.083 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-common_2.12-2.1.4.jar > 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include jar lib: streampark-shaded-jackson-1.0.0.jar > 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-shims-base_2.12-2.1.4.jar > 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-client-api_2.12-2.1.4.jar > 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-proxy_2.12-2.1.4.jar > 14:54:08.084 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-packer_2.12-2.1.4.jar > 14:54:08.085 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-kubernetes_2.12-2.1.4.jar > 14:54:08.085 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.proxy.FlinkShimsProxy - [StreamPark] include streampark lib: streampark-flink-core_2.12-2.1.4.jar > 14:54:08.926 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] > --------------------------------------- flink job start --------------------------------------- > userFlinkHome : /opt/flink-1.18.1 > flinkVersion : 1.18.1 > appName : mysql-to-paimon-s3 > devMode : CUSTOM_CODE > execMode : REMOTE > k8sNamespace : null > flinkExposedType : null > clusterId : null > applicationType : Apache Flink > savePoint : null > properties : rest.address -> 172.31.4.220 rest.port -> 8081 execution.checkpointing.interval -> 10s state.checkpoints.num-retained -> 10 execution.checkpointing.num-retained -> 5 classloader.resolve-order -> parent-first > args : mysql_sync_database > --mysql-conf hostname=172.31.4.149 > --mysql-conf port=3306 > --mysql-conf username=testuser > --mysql-conf password=*** > --mysql-conf database-name=testflink > --warehouse s3://flink/paimon/ > --catalog_conf s3.endpoint=https://ossapi-tst > --catalog_conf s3.access-key=ak_flink > --catalog_conf s3.secret-key=*** > --catalog_conf s3.path.style.access=true > --database ods > --including_tables 'o.*|product.?|shipments' > --table_prefix my_ > --table_suffix _001 > --table_conf source.checkpoint-align.enabled=true > --table_conf changelog-producer=input > --table_conf sink.parallelism=1 > appConf : json://{"$internal.application.main":"org.apache.paimon.flink.action.FlinkActions"} > flinkBuildResult : { workspacePath: /opt/streampark_ws/workspace/100001, shadedJarPath: /opt/streampark_ws/workspace/100001/streampark-flinkjob_mysql-to-paimon-s3.jar, pass: true } > ------------------------------------------------------------------------------------------- > > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.num-retained, 20 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: fs.allowed-fallback-filesystems, s3 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.execution.failover-strategy, region > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.address, 172.31.4.220 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.num-retained, 10 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.savepoints.dir, s3://flink/savepoints/ > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.aligned-checkpoint-timeout, 60s > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.bind-host, 0.0.0.0 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.secret-key, ****** > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.endpoint, https://ossapi-tst.hengrui.com > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.checkpoints-after-tasks-finish.enabled, true > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: parallelism.default, 1 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.numberOfTaskSlots, 4 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: env.java.opts.all, --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/j ava.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: pekko.ask.timeout, 30s > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.process.size, 3072m > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.mode, EXACTLY_ONCE > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.bind-host, 0.0.0.0 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: classloader.resolve-order, parent-first > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: web.timeout, 50000 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.memory.process.size, 2048m > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.port, 6123 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.bind-address, 0.0.0.0 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.interval, 1min > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.timeout, 3min > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.access-key, ak_flink > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.managed.size, 0 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.externalized-checkpoint-retention, DELETE_ON_CANCELLATION > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.address, 172.31.4.220 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.max-concurrent-checkpoints, 2 > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.dir, s3://flink/checkpoints/ > 2024-09-11 14:54:08 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.path.style.access, true > 14:54:09.280 [streampark-flink-app-bootstrap-0] WARN org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] param:$internal.application.main is error,skip it. > 14:54:09.321 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: rest.address :172.31.4.220 > 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: rest.port :8081 > 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: execution.checkpointing.interval :10s > 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: state.checkpoints.num-retained :10 > 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: execution.checkpointing.num-retained :5 > 14:54:09.322 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] submit application dynamicProperties: classloader.resolve-order :parent-first > 14:54:09.332 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - cliArgs: -t remote -Drest.address=172.31.4.220 -Drest.port=8081 -Dexecution.checkpointing.interval=10s -Dstate.checkpoints.num-retained=10 -Dexecution.checkpointing.num-retained=5 -Dclassloader.resolve-order=parent-first > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.num-retained, 20 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: fs.allowed-fallback-filesystems, s3 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.execution.failover-strategy, region > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.address, 172.31.4.220 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.num-retained, 10 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.savepoints.dir, s3://flink/savepoints/ > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.aligned-checkpoint-timeout, 60s > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.bind-host, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.secret-key, ****** > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.endpoint, https://ossapi-tst.hengrui.com > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.checkpoints-after-tasks-finish.enabled, true > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: parallelism.default, 1 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.numberOfTaskSlots, 4 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: env.java.opts.all, --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/j ava.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: pekko.ask.timeout, 30s > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.process.size, 3072m > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.mode, EXACTLY_ONCE > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.bind-host, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: classloader.resolve-order, parent-first > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: web.timeout, 50000 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.memory.process.size, 2048m > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.port, 6123 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.bind-address, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.interval, 1min > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.timeout, 3min > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.access-key, ak_flink > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.managed.size, 0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.externalized-checkpoint-retention, DELETE_ON_CANCELLATION > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.address, 172.31.4.220 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.max-concurrent-checkpoints, 2 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.dir, s3://flink/checkpoints/ > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.path.style.access, true > 14:54:09.344 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] Custom commandline: [org.apache.flink.client.cli.GenericCLI@53ad1d67, org.apache.flink.yarn.cli.FlinkYarnSessionCli@430e4fa6, org.apache.flink.client.cli.DefaultCLI@23c80632] > 14:54:09.345 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] Checking custom commandline org.apache.flink.client.cli.GenericCLI@53ad1d67, isActive: true > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.num-retained, 20 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: fs.allowed-fallback-filesystems, s3 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.execution.failover-strategy, region > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.address, 172.31.4.220 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.num-retained, 10 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.savepoints.dir, s3://flink/savepoints/ > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.aligned-checkpoint-timeout, 60s > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.bind-host, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.secret-key, ****** > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.endpoint, https://ossapi-tst.hengrui.com > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.checkpoints-after-tasks-finish.enabled, true > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: parallelism.default, 1 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.numberOfTaskSlots, 4 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: env.java.opts.all, --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/j ava.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: pekko.ask.timeout, 30s > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.process.size, 3072m > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.mode, EXACTLY_ONCE > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.bind-host, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: classloader.resolve-order, parent-first > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: web.timeout, 50000 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.memory.process.size, 2048m > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.port, 6123 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.bind-address, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.interval, 1min > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.timeout, 3min > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.access-key, ak_flink > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.managed.size, 0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.externalized-checkpoint-retention, DELETE_ON_CANCELLATION > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.address, 172.31.4.220 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.max-concurrent-checkpoints, 2 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.dir, s3://flink/checkpoints/ > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.path.style.access, true > 14:54:09.428 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] [flink-submit] Submit job with JobGraph Plan. > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.num-retained, 20 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: fs.allowed-fallback-filesystems, s3 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.execution.failover-strategy, region > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.address, 172.31.4.220 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.num-retained, 10 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.savepoints.dir, s3://flink/savepoints/ > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.aligned-checkpoint-timeout, 60s > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.bind-host, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.secret-key, ****** > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.endpoint, https://ossapi-tst.hengrui.com > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.checkpoints-after-tasks-finish.enabled, true > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: parallelism.default, 1 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.numberOfTaskSlots, 4 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: env.java.opts.all, --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/j ava.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: pekko.ask.timeout, 30s > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.process.size, 3072m > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.mode, EXACTLY_ONCE > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.bind-host, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: classloader.resolve-order, parent-first > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: web.timeout, 50000 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.memory.process.size, 2048m > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: jobmanager.rpc.port, 6123 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.bind-address, 0.0.0.0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.interval, 1min > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.timeout, 3min > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.access-key, ak_flink > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: taskmanager.memory.managed.size, 0 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.externalized-checkpoint-retention, DELETE_ON_CANCELLATION > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: rest.address, 172.31.4.220 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: execution.checkpointing.max-concurrent-checkpoints, 2 > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: state.checkpoints.dir, s3://flink/checkpoints/ > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.flink.configuration.GlobalConfiguration:160] Loading configuration property: s3.path.style.access, true > 2024-09-11 14:54:09 | INFO | streampark-flink-app-bootstrap-0 | org.apache.paimon.flink.action.ActionFactory:73] mysql_sync_database job args: --mysql-conf hostname=172.31.4.149 --mysql-conf port=3306 --mysql-conf username=testuser --mysql-conf password=testuser --mysql-conf database-name=testflink --warehouse s3://flink/paimon/ --catalog_conf s3.endpoint=https://ossapi-tst --catalog_conf s3.access-key=ak_flink --catalog_conf s3.secret-key=ak_flink --catalog_conf s3.path.style.access=true --database ods --including_tables o.*|product.?|shipments --table_prefix my_ --table_suffix _001 --table_conf source.checkpoint-align.enabled=true --table_conf changelog-producer=input --table_conf sink.parallelism=1 > 2024-09-11 14:54:09 | WARN | streampark-flink-app-bootstrap-0 | org.apache.paimon.utils.HadoopUtils:125] Could not find Hadoop configuration via any of the supported methods > 2024-09-11 14:54:10 | WARN | streampark-flink-app-bootstrap-0 | org.apache.hadoop.util.NativeCodeLoader:60] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable > 14:54:10.664 [streampark-flink-app-bootstrap-0] ERROR org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] REMOTE mode submit by jobGraph fail.org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme 's3' in the classpath. FlinkFileIOLoader also cannot access this path. Hadoop FileSystem also cannot access this path 's3://flink/paimon'. > at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) > at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:158) > at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:82) > at org.apache.streampark.flink.client.trait.FlinkClientTrait.getJobGraph(FlinkClientTrait.scala:252) > at org.apache.streampark.flink.client.trait.FlinkClientTrait.getJobGraph$(FlinkClientTrait.scala:231) > at org.apache.streampark.flink.client.impl.RemoteClient$.jobGraphSubmit(RemoteClient.scala:139) > at org.apache.streampark.flink.client.impl.RemoteClient$.$anonfun$doSubmit$1(RemoteClient.scala:48) > at org.apache.streampark.flink.client.trait.FlinkClientTrait.$anonfun$trySubmit$1(FlinkClientTrait.scala:207) > at scala.util.Try$.apply(Try.scala:209) > at org.apache.streampark.flink.client.trait.FlinkClientTrait.trySubmit(FlinkClientTrait.scala:205) > at org.apache.streampark.flink.client.trait.FlinkClientTrait.trySubmit$(FlinkClientTrait.scala:201) > at org.apache.streampark.flink.client.impl.RemoteClient$.doSubmit(RemoteClient.scala:49) > at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit(FlinkClientTrait.scala:123) > at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit$(FlinkClientTrait.scala:60) > at org.apache.streampark.flink.client.impl.RemoteClient$.submit(RemoteClient.scala:34) > at org.apache.streampark.flink.client.FlinkClientHandler$.submit(FlinkClientHandler.scala:40) > at org.apache.streampark.flink.client.FlinkClientHandler.submit(FlinkClientHandler.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.streampark.flink.client.FlinkClient$.$anonfun$proxy$1(FlinkClient.scala:87) > at org.apache.streampark.flink.proxy.FlinkShimsProxy$.$anonfun$proxy$1(FlinkShimsProxy.scala:60) > at org.apache.streampark.common.util.ClassLoaderUtils$.runAsClassLoader(ClassLoaderUtils.scala:38) > at org.apache.streampark.flink.proxy.FlinkShimsProxy$.proxy(FlinkShimsProxy.scala:60) > at org.apache.streampark.flink.client.FlinkClient$.proxy(FlinkClient.scala:82) > at org.apache.streampark.flink.client.FlinkClient$.submit(FlinkClient.scala:53) > at org.apache.streampark.flink.client.FlinkClient.submit(FlinkClient.scala) > at org.apache.streampark.console.core.service.impl.ApplicationServiceImpl.lambda$start$8(ApplicationServiceImpl.java:1655) > at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.UncheckedIOException: org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme 's3' in the classpath. FlinkFileIOLoader also cannot access this path. Hadoop FileSystem also cannot access this path 's3://flink/paimon'. > at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:92) > at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) > at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:80) > at org.apache.paimon.flink.action.ActionBase.initPaimonCatalog(ActionBase.java:71) > at org.apache.paimon.flink.action.ActionBase.<init>(ActionBase.java:58) > at org.apache.paimon.flink.action.cdc.SynchronizationActionBase.<init>(SynchronizationActionBase.java:77) > at org.apache.paimon.flink.action.cdc.SyncDatabaseActionBase.<init>(SyncDatabaseActionBase.java:61) > at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseAction.<init>(MySqlSyncDatabaseAction.java:108) > at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:52) > at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:31) > at org.apache.paimon.flink.action.cdc.SynchronizationActionFactoryBase.create(SynchronizationActionFactoryBase.java:45) > at org.apache.paimon.flink.action.cdc.SyncDatabaseActionFactoryBase.create(SyncDatabaseActionFactoryBase.java:44) > at org.apache.paimon.flink.action.ActionFactory.createAction(ActionFactory.java:82) > at org.apache.paimon.flink.action.FlinkActions.main(FlinkActions.java:38) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ... 33 more > Caused by: org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme 's3' in the classpath. FlinkFileIOLoader also cannot access this path. Hadoop FileSystem also cannot access this path 's3://flink/paimon'. > at org.apache.paimon.fs.FileIO.get(FileIO.java:420) > at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:89) > ... 51 more > Suppressed: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/ for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:515) > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409) > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) > at org.apache.paimon.flink.FlinkFileIO.getFileSystem(FlinkFileIO.java:127) > at org.apache.paimon.flink.FlinkFileIO.exists(FlinkFileIO.java:100) > at org.apache.paimon.fs.FileIOUtils.checkAccess(FileIOUtils.java:37) > at org.apache.paimon.fs.FileIO.get(FileIO.java:388) > ... 52 more > Suppressed: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3" > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) > at org.apache.paimon.fs.hadoop.HadoopFileIO.createFileSystem(HadoopFileIO.java:175) > at org.apache.paimon.fs.hadoop.HadoopFileIO.getFileSystem(HadoopFileIO.java:168) > at org.apache.paimon.fs.hadoop.HadoopFileIO.getFileSystem(HadoopFileIO.java:145) > at org.apache.paimon.fs.hadoop.HadoopFileIO.exists(HadoopFileIO.java:110) > at org.apache.paimon.fs.FileIOUtils.checkAccess(FileIOUtils.java:37) > at org.apache.paimon.fs.FileIO.get(FileIO.java:397) > ... 52 more > 14:54:14.550 [streampark-flink-app-bootstrap-0] INFO org.apache.streampark.flink.client.impl.RemoteClient - [StreamPark] REMOTE mode submit by restApi, WebInterfaceURL http://172.31.4.220:8081, jobId: 8f0fb2ab846651cf78dd908a5a0bb610 > 2024-09-11 14:54:14 | INFO | streampark-flink-app-bootstrap-0 | org.apache.streampark.console.core.task.FlinkAppHttpWatcher:635] [StreamPark][FlinkAppHttpWatcher] setOptioning > 2024-09-11 14:54:14 | INFO | streampark-flink-app-bootstrap-0 | org.apache.streampark.console.core.task.FlinkAppHttpWatcher:646] [StreamPark][FlinkAppHttpWatcher] add app to tracking,appId:100001 > ^C > ``` hello，我也遇到了这个问题，看起来是streampark提交flink任务的时候缺少了s3的插件，你那边解决了嘛？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
