njnu-seafish commented on PR #17824:
URL:
https://github.com/apache/dolphinscheduler/pull/17824#issuecomment-3691301811
> We don't need to import `hive-exec` here, have you install the
`dolphinscheduler-hdfs` plugin?
In my environment, if the hive-exec dependency is not added, SqlTask tasks
for Spark data sources will encounter the following error:
> 2025-12-25 16:56:33.130 ERROR [exclusive-task-executor-container-worker-1]
- PhysicalTaskExecutor{id=400, name=sql-spark, stat=RUNNING} execute failed
java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/thrift/TFilterTransport
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:493)
at java.net.URLClassLoader.access$100(URLClassLoader.java:75)
at java.net.URLClassLoader$1.run(URLClassLoader.java:389)
at java.net.URLClassLoader$1.run(URLClassLoader.java:383)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at
org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601)
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at
org.apache.dolphinscheduler.plugin.datasource.spark.param.SparkDataSourceProcessor.getConnection(SparkDataSourceProcessor.java:148)
at
org.apache.dolphinscheduler.plugin.datasource.api.client.BaseAdHocDataSourceClient.getConnection(BaseAdHocDataSourceClient.java:41)
at
org.apache.dolphinscheduler.plugin.datasource.api.plugin.DataSourceClientProvider.getAdHocConnection(DataSourceClientProvider.java:95)
at
org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:179)
at
org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:151)
at
org.apache.dolphinscheduler.server.worker.executor.PhysicalTaskExecutor.doTriggerTaskPlugin(PhysicalTaskExecutor.java:77)
at
org.apache.dolphinscheduler.task.executor.AbstractTaskExecutor.start(AbstractTaskExecutor.java:81)
at
org.apache.dolphinscheduler.task.executor.worker.TaskExecutorWorker.start(TaskExecutorWorker.java:65)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.thrift.TFilterTransport
at java.net.URLClassLoader.findClass(URLClassLoader.java:407)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 31 common frames omitted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
