[
https://issues.apache.org/jira/browse/PIG-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974479#comment-15974479
]
Adam Szita commented on PIG-5218:
---------------------------------
The problem is again the usage of spark-assembly fat jar but this time on the
frontend. It contains a version of jcl-over-slf4j and its LogFactory will
produce a logger which is not compatible with that of Hadoop's. (See exception
in description). {{org.apache.hadoop.fs.FileContextCreateMkdirBaseTest}} tries
to set a log level which fails because the logger is not an instance of
Log4JLogger.
I fixed this in [^PIG-5218.0.patch], whereby placing commons-logging into lib
folder results in that jar to be used over spark-assembly, and I also provided
a JVM arg setting in this test so that a proper type of LogFactory
implementation will be picked up
[here|https://github.com/apache/commons-logging/blob/trunk/src/main/java/org/apache/commons/logging/LogFactory.java#L484]
[~kellyzly] can you take a look please?
> Jyhton_Checkin_3 fails with spark exec type
> -------------------------------------------
>
> Key: PIG-5218
> URL: https://issues.apache.org/jira/browse/PIG-5218
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Adam Szita
> Assignee: Adam Szita
> Fix For: spark-branch
>
> Attachments: PIG-5218.0.patch
>
>
> Exception observed:
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.commons.logging.impl.SLF4JLocationAwareLog cannot be cast to
> org.apache.commons.logging.impl.Log4JLogger
> at
> org.apache.hadoop.test.GenericTestUtils.setLogLevel(GenericTestUtils.java:107)
> at
> org.apache.hadoop.fs.FileContextCreateMkdirBaseTest.<clinit>(FileContextCreateMkdirBaseTest.java:60)
> ... 29 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
