[
https://issues.apache.org/jira/browse/HADOOP-13386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826000#comment-16826000
]
Kalman commented on HADOOP-13386:
---------------------------------
Hi [~arpitagarwal]
Regarding the dependency, it is not really transitive dependency but run time
dependency:
Sqoop is using avro 1.8.1 and parquet 1.8.x for a while and Hadoop 3.1.1
contains avro 1.7.7.
During the execution avro 1.7.7 will be used and the following runtime error
occured when sqoop tried to import to hive in parquet format:
{code}
ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child :
java.lang.NoSuchMethodError:
org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType;
at
org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:178)
{code}
So the version update what is defined by this task would fix our problem.
As validation we executed the standard set of hdp-3.1.0 system tests with
Hadoop, Hive, Sqoop, HBase etc components and we haven't found any issue in
connection with the version update.
> Upgrade Avro to 1.8.x
> ---------------------
>
> Key: HADOOP-13386
> URL: https://issues.apache.org/jira/browse/HADOOP-13386
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
> Reporter: Ben McCann
> Assignee: Kalman
> Priority: Major
>
> Avro 1.8.x makes generated classes serializable which makes them much easier
> to use with Spark. It would be great to upgrade Avro to 1.8.x
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
