[
https://issues.apache.org/jira/browse/SPARK-56045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18066614#comment-18066614
]
Dongjoon Hyun commented on SPARK-56045:
---------------------------------------
This landed `branch-4.1` via https://github.com/apache/spark/pull/54885
> Parquet UNKNOWN Type Regression at Spark 4.1
> --------------------------------------------
>
> Key: SPARK-56045
> URL: https://issues.apache.org/jira/browse/SPARK-56045
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0, 4.1.1
> Reporter: Ziya Mukhtarov
> Assignee: Ziya Mukhtarov
> Priority: Major
> Labels: parquet, parquetReader, pull-request-available
> Fix For: 4.2.0
>
>
> Since Spark 4.1, which includes
> [SPARK-54220|https://issues.apache.org/jira/browse/SPARK-54220] NullType
> support in Parquet, the Parquet reader behavior for files written by external
> engines have changed. If the file contains a column type with UNKNOWN logical
> type annotation, before we were inferring a type based on the physical type,
> but since Spark 4.1, we are inferring NullType. As it is a regression, we
> need to introduce a new flag to control this behavior, with the default value
> resulting in the old behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
