[
https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700799#comment-16700799
]
Maxim Gekk commented on SPARK-23410:
------------------------------------
> Even if lineSeps is set, it is still necessary to identify the file bom
> charset.
For sure encoding should be inferred from BOM in any case. I just want to take
you attention on the case when when lineSep is not set. We infer lineSep in
UTF-8, and most likely should do the same for other encodings.
> In my opinion, we can try to read the first four bytes of the file on the
> executor side to identify the encoding of the file.Because once the charset
> of the file is determined, the charset of lineSeps is also determined.
For JSON, we create JacksonParser on the driver side before file reading. For
example:
https://github.com/apache/spark/blob/e9af9460bc008106b670abac44a869721bfde42a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala#L105-L107
. Need to take this into account.
[~x1q1j1] Please, open a PR, I am ready to review it.
> I know BOM is only the beginning of the file ..
[~hyukjin.kwon] I just double checked ;-)
> Unable to read jsons in charset different from UTF-8
> ----------------------------------------------------
>
> Key: SPARK-23410
> URL: https://issues.apache.org/jira/browse/SPARK-23410
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Maxim Gekk
> Priority: Major
> Attachments: utf16WithBOM.json
>
>
> Currently the Json Parser is forced to read json files in UTF-8. Such
> behavior breaks backward compatibility with Spark 2.2.1 and previous versions
> that can read json files in UTF-16, UTF-32 and other encodings due to using
> of the auto detection mechanism of the jackson library. Need to give back to
> users possibility to read json files in specified charset and/or detect
> charset automatically as it was before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
