Repository: spark Updated Branches: refs/heads/master 8cd6eea62 -> 90a6a46bd
[SPARK-4522][SQL] Parse schema with missing metadata. This is just a quick fix for 1.2. SPARK-4523 describes a more complete solution. Author: Michael Armbrust <[email protected]> Closes #3392 from marmbrus/parquetMetadata and squashes the following commits: bcc6626 [Michael Armbrust] Parse schema with missing metadata. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/90a6a46b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/90a6a46b Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/90a6a46b Branch: refs/heads/master Commit: 90a6a46bd11030672597f015dd443d954107123a Parents: 8cd6eea Author: Michael Armbrust <[email protected]> Authored: Thu Nov 20 20:34:43 2014 -0800 Committer: Michael Armbrust <[email protected]> Committed: Thu Nov 20 20:34:43 2014 -0800 ---------------------------------------------------------------------- .../scala/org/apache/spark/sql/catalyst/types/dataTypes.scala | 6 ++++++ 1 file changed, 6 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/90a6a46b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala ---------------------------------------------------------------------- diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala index ff1dc03..892b7e1 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala @@ -84,6 +84,12 @@ object DataType { ("nullable", JBool(nullable)), ("type", dataType: JValue)) => StructField(name, parseDataType(dataType), nullable, Metadata.fromJObject(metadata)) + // Support reading schema when 'metadata' is missing. + case JSortedObject( + ("name", JString(name)), + ("nullable", JBool(nullable)), + ("type", dataType: JValue)) => + StructField(name, parseDataType(dataType), nullable) } @deprecated("Use DataType.fromJson instead", "1.2.0") --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
