Repository: spark Updated Branches: refs/heads/branch-1.2 e445d3ce4 -> 668643b8d
[SPARK-4522][SQL] Parse schema with missing metadata. This is just a quick fix for 1.2. SPARK-4523 describes a more complete solution. Author: Michael Armbrust <[email protected]> Closes #3392 from marmbrus/parquetMetadata and squashes the following commits: bcc6626 [Michael Armbrust] Parse schema with missing metadata. (cherry picked from commit 90a6a46bd11030672597f015dd443d954107123a) Signed-off-by: Michael Armbrust <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/668643b8 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/668643b8 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/668643b8 Branch: refs/heads/branch-1.2 Commit: 668643b8de0958094766fa62e7e2a7a0909f11da Parents: e445d3c Author: Michael Armbrust <[email protected]> Authored: Thu Nov 20 20:34:43 2014 -0800 Committer: Michael Armbrust <[email protected]> Committed: Thu Nov 20 20:34:55 2014 -0800 ---------------------------------------------------------------------- .../scala/org/apache/spark/sql/catalyst/types/dataTypes.scala | 6 ++++++ 1 file changed, 6 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/668643b8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala ---------------------------------------------------------------------- diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala index ff1dc03..892b7e1 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala @@ -84,6 +84,12 @@ object DataType { ("nullable", JBool(nullable)), ("type", dataType: JValue)) => StructField(name, parseDataType(dataType), nullable, Metadata.fromJObject(metadata)) + // Support reading schema when 'metadata' is missing. + case JSortedObject( + ("name", JString(name)), + ("nullable", JBool(nullable)), + ("type", dataType: JValue)) => + StructField(name, parseDataType(dataType), nullable) } @deprecated("Use DataType.fromJson instead", "1.2.0") --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
