Repository: spark
Updated Branches:
refs/heads/branch-1.5 0887e5e87 -> 59747d085
[SPARK-10534] [SQL] ORDER BY clause allows only columns that are present in the
select projection list
Find out the missing attributes by recursively looking
at the sort order expression and rest of the code
takes care of projecting them out.
Added description from cloud-fan
I wanna explain a bit more about this bug.
When we resolve sort ordering, we will use a special method, which only
resolves UnresolvedAttributes and UnresolvedExtractValue. However, for
something like Floor('a), even the 'a is resolved, the floor expression may
still being unresolved as data type mismatch(for example, 'a is string type and
Floor need double type), thus can't pass this filter, and we can't push down
this missing attribute 'a
Author: Dilip Biswal <[email protected]>
Closes #9123 from dilipbiswal/SPARK-10534.
(cherry picked from commit 49ea0e9d7ce805d312d94a5b2936eec2053bc052)
Signed-off-by: Yin Huai <[email protected]>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/59747d08
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/59747d08
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/59747d08
Branch: refs/heads/branch-1.5
Commit: 59747d085ba072ede3b40c9680d2801dd868a844
Parents: 0887e5e
Author: Dilip Biswal <[email protected]>
Authored: Wed Oct 21 11:10:32 2015 -0700
Committer: Yin Huai <[email protected]>
Committed: Wed Oct 21 11:10:44 2015 -0700
----------------------------------------------------------------------
.../org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 2 +-
.../spark/sql/catalyst/analysis/AnalysisSuite.scala | 10 ++++++++++
2 files changed, 11 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/59747d08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
----------------------------------------------------------------------
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 47db448..0f459cd 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -482,7 +482,7 @@ class Analyzer(
val newOrdering = resolveSortOrders(ordering, grandchild, throws = true)
// Construct a set that contains all of the attributes that we need to
evaluate the
// ordering.
- val requiredAttributes = AttributeSet(newOrdering.filter(_.resolved))
+ val requiredAttributes = AttributeSet(newOrdering).filter(_.resolved)
// Figure out which ones are missing from the projection, so that we can
add them and
// remove them after the sort.
val missingInProject = requiredAttributes -- child.output
http://git-wip-us.apache.org/repos/asf/spark/blob/59747d08/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
----------------------------------------------------------------------
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
index 820b336..275295c 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
@@ -135,4 +135,14 @@ class AnalysisSuite extends AnalysisTest {
plan = testRelation.select(CreateStructUnsafe(Seq(a, (a +
1).as("a+1"))).as("col"))
checkAnalysis(plan, plan)
}
+
+ test("SPARK-10534: resolve attribute references in order by clause") {
+ val a = testRelation2.output(0)
+ val c = testRelation2.output(2)
+
+ val plan = testRelation2.select('c).orderBy(Floor('a).asc)
+ val expected = testRelation2.select(c,
a).orderBy(Floor(a.cast(DoubleType)).asc).select(c)
+
+ checkAnalysis(plan, expected)
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
