This is an automated email from the ASF dual-hosted git repository.
maxgekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new dabd21249f1e [SPARK-50904][SQL] Fix collation expression walker query
execution
dabd21249f1e is described below
commit dabd21249f1eae38e8e866521caee6cd387c8851
Author: Stefan Kandic <[email protected]>
AuthorDate: Tue Jan 21 18:47:56 2025 +0200
[SPARK-50904][SQL] Fix collation expression walker query execution
### What changes were proposed in this pull request?
Changing when we collect results in `CollationExpressionWalkerSuite` on
borders of changing session default collation.
### Why are the changes needed?
Because we are analyzing the query and then setting a different default
collation, this leads to problems for runtime replaceable expressions which
will only be resolved when we call `collect` and by then the session default
collation is changed. This is blocking the change in #49576.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
This is a test-only change.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #49586 from stefankandic/fixExpressionWalker.
Authored-by: Stefan Kandic <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
.../scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala
index 1f9589c1c9ce..4602c1a9cc1a 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/CollationExpressionWalkerSuite.scala
@@ -740,10 +740,10 @@ class CollationExpressionWalkerSuite extends
SparkFunSuite with SharedSparkSessi
for (funInfo <- funInfos.filter(f => !toSkip.contains(f.getName))) {
for (query <- "> .*;".r.findAllIn(funInfo.getExamples).map(s =>
s.substring(2))) {
try {
- val resultUTF8 = sql(query)
+ val resultUTF8 = sql(query).collect()
withSQLConf(SqlApiConf.DEFAULT_COLLATION -> "UTF8_LCASE") {
- val resultUTF8Lcase = sql(query)
- assert(resultUTF8.collect() === resultUTF8Lcase.collect())
+ val resultUTF8Lcase = sql(query).collect()
+ assert(resultUTF8 === resultUTF8Lcase)
}
} catch {
case e: SparkRuntimeException => assert(e.getCondition ==
"USER_RAISED_EXCEPTION")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
