This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new b70407eb815 [SPARK-43113][SQL][FOLLOWUP] Add comment about copying
steam-side variables
b70407eb815 is described below
commit b70407eb815ac97f5992b6cf961911e878ea5510
Author: Bruce Robbins <[email protected]>
AuthorDate: Fri Apr 21 10:24:56 2023 +0800
[SPARK-43113][SQL][FOLLOWUP] Add comment about copying steam-side variables
### What changes were proposed in this pull request?
Add a comment explaining a tricky situation involving the evaluation of
stream-side variables.
This is a follow-up to #40766.
### Why are the changes needed?
Make the code more clear.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
N/A
Closes #40881 from bersprockets/SPARK-43113_followup.
Authored-by: Bruce Robbins <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 6d4ed13c465ae7f35753a1f2b67f78c690881a9e)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala | 4 ++++
1 file changed, 4 insertions(+)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala
index ae91615da0f..a7d1edefcd6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala
@@ -42,6 +42,10 @@ trait JoinCodegenSupport extends CodegenSupport with
BaseJoinExec {
buildRow: Option[String] = None): (String, String, Seq[ExprCode]) = {
val buildSideRow = buildRow.getOrElse(ctx.freshName("buildRow"))
val buildVars = genOneSideJoinVars(ctx, buildSideRow, buildPlan,
setDefaultValue = false)
+ // We want to evaluate the passed streamVars. However, evaluation modifies
the contained
+ // ExprCode instances, which may surprise the caller to this method (in
particular,
+ // full outer join will want to evaluate streamVars in a different scope
than the
+ // condition check). Because of this, we first make a copy.
val streamVars2 = streamVars.map(_.copy())
val checkCondition = if (condition.isDefined) {
val expr = condition.get
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
