This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b13dc699f7d [SPARK-41990][SQL][FOLLOWUP] Add comments to explain why
`FieldReference.column` is used when `ParseException` happens
b13dc699f7d is described below
commit b13dc699f7df589663b1239e5e551815d9043f38
Author: huaxingao <[email protected]>
AuthorDate: Mon Jan 16 16:52:46 2023 +0800
[SPARK-41990][SQL][FOLLOWUP] Add comments to explain why
`FieldReference.column` is used when `ParseException` happens
### What changes were proposed in this pull request?
Add comments to explain why `FieldReference.column` is used when
`ParseException` happens
### Why are the changes needed?
To address this
[comment](https://github.com/apache/spark/pull/39564#discussion_r1070788648)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Only comments are added. No need to test.
Closes #39597 from huaxingao/followup.
Authored-by: huaxingao <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../src/main/scala/org/apache/spark/sql/sources/filters.scala | 6 ++++++
1 file changed, 6 insertions(+)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/sources/filters.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/sources/filters.scala
index 080d17b47fb..af5e4f5ef5a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/sources/filters.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/sources/filters.scala
@@ -80,6 +80,12 @@ sealed abstract class Filter {
try {
FieldReference(attribute)
} catch {
+ // The column name in V1 Filter is generated by PushableColumn. When
nested
+ // predicate push down is enabled, PushableColumn quotes the column name.
+ // However, when nested predicate push down is disabled, PushableColumn
+ // does not quote the column. If the column name is not a valid SQL
identifier,
+ // FieldReference.apply throws ParseException. We catch the
ParseException here
+ // and use FieldReference.column instead.
case _: ParseException =>
FieldReference.column(attribute)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
