[
https://issues.apache.org/jira/browse/SPARK-50126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17903274#comment-17903274
]
Pan Bingkun commented on SPARK-50126:
-------------------------------------
Okay, thank you!
Let's skip it in the the upcoming version `3.5.4`.
> PySpark expr() (expression) SQL Function returns None in Spark Connect
> ----------------------------------------------------------------------
>
> Key: SPARK-50126
> URL: https://issues.apache.org/jira/browse/SPARK-50126
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 3.5.0, 3.5.1, 3.5.2, 3.5.3
> Environment: Spark 3.5.3
> Python 3.9.20
> Reporter: Nick Herzig
> Priority: Major
> Labels: connect, expression, pyspark, pyspark.sql.functions,
> remote
>
> When PySpark is launched with remote Spark Connect configuration, calls to
> pyspark.sql.functions.expr always return `Column<None>`.
>
> PySpark without Spark Connect
> {code:java}
> $ pyspark
> ...
> >>> from pyspark.sql.functions import expr
> >>> e = expr("*")
> >>> print(e)
> Column<'unresolvedstar()'>
> {code}
> PySpark with Spark Connect
> {code:java}
> $ pyspark --remote "sc://localhost:15002"
> ...
> Client connected to the Spark Connect server at localhost:15002
> SparkSession available as 'spark'.
> >>> from pyspark.sql.functions import expr
> >>> e = expr("*")
> >>> print(e)
> Column<'None'>
> {code}
> Additional example
> {code:java}
> $ pyspark
> ...
> Spark context available as 'sc' (master = local[*], app id =
> local-1729876117745).
> SparkSession available as 'spark'.
> >>> from pyspark.sql.functions import expr
> >>> expr("*")
> Column<'unresolvedstar()'>
> >>> SparkSession.builder.master("local[*]").getOrCreate().stop()
> >>> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> >>> expr("*")
> Column<'None'> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
