[
https://issues.apache.org/jira/browse/SPARK-50126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17903251#comment-17903251
]
Pan Bingkun commented on SPARK-50126:
-------------------------------------
I verified on the master that it is correct, but it does have this issue on the
branch-3.5.
(pyspark) ➜ spark-community git:(master) ✗ ./bin/pyspark --remote
"sc://127.0.0.1:8888"
Python 3.12.7 | packaged by conda-forge | (main, Oct 4 2024, 15:57:01) [Clang
17.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 4.0.0.dev0
/_/
Using Python version 3.12.7 (main, Oct 4 2024 15:57:01)
Client connected to the Spark Connect server at 127.0.0.1:8888
SparkSession available as 'spark'.
>>> from pyspark.sql.functions import expr
>>> e = expr("*")
>>> print(e)
Column<'*'>
>>>
I wonder if we forgot to merge some PR into branch-3.5?
> PySpark expr() (expression) SQL Function returns None in Spark Connect
> ----------------------------------------------------------------------
>
> Key: SPARK-50126
> URL: https://issues.apache.org/jira/browse/SPARK-50126
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 3.5.0, 3.5.1, 3.5.2, 3.5.3
> Environment: Spark 3.5.3
> Python 3.9.20
> Reporter: Nick Herzig
> Priority: Major
> Labels: connect, expression, pyspark, pyspark.sql.functions,
> remote
> Fix For: 3.5.4
>
>
> When PySpark is launched with remote Spark Connect configuration, calls to
> pyspark.sql.functions.expr always return `Column<None>`.
>
> PySpark without Spark Connect
> {code:java}
> $ pyspark
> ...
> >>> from pyspark.sql.functions import expr
> >>> e = expr("*")
> >>> print(e)
> Column<'unresolvedstar()'>
> {code}
> PySpark with Spark Connect
> {code:java}
> $ pyspark --remote "sc://localhost:15002"
> ...
> Client connected to the Spark Connect server at localhost:15002
> SparkSession available as 'spark'.
> >>> from pyspark.sql.functions import expr
> >>> e = expr("*")
> >>> print(e)
> Column<'None'>
> {code}
> Additional example
> {code:java}
> $ pyspark
> ...
> Spark context available as 'sc' (master = local[*], app id =
> local-1729876117745).
> SparkSession available as 'spark'.
> >>> from pyspark.sql.functions import expr
> >>> expr("*")
> Column<'unresolvedstar()'>
> >>> SparkSession.builder.master("local[*]").getOrCreate().stop()
> >>> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> >>> expr("*")
> Column<'None'> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
