This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new bff980f6acc5 [SPARK-51234][PYTHON][DOCS] Document an import change in
`from pyspark.sql.functions import *`
bff980f6acc5 is described below
commit bff980f6acc58520fb0e5f9d246a411c9979fe6c
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Tue Feb 18 10:19:34 2025 +0800
[SPARK-51234][PYTHON][DOCS] Document an import change in `from
pyspark.sql.functions import *`
### What changes were proposed in this pull request?
Document an import change in `from pyspark.sql.functions import *`
### Why are the changes needed?
to notify users
### Does this PR introduce _any_ user-facing change?
doc-only
### How was this patch tested?
ci
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #49975 from zhengruifeng/py_funcs_wildcard_import.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 2c76dff9536d409e97ae3692dec2705f9de7f6ac)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/docs/source/migration_guide/pyspark_upgrade.rst | 1 +
1 file changed, 1 insertion(+)
diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst
b/python/docs/source/migration_guide/pyspark_upgrade.rst
index 55d067eb5fa2..6ba86d7a7041 100644
--- a/python/docs/source/migration_guide/pyspark_upgrade.rst
+++ b/python/docs/source/migration_guide/pyspark_upgrade.rst
@@ -74,6 +74,7 @@ Upgrading from PySpark 3.5 to 4.0
* In Spark 4.0, the schema of a map column is inferred by merging the schemas
of all pairs in the map. To restore the previous behavior where the schema is
only inferred from the first non-null pair, you can set
``spark.sql.pyspark.legacy.inferMapTypeFromFirstPair.enabled`` to ``true``.
* In Spark 4.0, `compute.ops_on_diff_frames` is on by default. To restore the
previous behavior, set `compute.ops_on_diff_frames` to `false`.
* In Spark 4.0, the data type `YearMonthIntervalType` in ``DataFrame.collect``
no longer returns the underlying integers. To restore the previous behavior,
set ``PYSPARK_YM_INTERVAL_LEGACY`` environment variable to ``1``.
+* In Spark 4.0, items other than functions (e.g. ``DataFrame``, ``Column``,
``StructType``) have been removed from the wildcard import ``from
pyspark.sql.functions import *``, you should import these items from proper
modules (e.g. ``from pyspark.sql import DataFrame, Column``, ``from
pyspark.sql.types import StructType``).
Upgrading from PySpark 3.3 to 3.4
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
