This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new d09d49d [SPARK-27870][PYTHON][FOLLOW-UP] Rename
spark.sql.pandas.udf.buffer.size to spark.sql.execution.pandas.udf.buffer.size
d09d49d is described below
commit d09d49df2ec457bbc4bcb6357920d16943bc4016
Author: HyukjinKwon <[email protected]>
AuthorDate: Wed Feb 5 11:38:33 2020 +0900
[SPARK-27870][PYTHON][FOLLOW-UP] Rename spark.sql.pandas.udf.buffer.size to
spark.sql.execution.pandas.udf.buffer.size
### What changes were proposed in this pull request?
This PR renames `spark.sql.pandas.udf.buffer.size` to
`spark.sql.execution.pandas.udf.buffer.size` to be more consistent with other
pandas configuration prefixes, given:
- `spark.sql.execution.pandas.arrowSafeTypeConversion`
- `spark.sql.execution.pandas.respectSessionTimeZone`
- `spark.sql.legacy.execution.pandas.groupedMap.assignColumnsByName`
- other configurations like `spark.sql.execution.arrow.*`.
### Why are the changes needed?
To make configuration names consistent.
### Does this PR introduce any user-facing change?
No because this configuration was not released yet.
### How was this patch tested?
Existing tests should cover.
Closes #27450 from HyukjinKwon/SPARK-27870-followup.
Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
---
python/pyspark/sql/tests/test_pandas_udf_scalar.py | 2 +-
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/sql/tests/test_pandas_udf_scalar.py
b/python/pyspark/sql/tests/test_pandas_udf_scalar.py
index 8e7e85f..b07de3c 100644
--- a/python/pyspark/sql/tests/test_pandas_udf_scalar.py
+++ b/python/pyspark/sql/tests/test_pandas_udf_scalar.py
@@ -868,7 +868,7 @@ class ScalarPandasUDFTests(ReusedSQLTestCase):
with QuietTest(self.sc):
with
self.sql_conf({"spark.sql.execution.arrow.maxRecordsPerBatch": 1,
- "spark.sql.pandas.udf.buffer.size": 4}):
+
"spark.sql.execution.pandas.udf.buffer.size": 4}):
self.spark.range(10).repartition(1) \
.select(test_close(col("id"))).limit(2).collect()
# wait here because python udf worker will take some time
to detect
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 2b5c68e..f6b0bbd 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -1819,7 +1819,7 @@ object SQLConf {
.createWithDefault(10000)
val PANDAS_UDF_BUFFER_SIZE =
- buildConf("spark.sql.pandas.udf.buffer.size")
+ buildConf("spark.sql.execution.pandas.udf.buffer.size")
.doc(
s"Same as `${BUFFER_SIZE.key}` but only applies to Pandas UDF
executions. If it is not " +
s"set, the fallback is `${BUFFER_SIZE.key}`. Note that Pandas
execution requires more " +
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
