This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 0a4f985c [SPARK-23098][SQL] Migrate Kafka Batch source to v2.
add 02f4763 [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an
iterator of DataFrames
No new revisions were added by this update.
Summary of changes:
.../org/apache/spark/api/python/PythonRunner.scala | 2 +
python/pyspark/rdd.py | 1 +
python/pyspark/sql/dataframe.py | 48 +++++++-
python/pyspark/sql/tests/test_pandas_udf_iter.py | 135 +++++++++++++++++++++
python/pyspark/worker.py | 62 ++++++----
.../plans/logical/pythonLogicalOperators.scala | 12 ++
.../main/scala/org/apache/spark/sql/Dataset.scala | 21 +++-
.../spark/sql/execution/SparkStrategies.scala | 2 +
.../python/MapPartitionsInPandasExec.scala | 95 +++++++++++++++
9 files changed, 353 insertions(+), 25 deletions(-)
create mode 100644 python/pyspark/sql/tests/test_pandas_udf_iter.py
create mode 100644
sql/core/src/main/scala/org/apache/spark/sql/execution/python/MapPartitionsInPandasExec.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
