This is an automated email from the ASF dual-hosted git repository.
weichenxu123 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from aed6a47580e [SPARK-43320][SQL][HIVE] Directly call Hive 2.3.9 API
add abd864766b0 [SPARK-43081][ML][CONNECT] Add torch distributor data
loader that loads data from spark partition data
No new revisions were added by this update.
Summary of changes:
dev/sparktestsupport/modules.py | 2 +
.../connect/test_parity_torch_data_loader.py} | 26 ++-
python/pyspark/ml/torch/data.py | 79 +++++++
python/pyspark/ml/torch/distributor.py | 259 ++++++++++++++++++---
python/pyspark/ml/torch/tests/test_data_loader.py | 136 +++++++++++
python/pyspark/ml/torch/tests/test_distributor.py | 2 +-
6 files changed, 461 insertions(+), 43 deletions(-)
copy python/pyspark/{pandas/tests/connect/test_parity_dataframe_spark_io.py =>
ml/tests/connect/test_parity_torch_data_loader.py} (61%)
create mode 100644 python/pyspark/ml/torch/data.py
create mode 100644 python/pyspark/ml/torch/tests/test_data_loader.py
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
