This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from 68928757c209 [SPARK-54062][SQL] MergeScalarSubqueries code cleanup
add 9b7ab21c9ef0 [SPARK-53975][PYTHON] Adds basic Python worker logging
support
No new revisions were added by this update.
Summary of changes:
.../spark/api/python/PythonWorkerFactory.scala | 7 +-
.../spark/api/python/PythonWorkerLogCapture.scala | 193 ++++++++++++++
.../scala/org/apache/spark/storage/BlockId.scala | 38 ++-
.../apache/spark/storage/LogBlockIdGenerator.scala | 11 +
.../scala/org/apache/spark/storage/LogLine.scala | 5 +
.../apache/spark/storage/BlockManagerSuite.scala | 51 ++++
python/pyspark/logger/logger.py | 10 +-
python/pyspark/logger/worker_io.py | 296 +++++++++++++++++++++
.../sql/tests/arrow/test_arrow_python_udf.py | 12 +
python/pyspark/sql/tests/test_udf.py | 102 +++++++
python/pyspark/testing/connectutils.py | 16 +-
python/pyspark/testing/sqlutils.py | 6 +
python/pyspark/worker.py | 14 +-
.../sql/catalyst/analysis/RelationResolution.scala | 20 +-
.../plans/logical/pythonLogicalOperators.scala | 46 +++-
.../org/apache/spark/sql/internal/SQLConf.scala | 11 +
.../spark/sql/connect/service/SessionHolder.scala | 2 +
.../spark/sql/api/python/PythonSQLUtils.scala | 18 ++
.../apache/spark/sql/classic/SparkSession.scala | 10 +
.../spark/sql/execution/SparkStrategies.scala | 2 +
.../sql/execution/python/BatchEvalPythonExec.scala | 11 +-
.../sql/execution/python/PythonUDFRunner.scala | 19 +-
.../execution/python/PythonWorkerLogsExec.scala | 63 +++++
.../execution/python/PythonWorkerLogsSuite.scala | 73 +++++
24 files changed, 1009 insertions(+), 27 deletions(-)
create mode 100644
core/src/main/scala/org/apache/spark/api/python/PythonWorkerLogCapture.scala
create mode 100644 python/pyspark/logger/worker_io.py
create mode 100644
sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerLogsExec.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonWorkerLogsSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
