This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new e3996f4 [SPARK-31767][PYTHON][CORE] Remove ResourceInformation in
pyspark module's namespace
e3996f4 is described below
commit e3996f4a8217244e276650b491ecbfd4611cfce8
Author: HyukjinKwon <[email protected]>
AuthorDate: Tue May 19 22:36:36 2020 -0700
[SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's
namespace
This PR proposes to only allow the import of `ResourceInformation` as below:
```
pyspark.resource.ResourceInformation
```
instead of
```
pyspark.ResourceInformation
pyspark.resource.ResourceInformation
```
because `pyspark.resource` is a separate module, and it is documented so.
The constructor of `ResourceInformation` isn't supposed to directly call
anyway.
To keep the code structure coherent.
No, it will be in the unreleased branches.
Manually tested via importing:
Before:
```python
>>> import pyspark
>>> pyspark.ResourceInformation
<class 'pyspark.resource.information.ResourceInformation'>
>>> pyspark.resource.ResourceInformation
<class 'pyspark.resource.information.ResourceInformation'>
```
After:
```python
>>> import pyspark
>>> pyspark.ResourceInformation
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'pyspark' has no attribute 'ResourceInformation'
>>> pyspark.resource.ResourceInformation
<class 'pyspark.resource.information.ResourceInformation'>
```
Also tested via
```bash
cd python
./run-tests --python-executables=python3
--modules=pyspark-core,pyspark-resource
```
Jenkins will test and existing tests should cover.
Closes #28589 from HyukjinKwon/SPARK-31767.
Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit dc3a606fbd37ede404a61551ff792fa94fe1780f)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/__init__.py | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/python/pyspark/__init__.py b/python/pyspark/__init__.py
index b485c3f..ee153af 100644
--- a/python/pyspark/__init__.py
+++ b/python/pyspark/__init__.py
@@ -54,7 +54,6 @@ from pyspark.files import SparkFiles
from pyspark.storagelevel import StorageLevel
from pyspark.accumulators import Accumulator, AccumulatorParam
from pyspark.broadcast import Broadcast
-from pyspark.resource import ResourceInformation
from pyspark.serializers import MarshalSerializer, PickleSerializer
from pyspark.status import *
from pyspark.taskcontext import TaskContext, BarrierTaskContext,
BarrierTaskInfo
@@ -119,5 +118,5 @@ __all__ = [
"SparkConf", "SparkContext", "SparkFiles", "RDD", "StorageLevel",
"Broadcast",
"Accumulator", "AccumulatorParam", "MarshalSerializer", "PickleSerializer",
"StatusTracker", "SparkJobInfo", "SparkStageInfo", "Profiler",
"BasicProfiler", "TaskContext",
- "RDDBarrier", "BarrierTaskContext", "BarrierTaskInfo",
"ResourceInformation",
+ "RDDBarrier", "BarrierTaskContext", "BarrierTaskInfo",
]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
