This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 77797db [SPARK-31780][K8S][TESTS] Add R test tag to exclude R K8s
image building and test
77797db is described below
commit 77797db60bc7207bc10fa01ebd5b55e61a870cb5
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed May 20 18:33:38 2020 -0700
[SPARK-31780][K8S][TESTS] Add R test tag to exclude R K8s image building
and test
### What changes were proposed in this pull request?
This PR aims to skip R image building and one R test during integration
tests by using `--exclude-tags r`.
### Why are the changes needed?
We have only one R integration test case, `Run SparkR on simple dataframe.R
example`, for submission test coverage. Since this is rarely changed, we can
skip this and save the efforts required for building the whole R image and
running the single test.
```
KubernetesSuite:
...
- Run SparkR on simple dataframe.R example
Run completed in 10 minutes, 20 seconds.
Total number of tests run: 20
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the K8S integration test and do the following manually. (Note that R
test is skipped)
```
$
resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
--deploy-mode docker-for-desktop --exclude-tags r --spark-tgz $PWD/spark-*.tgz
...
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Test basic decommissioning
Run completed in 10 minutes, 23 seconds.
Total number of tests run: 19
Suites: completed 2, aborted 0
Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
Closes #28594 from dongjoon-hyun/SPARK-31780.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit a06768ec4d5059d1037086fe5495e5d23cde514b)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
resource-managers/kubernetes/integration-tests/pom.xml | 3 +++
.../integration-tests/scripts/setup-integration-test-env.sh | 10 +++++++++-
.../spark/deploy/k8s/integrationtest/KubernetesSuite.scala | 1 +
.../apache/spark/deploy/k8s/integrationtest/RTestsSuite.scala | 4 ++--
4 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/resource-managers/kubernetes/integration-tests/pom.xml
b/resource-managers/kubernetes/integration-tests/pom.xml
index 5241257..3222ec7 100644
--- a/resource-managers/kubernetes/integration-tests/pom.xml
+++ b/resource-managers/kubernetes/integration-tests/pom.xml
@@ -121,6 +121,9 @@
<argument>--spark-tgz</argument>
<argument>${spark.kubernetes.test.sparkTgz}</argument>
+
+ <argument>--test-exclude-tags</argument>
+ <argument>"${test.exclude.tags}"</argument>
</arguments>
</configuration>
</execution>
diff --git
a/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
b/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
index 2e72ff8..beda56c 100755
---
a/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
+++
b/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
@@ -26,6 +26,7 @@ IMAGE_TAG="N/A"
JAVA_IMAGE_TAG="8-jre-slim"
SPARK_TGZ="N/A"
MVN="$TEST_ROOT_DIR/build/mvn"
+EXCLUDE_TAGS=""
# Parse arguments
while (( "$#" )); do
@@ -58,6 +59,10 @@ while (( "$#" )); do
SPARK_TGZ="$2"
shift
;;
+ --test-exclude-tags)
+ EXCLUDE_TAGS="$2"
+ shift
+ ;;
*)
break
;;
@@ -99,7 +104,10 @@ then
LANGUAGE_BINDING_BUILD_ARGS="-p
$DOCKER_FILE_BASE_PATH/bindings/python/Dockerfile"
# Build SparkR image
- LANGUAGE_BINDING_BUILD_ARGS="$LANGUAGE_BINDING_BUILD_ARGS -R
$DOCKER_FILE_BASE_PATH/bindings/R/Dockerfile"
+ tags=(${EXCLUDE_TAGS//,/ })
+ if [[ ! ${tags[@]} =~ "r" ]]; then
+ LANGUAGE_BINDING_BUILD_ARGS="$LANGUAGE_BINDING_BUILD_ARGS -R
$DOCKER_FILE_BASE_PATH/bindings/R/Dockerfile"
+ fi
# Unset SPARK_HOME to let the docker-image-tool script detect SPARK_HOME.
Otherwise, it cannot
# indicate the unpacked directory as its home. See SPARK-28550.
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
index 0d4fccc..00996df 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
@@ -419,6 +419,7 @@ class KubernetesSuite extends SparkFunSuite
private[spark] object KubernetesSuite {
val k8sTestTag = Tag("k8s")
+ val rTestTag = Tag("r")
val MinikubeTag = Tag("minikube")
val SPARK_PI_MAIN_CLASS: String = "org.apache.spark.examples.SparkPi"
val SPARK_DFS_READ_WRITE_TEST = "org.apache.spark.examples.DFSReadWriteTest"
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/RTestsSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/RTestsSuite.scala
index e81562a..b7c8886 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/RTestsSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/RTestsSuite.scala
@@ -19,9 +19,9 @@ package org.apache.spark.deploy.k8s.integrationtest
private[spark] trait RTestsSuite { k8sSuite: KubernetesSuite =>
import RTestsSuite._
- import KubernetesSuite.k8sTestTag
+ import KubernetesSuite.{k8sTestTag, rTestTag}
- test("Run SparkR on simple dataframe.R example", k8sTestTag) {
+ test("Run SparkR on simple dataframe.R example", k8sTestTag, rTestTag) {
sparkAppConf.set("spark.kubernetes.container.image", rImage)
runSparkApplicationAndVerifyCompletion(
appResource = SPARK_R_DATAFRAME_TEST,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
