This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new dcaa10a520f [SPARK-44110][BUILD] Propagate proxy settings to forked
JVMs
dcaa10a520f is described below
commit dcaa10a520ff75f4551e134d6edae0db422fcf5e
Author: Steve Vaughan Jr <[email protected]>
AuthorDate: Thu Jun 22 17:29:20 2023 -0700
[SPARK-44110][BUILD] Propagate proxy settings to forked JVMs
### What changes were proposed in this pull request?
Propagate proxy settings to forked unit test JVMs. System properties that
start with `http.` or `https.` are copied into the `jvmOptions` of the forked
processes, ensuring that required proxy settings are available during the tests.
### Why are the changes needed?
Running a subset of unit tests fail in enterprise environments that have
additional constraints on network access that are handled by using a proxy.
Examples of impacted tests:
- HadoopVersionInfoSuite, test name: SPARK-32256: Hadoop VersionInfo should
be preloaded.
- HiveClientUserNameSuite(2.0), test name: 2.0: username of HiveClient - no
UGI.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
A local CI build was modified to use Spark branches (3.3 and 3.4) that
included this change to verify it addressed the test failures. In addition,
logging changes were introduced to verify that the system properties were
absent when running the tests prior to the change.
Closes #41678 from snmvaughan/feature/proxy-on-fork-master.
Authored-by: Steve Vaughan Jr <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
project/SparkBuild.scala | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 761b8f905f5..196e3c3ff1b 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1580,6 +1580,19 @@ object TestSettings {
"SPARK_TESTING" -> "1",
"JAVA_HOME" ->
sys.env.get("JAVA_HOME").getOrElse(sys.props("java.home")),
"SPARK_BEELINE_OPTS" -> "-DmyKey=yourValue"),
+
+ // Copy system properties to forked JVMs so that tests know proxy settings
+ (Test / javaOptions) ++= {
+ val q = "\""
+ sys.props.toList
+ .filter {
+ case (key, value) => key.startsWith("http.") ||
key.startsWith("https.")
+ }
+ .map {
+ case (key, value) => s"-D$key=$q$value$q"
+ }
+ },
+
(Test / javaOptions) += s"-Djava.io.tmpdir=$testTempDir",
(Test / javaOptions) += "-Dspark.test.home=" + sparkHome,
(Test / javaOptions) += "-Dspark.testing=1",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
