Repository: spark
Updated Branches:
refs/heads/master 4700adb98 -> 0d51b6044
[SPARK-14072][CORE] Show JVM/OS version information when we run a benchmark
program
## What changes were proposed in this pull request?
This PR allows us to identify what JVM is used when someone ran a benchmark
program. In some cases, a JVM version may affect performance result. Thus, it
would be good to show processor information and JVM version information.
```
model name : Intel(R) Xeon(R) CPU E5-2697 v2 2.70GHz
JVM information : OpenJDK 64-Bit Server VM,
1.7.0_65-mockbuild_2014_07_14_06_19-b00
Int and String Scan: Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------
SQL Parquet Vectorized 981 / 994 10.7 93.5
1.0X
SQL Parquet MR 2518 / 2542 4.2 240.1
0.4X
```
```
model name : Intel(R) Xeon(R) CPU E5-2697 v2 2.70GHz
JVM information : IBM J9 VM, pxa6480sr2-20151023_01 (SR2)
String Dictionary: Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------
SQL Parquet Vectorized 693 / 740 15.1 66.1
1.0X
SQL Parquet MR 2501 / 2562 4.2 238.5
0.3X
```
## How was this patch tested?
Tested by using existing benchmark programs
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Author: Kazuaki Ishizaki <[email protected]>
Closes #11893 from kiszk/SPARK-14072.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0d51b604
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0d51b604
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0d51b604
Branch: refs/heads/master
Commit: 0d51b60443ae78fa46988a6aed2397db9c35f96d
Parents: 4700adb
Author: Kazuaki Ishizaki <[email protected]>
Authored: Tue Mar 22 21:01:52 2016 -0700
Committer: Reynold Xin <[email protected]>
Committed: Tue Mar 22 21:01:52 2016 -0700
----------------------------------------------------------------------
.../scala/org/apache/spark/util/Benchmark.scala | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/0d51b604/core/src/main/scala/org/apache/spark/util/Benchmark.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/Benchmark.scala
b/core/src/main/scala/org/apache/spark/util/Benchmark.scala
index b562b58..9e40baf 100644
--- a/core/src/main/scala/org/apache/spark/util/Benchmark.scala
+++ b/core/src/main/scala/org/apache/spark/util/Benchmark.scala
@@ -64,6 +64,7 @@ private[spark] class Benchmark(
val firstBest = results.head.bestMs
// The results are going to be processor specific so it is useful to
include that.
+ println(Benchmark.getJVMOSInfo())
println(Benchmark.getProcessorName())
printf("%-35s %16s %12s %13s %10s\n", name + ":", "Best/Avg Time(ms)",
"Rate(M/s)",
"Per Row(ns)", "Relative")
@@ -91,16 +92,31 @@ private[spark] object Benchmark {
* This should return something like "Intel(R) Core(TM) i7-4870HQ CPU @
2.50GHz"
*/
def getProcessorName(): String = {
- if (SystemUtils.IS_OS_MAC_OSX) {
+ val cpu = if (SystemUtils.IS_OS_MAC_OSX) {
Utils.executeAndGetOutput(Seq("/usr/sbin/sysctl", "-n",
"machdep.cpu.brand_string"))
} else if (SystemUtils.IS_OS_LINUX) {
Try {
val grepPath = Utils.executeAndGetOutput(Seq("which", "grep"))
Utils.executeAndGetOutput(Seq(grepPath, "-m", "1", "model name",
"/proc/cpuinfo"))
+ .replaceFirst("model name[\\s*]:[\\s*]", "")
}.getOrElse("Unknown processor")
} else {
System.getenv("PROCESSOR_IDENTIFIER")
}
+ cpu
+ }
+
+ /**
+ * This should return a user helpful JVM & OS information.
+ * This should return something like
+ * "OpenJDK 64-Bit Server VM 1.8.0_65-b17 on Linux 4.1.13-100.fc21.x86_64"
+ */
+ def getJVMOSInfo(): String = {
+ val vmName = System.getProperty("java.vm.name")
+ val runtimeVersion = System.getProperty("java.runtime.version")
+ val osName = System.getProperty("os.name")
+ val osVersion = System.getProperty("os.version")
+ s"${vmName} ${runtimeVersion} on ${osName} ${osVersion}"
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
