This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1e76dd8395ef [SPARK-55519][SQL][TESTS] Fix V2FunctionBenchmark
1e76dd8395ef is described below
commit 1e76dd8395ef82d2de3302d0baa0d39a9b1406f9
Author: Cheng Pan <[email protected]>
AuthorDate: Fri Feb 27 08:11:59 2026 -0800
[SPARK-55519][SQL][TESTS] Fix V2FunctionBenchmark
### What changes were proposed in this pull request?
Fix V2FunctionBenchmark, caused by SPARK-53968
(https://github.com/apache/spark/pull/52681)
```
$ build/sbt "sql/Test/runMain
org.apache.spark.sql.connector.functions.V2FunctionBenchmark"
...
[info] running (fork)
org.apache.spark.sql.connector.functions.V2FunctionBenchmark
[error] WARNING: Using incubator modules: jdk.incubator.vector
[info] 13:43:41.084 WARN org.apache.spark.util.Utils: Your hostname,
H27212-MAC-01.local, resolves to a loopback address: 127.0.0.1; using
10.242.159.140 instead (on interface en0)
[info] 13:43:41.087 WARN org.apache.spark.util.Utils: Set SPARK_LOCAL_IP if
you need to bind to another address
[info] 13:43:41.260 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to
load native-hadoop library for your platform... using builtin-java classes
where applicable
[info] Running benchmark: scalar function (long + long) -> long,
result_nullable = true codegen = true
[info] Running case: native_long_add
[error] Exception in thread "main" java.lang.NullPointerException: Cannot
invoke
"org.apache.spark.sql.catalyst.expressions.NumericEvalContext.evalMode()"
because the return value of
"org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.evalContext()" is
null
[error] at
org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.evalMode(arithmetic.scala:201)
[error] at
org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.failOnError(arithmetic.scala:209)
[error] at
org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.initQueryContext(arithmetic.scala:251)
[error] at
org.apache.spark.sql.catalyst.expressions.SupportQueryContext.$init$(Expression.scala:670)
[error] at
org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.<init>(arithmetic.scala:193)
[error] at
org.apache.spark.sql.connector.functions.V2FunctionBenchmark$NativeAdd.<init>(V2FunctionBenchmark.scala:111)
[error] at
org.apache.spark.sql.connector.functions.V2FunctionBenchmark$.$anonfun$scalarFunctionBenchmark$3(V2FunctionBenchmark.scala:88)
...
```
### Why are the changes needed?
Fix the benchmark.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Verified locally, GHA benchmark reports for JDK 17 and 21 are updated.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54540 from pan3793/SPARK-55744.
Lead-authored-by: Cheng Pan <[email protected]>
Co-authored-by: pan3793 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../V2FunctionBenchmark-jdk21-results.txt | 56 +++++++++++-----------
.../benchmarks/V2FunctionBenchmark-results.txt | 56 +++++++++++-----------
.../connector/functions/V2FunctionBenchmark.scala | 6 ++-
3 files changed, 60 insertions(+), 58 deletions(-)
diff --git a/sql/core/benchmarks/V2FunctionBenchmark-jdk21-results.txt
b/sql/core/benchmarks/V2FunctionBenchmark-jdk21-results.txt
index 363936391a63..38945637091f 100644
--- a/sql/core/benchmarks/V2FunctionBenchmark-jdk21-results.txt
+++ b/sql/core/benchmarks/V2FunctionBenchmark-jdk21-results.txt
@@ -1,44 +1,44 @@
-OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
+OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
scalar function (long + long) -> long, result_nullable = true codegen = true:
Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------
-native_long_add
9308 9710 686 53.7 18.6 1.0X
-java_long_add_default
22865 23994 1622 21.9 45.7 0.4X
-java_long_add_magic
11505 11983 804 43.5 23.0 0.8X
-java_long_add_static_magic
11395 11452 49 43.9 22.8 0.8X
-scala_long_add_default
23153 23807 968 21.6 46.3 0.4X
-scala_long_add_magic
11562 11585 38 43.2 23.1 0.8X
+native_long_add
9357 9828 787 53.4 18.7 1.0X
+java_long_add_default
23136 23301 258 21.6 46.3 0.4X
+java_long_add_magic
11555 11895 550 43.3 23.1 0.8X
+java_long_add_static_magic
11457 11505 52 43.6 22.9 0.8X
+scala_long_add_default
23402 23436 48 21.4 46.8 0.4X
+scala_long_add_magic
11533 11657 149 43.4 23.1 0.8X
-OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
+OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
scalar function (long + long) -> long, result_nullable = false codegen = true:
Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
-native_long_add
10290 10340 49 48.6 20.6 1.0X
-java_long_add_default
22328 22368 42 22.4 44.7 0.5X
-java_long_add_magic
11485 11572 118 43.5 23.0 0.9X
-java_long_add_static_magic
9962 9980 19 50.2 19.9 1.0X
-scala_long_add_default
22314 22377 104 22.4 44.6 0.5X
-scala_long_add_magic
11503 12078 705 43.5 23.0 0.9X
+native_long_add
10313 10344 34 48.5 20.6 1.0X
+java_long_add_default
22388 22880 536 22.3 44.8 0.5X
+java_long_add_magic
11491 11582 156 43.5 23.0 0.9X
+java_long_add_static_magic
9980 13846 6696 50.1 20.0 1.0X
+scala_long_add_default
22311 22344 30 22.4 44.6 0.5X
+scala_long_add_magic
11541 11627 77 43.3 23.1 0.9X
-OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
+OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
scalar function (long + long) -> long, result_nullable = true codegen = false:
Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
-native_long_add
22531 22615 108 22.2 45.1 1.0X
-java_long_add_default
26677 26707 32 18.7 53.4 0.8X
-java_long_add_magic
32551 32589 55 15.4 65.1 0.7X
-java_long_add_static_magic
30687 30774 95 16.3 61.4 0.7X
-scala_long_add_default
26292 26366 86 19.0 52.6 0.9X
-scala_long_add_magic
32777 32981 245 15.3 65.6 0.7X
+native_long_add
22692 22767 99 22.0 45.4 1.0X
+java_long_add_default
26553 26619 68 18.8 53.1 0.9X
+java_long_add_magic
32637 32804 285 15.3 65.3 0.7X
+java_long_add_static_magic
30751 30819 60 16.3 61.5 0.7X
+scala_long_add_default
26453 26489 42 18.9 52.9 0.9X
+scala_long_add_magic
32736 32891 263 15.3 65.5 0.7X
-OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
+OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
scalar function (long + long) -> long, result_nullable = false codegen =
false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns)
Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------
-native_long_add
22593 22807 331 22.1 45.2 1.0X
-java_long_add_default
26325 26370 46 19.0 52.7 0.9X
-java_long_add_magic
32796 32827 42 15.2 65.6 0.7X
-java_long_add_static_magic
30699 30874 163 16.3 61.4 0.7X
-scala_long_add_default
26341 26371 45 19.0 52.7 0.9X
-scala_long_add_magic
32794 32982 258 15.2 65.6 0.7X
+native_long_add
22670 22722 79 22.1 45.3 1.0X
+java_long_add_default
26424 26464 46 18.9 52.8 0.9X
+java_long_add_magic
32737 33464 1195 15.3 65.5 0.7X
+java_long_add_static_magic
30840 31079 380 16.2 61.7 0.7X
+scala_long_add_default
26384 26482 123 19.0 52.8 0.9X
+scala_long_add_magic
32628 32932 455 15.3 65.3 0.7X
diff --git a/sql/core/benchmarks/V2FunctionBenchmark-results.txt
b/sql/core/benchmarks/V2FunctionBenchmark-results.txt
index 142394b25ccf..adf993587564 100644
--- a/sql/core/benchmarks/V2FunctionBenchmark-results.txt
+++ b/sql/core/benchmarks/V2FunctionBenchmark-results.txt
@@ -1,44 +1,44 @@
-OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
+OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
scalar function (long + long) -> long, result_nullable = true codegen = true:
Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------------------------
-native_long_add
9211 9245 49 54.3 18.4 1.0X
-java_long_add_default
21996 22193 184 22.7 44.0 0.4X
-java_long_add_magic
10647 10743 91 47.0 21.3 0.9X
-java_long_add_static_magic
10559 10596 37 47.4 21.1 0.9X
-scala_long_add_default
23193 23214 20 21.6 46.4 0.4X
-scala_long_add_magic
10666 10686 30 46.9 21.3 0.9X
+native_long_add
9112 9184 81 54.9 18.2 1.0X
+java_long_add_default
22468 22546 89 22.3 44.9 0.4X
+java_long_add_magic
10635 10671 31 47.0 21.3 0.9X
+java_long_add_static_magic
10507 10579 63 47.6 21.0 0.9X
+scala_long_add_default
23158 23211 83 21.6 46.3 0.4X
+scala_long_add_magic
10646 10654 9 47.0 21.3 0.9X
-OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
+OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
scalar function (long + long) -> long, result_nullable = false codegen = true:
Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
-native_long_add
9988 10158 200 50.1 20.0 1.0X
-java_long_add_default
22418 22479 70 22.3 44.8 0.4X
-java_long_add_magic
10631 10657 23 47.0 21.3 0.9X
-java_long_add_static_magic
9864 9928 67 50.7 19.7 1.0X
-scala_long_add_default
22269 22354 76 22.5 44.5 0.4X
-scala_long_add_magic
10614 10670 52 47.1 21.2 0.9X
+native_long_add
9953 9984 28 50.2 19.9 1.0X
+java_long_add_default
22049 22151 105 22.7 44.1 0.5X
+java_long_add_magic
10611 10636 27 47.1 21.2 0.9X
+java_long_add_static_magic
9880 10707 1205 50.6 19.8 1.0X
+scala_long_add_default
22129 22175 78 22.6 44.3 0.4X
+scala_long_add_magic
10622 10712 95 47.1 21.2 0.9X
-OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
+OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
scalar function (long + long) -> long, result_nullable = true codegen = false:
Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------------------------------
-native_long_add
22860 23095 317 21.9 45.7 1.0X
-java_long_add_default
28130 28318 282 17.8 56.3 0.8X
-java_long_add_magic
30689 32298 2467 16.3 61.4 0.7X
-java_long_add_static_magic
30929 30969 35 16.2 61.9 0.7X
-scala_long_add_default
26355 27156 1356 19.0 52.7 0.9X
-scala_long_add_magic
30825 30861 33 16.2 61.7 0.7X
+native_long_add
23353 23427 116 21.4 46.7 1.0X
+java_long_add_default
28263 28280 15 17.7 56.5 0.8X
+java_long_add_magic
31114 31224 166 16.1 62.2 0.8X
+java_long_add_static_magic
31335 31963 1035 16.0 62.7 0.7X
+scala_long_add_default
26408 27086 882 18.9 52.8 0.9X
+scala_long_add_magic
31166 31292 136 16.0 62.3 0.7X
-OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
+OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
scalar function (long + long) -> long, result_nullable = false codegen =
false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns)
Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------
-native_long_add
21895 21957 74 22.8 43.8 1.0X
-java_long_add_default
25861 26173 537 19.3 51.7 0.8X
-java_long_add_magic
30646 30681 31 16.3 61.3 0.7X
-java_long_add_static_magic
30226 30256 32 16.5 60.5 0.7X
-scala_long_add_default
26041 26108 70 19.2 52.1 0.8X
-scala_long_add_magic
30668 30765 130 16.3 61.3 0.7X
+native_long_add
22149 22214 110 22.6 44.3 1.0X
+java_long_add_default
26026 26087 74 19.2 52.1 0.9X
+java_long_add_magic
30800 30875 67 16.2 61.6 0.7X
+java_long_add_static_magic
30936 31876 872 16.2 61.9 0.7X
+scala_long_add_default
26061 26084 35 19.2 52.1 0.8X
+scala_long_add_magic
30832 31138 451 16.2 61.7 0.7X
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/functions/V2FunctionBenchmark.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/functions/V2FunctionBenchmark.scala
index 592869968917..5d2cc1694eb7 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/functions/V2FunctionBenchmark.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/functions/V2FunctionBenchmark.scala
@@ -108,8 +108,10 @@ object V2FunctionBenchmark extends SqlBasedBenchmark {
case class NativeAdd(
left: Expression,
right: Expression,
- override val nullable: Boolean) extends BinaryArithmetic {
- override val evalContext: NumericEvalContext =
NumericEvalContext(EvalMode.LEGACY)
+ override val nullable: Boolean,
+ override val evalContext: NumericEvalContext =
NumericEvalContext(EvalMode.LEGACY))
+ extends BinaryArithmetic {
+
override def inputType: AbstractDataType = NumericType
override def symbol: String = "+"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
