This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8399dd321af [SPARK-44539][BUILD] Upgrade RoaringBitmap to 1.0.0
8399dd321af is described below
commit 8399dd321afce0cb0051501de55da296595fdf53
Author: panbingkun <[email protected]>
AuthorDate: Wed Sep 27 11:53:53 2023 -0500
[SPARK-44539][BUILD] Upgrade RoaringBitmap to 1.0.0
### What changes were proposed in this pull request?
- The pr aims to upgrade RoaringBitmap from 0.9.45 to 1.0.0.
- From version 1.0.0, the `ArraysShim` class has been moved from
`shims-x.x.x.jar` jar to `RoaringBitmap-x.x.x.jar` jar, so we no longer need to
rely on it.
### Why are the changes needed?
- The newest brings some improvments, eg:
Add zero-garbage deserialiser for ByteBuffer to RoaringBitmap by shikharid
in https://github.com/RoaringBitmap/RoaringBitmap/pull/650
More specialized method for value decrementation by xtonik in
https://github.com/RoaringBitmap/RoaringBitmap/pull/640
Duplicated small array sort routine by xtonik in
https://github.com/RoaringBitmap/RoaringBitmap/pull/638
Avoid intermediate byte array creation by xtonik in
https://github.com/RoaringBitmap/RoaringBitmap/pull/635
Useless back and forth BD bytes conversion by xtonik in
https://github.com/RoaringBitmap/RoaringBitmap/pull/636
- The full release notes:
https://github.com/RoaringBitmap/RoaringBitmap/releases/tag/1.0.0
https://github.com/RoaringBitmap/RoaringBitmap/releases/tag/0.9.49
https://github.com/RoaringBitmap/RoaringBitmap/releases/tag/0.9.48
https://github.com/RoaringBitmap/RoaringBitmap/releases/tag/0.9.47
https://github.com/RoaringBitmap/RoaringBitmap/releases/tag/0.9.46
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
Closes #42143 from panbingkun/SPARK-44539.
Authored-by: panbingkun <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
---
core/benchmarks/MapStatusesConvertBenchmark-jdk21-results.txt | 6 +++---
core/benchmarks/MapStatusesConvertBenchmark-results.txt | 8 ++++----
dev/deps/spark-deps-hadoop-3-hive-2.3 | 3 +--
pom.xml | 2 +-
4 files changed, 9 insertions(+), 10 deletions(-)
diff --git a/core/benchmarks/MapStatusesConvertBenchmark-jdk21-results.txt
b/core/benchmarks/MapStatusesConvertBenchmark-jdk21-results.txt
index 48dbc8e0241..416aaf5b7aa 100644
--- a/core/benchmarks/MapStatusesConvertBenchmark-jdk21-results.txt
+++ b/core/benchmarks/MapStatusesConvertBenchmark-jdk21-results.txt
@@ -6,8 +6,8 @@ OpenJDK 64-Bit Server VM 21+35 on Linux 5.15.0-1046-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
MapStatuses Convert: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Num Maps: 50000 Fetch partitions:500 813 900
129 0.0 812807240.0 1.0X
-Num Maps: 50000 Fetch partitions:1000 2226 2238
17 0.0 2226321250.0 0.4X
-Num Maps: 50000 Fetch partitions:1500 3149 3300
133 0.0 3148506179.0 0.3X
+Num Maps: 50000 Fetch partitions:500 899 949
74 0.0 898941184.0 1.0X
+Num Maps: 50000 Fetch partitions:1000 1947 2043
115 0.0 1947362412.0 0.5X
+Num Maps: 50000 Fetch partitions:1500 3079 3122
75 0.0 3078809212.0 0.3X
diff --git a/core/benchmarks/MapStatusesConvertBenchmark-results.txt
b/core/benchmarks/MapStatusesConvertBenchmark-results.txt
index 5ed55c839eb..bd87f4876e4 100644
--- a/core/benchmarks/MapStatusesConvertBenchmark-results.txt
+++ b/core/benchmarks/MapStatusesConvertBenchmark-results.txt
@@ -3,11 +3,11 @@ MapStatuses Convert Benchmark
================================================================================================
OpenJDK 64-Bit Server VM 17.0.8+7-LTS on Linux 5.15.0-1046-azure
-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
MapStatuses Convert: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Num Maps: 50000 Fetch partitions:500 1127 1138
13 0.0 1127479807.0 1.0X
-Num Maps: 50000 Fetch partitions:1000 2146 2183
49 0.0 2146214882.0 0.5X
-Num Maps: 50000 Fetch partitions:1500 3138 3204
69 0.0 3138378370.0 0.4X
+Num Maps: 50000 Fetch partitions:500 1062 1065
3 0.0 1061535322.0 1.0X
+Num Maps: 50000 Fetch partitions:1000 2110 2124
20 0.0 2109563282.0 0.5X
+Num Maps: 50000 Fetch partitions:1500 3166 3228
70 0.0 3165927840.0 0.3X
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 5c17d727b0a..8c53a71c76a 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -1,7 +1,7 @@
HikariCP/2.5.1//HikariCP-2.5.1.jar
JLargeArrays/1.5//JLargeArrays-1.5.jar
JTransforms/3.1//JTransforms-3.1.jar
-RoaringBitmap/0.9.45//RoaringBitmap-0.9.45.jar
+RoaringBitmap/1.0.0//RoaringBitmap-1.0.0.jar
ST4/4.0.4//ST4-4.0.4.jar
activation/1.1.1//activation-1.1.1.jar
aircompressor/0.25//aircompressor-0.25.jar
@@ -236,7 +236,6 @@
scala-parallel-collections_2.13/1.0.4//scala-parallel-collections_2.13-1.0.4.jar
scala-parser-combinators_2.13/2.3.0//scala-parser-combinators_2.13-2.3.0.jar
scala-reflect/2.13.11//scala-reflect-2.13.11.jar
scala-xml_2.13/2.2.0//scala-xml_2.13-2.2.0.jar
-shims/0.9.45//shims-0.9.45.jar
slf4j-api/2.0.9//slf4j-api-2.0.9.jar
snakeyaml-engine/2.6//snakeyaml-engine-2.6.jar
snakeyaml/2.0//snakeyaml-2.0.jar
diff --git a/pom.xml b/pom.xml
index 28199ff3d25..20ca97dc498 100644
--- a/pom.xml
+++ b/pom.xml
@@ -826,7 +826,7 @@
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
- <version>0.9.45</version>
+ <version>1.0.0</version>
</dependency>
<!-- Netty Begin -->
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
