This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b4d88beb65f5 [SPARK-54059][CORE] Reduce default page size by
LONG_ARRAY_OFFSET if ZGC or ShenandoahGC and ON_HEAP are used
b4d88beb65f5 is described below
commit b4d88beb65f59e2041cb7fc8379c4212d83640b6
Author: WanKun <[email protected]>
AuthorDate: Tue Oct 28 08:40:00 2025 -0700
[SPARK-54059][CORE] Reduce default page size by LONG_ARRAY_OFFSET if ZGC or
ShenandoahGC and ON_HEAP are used
### What changes were proposed in this pull request?
If ZGC or ShenandoahGC and ON_HEAP are used, we should consider the long
array object header size.
Test ZGC and ShenandoahGC with and without this optimization:
Test code:
https://gist.githubusercontent.com/wankunde/b7cf073b31f3a4bf5b525ab4a6b1f700/raw/85db977f5ef8609db27ced49d10ace9a248433c1/TestBlockSize.java
Test result:
| | Max allocated size without optimization | Max allocated
size with optimization |
|--------------|-----------------------------------------|---------------------------------------|
| G1GC | 817889280 | 1019211984
|
| ZGC | 671088640 | 1006629120
|
| ShenandoahGC | 855638016 | 964686240
|
Detail test result:
https://gist.github.com/wankunde/6f3b6ed8abaa247307469ab5a4fae2bb
### Why are the changes needed?
Optimize spark memory usage.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #52754 from wankunde/zgc2.
Lead-authored-by: WanKun <[email protected]>
Co-authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/memory/MemoryManager.scala | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/memory/MemoryManager.scala
b/core/src/main/scala/org/apache/spark/memory/MemoryManager.scala
index c33fca35764e..639b82b6080b 100644
--- a/core/src/main/scala/org/apache/spark/memory/MemoryManager.scala
+++ b/core/src/main/scala/org/apache/spark/memory/MemoryManager.scala
@@ -244,9 +244,9 @@ private[spark] abstract class MemoryManager(
* by looking at the number of cores available to the process, and the total
amount of memory,
* and then divide it by a factor of safety.
*
- * SPARK-37593 If we are using G1GC, it's better to take the
LONG_ARRAY_OFFSET
- * into consideration so that the requested memory size is power of 2
- * and can be divided by G1 heap region size to reduce memory waste within
one G1 region.
+ * SPARK-37593 If we are using G1GC, ZGC or ShenandoahGC, it's better to
take the
+ * LONG_ARRAY_OFFSET into consideration so that the requested memory size is
power of 2
+ * and can be divided by heap region size to reduce memory waste.
*/
private lazy val defaultPageSizeBytes = {
val minPageSize = 1L * 1024 * 1024 // 1MB
@@ -260,7 +260,8 @@ private[spark] abstract class MemoryManager(
}
val size = ByteArrayMethods.nextPowerOf2(maxTungstenMemory / cores /
safetyFactor)
val chosenPageSize = math.min(maxPageSize, math.max(minPageSize, size))
- if (Utils.isG1GC && tungstenMemoryMode == MemoryMode.ON_HEAP) {
+ if ((Utils.isG1GC || Utils.isZGC || Utils.isShenandoahGC) &&
+ tungstenMemoryMode == MemoryMode.ON_HEAP) {
chosenPageSize - Platform.LONG_ARRAY_OFFSET
} else {
chosenPageSize
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
