>From Ritik Raj <[email protected]>: Ritik Raj has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20283 )
Change subject: [ASTERIXDB-3636][STO] Fix buffer reservations understimation ...................................................................... Patch Set 16: (2 comments) File asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/StorageProperties.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20283/comment/33cea28e_2856df9e PS11, Line 266: StorageUtil.getLongSizeInBytes(100, MEGABYTE) > Sorry, I mean what use cases in general does 100MiB allow? I want to make > sure that the OOTB configu […] 100MiB is the upper cap, the pool starts with columnBufferPoolMaxSize/2 = 8000/2 = 4000. hence a total of 4000 * 4KiB = 16MiB, will be allocated. After that based on the usage pool size will grow till 8000 buffers. Once it crosses the 8000 buffers, rest of them won't be pooled and are subject to Garbage collection. so in the case where the usage is minimal the column pool won't reach is 100MiB. File asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/StorageProperties.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20283/comment/6935ecf7_6a7023db PS14, Line 80: getRangedLongByteUnit(0, MAX_HEAP_BYTES) > We should select a more sane upper bound- clearly we can't use 100% of the > max heap for the storage […] Setting the value to 100% does not mean that the entire heap will be reserved by the column buffer pool. Instead, it specifies that, in scenarios such as flush operations or bulk loading datasets with a large number of columns, the pool keep allocating buffers till it reach 100% of memory, which was the case before the column buffer pool change. However, the pool will retain at most 8,000 buffers of 4 KiB each (controlled by storageColumnBufferPoolSize). Any additional buffers beyond this limit will eventually be reclaimed by the garbage collector. In other words, the “100%” setting behaves similarly to the pre–column buffer pool implementation, where a new buffer was allocated whenever requested. The key difference is that now the pool will keep up to 8,000 buffers permanently, provided that at some point more than 8,000 buffers were allocated. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20283 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: ionic Gerrit-Change-Id: I3bfb9e9ac4b908ee2e7d84cab4781f6e0f7444e8 Gerrit-Change-Number: 20283 Gerrit-PatchSet: 16 Gerrit-Owner: Ritik Raj <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Michael Blow <[email protected]> Gerrit-Reviewer: Murtadha Hubail <[email protected]> Gerrit-Reviewer: Ritik Raj <[email protected]> Gerrit-Comment-Date: Wed, 03 Sep 2025 06:09:37 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Michael Blow <[email protected]> Comment-In-Reply-To: Ritik Raj <[email protected]> Gerrit-MessageType: comment
