>From Ritik Raj <[email protected]>: Attention is currently required from: Murtadha Hubail. Ritik Raj has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249 )
Change subject: [ASTERIXDB-3636][STO] Add ColumnBufferPool to prevent OOM during high-volume column ingestion ...................................................................... Patch Set 7: (20 comments) File asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/NCAppRuntimeContext.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/20716294_acabcc3b PS6, Line 332: 0.0 > make a constant indicating that column buffer memory is unlimited Done File asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/StorageProperties.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/511069c6_d673caf1 PS6, Line 77: STORAGE_COLUMN_BUFFER_MAX_SIZE_PERCENTAGE > Maybe STORAGE_COLUMN_BUFFER_POOL_MEMORY_PERCENTAGE Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/a9e2f41f_46878ca2 PS6, Line 78: STORAGE_COLUMN_BUFFER_POOL_MAX_SIZE > make ranged (1, xxxxxxx) Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/d54c77b9_922c184b PS6, Line 258: double poolSizePercentage = accessor.getDouble(Option.STORAGE_COLUMN_BUFFER_MAX_SIZE_PERCENTAGE); : long maxMemoryForColumnBuffers = (long) (MAX_HEAP_BYTES * (poolSizePercentage / 100)); : long maxSpaceAcquiredByPool = (long) getColumnBufferPoolMaxSize() * getColumnBufferSize(); : if (maxSpaceAcquiredByPool > maxMemoryForColumnBuffers) { : throw new IllegalStateException( : "Invalid column buffer pool configuration, more memory budgeted than available in JVM. JVM max memory: " : + MAX_HEAP_BYTES + ", Column buffer size: " + getColumnBufferSize() : + ", Column buffer pool size: " + getColumnBufferPoolMaxSize() : + ", Column buffer pool percentage: " + poolSizePercentage); : } > I don't think you will need this once you make the configuration accept only > valid values Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/3dded491_d3e39ac6 PS6, Line 295: // Should we consider only the column buffer pool size? > account for the reserved percentage Done File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/AbstractColumnTupleWriter.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/7d9b300f_d4fe0d43 PS6, Line 102: closeOnAbort > abort Done File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/btree/ColumnBTreeBulkloader.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/3e21dc2c_f72f2287 PS6, Line 63: writersInitialized > writerInitialized Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/653f4c68_c722105f PS6, Line 67: columnCredits > just called reservedBufferCount or something like that Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/23554d23_f5871cc7 PS6, Line 155: if (columnCredits > 0) { : columnBufferPool.release(columnCredits); : columnCredits = 0; : } > refactor Done File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/ColumnBufferPool.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/59d3a35c_0ce3d962 PS6, Line 36: > remove Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/a37b9241_9e621291 PS6, Line 46: numFailedRecycle > remove if not too useful Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/bdb8ad26_009be7a9 PS6, Line 83: nitialized: poolSize={}, cappedMemoryInBytes={}, maxCredits={}, bufferGranuleInBytes={}, reserveTimeoutMillis={}", > remember to update with the new APIs names Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/f39240f4_e43c0367 PS6, Line 129: permits > use same language as reserve Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/d0b58a7b_dd212a6d PS6, Line 168: buffer.clear(); > remove one of the clears Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/746f5635_ff21bfab PS6, Line 192: LOGGER.info > error Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/d7b1bc7f_c87797af PS6, Line 194: columnBufferGranuleInBytes > update wording Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/33abf245_730c3ff7 PS6, Line 200: memory limit reached > be specific about which memory limit was reached Done File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/IBufferCache.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/2384c417_37e6ef7b PS6, Line 277: > revert Done File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/IColumnBufferPool.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/0bf70159_97c1124f PS6, Line 25: > remove Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249/comment/db030c37_e6772675 PS6, Line 29: Credits > unify the wording Done -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20249 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: ionic Gerrit-Change-Id: I62437839d89b11d950e6715a00e844aedd0dab8e Gerrit-Change-Number: 20249 Gerrit-PatchSet: 7 Gerrit-Owner: Ritik Raj <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-CC: Murtadha Hubail <[email protected]> Gerrit-Attention: Murtadha Hubail <[email protected]> Gerrit-Comment-Date: Tue, 26 Aug 2025 10:50:51 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Murtadha Hubail <[email protected]> Gerrit-MessageType: comment
