github-actions[bot] commented on code in PR #69770:
URL: https://github.com/apache/doris/pull/69770#discussion_r4400239600
##########
be/src/storage/index/inverted/snii/encoding/zstd_codec.cpp:
##########
@@ -49,4 +49,36 @@ Status zstd_decompress(Slice input, size_t
expected_uncomp_len, std::vector<uint
return Status::OK();
}
+ZstdDecompressor::~ZstdDecompressor() {
+ ZSTD_freeDCtx(static_cast<ZSTD_DCtx*>(_context));
+}
+
+Status ZstdDecompressor::decompress(Slice input, size_t expected_uncomp_len,
Slice* out) {
+ if (_context == nullptr) {
+ _context = ZSTD_createDCtx();
+ if (_context == nullptr) {
+ return Status::Error<ErrorCode::MEM_ALLOC_FAILED, false>(
+ "zstd decompress: cannot create a context");
+ }
+ }
+ if (expected_uncomp_len > _capacity) {
+ // Every byte is written by the decompression before it is read, so
the buffer is not
+ // zero-filled.
+ _buffer.reset(new uint8_t[expected_uncomp_len]);
Review Comment:
[P2] Track the retained decompression buffer in the query MemTracker
This raw `new[]` now keeps the largest decoded PRX buffer until
`SniiIndexSource` is destroyed, but neither the buffer nor its owner charges
those bytes to a MemTracker. On this head, `runtime/memory/jemalloc_hook.cpp`
only forwards malloc/free to jemalloc; the limit checks and consume/release
calls live in Doris `Allocator`. Thus a phrase/scored query retains its
high-water plaintext allocation outside query accounting even while later
windows or other SEARCH fields are evaluated (the reader accepts up to 256 MiB
per frame). The previous local vector was also untracked, but released that
memory after each frame; extending its lifetime makes this a sustained
query-limit bypass. Use `DorisUniqueBufferPtr`/`make_unique_buffer` or
equivalent balanced tracked allocation for the retained capacity;
`Allocator<false>` preserves the intended lack of zero-fill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
