morningman opened a new issue, #70200: URL: https://github.com/apache/doris/issues/70200
### Background Every JNI reader in BE runs its Java scanner in the one JVM that BE embeds, with a 2 GB heap by default (`-Xmx` in `JAVA_OPTS_FOR_JDK_17` of be.conf). BE's memory trackers don't see that heap, and a scan's concurrency, up to 16 scanners per instance (`max_file_scanners_concurrency`), counts blocks, not Java objects. Most readers stream a few megabytes, but three kinds hold hundreds of megabytes from opening until they close: - a paimon split read through JNI that has to be merged (a primary-key table written since its last compaction) keeps a row group of every file it merges, about 130 MB per split on a 30M-row table; - a fluss primary-key bucket read whole (`PK_FULL`) replays the change log after the bucket's kv snapshot into a sorted map; - the log tail of a fluss union read of a primary-key table (`PK_TAIL`) keeps the last row of every key in its slice. ### Current behavior On a Release BE with the default 2 GB heap and 16 scanners per instance, each of these fails every time with `OutOfMemoryError: Java heap space`: - `SELECT *` over a 30M-row paimon primary-key table that is not compacted (32 JNI merge splits), through the paimon catalog or fluss's `$lake`; - `SELECT *` over a fluss primary-key table of 2M keys, 1M of them updated since its last kv snapshot, alone or four at once; - eight concurrent union reads of a primary-key table with 100K updated keys per bucket in its log tail. The error, `[JNI_ERROR]OutOfMemoryError: Java heap space`, names neither the JVM nor where its size is set, and the statements right after it fail with `[JNI_ERROR]Failed to attach the current thread to the JVM, code=-1` until the failed statement's scanners have exited. ### Expected behavior A user can have such reads wait for room in the heap instead of failing, without slowing the reads that hold little, and the out-of-memory error says which heap ran out and how to give it more. ### Proposed design As implemented in #68713: - A session variable, `enable_jni_heap_admission` (default `false`). When it is set, the connector that plans a range declares the heap its reader will hold in `TFileRangeDesc.jni_heap_bytes`: paimon for every split it reads through JNI, by its files' row groups and dictionary pages, laid out by the schema version each file was written under; fluss for `PK_FULL` and `PK_TAIL` ranges, by their offset span and the row layout of the columns read. - `JniScanHeapGate` in BE admits a declaring reader, before it creates its Java scanner, while the admitted declarations plus its own fit in `jni_scanner_heap_budget_ratio` (new BE config, default 0.5) of `-Xmx`: in arrival order, and always when no other reader holds a share. The share comes back when the Java scanner closes, since these readers hold their heap until then. A wait ends after `jni_scanner_heap_max_wait_ms` (new BE config, default 60000) per split. - A reader that doesn't fit opens nothing. Its scan task is parked off the scan workers and out of its context's in-flight count, and the gate's own thread resumes it once the reader is admitted. - The out-of-memory error, and the attach failure after it, end with a hint: the heap's maximum, where `-Xmx` is set, and the ways out. Alternatives considered and rejected: - Admitting by the measured heap (`totalMemory() - freeMemory()`): it counts uncollected garbage and every other user of the JVM, so readers waited the full 60 s while the live heap was 220-250 MB, and eight concurrent `SELECT *` over a fluss log table took a median 62 s against 28 s. - The same, plus `Runtime.gc()` once a reader has waited: each collection is a full GC that stops every JNI reader, Java UDF and libhdfs stream in the BE. - Waiting at the gate on the scan worker: the waiters can take the workers and context slots that admitted readers need to finish. With one reader fitting at a time, two concurrent queries stalled for 128-131 s and then ran out of heap. ### Impact Users who read paimon tables through JNI or fluss primary-key tables, on BEs with the default or a small JVM heap, in both shared-nothing and cloud mode. Modules: BE scan scheduling and JNI readers, and the paimon and fluss connectors in FE. ### Reproduction or evidence The reads under Current behavior, measured in #68713 with the variable off and on: for example, the fluss primary-key table takes 1.68-1.72 s and the paimon table 11.2-11.7 s with it on, while reads that declare nothing take as long as with it off. ### Scope and non-goals - In scope: the three kinds of reader above, and the out-of-memory error text. - Not in scope: bounding what readers actually hold as they read (the declarations are estimates made while planning); other JNI readers (hudi, JDBC, trino, max-compute, iceberg system tables), which declare nothing; native readers, which don't use the JVM heap. ### Acceptance criteria - With the variable on, the reads under Current behavior finish without running the heap out. - Every read returns the same rows with the variable on and off. - Reads that declare nothing are as fast with the variable on as off. - A waiting reader holds no scan thread, and the readers that hold shares finish without relying on the wait limit. - With the variable off, only the error text changes. ### Risks and compatibility - The declarations are estimates, which is why the variable is off by default. Fluss prices strings, bytes and nested values at 64 bytes, since its metadata carries no lengths, so tables with long values or keys are not protected; a paimon merge briefly holds about three times its declaration while decoding. - `TFileRangeDesc.jni_heap_bytes` is a new optional thrift field, 18 (branch-4.1 and branch-4.2 use 17). An old BE ignores it, and an old FE declares nothing. No metadata or data format changes. ### Related - #68713, the implementation - #68712 (scans run up to 16 scanners per instance again) and #68710 (an out-of-memory error no longer exits BE) - apache/doris-website#4186, the user docs on the Fluss catalog page -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
