matanper opened a new issue, #68970: URL: https://github.com/apache/doris/issues/68970
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, arm64). master is the same: - `be/src/io/fs/s3_file_bufferpool.cpp:68-74`: `FileBuffer::PartData` allocates `Memory<>(config::s3_write_buffer_size)` in its constructor, i.e. a whole part, when the buffer is built. - `be/src/io/fs/s3_file_writer.cpp:340-342`: `S3FileWriter::appendv` builds that buffer on the first append of every part. - `be/src/io/fs/s3_file_writer.cpp:269-280`: `_submit_upload_buffer` submits the full part to `S3FileUploadThreadPool` with no limit. - `be/src/runtime/exec_env_init.cpp:290-293`: that pool is built without `set_max_queue_size`, so its queue is `INT_MAX` (`be/src/util/threadpool.cpp:71`). ### Description Load S3 upload buffer memory has no bound. It grows in three ways: 1. **Each buffer is a whole part from the first byte.** `PartData` allocates `s3_write_buffer_size` up front, and the `S3FileBuffer` tracker is charged the full size before the malloc (`FileBufferBuilder::build`, `s3_file_bufferpool.cpp:244-245`). So every open segment, index or packed-file writer holds a full part, even when the file is a few hundred KB. 2. **Submitted parts are not capped.** Once a part is full it is queued on the upload pool. When S3 uploads fall behind, parts pile up in the queue with no limit, and each one still holds its full buffer. 3. **Load writes are not throttled on this memory.** `MemTableMemoryLimiter` only looks at memtable memory and process memory. Upload buffers are freed only when their upload (and file-cache copy) finishes, so loads keep producing parts while uploads are behind. Many buffers can exist at once. There is one per concurrent memtable flush; with adaptive flush threads that is up to 4 × CPUs, plus the high-priority pool. There is one per async index-file close, one per packed-file writer, and one per queued part. What we measured: a 192-vCPU BE with `s3_write_buffer_size = 64 MiB` and a write-through file cache, under heavy concurrent Arrow Stream Load into a table with a large VARIANT column. `S3FileBuffer` peaked at 210 GB tracked (78 GB current), process memory reached the limit, and loads were cancelled with `MEM_LIMIT_EXCEEDED`. In an earlier run, `S3FileBuffer` reached ~197 GB and the container hit cgroup OOM pressure. The 210 GB is the tracked figure, not RSS. Tracked bytes still matter: they decide which task is cancelled, and `Allocator::sys_memory_check` adds every full-part request to process memory before it compares against the limit. With the default 5 MiB part, each buffer is 13× smaller, but the count is still unbounded. ### Solution We have a patch in three parts. All the new configs are mutable, and with the defaults behaviour does not change. 1. **Grow upload buffers on demand.** A new config `s3_write_buffer_initial_size` (default 256 KiB) sets the starting size. `UploadFileBuffer::append_data` doubles the buffer up to the part capacity, using `Allocator::realloc` under the `S3FileBuffer` tracker, so the tracker charges what is actually allocated. Growth happens only on the appending thread, before submit, so the upload body, CRC32C/MD5 and the file-cache copy all see the final memory. Part boundaries and part count do not change. A value `<= 0` or `>= s3_write_buffer_size` restores the full up-front allocation. Download buffers are unchanged. 2. **Cap in-flight load parts.** A new config `load_s3_upload_max_inflight_parts` (default 0 = unbounded) sets a counting limiter. A load writer takes a permit before it submits a part and returns it when the buffer is freed. The packed-file writer is exempt, because its appends run under a BE-wide mutex and its single writer already bounds its parts. Compaction can use a separate limiter of the same kind. 3. **Back-pressure load writes on upload buffer bytes.** Load writers count their upload buffers in an atomic: buffers being filled, parts waiting for a permit, parts uploading, and parts waiting for a cache copy. The count covers load output only, not compaction or schema change. At the end of `MemTableMemoryLimiter::handle_memtable_flush`, load writes wait while the count is above `load_s3_upload_buffer_backpressure_bytes`, until it falls to `..._resume_percent` (hysteresis). A single wait is bounded by `..._max_wait_ms`, and the wait honours load cancel. Buffer allocation itself never blocks. A blocking cap there can deadlock, because a flush holds its segment buffer while it needs another one for the index file or next segment. The threads that wait at the gate hold no upload buffers, and the threads that own buffers never pass through the gate, so the counted bytes keep draining. ### How to Reproduce? Cloud mode with an S3 vault. Raise `s3_write_buffer_size` (e.g. 64 MiB) to make it obvious. Run many concurrent stream loads into a table that produces many small segment and index files (for example a wide VARIANT column), and slow down or saturate S3 uploads. Then watch `S3FileBuffer` in the BE `/profile` memory report. ### Anything Else? The 210 GB and 197 GB peaks are production measurements. The per-cause breakdown above comes from reading the code, not from instrumentation. Unit tests cover the patch, including a stress test where 24 writers feed a 16-thread flush pool, each flush holds two buffers, and the wait is unbounded; it does not deadlock. We have not yet measured the patch end-to-end on a cluster. Related: #68903 (compaction and load share the upload pool) and #68910 (runtime changes to `s3_write_buffer_size`). ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
