matanper opened a new issue, #68977: URL: https://github.com/apache/doris/issues/68977
## Background We run Doris 4.1.4 in cloud (storage-compute decoupled) mode on arm64 BEs with an S3 storage vault and write-through file cache, under high sustained Stream Load rates (millions of rows/s across a few BEs) with heavy concurrent compaction. This is an **umbrella / tracking issue** for the cloud storage and load-path issues we found under that load, so maintainers and contributors have one place to see status. Each item has a patch on 4.1.4 that we've run under that load; we're willing to send PRs for all of them, and comments on direction before we do are welcome. --- ## S3 writes - [ ] **#68790** — Storage vault `path_version` / `shard_num` unusable for S3 vaults (FE `UnsupportedOperationException`, dropped by meta-service/BE). *(patch ready)* - [ ] **#68905** — S3 PutObject/UploadPart bodies are MD5-hashed twice (Doris `Content-MD5` plus the SDK's own checksum). *(patch ready)* - [ ] **#68910** — Changing mutable `s3_write_buffer_size` at runtime overflows in-flight `S3FileWriter` buffers. *(patch ready)* - [ ] **#68903** — Compaction S3 uploads share the load upload and close pools, so high compaction concurrency stalls load closes. *(patch ready)* - [ ] **#68970** — Load S3 upload buffer memory is unbounded: every buffer allocates a full part up front, and nothing caps queued or filling parts. *(patch ready)* ## Load / commit path - [ ] **#68907** — Cloud tablet sink serializes and sends batches over brpc even when the target is the local BE. *(patch ready)* - [ ] **#68913** — Cloud load close builds and discards a rowset writer for every tablet it never wrote. *(patch ready)* - [ ] **#68914** — Each version hole gets its own empty rowset; coalesce a run of holes into one range rowset. *(patch ready)* - [ ] **#68909** — Meta-service retries `KV_TXN_CONFLICT` with 0.5–4 s exponential sleeps, so hot-partition commits stall and fail. *(patch ready)* - [ ] **#68973** — Stream Load blocks its libevent worker in `on_header` (`begin_txn` + FE plan) and while the body pipe is full. *(patch ready)* - [ ] **#68974** — Loads near the process memory limit are cancelled by memory GC instead of being throttled. *(patch ready)* - [ ] **#68975** — Setting jemalloc `dirty_decay_ms` to 0 above the soft memory limit makes `free()` purge synchronously and stalls BE threads. *(patch ready)* ## File cache - [ ] **#68892** — `BlockFileCache::find_evict_candidates` scans in-use entries without bound under the global cache lock. *(patch ready)* - [ ] **#68893** — Reactive eviction unlinks block files while holding the global cache lock. *(patch ready)* - [ ] **#68894** — File cache writes leave written blocks in the page cache, pushing containers to `memory.max`. *(patch ready)* - [ ] **#68971** — FDCache uses one `shared_mutex` for all file cache readers, so cache-hit lookups queue behind inserts and evictions. *(patch ready)* - [ ] **#68972** — `CachedRemoteFileReader` keeps evicted blocks in its direct-read map, so later direct reads of that range fail to open the deleted file. *(patch ready)* ## Compaction / rowsets - [ ] **#68904** — Vertical compaction copies value columns run by run, which dominates CPU when input rowsets interleave. *(patch ready)* - [ ] **#68911** — Cloud compaction keeps running on dropped/truncated tablets because lease `TABLET_NOT_FOUND` is ignored. *(patch ready)* - [ ] **#68915** — BE aborts when `BetaRowsetReader` iterator init throws a `std::exception`. *(patch ready)* ## Auto analyze - [ ] **#68895** — Sampled auto analyze still runs an unsampled MIN/MAX full scan per column. *(patch ready)* - [ ] **#68896** — Failed auto analyze jobs don't record an analyze time, so they restart every cycle. *(patch ready)* --- Related: VARIANT and Arrow Stream Load issues from the same workload are tracked in #68976. ## How to contribute - Comment on an item to discuss the approach or to pick it up. - We'll link each PR here as it opens and check items off as they merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
