matanper opened a new issue, #68976: URL: https://github.com/apache/doris/issues/68976
## Background We run Doris 4.1.4 in cloud (storage-compute decoupled) mode on arm64 BEs, ingesting at high sustained rates with Stream Load (`format=arrow`) into tables whose main payload is a large VARIANT V2 column (Parquet-Variant binary input, typed schema-template paths, thousands of subcolumns). This is an **umbrella / tracking issue** for the VARIANT- and Arrow-specific issues we found, so maintainers and contributors have one place to see status. Each item has a patch on 4.1.4 that we've run under that load; we're willing to send PRs for all of them, and comments on direction before we do are welcome. --- ## Arrow Stream Load - [ ] **#68890** — Support Stream Load `format=arrow` into VARIANT columns (Arrow binary Variant input). *(feature, patch ready; related: #68883 added native Variant on the Arrow Flight output side)* - [ ] **#68906** — Arrow Stream Load decodes the whole body and converts it into one block, so per-load memory scales with request size. *(patch ready)* - [ ] **#68891** — Arrow Stream Load decompresses record batches on Arrow's global thread pool. *(patch ready)* ## VARIANT V2 load path - [ ] **#68912** — VARIANT V2 load path over-allocates: power-of-two column growth and a full-block copy at memtable flush. *(patch ready)* - [ ] **#68908** — VARIANT V2 payloads are fully re-validated when a tablets channel decodes a block from a peer BE. *(patch ready)* - [ ] **#68898** — BE aborts via `std::terminate` when a VARIANT read exceeds the query memory limit (allocation in a noexcept release path). *(patch ready)* ## VARIANT statistics - [ ] **#68940** — Collect statistics for VARIANT typed (schema-template) paths. *(feature, no patch yet)* --- Related: storage, file cache, load and compaction issues found under the same workload are tracked in a companion tracking issue (linked below once filed). ## How to contribute - Comment on an item to discuss the approach or to pick it up. - We'll link each PR here as it opens and check items off as they merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
