freemandealer opened a new pull request, #14014:
URL: https://github.com/apache/doris/pull/14014
For table with hundreds of text type columns, flushing its memtable may cost
huge memory. These memory are consumed when initializing page builder, as it
reserves 1MB for each column. So memory consumption grows in proportion with
column number. Shrinking the reservation may reduce memory substantially in
load process.
Signed-off-by: freemandealer <freeman.zhang1...@gmail.com>
# Proposed changes
Issue Number: close #14012
For table with hundreds of text type columns, flushing its memtable may cost
huge memory. These memory are consumed when initializing page builder, as it
reserves 1MB for each column. So memory consumption grows in proportion with
column number. Shrinking the reservation may reduce memory substantially in
load process, as shown in the test results listed below.
Shrinking will not limit the size of dict page size -- it could still grow
when needed. Thus, we can make the most of dict encoding while achive
reasonable memory consumption. No performance degrading is observed in the
tests, proving buffer growth brings little overheads.
We believe this commit will make Doris more capable for larger load
concurrency.
Test config:
| CPU | Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz 96 core |
| ---- | ------------------------------------------------------------ |
| RAM | 375G |
| SSD | Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO |
| Data | GithubEvent, 3.1G single json, 664 columns, dict size will trigger
buffer growth |
Test results:
- single load
| | Resv 1M (original) | Resv 64K + Grow |
| ---------------- | ------------------- | --------------- |
| LoadTime: | 99.4s | 94.7s (-5%) |
| Peak ProcessMem: | 12G | 11G (-1G) |
| Peak LoadMem: | 17G | 13G (-4G) |
| Peak Orphan: | 11G | 8G (-3G) |
- multiple loads (concurrency = 10)
| | Resv 1M (original) | Resv 64K + Grow |
| ---------------- | ------------------- | --------------- |
| LoadTime: | 183.2s | 182.9s (-1%) |
| Peak ProcessMem: | 80G | 73G (-7G) |
| Peak LoadMem: | 55G | 51G (-4G) |
| Peak Orphan: | 57G | 54G (-4G) |
## Checklist(Required)
1. Does it affect the original behavior:
- [X] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [X] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [X] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [X] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [X] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
