HappenLee opened a new pull request, #68961:
URL: https://github.com/apache/doris/pull/68961
### What problem does this PR solve?
Related PR: #63258
Remote broadcast channels share the same `PBlock`, but brpc normally
serializes its large `column_values` into a separate request buffer for every
destination. Broadcasting an S-byte block to N destinations therefore copies
roughly N*S bytes while preparing the protobuf requests.
Encode the `block` or repeated `blocks` field once in
`BroadcastPBlockHolder`, before publishing it to the destination queues. Each
RPC serializes its routing fields and appends references to the cached encoding
through `brpc::SerializedRequest`. The receiver continues to parse the existing
`PTransmitDataParams` wire format. Queue order, packet sequences, EOS, and
per-RPC callback ownership are preserved.
The original PBlock remains available to the existing send paths. Queue
admission accounts for both it and the cache once per holder.
Single-destination broadcasts and oversized fragments retain the original path;
assembled requests retain required-field validation and the protobuf size
limit. The HTTP borrowed-data fix remains in #63258.
The earlier standalone prototype measured packing CPU at 1 MiB x 8
destinations dropping from 3370.81 us to 402.43 us. This is motivation for the
change, not an end-to-end measurement of this production patch. Network bytes
and receiver processing are unchanged.
### Release note
Reduce repeated protobuf serialization and payload copies when broadcasting
exchange blocks to multiple remote destinations.
### Check List (For Author)
- Test
- [x] Unit Test: `broadcast_protobuf_test.cpp` covers wire equivalence,
shared backing buffers, batching/EOS, the actual send branch, queue accounting,
size limits, and asynchronous RPC completion/cancellation.
- ASAN: `ENABLE_PCH=OFF ENABLE_UNITY_BUILD=ON ./run-be-ut.sh -j 48 --run
--filter='*BroadcastProtobufTest*:*ExchangeSinkOperatorTest*:*ExchangeSinkTest*'`
— 24 tests passed (14 new and 10 existing).
- `build-support/clang-format.sh` / `build-support/check-format.sh` with
clang-format 16.0.6; `build-support/check-build-hygiene.sh` and `git diff
--check` passed.
- `build-support/run-clang-tidy.sh`: all changed production lines and
the entire new test file passed. The local LLVM 20 package needed an explicit
resource-directory argument.
- Behavior changed:
- [x] No. SQL results and the exchange wire protocol are unchanged.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
