airborne12 opened a new pull request, #69770:
URL: https://github.com/apache/doris/pull/69770
### What problem does this PR solve?
Issue Number: None
Related PR: #69649
Problem Summary: A phrase query decodes the positions of the selected
documents of every .prx window it reads. Timed per window on a 10M-row
AgentLogs load, a zstd window cost 18.6 us against 1.4 us for a PFOR window:
11.7 us to decode the plain payload, 4.8 us to decompress, and about 2 us more
because every window created and freed its own zstd context (about 96 KB) and a
zero-filled output vector. Two commits:
1. **Decode the plain payload without a Status per document.**
`decode_payload_csr_selective` made at least one out-of-line call returning a
Status for every document of the window, selected or not (`get_varint32_fast`,
`skip_varints`, `decode_delta_run`). The loop now keeps the cursor in a
register and handles the usual cases inline: a single-byte count, walking past
the deltas of a document that is not selected, and a run of single-byte deltas.
Anything else -- a multi-byte value, a run that does not fit, a prefix sum that
would overflow -- goes to the checked decoders from the same cursor, so they
decide and report as before. The output vector grows as before; nothing is
reserved up front.
2. **Reuse one zstd context and buffer per index source.**
`ZstdDecompressor` keeps one context, created at the first zstd frame, and one
output buffer that only grows and is not zero-filled (decompression writes
every byte before it is read). `SniiIndexSource` owns one, and the cursors it
opens pass it to the .prx decoder through `PrxDecodeContext`. The cursors of a
source decode on its thread one window at a time, and the decoded payload is
consumed before the next window is read. A source lives for one query on one
segment, so a query holds at most one context and one buffer per segment it
reads, and only if it decodes a zstd window; before, each window created and
released them. Compaction decodes through an allocation gate, which keeps the
old path, as does a cursor given no decompressor.
Hot queries, old against new read path in one process (run-time switch in an
experiment build, 6 paired cycles), on 99M AgentLogs rows written with the
smallest frames, as every segment written before #69649 is (mostly zstd
windows): DS7 (unfiltered phrase) 0.87x, A6 0.89x, A10 0.93x, DS4 (phrase
prefix) 0.94x, A9 0.95x, DS1 0.96x; repeating the old path against itself gave
0.93-1.05. The gain follows the zstd windows a query reads: the phrase queries
of AgentLogs written with #69649 read PFOR windows and do not change, and on
TextBench, whose large windows stay zstd, the time per window is the
decompression itself and the change is within noise. Every query returns the
same result.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- PrxPodTest.SelectivePlainDecodeMatchesInputForRandomWindows: 300
random windows of mixed varint widths, raw and zstd, decoded selectively and in
full, also through one shared decompressor, compared with the input.
- SniiZstdDecompressor: frames of different sizes through one
decompressor; a corrupt frame is reported and the next one decodes.
- Manual test: interleaved hot A/B in one process on 10M and 99M AgentLogs
and 12M TextBench rows.
- Behavior changed: No
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
