Gabriel39 opened a new pull request, #69646:
URL: https://github.com/apache/doris/pull/69646
### What problem does this PR solve?
Problem Summary:
Parquet V2 currently visits row groups in physical file order. For a small
`ORDER BY ... LIMIT`, the best values can be near the end of a file. Earlier
groups are then decoded before the TopN sink publishes a useful bound. Even
after a bound arrives, the deferred row-group path does not recheck the footer
unless a new scan request is published; TopN updates its bound in place.
`VTopNPred` also lacks the ZoneMap interface needed to reject whole row groups.
For example, consider four groups with key ranges `[300, 399]`, `[200,
299]`, `[100, 199]`, `[0, 99]` and `ORDER BY k ASC LIMIT 3`. Reading the last
group first produces a bound of `2`; the other three groups can be rejected
using their footer minimum before dictionary/index probes, prefetch, or data
decoding. The descending case uses the footer maximum.
### Optimization approach
- Add conservative ZoneMap evaluation to `VTopNPred`. Snapshot the current
runtime bound at each evaluation, retain equality for secondary sort keys,
honor NULL ordering, and fall back when types, ranges, or floating-point NaN
information are unsuitable.
- Prioritize selected row groups by the leading TopN slot's sanitized footer
minimum/maximum. Resolve the file-local slot through the request's column
mapping. Keep execution indices separate from the physical plans so row IDs,
delete coordinates, and Condition Cache boundaries remain valid.
- Recheck the latest TopN bound at every row-group boundary before expensive
metadata probes and I/O. Files without statistics retain the existing path.
Reordering is disabled for row-ID fetches and evaluation-sensitive predicate
fences; missing or unusable statistics never exclude a group by themselves.
- Stop FE TopN-filter pushdown at projects containing volatile or
non-movable expressions. For example, crossing `assert_true(k < 3, 'out of
range')` could suppress an error when TopN prunes input below the project.
### Measured optimization effect
The included opt-in `ParquetScanTest.TopNRowGroupOrderMicrobench` exercises
the actual Parquet V2 reader plus a bounded TopK heap and per-block updates of
the real runtime predicate. Its workload corresponds to:
```sql
SELECT k FROM parquet_source ORDER BY k ASC LIMIT 10;
SELECT k FROM parquet_source ORDER BY k DESC LIMIT 10;
```
Fixture: one required INT32 column, 128 row groups x 4,096 rows (524,288
rows), disjoint key ranges, and the best group physically last for each
direction. Files use uncompressed PLAIN DataPageV2 with dictionary and
PageIndex disabled, isolating footer pruning. Every iteration verifies all ten
final values.
| Per reader query | Before | After |
| --- | ---: | ---: |
| Row groups decoded | 128 | 1 |
| RawRowsRead | 524,288 | 4,096 (-99.22%) |
| RowGroupsFilteredByMinMax | 0 | 127 |
| Actual local read bytes | 2,149,888 | 65,602 (-96.95%) |
| ASC median latency | 10.698 ms | 0.669 ms (~16.0x) |
| DESC median latency | 8.590 ms | 0.660 ms (~13.0x) |
Method: Clang 20 RELEASE builds, warm local filesystem cache, the same
pinned CPU, nine paired samples alternating baseline/candidate order, and 32
reader queries per sample after warmup. The baseline restores the four modified
TopN/Parquet implementation files from the parent commit while retaining the
identical benchmark harness and request plumbing; the benchmark calls
ParquetReader directly. All paired checksums match. Work counters are identical
across all samples. ASC candidate samples range from 0.647 to 1.097 ms and DESC
from 0.642 to 0.745 ms; the shared host has timing outliers. These are reader
microbenchmark results; complete SQL gains depend on file splits, parallelism,
feedback latency, and storage.
### Validation and reproduction
- Targeted BE suites: 370 tests passed, including Parquet
scanning/statistics, column mapping, and TableReader; the opt-in benchmark is
skipped in the ordinary run and was executed separately for both baseline and
candidate.
- FE `TopNRuntimeFilterTest`: all 8 tests passed, including the new
error/volatile project fence test.
- New reader tests cover ASC/DESC late bounds, exact TopK results, physical
row IDs, localized slot mapping, missing statistics, predicate fences, row-ID
fetch order, NULL placement, and equality boundaries.
- FE Checkstyle, clang-format 16, BE build-hygiene checks, and clang-tidy on
all nine affected C++ files passed.
Correctness suites:
```bash
./run-be-ut.sh -j8 --run \
--filter='*Parquet*:*ColumnMapper*:*TableReader*'
./run-fe-ut.sh --run
org.apache.doris.nereids.postprocess.TopNRuntimeFilterTest
```
Opt-in benchmark (use a RELEASE UT build):
```bash
DORIS_TOPN_ROW_GROUP_BENCHMARK=1 BUILD_TYPE_UT=RELEASE \
./run-be-ut.sh -j8 --run
--filter=ParquetScanTest.TopNRowGroupOrderMicrobench
```
For paired comparison, build and save baseline/candidate UT executables with
the same harness, then use the existing runner:
```bash
DORIS_TOPN_ROW_GROUP_BENCHMARK=1 \
DORIS_BE_UT_EXECUTABLE=build-support/run-query-engine-paired-benchmarks.py \
QUERY_ENGINE_BENCH_BASELINE=./baseline/doris_be_test \
QUERY_ENGINE_BENCH_CANDIDATE=./candidate/doris_be_test \
QUERY_ENGINE_BENCH_CPU=0 \
QUERY_ENGINE_BENCH_OUTPUT=./paired-results \
BUILD_TYPE_UT=RELEASE ./run-be-ut.sh -j8 --run \
--filter=ParquetScanTest.TopNRowGroupOrderMicrobench
```
Select an available idle CPU for `QUERY_ENGINE_BENCH_CPU`. The runner emits
paired timings/checksums; `PARQUET_TOPN_WORK` lines report per-query raw rows,
groups filtered by min/max, and actual local read bytes.
### Release note
Improve Parquet V2 TopN scans by reading promising row groups first and
applying the latest runtime bound before opening subsequent groups.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (reader microbenchmark and commands above)
- Behavior changed:
- [x] Yes. Eligible Parquet V2 TopN scans may read row groups in a
different order and skip more groups. SQL ordering remains defined by the TopN
operator.
- Does this need documentation?
- [x] No. Internal scan optimization with no new user-facing option.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
