airborne12 opened a new pull request, #70388: URL: https://github.com/apache/doris/pull/70388
### Background TopN lazy materialization reads only the sort keys and a row location in the scan, and fetches the other returned columns for the rows that survive the TopN. A column a filter above the scan reads stayed in the scan output, unless the experimental `topn_lazy_materialization_using_index` deferred every filter column. ### What problem does this PR solve? Issue Number: None Related PR: #59572 PR Type: Performance Problem Summary: A TopN that filters a column with search() or MATCH and returns it read that column for every match. Such a column is now fetched after the TopN, so the scan answers the predicate from the inverted index without reading it: a search-box query over 100M logs uses 59% less CPU and 38% less time. ### Details - **Why it was slow:** Body was read for 5.49M matches and carried through the sort to return 200 rows. - **Optimization:** a filter directly above a base-index OLAP scan keeps lazy the columns only search() and MATCH (alone or under AND/OR/NOT) read; the probe, the pruning and the consumed-slot check share that rule. - **Results:** branch-4.2 port, old and new planning alternating on one cluster: search-box TopN queries 37-59% less CPU, TextBench and AgentLogs MATCH TopN queries 23-56% less; other queries within 4%; results identical. - **Applicability and cost:** surviving rows fetch the column by row location; a segment without the index reads it in the scan as before. No change to writing or storage. ### Testing FE unit tests in `TopnLazyMaterializeTest` (fails before the change) and `MaterializeProbeVisitorTest`; regression test `test_topn_lazy_index_predicate_slots`, with a debug point proving the scan does not read the column. ### Release note TopN queries that return a column filtered by search() or MATCH read that column only for the rows they return. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: local unit and regression tests; CI not run ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, tests, this description - Human review: not yet reviewed 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
