matanper opened a new issue, #70381: URL: https://github.com/apache/doris/issues/70381
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, arm64). master is the same: - `be/src/storage/segment/segment_iterator.cpp:3315` / `:3329`: `_process_common_expr` runs before `_materialization_of_virtual_column` (`:3370`). - `be/src/exprs/virtual_slot_ref.cpp:123-131`: the fallback that computes the expression into a temporary column. - `fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVirtualColumnsIntoOlapScan.java:286`: the lambda skip. - `PushDownVirtualColumnsIntoOlapScan.java:136`: scalar-only `SUPPORTED_VIRTUAL_COLUMN_TYPES`. ### Description With `experimental_enable_virtual_slot_for_cse = true` (#52701, #56340), `PushDownVirtualColumnsIntoOlapScan` replaces a repeated sub-expression with a virtual column, and EXPLAIN shows every reference reading the same virtual slot. Query CPU does not go down. Why: in `SegmentIterator::_next_batch_internal` the pushed-down common exprs run in `_process_common_expr` (`:3315` / `:3329`), before `_materialization_of_virtual_column` (`:3370`). When a `VirtualSlotRef` finds its block column still `ColumnNothing`, `VirtualSlotRef::execute_column_impl` (`virtual_slot_ref.cpp:123-131`) computes the expression into a temporary column. The block is const there, so the result is not kept. N references in the filter therefore cost N evaluations, the same as without the rewrite. A second limitation: the rule skips any expression that contains a lambda (`:286`) and accepts only scalar result types (`:136`). A repeated `ARRAY_MAP(x -> ..., <array expr>)`, or its array argument such as `REGEXP_EXTRACT_ALL(s, ...)`, is never extracted, even though these are typically the most expensive repeated expressions in a filter. ### Solution 1. **BE:** compute each virtual column that a pushed-down common expr reads once per block and keep it in the block, so later references reuse it. It must keep today's semantics: - Skip columns an index already supplies, such as ANN distance and score. - Don't compute it eagerly when it is referenced only under a short-circuit branch (`IF` / `CASE` / `AND` / `OR`). Otherwise a guarded expression, for example a strict `CAST` inside `IF(s REGEXP '^[0-9]+$', ...)`, would run on rows the guard excludes. - Compute it just before the first conjunct that reads it, so an earlier conjunct that empties the block still skips it. 2. **FE:** - Allow lambda-containing expressions and a lambda's array arguments as candidates, plus ARRAY result types. - Never count or replace anything inside a lambda body. - Match copies of a lambda expression with a key that renumbers the lambda-argument ExprIds and still compares the array arguments in order, so `array_map(x -> f(x), a)` and `array_map(x -> f(x), b)` never share a column. - Make sure nested-column / VARIANT pruning tolerates a virtual slot that has no original column. ### How to Reproduce? ```sql -- t: DUPLICATE KEY table, s: STRING column of ~700 bytes, a few million rows SET experimental_enable_virtual_slot_for_cse = true; SELECT COUNT(*) FROM t WHERE LOWER(s) LIKE '%a%' OR LOWER(s) LIKE '%b%' OR LOWER(s) LIKE '%c%' OR LOWER(s) LIKE '%d%' OR LOWER(s) LIKE '%e%' OR LOWER(s) LIKE '%f%' OR LOWER(s) LIKE '%g%' OR LOWER(s) LIKE '%h%'; ``` EXPLAIN shows one virtual column. Compare the audit log `cpu_time_ms` with the variable off and on: they are the same. ### Anything Else? Measured with a patch on a single-node test cluster: 4M rows, ~720-byte string, query CPU from the audit log, median of 3 runs, query and page caches off, variable on. | Query | Without the patch | With the patch | |---|---|---| | 1× `LOWER(s) LIKE` | 2.4 s | 2.5 s | | 8× `LOWER(s) LIKE` | 8.2 s | 4.7 s | | 13× `ARRAY_MAP(x -> LOWER(REGEXP_REPLACE(x, ...)), REGEXP_EXTRACT_ALL(s, ...))` (50K rows) | 47.8 s | 3.8 s | Results were identical in every case. Without the patch the variable makes no difference; for example, the 8× query took 8.20 s with it off and 8.16 s with it on. The patch is still being hardened for the short-circuit and lambda-identity points listed in Solution. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
