matanper opened a new issue, #68940: URL: https://github.com/apache/doris/issues/68940
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Collect column statistics (NDV, null count, min/max, avg size) for typed paths of a VARIANT column, i.e. the paths declared in the column's schema template (`VARIANT<'a.b': INT, 'c': STRING>`), so the Nereids optimizer can estimate predicates and aggregations on them. Today a VARIANT column is excluded from statistics as a whole: `StatisticsUtil.isUnsupportedType` returns true for `type.isVariantType()` (4.1.4 `StatisticsUtil.java:777-786`, master `:525`), so neither manual nor auto analyze collects anything for it. A query on `v['a']['b']` is rewritten into a sub-path slot, and `StatsCalculator` sees no statistics for it, so it falls back to `ColumnStatistic.UNKNOWN` (default selectivity, unknown NDV). That is reasonable for dynamic paths, whose set and types change between segments. But typed paths are a fixed, user-declared set with a fixed type, which makes them statistically equivalent to ordinary columns. ### Use case Semi-structured event tables where nearly all fields live in one VARIANT column (e.g. security, observability or clickstream events following a common schema), and only a handful of envelope columns (tenant id, event time, ids) are real columns. Users declare typed paths for the fields they filter and group by most, and add inverted indexes on them with `field_pattern`. - **Filters:** indexes make filters fast without statistics. - **Aggregations and joins:** these are still planned blind. - `GROUP BY v['class_id']` or `GROUP BY v['src']['ip']` gets an unknown NDV, which drives one- vs two-phase aggregation and parallelism choices. - Joins between such a table and dimension or lookup tables on a typed path (e.g. an IP or a user id) have no NDV for join reordering or broadcast vs shuffle. - **Workaround:** materialize the path as a separate column, purely to get statistics. That duplicates the data and gives up the single-VARIANT design. ### Proposal Scope it to typed paths only. No dynamic-path discovery. 1. **Collection:** treat each typed path as an analyzable column whose stats key is the VARIANT column name plus the path, e.g. `v.a.b`. - The SQL computes over `CAST(element_at(...) AS <declared type>)`, or the typed sub-column read directly, reusing the existing full and sampled templates. - Unsupported declared types (complex types) are skipped, as for ordinary columns. 2. **Auto analyze:** include typed paths in column eligibility and change tracking. Health or updated rows could initially be tracked at the parent column level. 3. **Optimizer:** in `StatsCalculator` / `ExpressionEstimation`, map a sub-path slot whose path matches a typed path to that path's statistics, and fall back to UNKNOWN otherwise. 4. **Lifecycle:** drop or invalidate a path's statistics when the schema template changes (path removed or retyped), the same way dropping a column does. Open questions: - Should `auto_analyze_table_width_threshold` count typed paths? - Should paths matched by a glob (`MATCH_NAME_GLOB`) be in scope, or only exact names? ### Related issues None found. #66911 (a BE crash in `VStatisticsIterator` on VARIANT segments) is related only in that statistics reads currently avoid VARIANT. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
