yujun777 opened a new pull request, #67575: URL: https://github.com/apache/doris/pull/67575
## Summary Tracked in https://github.com/apache/doris/issues/65418 MVs containing `ARRAY_AGG` or `COLLECT_LIST` can now be refreshed incrementally (IVM): the visible array is the full aggregate state, so no hidden state column is needed. The delta aggregate emits insert-side and delete-side arrays and apply merges them as a multiset `except_all(concat(coalesce(old, []), ins), del)`, which keeps duplicate elements and NULLs correct. `COLLECT_LIST` skips NULL rows, so its polarity columns use the conditional-argument idiom. `ARRAY_AGG` keeps NULL elements, so NULL-filtering cannot build its polarity arrays; instead every change row is packed into a single `array_agg(struct(dml_factor, elem))` aggregate output and the delta top project derives the insert/delete polarity columns above the aggregate with `array_filter`/`array_map`, keeping the same two transient polarity slots for the multiset apply. `ARRAY_AGG` over JSONB/VARIANT elements (struct/map/array constructors all reject those types) is rejected during analysis with a precise reason, surfaced to the user at `CREATE MATERIALIZED VIEW` time. DISTINCT and the LIMIT variant of `collect_list` fall back to complete refresh. ## Key changes - add ARRAY_AGG and COLLECT_LIST aggregate kinds sharing one `IvmAggArrayProcessor` (two transient polarity delta slots, multiset apply, no hidden state); ARRAY_AGG derives its polarity columns above the delta aggregate through a new per-processor top-project hook - centralize polarity condition expressions and the typed empty array literal in `IvmAggExpressionBuilder` - regression suite `test_ivm_agg_array_1` covering grouped, scalar and mixed (array_agg + count + sum) MVs over insert/update/delete windows with NULL and non-NULL values ## Unit Test - `IvmAggArrayAggProcessorTest` / `IvmAggCollectListProcessorTest` and the IVM agg delta handler suite: 34 cases pass - `CreateMTMVCommandTest`: CREATE of an ARRAY_AGG-over-JSONB incremental MV fails with the precise unsupported-element-type reason - `test_ivm_agg_array_1` end-to-end on a local cluster: every window equals the base-table recompute and the COMPLETE result -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
