jacktengg opened a new pull request, #69304: URL: https://github.com/apache/doris/pull/69304
### What problem does this PR solve? Issue Number: None Related PR: #60228, #68312 Problem Summary: Nullable aggregation still selected V1 for variadic arguments and persisted AGG_STATE, while unary V2/native readers had separate NULL-only, lifecycle and scratch behavior. This change uses V2 for all normal unary/variadic nullable execution, preserves internal validity independently of SQL result nullability, and skips NULL-only partials before nested merge. Specialized COUNT/Array/Map implementations keep their own NULL semantics. A concrete V2 byte-codec adapter preserves the published AGG_STATE String format independently of query settings; native COUNT/Bitmap/Array/Map codecs remain supported. Execution version 20 identifies the unified intermediate-column protocol. Older requests use an isolated wire codec adapter, and the old session variable stays accepted as a hidden deprecated no-op. Generic readers retain aligned scratch only when their actual reader requires it; native SUM/AVG/COUNT avoid unused scratch. Temporary streaming payloads are bounded per row. The bitmap regression coverage distinguishes a NULL-only partial from a valid empty bitmap across aggregation phases. Old-schema no-input markers survive full compaction (3 rowsets to 1), and eight historical/new state types survive mixed-rowset full compaction (4 to 1). Real old/new BE replicas are exercised under both old FE and new FE pinned to execution version 19, including new-FE writes to both binary versions. ### Release note Nullable aggregate execution uses V2. `enable_aggregate_function_null_v2` remains accepted as a deprecated no-op; its published Thrift id is retained. Keep FE `be_exec_version=19` until every executing/reading/compacting BE supports version 20. New version-20 AGG_STATE metadata and bitmap no-input markers require upgraded readers. Markers can be persisted in version-19 schemas, and changing the current query setting cannot downgrade existing types or bytes. Downgrade after new-format writes requires compatible metadata and data. Historical bitmap NULL-only states already collapsed to an empty bitmap cannot be reconstructed. ### Check List (For Author) - Test: 406 focused BE ASAN tests; 19 AGG_STATE regression suites; standard-generated and compared expectations; real historical rowsets, mixed old/new replicas and full compaction; RELEASE BE/FE builds; clang-format 16 and header hygiene. Whole-header clang-tidy has eight pre-existing diagnostics, zero on changed lines; new codec/UT checks pass. - Full UT limitation: unchanged Variant fixture constructor/API mismatches block the complete baseline test target; the optional standard-runner source filter leaves the default full target unchanged. - Behavior changed: Yes (unified V2 execution, preserved NULL-only partials, retired runtime switch). - Does this need documentation: Yes (included below). ### Performance and reproducibility 550 warmed recorded SQL executions cover 21 cases on the same shared machine/data/settings, comparing the original target branch V1/V2 and migrated execution. Results agree. For high-cardinality SUM/AVG, the migrated mean is about 6%/16%/27% lower than original V2 at 1%/50%/100% NULL. Weighted AVG improves about 22%–27%; Q78 SF1 is reported separately. Bulk persisted MERGE is +3.9% within observed ranges, and UNION/FINALIZE is +0.6%. Small changes are not claimed as stable gains on this shared host. 48 native kernel cases have five final repetitions with CV below 5%; all earlier repetitions remain included. Ordinary heap request counters, CPU samples, operator avg/min/max and query wall time are reported separately. Query memory peaks do not uniformly fall: weighted AVG/50%-NULL mean is 254.1→262.3 MiB (max 256.7→277.6), and wide string/set mean is 2565.2→2577.6 MiB (max 2603.1→2776.6). Raw ranges, state-memory evidence and attribution limits are disclosed. - [Execution matrix, format contract and upgrade boundaries](https://github.com/jacktengg/incubator-doris/blob/DORIS-29641/docs/nullable-aggregation-v2-migration.md) - [Reproduction, new measurements and raw profile/perf/build archives](https://github.com/jacktengg/incubator-doris/blob/DORIS-29641/docs/performance/nullable-aggregation/README.md) - [Exact commits, binary hashes, environment and archive SHA256](https://github.com/jacktengg/incubator-doris/blob/DORIS-29641/docs/performance/nullable-aggregation/manifest.json) Historical PR timings are background only and are not presented as this migration's results. The external bitmap issue's exact original SQL/status was unavailable; an equivalent sparse partial-state scenario is covered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
