eldenmoon opened a new pull request, #70363: URL: https://github.com/apache/doris/pull/70363
### What problem does this PR solve? cherry-pick #70359 (adapted for branch-4.1, with the scan-filter correction described below) Related PR: #70359 Problem Summary: backport the Variant-to-non-Variant nullable fix to branch-4.1 and retain the resulting null tests during scan normalization. A NOT NULL Variant column can contain Variant nulls or values that cannot be converted to a scalar. The existing BE CAST can return SQL NULL, but `Cast.castNullable` only accounted for Variant V2 and legacy casts to JSON. This causes both wrong optimizer rewrites and execution contract failures: `<=> NULL` becomes FALSE, COALESCE/NVL lose their fallback, and direct CAST queries can fail with a declared non-nullable / actual nullable mismatch. End-to-end validation exposed a second issue: after correcting FE metadata, scan normalization still replaced `CAST(v AS INT) IS NULL` with a null test on the source Variant column. This loses NULLs introduced by conversion; IS NOT NULL can retain the wrong rows. ### Fix - Reuse the existing `Cast.castNullable` rule to declare Variant-to-non-Variant casts nullable for both representations. Identity casts retain input nullability; no optimizer-specific nullable rule is added. - Keep NULL tests over casts as expressions through the existing scan pushdown rejection path. The scan eligibility check only admits casts from Variant here, so ordinary slot NULL predicates keep their existing path. - Add focused FE optimizer coverage and SQL regression for projections, filters, COALESCE/NVL, nullable inputs, incompatible values, identity casts, an aliased CTE, and all-Variant-null roots. ### Validation - Before the fix: FE tests reproduce the wrong nullable flag, FALSE folding of `<=> NULL`, and removal of conditional fallbacks. SQL fails with a nullable column mismatch. With the FE fix alone, scan filters still incorrectly return no rows; the captured projection results identify the rows that should survive. - After the fix: 59 FE unit tests passed (CastTest, CheckCastTest, NullSafeEqualToEqualTest, SimplifyConditionalFunctionTest, VariantCastNullableTest). - FE build with Checkstyle and BE ASAN build passed using `build.sh`. - Regression goldens were generated by the repository runner and inspected. Normal comparison: test_variant_cast_nullable and existing test_variant_cast passed (2 suites). - C++ formatting: repository wrapper with clang-format 16. Static analysis: running; the initial invocation was blocked by the toolchain resource path and an existing header NOLINT diagnostic, and a corrected invocation is in progress. ### Compatibility and performance BE CAST conversion semantics, storage formats, and Thrift definitions are unchanged. The fix corrects metadata, optimizer simplification, and filtering against the existing conversion result. Existing legacy Variant conversion quirks are preserved by the golden tests. The FE change adds no per-row conversion work. Unsafe NULL-predicate pushdown over a Variant cast is replaced by expression evaluation, which may scan/evaluate more rows for those predicates; ordinary source-column NULL pushdown is unchanged. No performance benchmark is claimed. No new public API or dependency is added. ### Release note Fix incorrect NULL predicates, conditional-function simplification, and nullable result type mismatches when casting Variant values to non-Variant types with legacy Variant execution. ### Check List (For Author) - Test: FE Unit Test: 59 passed; SQL regression: 2 suites passed; FE/BE ASAN builds passed - Behavior changed: Yes. Preserve NULL/fallback results and filter rows according to the CAST result; conversion semantics are unchanged. - Does this need documentation: No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
