mbutrovich commented on issue #22547: URL: https://github.com/apache/datafusion/issues/22547#issuecomment-4971811505
> I recommend we backport regression fixes too (made with Claude) -- I can review later in the week > > Wrong results: > > 1. [fix: handle `IS TRUE` correctly in `EliminateOuterJoin` #22444](https://github.com/apache/datafusion/pull/22444) — IS TRUE in EliminateOuterJoin wrongly converts outer joins to inner joins ([Outer join elimination does not handle `IS TRUE` correctly #22441](https://github.com/apache/datafusion/issues/22441)); introduced by [EliminateOuterJoin with Like, IsTrue, IsFalse, IsNotUnknown #21549](https://github.com/apache/datafusion/pull/21549). Empirically verified: 53.1.0 returns correct rows, 54.0.0 doesn't. > 2. [fix: preserve no-filter SMJ matches across pending outer batches #23049](https://github.com/apache/datafusion/pull/23049) — sort-merge join semi/anti drops matches across pending outer batches ([CI failure: `fuzz_cases::join_fuzz::test_left_anti_join_1k` #23048](https://github.com/apache/datafusion/issues/23048)); introduced by the SMJ bitwise-path rewrite [perf: specialized SemiAntiSortMergeJoinStream #20806](https://github.com/apache/datafusion/pull/20806). > 3. [fix: NestedLoopJoinExec emits spurious unmatched-left rows with multiple probe partitions #22791](https://github.com/apache/datafusion/pull/22791) — NestedLoopJoin emits spurious NULL-padded unmatched-left rows with multiple probe partitions; introduced by [feat: Add memory-limited execution for NestedLoopJoinExec #21448](https://github.com/apache/datafusion/pull/21448). > 4. [fix: regex simplification of anchored patterns produces wrong results #22727](https://github.com/apache/datafusion/pull/22727) — anchored-regex simplification produces wrong results/errors on Utf8View and LargeUtf8 columns ([Regex simplification of anchored patterns produces wrong results #22726](https://github.com/apache/datafusion/issues/22726)); introduced by [perf: Apply logical regexp optimizations to Utf8View and LargeUtf8 inputs #20581](https://github.com/apache/datafusion/pull/20581), which extended the rewrite to those types. > 5. [fix: approx_distinct over-counts for utf8view #22815](https://github.com/apache/datafusion/pull/22815) — approx_distinct over-counts Utf8View values ([`approx_distinct` over-counts `Utf8View` because the hash strategy is chosen per batch instead of per value #22796](https://github.com/apache/datafusion/issues/22796)); introduced by the inline-u128 hashing fast path in [perf: Optimize `approx_distinct` for inline Utf8View #21064](https://github.com/apache/datafusion/pull/21064). > > Panics: > > 6. [fix: Avoid panicing when stats are not available for a file group split #23277](https://github.com/apache/datafusion/pull/23277) — panic ("len is 0 but index is 0") ordering a parquet scan by a computed projection alias ([Panic in DataFusion 54.0.0 when ordering Parquet scan by computed projection alias #23219](https://github.com/apache/datafusion/issues/23219)). The panicking code predates 53, but [feat: sort file groups by statistics during sort pushdown (Sort pushdown phase 2) #21182](https://github.com/apache/datafusion/pull/21182) (sort pushdown phase 2) made it reachable in default configuration — queries that ran on 53.x panic on 54.0.0. Already on the release issue [Release DataFusion `54.1.0` (minor/patch) Release #22547](https://github.com/apache/datafusion/issues/22547). > 7. [fix: isolate anonymous file statistics cache #22950](https://github.com/apache/datafusion/pull/22950) — shared statistics cache reused across schemas for anonymous file scans → panic ([panic: `ProjectionExprs::project_statistics` index out of bounds #22935](https://github.com/apache/datafusion/issues/22935), reporter explicitly bisected to "regression between 53.1.0 and 54.0.0"); introduced by [Add a memory bound FileStatisticsCache for the Listing Table #20047](https://github.com/apache/datafusion/pull/20047). > > Queries that fail to plan: > > 8. [Fix:22477 any all schema error #22915](https://github.com/apache/datafusion/pull/22915) — = ANY (SELECT ...) / <> ALL (SELECT ...) fail with "duplicate unqualified field name mark" ([`= ANY` / `<> ALL` subquery fails to plan: `Schema contains duplicate unqualified field name mark` #22477](https://github.com/apache/datafusion/issues/22477)); introduced by [fix: `optimize_projections` failure after mark joins created by `EXISTS OR EXISTS` #21265](https://github.com/apache/datafusion/pull/21265). Empirically verified working on the published 53.1.0 crate. > 9. [fix: Correctly compute nullability in recursive CTE schemas #22552](https://github.com/apache/datafusion/pull/22552) — recursive CTE queries with certain nullability patterns now fail at execution ([Recursive CTE Nullability Handling Should Preserve Logical Schema Without Requiring SQL Rewrites #22034](https://github.com/apache/datafusion/issues/22034)); [Add reusable plan-time schema alignment helper and apply to RecursiveQueryExec #21912](https://github.com/apache/datafusion/pull/21912) made nullability checking strict without fixing the schema derivation. Caveat from the first pass: the fix is invasive (adds a field to LogicalPlan::RecursiveQuery + proto changes). One of these is backported already. I am opening PRs for the remaining eight, however only six were clean cherry-picks. I will call them out in the reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
