comphead commented on code in PR #21824: URL: https://github.com/apache/datafusion/pull/21824#discussion_r3138648704
########## docs/source/library-user-guide/optimizer-rules.md: ########## @@ -0,0 +1,607 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +<!--- +This file was generated by the dev/update_optimizer_rule_docs.sh script. +Do not edit it manually as changes will be overwritten. +Instead, edit the rule documentation metadata in +datafusion/core/src/optimizer_rule_docs.rs or the +dev/update_optimizer_rule_docs.sh file for updating surrounding text. +--> + +# Optimizer Rules Reference + +This page lists the built-in rules in DataFusion's default analyzer, logical optimizer, and physical optimizer pipelines. + +The rule names shown here match the names shown by `EXPLAIN VERBOSE`. + +Rule order matters. The default pipeline may change between releases, so this page is a release-specific reference. + +## Analyzer Rules + +These rules run before logical optimization. They prepare the logical plan for later planning and optimization stages. + +| order | rule name | phase | summary | +| ----- | ----------------------------------------------- | ----- | --------------------------------------------------------------------------------------- | +| 1 | [`resolve_grouping_function`](#analyzer-rule-1) | - | Rewrites `GROUPING(...)` calls into expressions over DataFusion's internal grouping id. | +| 2 | [`type_coercion`](#analyzer-rule-2) | - | Adds implicit casts so operators and functions receive valid input types. | + +(analyzer-rule-1)= + +### `resolve_grouping_function` + +Rewrites `GROUPING(...)` calls into expressions over DataFusion's internal grouping id. + +#### Applies When + +The plan contains `GROUPING` expressions produced by `GROUPING SETS`, `ROLLUP`, or `CUBE`. + +(analyzer-rule-2)= + +### `type_coercion` + +Adds implicit casts so operators and functions receive valid input types. + +#### Applies When + +Expressions mix types that must be coerced to a common or expected type before planning can continue. + +## Logical Optimizer Rules + +These rules rewrite `LogicalPlan` nodes while preserving query results. + +| order | rule name | phase | summary | +| ----- | ------------------------------------------------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------- | +| 1 | [`rewrite_set_comparison`](#logical-rule-1) | - | Rewrites `ANY` and `ALL` set-comparison subqueries into `EXISTS`-based boolean expressions with correct SQL NULL semantics. | +| 2 | [`optimize_unions`](#logical-rule-2) | - | Flattens nested unions and removes unions with a single input. | +| 3 | [`simplify_expressions`](#logical-rule-3) | - | Constant-folds and simplifies expressions while preserving output names. | +| 4 | [`replace_distinct_aggregate`](#logical-rule-4) | - | Rewrites `DISTINCT` and `DISTINCT ON` operators into aggregate-based plans that later rules can optimize further. | +| 5 | [`eliminate_join`](#logical-rule-5) | - | Replaces keyless inner joins with a literal `false` filter by an empty relation. | +| 6 | [`decorrelate_predicate_subquery`](#logical-rule-6) | - | Converts eligible `IN` and `EXISTS` predicate subqueries into semi or anti joins. | +| 7 | [`scalar_subquery_to_join`](#logical-rule-7) | - | Rewrites eligible scalar subqueries into joins and adds schema-preserving projections. | +| 8 | [`decorrelate_lateral_join`](#logical-rule-8) | - | Rewrites eligible lateral joins into regular joins. | +| 9 | [`extract_equijoin_predicate`](#logical-rule-9) | - | Splits join filters into equijoin keys and residual predicates. | +| 10 | [`eliminate_duplicated_expr`](#logical-rule-10) | - | Removes duplicate expressions from projections, aggregates, and similar operators. | +| 11 | [`eliminate_filter`](#logical-rule-11) | - | Drops always-true filters and replaces always-false or NULL filters with empty relations. | +| 12 | [`eliminate_cross_join`](#logical-rule-12) | - | Uses filter predicates to replace cross joins with inner joins when join keys can be found. | +| 13 | [`eliminate_limit`](#logical-rule-13) | - | Removes no-op limits and simplifies trivial limit shapes. | +| 14 | [`propagate_empty_relation`](#logical-rule-14) | - | Pushes empty-relation knowledge upward so operators fed by no rows collapse early. | +| 15 | [`filter_null_join_keys`](#logical-rule-15) | - | Adds `IS NOT NULL` filters to nullable equijoin keys that can never match. | +| 16 | [`eliminate_outer_join`](#logical-rule-16) | - | Rewrites outer joins to inner joins when later filters reject the NULL-extended rows. | +| 17 | [`push_down_limit`](#logical-rule-17) | - | Moves literal limits closer to scans and unions and merges adjacent limits. | +| 18 | [`push_down_filter`](#logical-rule-18) | - | Moves filters as early as possible through filter-commutative operators. | +| 19 | [`single_distinct_aggregation_to_group_by`](#logical-rule-19) | - | Rewrites single-column `DISTINCT` aggregations into two-stage `GROUP BY` plans. | +| 20 | [`eliminate_group_by_constant`](#logical-rule-20) | - | Removes constant or functionally redundant expressions from `GROUP BY`. | +| 21 | [`common_sub_expression_eliminate`](#logical-rule-21) | - | Computes repeated subexpressions once and reuses the result. | +| 22 | [`extract_leaf_expressions`](#logical-rule-22) | - | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. | +| 23 | [`push_down_leaf_projections`](#logical-rule-23) | - | Pushes the helper projections created by leaf extraction toward leaf inputs. | +| 24 | [`optimize_projections`](#logical-rule-24) | - | Prunes unused columns and removes unnecessary logical projections. | + +(logical-rule-1)= + +### `rewrite_set_comparison` + +Rewrites `ANY` and `ALL` set-comparison subqueries into `EXISTS`-based boolean expressions with correct SQL NULL semantics. + +#### Applies When + +A filter or expression contains a set comparison such as `= ANY`, `< ALL`, or similar constructs. + +(logical-rule-2)= + +### `optimize_unions` + +Flattens nested unions and removes unions with a single input. + +#### Applies When + +The logical plan contains `UNION` nodes, including nested unions or distinct-over-union shapes. + +(logical-rule-3)= + +### `simplify_expressions` + +Constant-folds and simplifies expressions while preserving output names. + +#### Applies When + +Expressions contain literals, redundant boolean logic, removable casts, or other simplifiable patterns. + +(logical-rule-4)= + +### `replace_distinct_aggregate` + +Rewrites `DISTINCT` and `DISTINCT ON` operators into aggregate-based plans that later rules can optimize further. + +#### Applies When + +The plan contains logical `Distinct` operators. + +(logical-rule-5)= + +### `eliminate_join` + +Replaces keyless inner joins with a literal `false` filter by an empty relation. + +#### Applies When + +An inner join has no join keys and its join filter is the literal boolean `false`. + +(logical-rule-6)= + +### `decorrelate_predicate_subquery` + +Converts eligible `IN` and `EXISTS` predicate subqueries into semi or anti joins. + +#### Applies When + +A filter contains predicate subqueries that can be decorrelated into joins. + +(logical-rule-7)= + +### `scalar_subquery_to_join` + +Rewrites eligible scalar subqueries into joins and adds schema-preserving projections. + +#### Applies When + +A filter or projection contains scalar subqueries that can be decorrelated into joins. + +(logical-rule-8)= + +### `decorrelate_lateral_join` + +Rewrites eligible lateral joins into regular joins. + +#### Applies When + +The plan contains lateral joins with correlated input that can be pulled up safely. + +(logical-rule-9)= + +### `extract_equijoin_predicate` + +Splits join filters into equijoin keys and residual predicates. + +#### Applies When + +A join filter contains equality predicates that can become explicit join keys. + +(logical-rule-10)= + +### `eliminate_duplicated_expr` + +Removes duplicate expressions from projections, aggregates, and similar operators. + +#### Applies When + +The same logical expression appears more than once in a plan node input list. + +(logical-rule-11)= + +### `eliminate_filter` + +Drops always-true filters and replaces always-false or NULL filters with empty relations. + +#### Applies When + +A filter predicate simplifies to a constant boolean value or NULL. + +(logical-rule-12)= + +### `eliminate_cross_join` + +Uses filter predicates to replace cross joins with inner joins when join keys can be found. + +#### Applies When + +A filter above a cross join exposes equijoin conditions between the inputs. + +(logical-rule-13)= + +### `eliminate_limit` + +Removes no-op limits and simplifies trivial limit shapes. + +#### Applies When + +The plan contains limits that can be proven redundant or can collapse to simpler forms. + +(logical-rule-14)= Review Comment: what are those logical rule entries? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
