raghav-reglobe opened a new pull request, #69273:
URL: https://github.com/apache/doris/pull/69273
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
A statement that must run on the master FE (`INSERT INTO ... SELECT`,
`CREATE TABLE ... AS SELECT`, ...) and arrives at a FOLLOWER or OBSERVER is
forwarded through `MasterOpExecutor`. The master plans it in a fresh
`ConnectContext` seeded from its own global variables plus the session
variables that `SessionVariable.getForwardVariables()` sends, i.e. only the
fields whose `@VarAttr` carries `needForward = true` (or one of the
`affectQueryResult*` attributes).
`disable_join_reorder` and 37 other planner/executor knobs carried no
`needForward` flag, so a session-level `SET` of any of them was silently
dropped on the forwarded path:
- the master could pick a different join order and distribution than the
same statement run on the master FE directly, and a query that fits under its
memory limit on the master fails with `MEM_LIMIT_EXCEEDED` when it enters
through a follower;
- `EXPLAIN` on the follower shows the plan the follower would run, not the
one the master executes, so the difference is hard to diagnose.
Reproduce on any cluster with at least two FEs: connect to a non-master FE,
`SET disable_join_reorder = true;`, run an `INSERT INTO t SELECT ... FROM a
JOIN b JOIN c ...` whose written join order differs from the optimizer's
choice, and read the query profile: the executed plan was reordered. The same
session on the master FE honours the setting.
Fix: flag the 38 knobs `needForward = true`, by family:
- join order: `disable_join_reorder`, `max_join_number_of_reorder`,
`max_join_number_bushy_tree`, `enable_cost_based_join_reorder`,
`enable_dphyp_optimizer`, `dphyper_limit`
- distribution: `enable_bucket_shuffle_join`, `disable_colocate_plan`,
`broadcast_right_table_scale_factor`, `enable_broadcast_join_force_passthrough`
- CTE: `enable_cte_materialize`, `cte_inline_mode`,
`inline_cte_referenced_threshold`
- TopN: `enable_partition_topn`, `global_partition_topn_threshold`,
`topn_opt_limit_threshold`, `topn_filter_ratio`
- aggregation: `disable_streaming_preaggregations`,
`enable_distinct_streaming_aggregation`, `enable_push_down_no_group_agg`,
`agg_phase`
- exchange: `enable_local_exchange`, `enable_local_exchange_before_agg`,
`parallel_exchange_instance_num`, `exchange_multi_blocks_byte_size`,
`enable_shared_exchange_sink_buffer`, `local_exchange_free_blocks_limit`
- runtime filter: `enable_decoupled_runtime_filter`,
`expand_runtime_filter_by_inner_join`, `ignore_runtime_filter_ids`
- spill / memory: `spill_min_revocable_mem`,
`spill_aggregation_partition_count`, `spill_hash_join_partition_count`,
`low_memory_mode_buffer_limit`
- skew join: `skew_rewrite_join_salt_explode_factor`
- parallelism: `parallel_prepare_threshold`, `enable_parallel_outfile`
- cache: `enable_sql_cache`
`agg_shuffle_use_parent_key` already carries an explicit `needForward =
false` and is left as it is.
`SessionVariablesTest.testForwardPlanAndExecutionKnobs` checks that every
knob above is present in `getForwardVariables()` and that
`disable_join_reorder` round-trips through `setForwardedSessionVariables`.
### Release note
Session variables that change how a statement is planned or executed
(`disable_join_reorder`, the join-reorder limits, DPhyp, broadcast /
bucket-shuffle / colocate switches, CTE inlining, TopN thresholds, streaming
and distinct aggregation, local exchange, runtime-filter switches, spill
partition counts, the salt-join explode factor, `enable_sql_cache`, ...) are
now forwarded to the master FE when a statement is forwarded from a follower or
observer. Previously a session-level `SET` of one of them was ignored on that
path.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- On a cluster with two FEs, `SET disable_join_reorder = true` on
the non-master FE and run an `INSERT INTO ... SELECT` with a three-way join.
Before the fix the master's query profile shows the reordered plan; after the
fix it shows the written join order, identical to running the statement on the
master FE.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. A statement forwarded to the master now honours the session
variables above as if it had been run on the master directly.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
