tuantran0910 opened a new pull request, #69954:
URL: https://github.com/apache/doris/pull/69954
### What problem does this PR solve?
Issue Number: close #69902
Related PR: #55942
Problem Summary:
A shuffled hash join whose build side matches very few rows can lose its
global runtime filter, so every probe-side scan reads the whole table. The
result stays correct, but latency regresses (in the report, from about 0.2s
to 10-30s, reading 74M rows instead of 1). It is intermittent; under
concurrent load about 1-6% of the queries were affected.
Reproduce: run a `RIGHT OUTER JOIN(PARTITIONED)` (or an inner/semi/anti join
that short-circuits on an empty build side) with two join instances and a
build side of one row. The probe-side scan profile then shows
`state: [DISABLED] ... reason: get disabled from remote` with
`reached_timeout: false`. Raising `runtime_filter_wait_time_ms` does not
help, and `enable_sync_runtime_filter_size=false` only narrows the window.
Root cause: the hash join build sink released its probe at eos, but built
and published its runtime filters only later in close(), after waiting for
the global filter size sync. On the instance with an empty build side (or a
null key for a null-aware anti join), `short_circuit_for_probe` makes the
probe return eos at once without reading its input. Its pipeline then
closes, `Pipeline::make_all_runnable` wakes up the build task early, and
`HashJoinBuildSinkLocalState::terminate` calls `skip_process()`, which
publishes the filters as DISABLED. `RuntimeFilterWrapper::merge` treats a
DISABLED input as absorbing, so the merged global filter is disabled for the
probe-side scans of all instances.
Fix: when the sink builds the hash table, has runtime filters and its probe
short-circuits, release the probe in close(), after the runtime filters are
published (also when close() fails). An empty build then publishes a READY,
empty filter and the merged filter keeps the keys of the other instances.
Do the same for the partitioned (spill) hash join sink when it is not
spilled. A sink terminated before eos still falls back to skip_process(), so
the guarantee of #55942 is kept. Cancellation and early wake-up still
unblock the probe.
Before: the probe-side scans intermittently got a disabled filter and read
the whole table. After: they get the READY filter of the build side.
### Release note
Fix a shuffled hash join intermittently disabling its global runtime filter
when one join instance has an empty build side, which made the probe side
scan the whole table.
### Check List (For Author)
- Test: Regression test / Unit Test
- BE UT `HashJoinBuildSinkTest.ReleaseShortCircuitProbeOnClose`
- Regression test `query_p0/join/test_short_circuit_probe_runtime_filter`
with a debug point that widens the race window
- Behavior changed: Yes. A short-circuited probe now finishes after its
build sink publishes the runtime filters, instead of at the build eos.
- Does this need documentation: No
Draft: not built locally yet; opened to run CI on the change and the new
tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
