fwojciec commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-6116390085
@jayzhan211 [Item F4](https://github.com/apache/datafusion/issues/24768#issuecomment-5552303741) (the build hash map shared as `Arc` with the join dynamic filter) also happens today in normal execution, without any spilling or fallback path. The join's reservation is freed when its stream ends, but `HashTableLookupExpr` in the dynamic filter keeps the map until the plan is dropped. In a multi-stage query, every finished join keeps its map unreserved while later stages reserve the full pool on top of it. In our workload this left about 10 GiB unreserved above a 15–18 GiB pool after a stage change, and the process was OOM-killed instead of returning `ResourcesExhausted`. Minimal repro (one join, drain and drop the stream, keep the plan): 61 MiB live with 0 reserved when `enable_join_dynamic_filter_pushdown = true`, 0 MiB when it is false. Reproduced on 55.1.0, 55.2.0 and main @ 454ca08b43: https://github.com/fwojciec/datafusion-join-map-retention-repro As a local workaround, we push bounds only (no map) in the dynamic filter. This removed the gap in our workload (from 10–12 GiB to 1–2 GiB after the stage change). That workaround drops the membership filter, so it is not a general fix. Options that keep the filter: hold a reservation together with the map inside the `Arc`, or replace the filter with bounds only when the join completes. If one of these looks right to you, I can try a PR on main. I can also open a separate issue, but this epic seemed the better place for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
