comphead opened a new issue, #26537: URL: https://github.com/apache/datafusion/issues/26537
### Is your feature request related to a problem or challenge? `NthValueEvaluator::memoize` returns early for `LAST_VALUE` and `NTH_VALUE` when `IGNORE NULLS` is set ([nth_value.rs#L415-L416](https://github.com/apache/datafusion/blob/3ee3e394c19c97ccc7cca7ed3becb75cf364746a/datafusion/functions-window/src/nth_value.rs#L414-L415)). For a frame with an `UNBOUNDED PRECEDING` start, such as the default `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`, the frame start then never moves. Reading the code, this means `BoundedWindowAggExec` cannot prune its input buffer and keeps the whole partition in memory, while `RESPECT NULLS` keeps only the last row (`LAST_VALUE`) or the last `n` rows (`NTH_VALUE(expr, -n)`). #26533 fixes the CPU cost of these frames (#26532), but not the buffering. I have not measured the memory use. ### Describe the solution you'd like Let `memoize` move the frame start under `IGNORE NULLS` too, keeping the rows the result can still depend on: - `LAST_VALUE`: from the last non-NULL row of the frame, or nothing before the current row if the frame has none. - `NTH_VALUE(expr, -n)`: from the `n`th non-NULL row from the end of the frame. - `NTH_VALUE(expr, n)` with `n > 0`: the result is final once the frame holds `n` non-NULL rows, as `FIRST_VALUE` already does. ### Describe alternatives you've considered None. ### Additional context Raised by @andygrove in the review of #26533. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
