0lai0 commented on issue #5022: URL: https://github.com/apache/datafusion-comet/issues/5022#issuecomment-6252227198
An update on where this stands, and a correction to the description. **Correction.** The description says Spark returns `NULL` under non-ANSI and throws under ANSI when a `DECIMAL` bound overflows. That is not what Spark does. `WindowEvaluatorFactoryBase` builds a `DECIMAL` bound with `DecimalAddNoOverflowCheck`, which neither returns `NULL` nor throws: it keeps the exact value and compares against it. So my earlier comment here, that `DECIMAL` needs the ANSI-mode work in apache/datafusion#23929, was based on the wrong premise. The actual divergence is the same for both types: when the native bound overflows, DataFusion takes the partition edge as the bound, which also takes in the `NULL` keys sorted at that edge, while Spark's exact bound leaves them out. For example, `DECIMAL(38,0)` values `[0, N, NULL]` with `N = 9e37` and `ORDER BY d ASC NULLS LAST RANGE BETWEEN N PRECEDING AND N FOLLOWING` give `[2,2,1]` in Spark and `[2,3,1]` natively. **DECIMAL.** #4987 falls back when the largest value of the `ORDER BY` type plus the offset can exceed `2^127 - 1` unscaled, which only `DECIMAL(38, s)` with a very large offset reaches. Everything else runs natively. **DATE.** #4974 (merged) falls back for offsets that can wrap Spark's `Int` addition, but the native bound already overflows once it leaves chrono's date range (about year 262142), and that pulls in `NULL` keys the same way. #<DATE PR> lowers the threshold to 92093340 days, the largest offset that keeps the bound inside chrono's range for every date in Spark's supported range. @andygrove I can't edit the issue body. Could you update it to the corrected Spark behavior? Once #4987 and #<DATE PR> merge, the last one can close this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
