sunchao commented on PR #5472: URL: https://github.com/apache/datafusion-comet/pull/5472#issuecomment-5455261770
Addressed the pruning follow-up in [d48800c93](https://github.com/apache/datafusion-comet/commit/d48800c93b089088724cb43af7dc5584e5756439). Floating `IN`/`InSet` lists containing only finite nonzero literals (and optional nulls) now keep the column unwrapped, so Parquet can use its statistics. NaNs, signed zeros, infinities, dynamic candidates, and existing normalization keep their previous handling. The change stays in the existing serializer; native code is unchanged. Validation: 39 tests passed on Spark 4.1 and 37 on Spark 3.4, with two expected version-based cancellations on 3.4. Both new serialization tests fail against the previous serializer. A fresh controlled Parquet probe confirms that the restored expression shape prunes 15 of 16 row groups instead of zero while returning the same two rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
