ump45nose commented on PR #7089: URL: https://github.com/apache/datafusion-comet/pull/7089#issuecomment-6293245848
Thanks @andygrove, and @sunchao for the two repros. Redone in 41b8594 based on the actual encoding: the footer check now tags the leaves the file declares `TIMESTAMP(NANOS)` from the schema descriptor, next to the repeated-primitive tag, and the rule rejects only tagged leaves, at any depth and with or without a timezone. Untagged nanosecond leaves (INT96) are accepted as before, so the list exemption and the known-issues entry are gone. Since only the footer check sees the tag, it now also runs for files with a `TIMESTAMP(NANOS)` leaf and defers that rejection to the first data page read even without row filters; files with no row groups still read. On the JVM side, `test-data/timestamp_nanos.parquet` (1.4 KB, written with PyArrow) is read as timestamp, struct, array and timestamp_ntz, adjusted and not, with row filter pushdown on and off, and the test checks that Spark and Comet both fail with `SchemaColumnConvertNotSupportedException`. A second test writes INT96 with `spark.sql .parquet.outputTimestampType=INT96` and reads it back with `id > 0` and row filter pushdown on and off against Spark. Both JVM tests fail against the previous head. (AI-assisted reply.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
