ump45nose commented on PR #7089: URL: https://github.com/apache/datafusion-comet/pull/7089#issuecomment-6270975928
Thanks @andygrove. I rebased onto main and moved the docs change into the Known Issues section of `parquet.md`, since #7090 removed the old page. I added `legacy_int96_array_still_reads_as_timestamp` (737d58d). It writes an INT96 array in the legacy `bag` layout with the low-level Parquet writer, reads it as `array<timestamp>` through the scan with the conversion check on, and compares the values. If DataFusion ever hands that leaf over with a timezone, the new rule rejects it and this test fails. I checked that by forcing the rule to also reject below lists. On the JVM side, Spark's writer can't produce `TIMESTAMP(NANOS)` (it writes INT96, TIMESTAMP_MICROS or TIMESTAMP_MILLIS), so a `CometTestBase` test would need a checked-in file or a parquet-mr writer in the test. I haven't added one yet. If you'd like it, I can add a small resource file and a test that checks Spark and Comet both fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
