sunchao commented on PR #5457:
URL:
https://github.com/apache/datafusion-comet/pull/5457#issuecomment-5448899045
Fixed the remaining Parquet reader-boundary issue in
`6d016e289097a0fa8e63dbc9659d5b21c23a35bf`.
The implicit conversion from a physical DATE column to a requested
TIMESTAMP_NTZ column bypassed the Catalyst cast guard. On Spark 4.x, a file
with an overflowing date at row 5000 should still let `filter("id =
0").limit(1)` return its valid first row when Spark reads batches of 4096.
Native conversion could instead consume that overflow. Blocking only the native
scan was insufficient: a native filter could request another Spark batch
through the columnar Arrow bridge before returning its first selected row.
The fix keeps V1 Parquet scans requesting top-level NTZ data columns on
Spark and blocks both Arrow bridge paths, including with native scans disabled
or AQE enabled. Ordinary NTZ files conservatively fall back too because the
logical schema cannot prove each file's physical type. Pruned, partition-only,
nested, and Iceberg cases retain their existing scope. The guide explains this
tradeoff and the Spark 3.x schema rejection versus Spark 4.x conversion
behavior. Existing native kernel/operator assertions remain, using materialized
or nested fixtures where needed.
Validation passed:
- Spark 4.1.3: 221 selected tests, including the complete cast and
error-converter suites; 3 expected cancellations and 8 existing ignored tests.
- Spark 3.4.3, 3.5.9, 4.0.4, and 4.2.0: 28 focused reader/NTZ tests per
profile, with 3 expected cancellations each.
- 145 broader fuzz/map/writer tests, all 33 temporal tests on unchanged
suite/helper/production sources, and 129 executed SQL cases. One additional SQL
case is intentionally bypassed outside Spark 3.4.
- Spotless, Scalastyle, and CI's syntactic Scalafix 0.14.6 check. Actual
loaded and bundled native-library hashes match the unchanged native source
tree; no native rebuild or new performance measurement is claimed.
The PR description now includes the new behavior and validation, while
retaining earlier benchmark revisions. Hosted CI for this commit is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
