sunchao opened a new pull request, #5556: URL: https://github.com/apache/datafusion-comet/pull/5556
## Which issue does this PR close? Follow-up to #5368. No separate issue. ## Rationale for this change The schema check added in #5368 rejects valid timestamp batches when native producers use the equivalent timezone labels `UTC` and `Etc/UTC`. With `spark.sql.session.timeZone=Etc/UTC`, a native Parquet scan produces `Timestamp(MICROSECOND, UTC)`, while native `date_trunc` preserves `Etc/UTC`. Unioning those branches and coalescing them into one partition makes `mapInArrow` and `mapInPandas` fail with `Arrow input schema changed between batches` when the second schema reaches the same Python input stream. Confirmed on the merged commit `e0fa3ccb16d4c9b3a03a38b09af6ed0fb242e027`, in both branch orders. ## What changes are included in this PR? - Accept `UTC` and `Etc/UTC` timestamp types interchangeably only when their units match, including in nested fields. - Preserve the first batch's stream schema and existing buffers. Keep all other type, dictionary, extension-metadata, and child-layout checks unchanged. - Add positive and negative Arrow compatibility tests and a 16-case Python regression covering both APIs, timezone labels, branch orders, and accelerated/fallback modes, with nulls and fractional timestamps. ## How are these changes tested? - Merged-code control: the 16 new Python cases produced **4 failures and 12 passes**. The failures were exactly the accelerated `Etc/UTC` cases for both APIs and branch orders. - Fixed code: the full `test_pyarrow_udf.py` suite passed **133/133**, including all 16 new cases, with no skips. This used Spark 4.1.3, real native execution, and real Python workers; accelerated cases assert the native Python, union, coalesce, and projection plan nodes. - `CometArrowPythonRunnerSuite`: **11/11 passed**, including Arrow C Data JNI ownership/cleanup checks. Running the same tests on the merged control gives the expected single alias-compatibility failure. - Full Spark 4.1 Spotless check and `git diff --check` passed. Local execution used all production JVM sources compiled into an isolated unshaded test jar and a native library built from the exact merged commit. Standard Maven packaging could not finish because the inherited Maven mirror timed out; shaded packaging and the Spark 4.0 matrix remain for CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
