pingzh opened a new pull request, #5697:
URL: https://github.com/apache/datafusion-comet/pull/5697
## Which issue does this PR close?
Fixes the six shuffle CI failures in #5262. This PR targets `df55`.
## Rationale for this change
Arrow 59.2 rejects nullable map-key fields during IPC validation with `Map
key field must not be nullable`, before Comet's schema compatibility check. The
Celeborn reader tests still expect `type mismatch`, so both reader variants
fail despite rejecting the malformed input correctly.
## What changes are included in this PR?
Update the shared nullable-map-key assertion for the JVM reader and native
ShuffleScan to expect Arrow's diagnostic. Keep the assertions that no rows are
returned and the fetch failure is reported exactly once.
## How are these changes tested?
- Full `CometCelebornShuffleReaderSuite` on Spark 4.1.3 / JDK 21: 58 tests
passed, including both nullable-map-key reader variants.
- Used the native library built by CI run 33891332915 for the unchanged base
commit `3ee7fa65b2f5e7018b5c60f4088a57eba2b9934f`.
- Maven Scala style and Spotless checks, plus `git diff --check`.
Test command (from the repository root):
```sh
SPARK_LOCAL_HOSTNAME=localhost SPARK_LOCAL_IP=127.0.0.1 SPARK_HOME="$PWD" \
./mvnw -B -Prelease -Pspark-4.1 test -Dtest=none \
-Dsuites=org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleReaderSuite
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
