LinSimon-901101 opened a new pull request, #6074:
URL: https://github.com/apache/datafusion-comet/pull/6074
## Which issue does this PR close?
Closes #4384.
## Rationale for this change
Add reproducible randomized coverage for the Arrow IPC path between Comet
and Python, exercising combinations of nested types, null densities, and
multiple batches alongside the existing targeted tests.
## What changes are included in this PR?
- Add seeded generators for primitive and bounded nested array, struct,
and map types, including variable-length values and decimal types.
- Exercise randomized per-column null densities and explicit null fractions
of 0, 0.01, 0.5, 0.99, and 1.
- Compare accelerated and fallback mapInArrow results against a vanilla
Spark Parquet baseline that is also checked against the generated input.
- Assert execution plans and verify actual Python input batch indices
and sizes, including partial final batches.
- Handle the PySpark 4.2 Arrow schema conversion API change.
- Add the module to the PyArrow UDF workflow and CI change detection,
retaining the existing tests.
## How are these changes tested?
Local testing on macOS with JDK 21, Python 3.11.15, PyArrow 25.0.1,
pandas 3.0.6, and pytest 9.1.1:
- Spark 4.0.4: 39 passed before the schema API compatibility change;
all 3 affected Arrow IPC tests passed again after the change.
- Spark 4.1.3: 39 passed before the schema API compatibility change;
all 3 affected Arrow IPC tests passed again after the change.
- Spark 4.2.0: all 39 tests passed after the compatibility change.
PySpark emitted pandas 3.x compatibility warnings.
- `python dev/ci/check-ci-config.py`: passed after rebasing onto main.
- Verified that changes to the new test select the PyArrow UDF job for
merge queue events, but not ordinary unlabeled PR updates.
The end-to-end cases cover three seeds, two batch sizes, and six null
configurations, checking both accelerated and fallback execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
