andygrove opened a new pull request, #7123: URL: https://github.com/apache/datafusion-comet/pull/7123
## Which issue does this PR close? Closes #7118. ## Rationale for this change Since 1.1.0, two changes send strings through Comet's invalid UTF-8 decode that used to stay in Spark: a Text scan converted on a broadcast join's build side (#6786) and `min`/`max` over a string cast from binary (#6927). Arrow strings must be valid UTF-8, so Comet replaces each ill-formed sequence with `U+FFFD`, and `hex`, `length`, hashing, equality and ordering can disagree with Spark. Text is the source most likely to hold such bytes (for example a Latin-1 file), and the compatibility guide said invalid UTF-8 "does not occur for text read from Parquet", so 1.2.0 could return different results from 1.1.0 for the same query without the guide saying so. ## What changes are included in this PR? - `spark.comet.convert.broadcastBuildSide.enabled` now defaults to `false`, which restores the 1.1.0 behavior for Text build sides. The config doc says why. - `compatibility/index.md`: replaces the "does not occur" sentence with the cases where invalid UTF-8 can reach Comet, with the `hex`/`length` and `min`/`max` examples from the issue. - `datasources.md`: the build-side paragraph now says the conversion is opt-in, why, and links to that section. - Tests that exercise the bridge now enable the config explicitly. `CometBroadcastBuildSideSuite` sets it for the whole suite, and `CometExecSuite` uses a small `bridgeTest` helper for the tests that rely on it. The Text test also asserts that the default is off. The `min`/`max` case is documented and not changed, as it falls under the ordering difference the guide already lists. ## How are these changes tested? `CometBroadcastBuildSideSuite` and the broadcast build side, per-format opt-out and AQE DPP tests in `CometExecSuite`, run on the default Spark 4.1 profile (49 passed, 0 failed, 1 canceled). `npx prettier --check` passes on both docs files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
