andygrove opened a new pull request, #7121: URL: https://github.com/apache/datafusion-comet/pull/7121
## Which issue does this PR close? Closes #7118. ## Rationale for this change Two changes since 1.1.0 send strings through Comet's invalid UTF-8 decode that used to stay in Spark: a Text scan converted on a broadcast build side (#6786), and `min`/`max` over a string cast from binary (#6927). The compatibility guide says invalid UTF-8 "does not occur for text read from Parquet or produced by string expressions", which leads a reader to conclude that queries over Text files are unaffected, so 1.2.0 can return different results from 1.1.0 without the guide saying so. ## What changes are included in this PR? - `compatibility/index.md`: replace that sentence with the two cases where invalid UTF-8 can occur, with the `hex`/`length` and `min`/`max` examples from the issue and the workaround for each (`spark.comet.convert.broadcastBuildSide.enabled=false`, or aggregating over the `BINARY` column). - `datasources.md`: the broadcast build-side paragraph now mentions the `U+FFFD` replacement and links to the compatibility section. This takes the first option in the issue (document the behavior). It does not change the default of `spark.comet.convert.broadcastBuildSide.enabled`, which #6786 set to `true` on purpose. ## How are these changes tested? Documentation only. Checked with `npx prettier --check` on both files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
