andygrove commented on PR #7123: URL: https://github.com/apache/datafusion-comet/pull/7123#issuecomment-6295661138
@parthchandra pinging you as well, since this changes the default for the case in #6008. The build-side bridge from #6786 stays on by default, but a Text build side is now bridged only when `spark.comet.convert.broadcastBuildSide.text.enabled=true`. Spark's Text reader keeps the raw bytes of each line, and the conversion replaces bytes that are not valid UTF-8 with `U+FFFD`, so a Latin-1 lookup file could return different results from 1.1.0 (#7118). That means an allowlist read from a Text file, like the one in #6008, now needs that setting before the join runs natively. A new `spark.comet.convert.text.enabled` converts every Text scan, if that suits you better. Other build sides that Comet can't read natively, such as ORC or an Iceberg table the native reader declines, are still bridged by default. Does this work for your use case? I'd appreciate a review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
