parthchandra opened a new issue, #7429: URL: https://github.com/apache/datafusion-comet/issues/7429
### What is the problem the feature request solves? JSON sources (`FileSourceScanExec` with `JsonFileFormat`, and V2 `JsonScan`) are read by Spark. `spark.comet.convert.json.enabled` can convert Spark's rows to Arrow, but parsing still runs on the JVM. ### Describe the potential solution Add a native JSON scan that matches Spark's JSON reader: - user-supplied schema - `PERMISSIVE`, `DROPMALFORMED` and `FAILFAST` modes, and `columnNameOfCorruptRecord` - timestamp and date formats with the session timezone - multi-line JSON and compression codecs - fall back for the options it can't match ### Additional context Native `from_json` already exists (opt-in). It may share parsing code with the scan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
