dwsmith1983 opened a new issue, #6967: URL: https://github.com/apache/datafusion-comet/issues/6967
### What is the problem the feature request solves? Found in #6838 ([review comment](https://github.com/apache/datafusion-comet/pull/6838#discussion_r4266138898)). A native scan routed to libhdfs builds its client with `create_hdfs_operator` (`native/core/src/parquet/parquet_support.rs`), which sets only the name node on opendal's HDFS builder. libhdfs then creates a fresh Hadoop `Configuration` from the executor classpath, so it sees `core-site.xml` and `hdfs-site.xml` but none of the scan's Hadoop options: not the reader options, and not `spark.hadoop.*` settings. A filesystem whose `fs.<scheme>.impl` or other settings come only from those sources fails natively (`No FileSystem for scheme ...`) even though Spark can read it. #6838 keeps the scan rule from claiming a libhdfs scheme that only a reader option lists, so those scans fall back to Spark. Settings for a libhdfs scheme still have to live in the executor's `core-site.xml`. ### Describe the potential solution Pass the scan's `fs.*` Hadoop options to libhdfs when building the client, for example through `hdfsBuilderConfSetStr`. opendal's HDFS builder has no way to set arbitrary configuration today, so this needs either an opendal change or building the libhdfs client directly. The forwarded keys should be limited to filesystem settings, and secrets among them handled the way the object_store path handles them. ### Additional context Once options reach libhdfs, the scan rule could again claim schemes listed only in reader options. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
