Xuanwo opened a new issue, #1130: URL: https://github.com/apache/iceberg-rust/issues/1130
There are several ways to access HDFS in remote areas: - libhdfs + JNI: needs java runtime and libhdfs - WebHDFS: http based, may have performance issues - [HDFS Native](https://github.com/Kimahriman/hdfs-native): rust native but not well adopted. OpenDAL supports all these methods. However, from Iceberg's perspective, it's better to expose a single `hdfs` storage option and allow users to choose the method via feature flags to avoid confusion. This is because we write service names in our metadata files, and all implementations must be able to read `hdfs://host:port/files`. So my current plan is to introduce `webhdfs` and `hdfs_native` support in iceberg-rust. They will hide under `storage-webhdfs` and `storage-hdfs-native` flag: Users can choose which implementations to use: ```rust fn parse_scheme(scheme: &str) -> crate::Result<Scheme> { match scheme { "memory" => Ok(Scheme::Memory), "file" | "" => Ok(Scheme::Fs), "s3" | "s3a" => Ok(Scheme::S3), "gs" | "gcs" => Ok(Scheme::Gcs), #[cfg(feature = "storage-webhdfs")] "hdfs" => Ok(Scheme::Webhdfs), #[cfg(feature = "storage-hdfs-native")] "hdfs" => Ok(Scheme::HdfsNative), s => Ok(s.parse::<Scheme>()?), } } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
