ydj960301 opened a new issue, #12857: URL: https://github.com/apache/iceberg/issues/12857
### Apache Iceberg version 1.8.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 ## Environment - Image: spark-iceberg:3.5.5_1.8.1 (custom Docker container) - - Spark version: 3.5.5 - - Iceberg version: 1.8.1 - **Relevant spark-defaults.conf:** spark-defaults.conf ``` spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions spark.sql.catalog.hive=org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.hive.type=hive spark.sql.catalog.hive.uri=thrift://hive-metastore:9083 spark.sql.catalog.hive.io-impl=org.apache.iceberg.aws.s3.S3FileIO spark.sql.catalog.hive.warehouse=s3://bucket/ spark.sql.catalog.hive.s3.endpoint=http://~~ spark.sql.defaultCatalog=hive spark.eventLog.enabled=true spark.eventLog.dir=/home/iceberg/spark-events spark.history.fs.logDirectory=/home/iceberg/spark-events spark.sql.catalogImplementation=hive spark.sql.hive.thriftServer.singleSession=false spark.serializerorg.apache.spark.serializer.KryoSerializer spark.hive.metastore.uris=thrift://hive-metastore:9083 spark.hive.metastore.schema.verification=false spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.s3a.endpoint=http://~~~ spark.hadoop.fs.s3a.path.style.access=true spark.hadoop.fs.s3a.access.key=access spark.hadoop.fs.s3a.secret.key=secret ``` **Problem** When attempting to register a table using the following procedure in spark-sql: ``` call hive_prod.system.register_table( table => 'log.test', metadata_file => 's3a://bucket/log/test/metadata/00832-fe78910f-e1b7-4ae0-bfc2-f23b34559f2b.metadata.json') ``` I encounter the following error: ``` Caused by: MetaException(message:Got exception: org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme "s3") at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:42225) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:42193) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:42119) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1203) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1189) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2396) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:750) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:569) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169) at jdk.proxy2/jdk.proxy2.$Proxy31.createTable(Unknown Source) at org.apache.iceberg.hive.HiveOperationsBase.lambda$persistTable$2(HiveOperationsBase.java:159) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:72) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:65) at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122) at org.apache.iceberg.hive.HiveOperationsBase.persistTable(HiveOperationsBase.java:157) at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:248) ... 63 more ``` **Notes** - Reading data from s3a:// works fine using spark.read.parquet(). - - The issue only occurs during the execution of the register_table procedure. - - - I have tried both s3a:// and s3:// in the metadata_file path, but the same error occurs in both cases. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
