vinitamaloo-asu opened a new issue, #8277:
URL: https://github.com/apache/iceberg/issues/8277
### Query engine
Spark - 3.1
Iceberg - 1.3.1
### Question
I am using HiveCatalog to create Iceberg tables with Spark as the execution
engine. Everytime I try to run hiveCatalog.createTable command it gives me an
exception.
This is my SparkSession configuration:
` conf.set("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.set("spark.sql.catalog.spark_catalog",
"org.apache.iceberg.spark.SparkSessionCatalog")
.set("spark.sql.catalog.spark_catalog.type", "hive")
.set("spark.sql.catalog.spark_catalog.warehouse", warehouseDir +
"/spark_catalog/")
.set("spark.sql.catalog.iceberg_catalog",
"org.apache.iceberg.spark.SparkCatalog")
.set("spark.sql.catalog.iceberg_catalog.type", "hive")
.set("spark.sql.catalog.iceberg_catalog.warehouse", warehouseDir +
"/iceberg_catalog/")
.set("iceberg.engine.hive.enabled", "true")`
This is how I am using HiveCatalog to create table:
` val hiveCatalog = new HiveCatalog()
val warehouse_uri =
sparkSession.sparkContext.getConf.get("spark.sql.catalog.iceberg_catalog.warehouse")
hiveCatalog.setConf(sparkSession.sparkContext.hadoopConfiguration)
val catalogProperties = Map(
CatalogProperties.CATALOG_IMPL ->
"org.apache.iceberg.hive.HiveCatalog",
CatalogProperties.URI-> "",
CatalogProperties.WAREHOUSE_LOCATION -> warehouse_uri,
CatalogProperties.APP_ID->sparkSession.sparkContext.getConf.get("spark.app.id"),
CatalogProperties.USER -> "###")
hiveCatalog.initialize("iceberg_catalog",
JavaConverters.mapAsJavaMap(catalogProperties))`
Below is the exception that I am seeing:
org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive
Metastore
org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:84)
org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:34)
org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:125)
org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:56)
org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122)
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:158)
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97)
org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:80)
org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:189)
I am using iceberg-spark-runtime-3.1_2.12 for Iceberg version 1.3.1.
Please let me know what I might be doing wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
