vinitamaloo-asu commented on issue #8277:
URL: https://github.com/apache/iceberg/issues/8277#issuecomment-1672685374
I am trying to connect to a local embedded metastore that's why kept this
uri empty.
From my testing, I observed that when I am initializing a new HiveCatalog,
it does not create HiveConf with SparkConf and only has hadoop configuration.
When SparkSession is initially configured, it creates a new HiveConf which
has both sparkconf and hadoopConfiguration.
Snippet from HiveClientImpl(newHiveConf):
` val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey ->
kv.getValue) ++
sparkConf.getAll.toMap ++ extraConfig).toMap
confMap.foreach { case (k, v) => hiveConf.set(k, v) }
SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
logDebug(s"Applying Hadoop/Hive/Spark and extra properties to Hive
Conf:$k=$v")
}`
And since my new hivecatalog does not have sparkConfig attached it also does
not have all the hive configurations which I set during the sparkSession
builder phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
