[
https://issues.apache.org/jira/browse/ATLAS-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861107#comment-15861107
]
Greg Senia commented on ATLAS-1546:
-----------------------------------
I accomplished this by adding a new definition to Ambari
Here is a workaround to make HiveCLI work from Ambari... A bit involved but it
works..
The work around to make Ambari generate a hive-cli-atlas-application.properties
file is as follows:
curl -u username -H "X-Requested-By: ambari" -X PUT -d @atlas-hivecli.json
"http://localhost:8080/api/v1/clusters/tech";
atkas-hivecli.json:
[{"Clusters":{
"desired_config":[{
"type" : "hive-cli-atlas-application.properties",
"properties" : {
"atlas.hook.hive.keepAliveTime" : "10",
"atlas.hook.hive.maxThreads" : "5",
"atlas.hook.hive.minThreads" : "5",
"atlas.hook.hive.numRetries" : "3",
"atlas.hook.hive.queueSize" : "1000",
"atlas.hook.hive.synchronous" : "false",
"atlas.jaas.KafkaClient.loginModuleControlFlag" : "required",
"atlas.jaas.KafkaClient.loginModuleName" :
"com.sun.security.auth.module.Krb5LoginModule",
"atlas.jaas.KafkaClient.option.serviceName" : "kafka",
"atlas.jaas.KafkaClient.option.renewTicket" : "True",
"atlas.jaas.KafkaClient.option.storeKey" : "false",
"atlas.jaas.KafkaClient.option.useKeyTab" : "false",
"atlas.jaas.KafkaClient.option.useTicketCache" : "True"
}
}
]
}
}
]
Apply the following to /var/lib/ambari-server/resources/stacks/HDP/HDP-2.5
[username@hadoop1 ~]$ cat stacks_ambari.patch
--- /dev/null
+++
/var/lib/ambari-server/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-cli-atlas-application.properties.xml
2017-02-09 14:10:05.000000000 -0500
@@ -0,0 +1,61 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+-->
+<configuration supports_final="false">
+ <!-- These are the Atlas Hooks properties specific to this service. This file
is then merged with common properties
+ that apply to all services. -->
+ <property>
+ <name>atlas.hook.hive.synchronous</name>
+ <value>false</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.numRetries</name>
+ <value>3</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.minThreads</name>
+ <value>5</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.maxThreads</name>
+ <value>5</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.keepAliveTime</name>
+ <value>10</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.queueSize</name>
+ <value>1000</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+</configuration>
--- /var/lib/ambari-server/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml
2016-11-23 02:27:15.000000000 -0500
+++ /var/lib/ambari-server/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml
2017-02-09 09:46:52.000000000 -0500
@@ -243,6 +243,7 @@
<configuration-dependencies>
<config-type>application-properties</config-type>
<config-type>hive-atlas-application.properties</config-type>
+ <config-type>hive-cli-atlas-application.properties</config-type>
</configuration-dependencies>
</service>
</services>
/var/lib/ambari-server/resources/common-services:
[username@hadoop1 ~]$ cat common_ambari.patch
diff -Naur -x '*.pyc' -x '*.zip' -x '*.pyo'
/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
/tmp/amb/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
---
/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
2016-11-23 02:27:10.000000000 -0500
+++
/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
2017-02-09 14:42:00.000000000 -0500
@@ -228,12 +228,15 @@
group=params.user_group,
mode=0644)
- # Generate atlas-application.properties.xml file
if has_atlas_in_cluster():
atlas_hook_filepath = os.path.join(params.hive_config_dir,
params.atlas_hook_filename)
- setup_atlas_hook(SERVICE.HIVE, params.hive_atlas_application_properties,
atlas_hook_filepath, params.hive_user, params.user_group)
+ setup_atlas_hook(SERVICE.HIVE, params.hive_cli_atlas_application_properties,
atlas_hook_filepath, params.hive_user, params.user_group)
if name == 'hiveserver2':
+ if has_atlas_in_cluster():
+ atlas_hook_filepath = os.path.join(params.hive_server_conf_dir,
params.atlas_hook_filename)
+ setup_atlas_hook(SERVICE.HIVE, params.hive_atlas_application_properties,
atlas_hook_filepath, params.hive_user, params.user_group)
+
XmlConfig("hiveserver2-site.xml",
conf_dir=params.hive_server_conf_dir,
configurations=params.config['configurations']['hiveserver2-site'],
diff -Naur -x '*.pyc' -x '*.zip' -x '*.pyo'
/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
/tmp/amb/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
---
/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
2016-11-23 02:27:10.000000000 -0500
+++
/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
2017-02-09 14:42:41.000000000 -0500
@@ -520,6 +520,7 @@
########################################################
#region Atlas Hooks
hive_atlas_application_properties =
default('/configurations/hive-atlas-application.properties', {})
+hive_cli_atlas_application_properties =
default('/configurations/hive-cli-atlas-application.properties', {})
if has_atlas_in_cluster():
atlas_hook_filename =
default('/configurations/atlas-env/metadata_conf_file',
'atlas-application.properties')
diff -Naur -x '*.pyc' -x '*.zip' -x '*.pyo'
/var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
/tmp/amb/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
---
/var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
2016-11-23 02:27:06.000000000 -0500
+++
/var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
2017-02-09 15:09:15.000000000 -0500
@@ -314,7 +314,7 @@
if has_atlas_in_cluster():
atlas_hook_filepath = os.path.join(params.hive_conf_dir,
params.atlas_hook_filename)
Logger.info("Has atlas in cluster, will save Atlas Hive hook into
location %s" % str(atlas_hook_filepath))
- setup_atlas_hook(SERVICE.HIVE, params.hive_atlas_application_properties,
atlas_hook_filepath, params.oozie_user, params.user_group)
+ setup_atlas_hook(SERVICE.HIVE, params.hive_cli_atlas_application_properties,
atlas_hook_filepath, params.oozie_user, params.user_group)
Directory(params.oozie_server_dir,
owner = params.oozie_user,
diff -Naur -x '*.pyc' -x '*.zip' -x '*.pyo'
/var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
/tmp/amb/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
---
/var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
2016-11-23 02:27:06.000000000 -0500
+++
/var/lib/ambari-server/resources//common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
2017-02-09 15:10:05.000000000 -0500
@@ -308,6 +308,7 @@
########################################################
#region Atlas Hooks needed by Hive on Oozie
hive_atlas_application_properties =
default('/configurations/hive-atlas-application.properties', {})
+hive_cli_atlas_application_properties =
default('/configurations/hive-cli-atlas-application.properties', {})
if has_atlas_in_cluster():
atlas_hook_filename =
default('/configurations/atlas-env/metadata_conf_file',
'atlas-application.properties')
No API call avaialble to allow linking a config property to the
serviceconfigmapping DB so it can be performed as follows:
Update Ambari DB !!! WORKING EXAMPLE!!!
insert into clusterconfigmapping (cluster_id, type_name,
version_tag,create_timestamp, selected, user_name) VALUES ('2',
'hive-cli-atlas-application.properties', 'generatedTag_1', '1484066089143',
'1', 'username');
select max(config_id) from clusterconfig where type_name =
'hive-cli-atlas-application.properties';
select max(version) from serviceconfig where service_name = 'HIVE';
insert into serviceconfigmapping (service_config_id, config_id) VALUES ('1203',
'1552');
ambari-server restart
Restart Hive Services and Oozie ]
> Hive hook should choose appropriate JAAS config if host uses kerberos
> ticket-cache
> ----------------------------------------------------------------------------------
>
> Key: ATLAS-1546
> URL: https://issues.apache.org/jira/browse/ATLAS-1546
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-intg
> Affects Versions: 0.7-incubating, 0.8-incubating
> Reporter: Madhan Neethiraj
> Assignee: Nixon Rodrigues
>
> In a kerberized environment, Atlas hook uses JAAS configuration section named
> "KakfaClient" to authenticate with Kafka broker. In a typical Hive deployment
> this configuration section is set to use the keytab and principal of
> HiveServer2 process. The hook running in HiveCLI might fail to authenticate
> with Kafka if the user can't read the configured keytab.
> Given that HiveCLI users would have performed kinit, the hook in HiveCLI
> should use the ticket-cache generated by kinit. When ticket cache is not
> available (for example in HiveServer2), the hook should use the configuration
> provided in KafkaClient JAAS section.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
