Repository: kylin Updated Branches: refs/heads/document cace0b5d0 -> b68aa9a2f
update blog standalone-hbase-cluster with "Enable NN HA" section Project: http://git-wip-us.apache.org/repos/asf/kylin/repo Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/b68aa9a2 Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/b68aa9a2 Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/b68aa9a2 Branch: refs/heads/document Commit: b68aa9a2fc7f219df6dc0e7c7f32591ef14e198d Parents: cace0b5 Author: sunyerui <sunye...@gmail.com> Authored: Thu Jul 21 20:14:36 2016 +0800 Committer: sunyerui <sunye...@gmail.com> Committed: Thu Jul 21 20:14:36 2016 +0800 ---------------------------------------------------------------------- .../blog/2016-06-10-standalone-hbase-cluster.md | 25 ++++++++++++++++---- 1 file changed, 20 insertions(+), 5 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kylin/blob/b68aa9a2/website/_posts/blog/2016-06-10-standalone-hbase-cluster.md ---------------------------------------------------------------------- diff --git a/website/_posts/blog/2016-06-10-standalone-hbase-cluster.md b/website/_posts/blog/2016-06-10-standalone-hbase-cluster.md index d01fce9..5099070 100644 --- a/website/_posts/blog/2016-06-10-standalone-hbase-cluster.md +++ b/website/_posts/blog/2016-06-10-standalone-hbase-cluster.md @@ -24,14 +24,29 @@ Update the config `kylin.hbase.cluster.fs` in kylin.properties, with a value as Notice that the value should keep consistent with the Namenode address of `root.dir` on HBase Master node, to ensure bulkload into hbase. -## Using NN HA -HDFS Namenode HA improved the availablity of cluster significantly, and maybe the HBase cluster enabled it. Apache Kylin doesn't support the HA perfectly for now, and here's the workaroud: +## Enable NN HA +HDFS Namenode HA improved the availablity of cluster significantly, and maybe the HBase cluster enabled it. +To enable NN HA on HBase cluster, set `kylin.hbase.cluster.fs` as NN-HA format path of HBase cluster in kylin.properties, like `hdfs://hbase-cluster:8020`. - - Add all `dfs.nameservices` related configs of HBase Cluster into `hadoop/etc/hadoop/hdfs-site.xml` in Kylin Server, to make sure that can access HBase Cluster using hdfs shell with nameservice path - - Add all `dfs.nameservices` related configs of both two clusters into `kylin_job_conf.xml`, to make sure that the MR job can access hbase cluster with nameservice path +Since the MR Job of cube building access both main cluster and hbase cluster, exceptions may be threw during task running. The reason is task need to access HBase cluster with NN-HA related configs, which usually not deployed in the main cluster. + +These exceptions could be resolved by configs updating. In details, merge NN-HA related configs of two clusters, including `dfs.nameservices`, `dfs.ha.namenodes.`, `dfs.client.failover.proxy.provider.` and other related. These configs should be updated into hdfs-site.xml in Kylin Server and Resource Manager of main cluster, also be added into kylin_job_conf.xml. + +Howerver, it may be difficult to update configs in production environments. Here's another way which avoing configs updating: + + - Update HBase client in Kylin Server with patch HBASE-14347, and set `hbase.use.dynamic.jar=false` in hbase-site.xml + - Update Kylin Server with patch KYLIN-1910, and set `kylin.hbase.cluster.hdfs.config.file=hbase.hdfs.xml` in kylin.properties. The `hbase.hdfs.xml` meaning the hdfs-site.xml of HBase cluster, put it in the same dir of kylin.properties + - Update Hadoop of Kylin Server and Resource Manager with patch YARN-3021 ## TroubleShooting + - UnknownHostException occurs during Cube Building - It usually occurs with HBase HA nameservice config, please refer the above section "Using NN HA" + It usually occurs with HBase HA nameservice config, please refer the above section "Enable NN HA" + - 'Error when open connection hbase' during Kylin Server startup + HBase tries to mkdir tmp dirs on hdfs during connection setup, failed with lack of NN-HA related configs, refer "Enable NN HA" + - Failed to submit cube building job + HBase cluster path will be parsed during generate cube building job, failed with lack of NN-HA related configs, refer "Enable NN HA" + - Cube Building Step 'Convert Cuboid Data to HFile' failed + Usually occurred with Kerberos Authentication. Resource Manager tries to renew all tokens when job submitting, the renew of HBase cluster token will be failed, with lack of NN-HA related configs, refer "Enable NN HA" - HFile BulkLoading Stucks for long time Check the regionserver log, there should be lots of error log, with WrongFS exception. Make sure the namenode address in `kylin.properites/kylin.hbase.cluster.fs` and hbase master node `hbase-site.xml/root.dir` is same
