svn commit: r1839718 - in /kylin/site: ./ cn/docs/install/ cn/docs23/install/ docs/howto/ docs/install/ docs23/howto/ docs23/install/

[lidong]

Author: lidong
Date: Fri Aug 31 01:53:20 2018
New Revision: 1839718

URL: http://svn.apache.org/viewvc?rev=1839718&view=rev
Log:
Update document for EMR, jobengine HA, etc

Modified:
    kylin/site/cn/docs/install/advance_settings.html
    kylin/site/cn/docs/install/kylin_aws_emr.html
    kylin/site/cn/docs/install/kylin_cluster.html
    kylin/site/cn/docs23/install/advance_settings.html
    kylin/site/docs/howto/howto_ldap_and_sso.html
    kylin/site/docs/install/advance_settings.html
    kylin/site/docs/install/kylin_aws_emr.html
    kylin/site/docs/install/kylin_cluster.html
    kylin/site/docs23/howto/howto_ldap_and_sso.html
    kylin/site/docs23/install/advance_settings.html
    kylin/site/feed.xml

Modified: kylin/site/cn/docs/install/advance_settings.html
URL: 
http://svn.apache.org/viewvc/kylin/site/cn/docs/install/advance_settings.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/cn/docs/install/advance_settings.html (original)
+++ kylin/site/cn/docs/install/advance_settings.html Fri Aug 31 01:53:20 2018
@@ -259,7 +259,7 @@ var _hmt = _hmt || [];
 <p>使用多任务引擎，你可以在多个 Kylin 节点上é…
ç½®å®ƒçš„角色为 <code class="highlighter-rouge">job</code> 或 <code 
class="highlighter-rouge">all</code>。为了避å…
å®ƒä»¬ä¹‹é—´äº§ç”Ÿç«žäº‰ï¼Œéœ€è¦å¯ç”¨åˆ†å¸ƒå¼ä»»åŠ¡é”ï¼Œè¯·åœ¨ <code 
class="highlighter-rouge">kylin.properties</code> 里配置：</p>
 
 <div class="highlighter-rouge"><pre 
class="highlight"><code>kylin.job.scheduler.default=2
-kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperDistributedJobLock
+kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperJobLock
 </code></pre>
 </div>
 <p>并记得将所有任务和查询节点的地址注册到 <code 
class="highlighter-rouge">kylin.server.cluster-servers</code>.</p>

Modified: kylin/site/cn/docs/install/kylin_aws_emr.html
URL: 
http://svn.apache.org/viewvc/kylin/site/cn/docs/install/kylin_aws_emr.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/cn/docs/install/kylin_aws_emr.html (original)
+++ kylin/site/cn/docs/install/kylin_aws_emr.html Fri Aug 31 01:53:20 2018
@@ -315,6 +315,22 @@ $KYLIN_HOME/bin/kylin.sh start
 
 <p>Build 同一个 Cube，当 Cube 准备好后运行查询。您可以浏览 
S3 查看数据是否安全的持久化了。</p>
 
+<h3 id="spark-">Spark 配置</h3>
+
+<p>EMR 的 Spark 版本很可能与 Kylin 编译的版本不一致，因
此您通常不能直接使用 EMR 打包的 Spark 用于 Kylin 的任务。 
您需要在启动 Kylin 之前，将 “SPARK_HOME” 环境变量设置指向 
Kylin 的 Spark 子目录 (KYLIN_HOME/spark) 。此外，为了从 Spark 
中访问 S3 或 EMRFS 上的文件，您需要将 EMR 的扩展类从 EMR 
的目录拷贝到 Kylin 的 Spark 下。</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>export 
SPARK_HOME=$KYLIN_HOME/spark
+
+cp /usr/lib/hadoop-lzo/lib/*.jar $KYLIN_HOME/spark/jars/
+cp /usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-*.jar 
$KYLIN_HOME/spark/jars/
+cp /usr/lib/hadoop/hadoop-common*-amzn-*.jar $KYLIN_HOME/spark/jars/
+
+$KYLIN_HOME/bin/kylin.sh start
+</code></pre>
+</div>
+
+<p>您也可以参考 EMR Spark 的 spark-defauts 来设置 Kylin 的 Spark é…
ç½®ï¼Œä»¥èŽ·å¾—更好的对集群资源的适配。</p>
+
 <h3 id="emr--1">关闭 EMR 集群</h3>
 
 <p>关闭 EMR 集群前，我们建议您为 Kylin metadata 做备份且将å…
¶ä¸Šä¼ åˆ° S3。</p>

Modified: kylin/site/cn/docs/install/kylin_cluster.html
URL: 
http://svn.apache.org/viewvc/kylin/site/cn/docs/install/kylin_cluster.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/cn/docs/install/kylin_cluster.html (original)
+++ kylin/site/cn/docs/install/kylin_cluster.html Fri Aug 31 01:53:20 2018
@@ -198,18 +198,31 @@ var _hmt = _hmt || [];
 </div>
 
 <ul>
-  <li><code class="highlighter-rouge">kylin.server.mode</code><br />
-确保只有一个实例的 <code 
class="highlighter-rouge">kylin.server.mode</code> 设置为 “all” 或 
“job”, 其余的应该为 “query”</li>
+  <li><code class="highlighter-rouge">kylin.server.mode</code></li>
 </ul>
 
+<p>默认情况下，只有一个实例的 <code 
class="highlighter-rouge">kylin.server.mode</code> 设置为 “all” 或 
“job”, 其余的为 “query”。</p>
+
 <div class="highlighter-rouge"><pre 
class="highlight"><code>kylin.server.mode=all
 </code></pre>
 </div>
 
+<p>也即默认情
况下，只有一个节点用于调度构建任务的执行。如果您需要é…
ç½®å¤šä¸ªèŠ‚点同时执行任务构建，以满足高可用和高并发的需求，请参考
 “启用多个任务引擎” 的内容，在 <a 
href="advance_settings.html">高级设置</a> 页.</p>
+
 <h3 id="section">安装负载均衡器</h3>
 
 <p>为确保 Kylin 服务器的高可用性, 
您需要在这些服务器之前安装负载均衡器, 让其将传å…
¥çš„请求路由至集群。客户端和负载均衡器通信代替和特定的 
Kylin 实例通信。安装负载均衡器超出了范围，您可以选择像 
Nginx, F5 或 cloud LB 服务这样的实现。</p>
 
+<h3 id="section-1">读／写分离的双集群配置</h3>
+
+<p>Kylin 可以连接两个集群以获得更好的稳定性和性能：</p>
+
+<ul>
+  <li>一个 Hadoop 集群用作 Cube 构建; 
这个集群可以是一个大的、与其它应用共享的集群；</li>
+  <li>一个 HBase 集群用作 SQL 查询；通常这个集群是专门为 
Kylin 配置的，节点数不用像 Hadoop 集群那么多。HBase 的é…
ç½®å¯ä»¥æ›´åŠ é’ˆå¯¹ Kylin Cube 只读的特性而进行优化。</li>
+</ul>
+
+<p>这种部署策略已经被很多大企业所采纳并得到验证。它是迄今我们知道适合生产环境的最佳部署方案。å
…³äºŽå¦‚何配置这种架构，请参考 <a 
href="/blog/2016/06/10/standalone-hbase-cluster/">Deploy Apache Kylin with 
Standalone HBase Cluster</a></p>
 
                                                        </article>
                                                </div>

Modified: kylin/site/cn/docs23/install/advance_settings.html
URL: 
http://svn.apache.org/viewvc/kylin/site/cn/docs23/install/advance_settings.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/cn/docs23/install/advance_settings.html (original)
+++ kylin/site/cn/docs23/install/advance_settings.html Fri Aug 31 01:53:20 2018
@@ -259,7 +259,7 @@ var _hmt = _hmt || [];
 <p>使用多任务引擎，你可以在多个 Kylin 节点上é…
ç½®å®ƒçš„角色为 <code class="highlighter-rouge">job</code> 或 <code 
class="highlighter-rouge">all</code>。为了避å…
å®ƒä»¬ä¹‹é—´äº§ç”Ÿç«žäº‰ï¼Œéœ€è¦å¯ç”¨åˆ†å¸ƒå¼ä»»åŠ¡é”ï¼Œè¯·åœ¨ <code 
class="highlighter-rouge">kylin.properties</code> 里配置：</p>
 
 <div class="highlighter-rouge"><pre 
class="highlight"><code>kylin.job.scheduler.default=2
-kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperDistributedJobLock
+kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperJobLock
 </code></pre>
 </div>
 <p>并记得将所有任务和查询节点的地址注册到 <code 
class="highlighter-rouge">kylin.server.cluster-servers</code>.</p>

Modified: kylin/site/docs/howto/howto_ldap_and_sso.html
URL: 
http://svn.apache.org/viewvc/kylin/site/docs/howto/howto_ldap_and_sso.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/docs/howto/howto_ldap_and_sso.html (original)
+++ kylin/site/docs/howto/howto_ldap_and_sso.html Fri Aug 31 01:53:20 2018
@@ -5610,7 +5610,7 @@ var _hmt = _hmt || [];
 <p>Firstly, provide LDAP URL, and username/password if the LDAP server is 
secured; The password in kylin.properties need be encrypted; You can run the 
following command to get the encrypted value:</p>
 
 <div class="highlighter-rouge"><pre class="highlight"><code>cd 
$KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/lib
-java -classpath 
kylin-server-base-\&lt;versioin\&gt;.jar:spring-beans-3.2.17.RELEASE.jar:spring-core-3.2.17.RELEASE.jar:commons-codec-1.7.jar
 org.apache.kylin.rest.security.PasswordPlaceholderConfigurer AES 
&lt;your_password&gt;
+java -classpath 
kylin-server-base-\&lt;versioin\&gt;.jar:kylin-core-common-\&lt;versioin\&gt;.jar:spring-beans-4.3.10.RELEASE.jar:spring-core-4.3.10.RELEASE.jar:commons-codec-1.7.jar
 org.apache.kylin.rest.security.PasswordPlaceholderConfigurer AES 
&lt;your_password&gt;
 </code></pre>
 </div>
 

Modified: kylin/site/docs/install/advance_settings.html
URL: 
http://svn.apache.org/viewvc/kylin/site/docs/install/advance_settings.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/docs/install/advance_settings.html (original)
+++ kylin/site/docs/install/advance_settings.html Fri Aug 31 01:53:20 2018
@@ -5694,7 +5694,7 @@ The compression settings define in <code
 <p>To enable the distributed job scheduler, you need to set or update the 
configs in the kylin.properties:</p>
 
 <div class="highlighter-rouge"><pre 
class="highlight"><code>kylin.job.scheduler.default=2
-kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperDistributedJobLock
+kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperJobLock
 </code></pre>
 </div>
 <p>Please add all job servers and query servers to the <code 
class="highlighter-rouge">kylin.server.cluster-servers</code>.</p>

Modified: kylin/site/docs/install/kylin_aws_emr.html
URL: 
http://svn.apache.org/viewvc/kylin/site/docs/install/kylin_aws_emr.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/docs/install/kylin_aws_emr.html (original)
+++ kylin/site/docs/install/kylin_aws_emr.html Fri Aug 31 01:53:20 2018
@@ -5750,6 +5750,22 @@ $KYLIN_HOME/bin/kylin.sh start
 
 <p>Build the sample Cube, and then run queries when the Cube is ready. You can 
browse S3 to see whether the data is safely persisted.</p>
 
+<h3 id="spark-configuration">Spark Configuration</h3>
+
+<p>EMR’s Spark version may be incompatible with Kylin, so you couldn’t 
directly use EMR’s Spark. You need to set “SPARK_HOME” environment 
variable to Kylin’s Spark folder (KYLIN_HOME/spark) before start Kylin. To 
access files on S3 or EMRFS, we need to copy EMR’s implementation jars to 
Spark.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>export 
SPARK_HOME=$KYLIN_HOME/spark
+
+cp /usr/lib/hadoop-lzo/lib/*.jar $KYLIN_HOME/spark/jars/
+cp /usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-*.jar 
$KYLIN_HOME/spark/jars/
+cp /usr/lib/hadoop/hadoop-common*-amzn-*.jar $KYLIN_HOME/spark/jars/
+
+$KYLIN_HOME/bin/kylin.sh start
+</code></pre>
+</div>
+
+<p>You can also copy EMR’s spark-defauts configuration to Kylin’s spark 
for a better utilization of the cluster resources.</p>
+
 <h3 id="shut-down-emr-cluster">Shut down EMR Cluster</h3>
 
 <p>Before you shut down EMR cluster, we suggest you take a backup for Kylin 
metadata and upload it to S3.</p>

Modified: kylin/site/docs/install/kylin_cluster.html
URL: 
http://svn.apache.org/viewvc/kylin/site/docs/install/kylin_cluster.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/docs/install/kylin_cluster.html (original)
+++ kylin/site/docs/install/kylin_cluster.html Fri Aug 31 01:53:20 2018
@@ -5613,7 +5613,9 @@ var _hmt = _hmt || [];
   <li><strong>all</strong> : run both job engine and query engines in this 
instance.</li>
 </ul>
 
-<p>Notice that only one instance can run the job engine (“all” or 
“job” mode), the others must be “query” mode.</p>
+<p>By default only one instance can run the job engine (“all” or “job” 
mode), the others should be in the “query” mode.</p>
+
+<p>If you want to run multiple job engines to get high availability or handle 
heavy concurrent jobs, please check “Enable multiple job engines” in <a 
href="advance_settings.html">Advanced settings</a> page.</p>
 
 <p>A typical scenario is depicted in the following chart:</p>
 
@@ -5645,6 +5647,16 @@ Make sure there is only one instance who
 
 <p>To enable Kylin service high availability, you need setup a load balancer 
in front of these servers, letting it routes the incoming requests to the 
cluster. Client side communicates with the load balancer, instead of with a 
specific Kylin instance. The setup of load balancer is out of the scope; you 
may select an implementation like Nginx, F5 or cloud LB service.</p>
 
+<h3 id="configure-readwrite-separated-deployment">Configure Read/Write 
separated deployment</h3>
+
+<p>Kylin can work with two clusters to gain better stability and 
performance:</p>
+
+<ul>
+  <li>A Hadoop cluster for Cube building; This can be a shared, large 
cluster.</li>
+  <li>A HBase cluster for SQL queries; Usually this is a dedicated cluster 
with less nodes. The HBase configurations can be tuned for better read 
performance as Cubes are immutable after built.</li>
+</ul>
+
+<p>This deployment has been adopted and verified by many large companies. It 
is the best solution for production deployment as we know. For how to do this, 
please refer to <a href="/blog/2016/06/10/standalone-hbase-cluster/">Deploy 
Apache Kylin with Standalone HBase Cluster</a></p>
 
                                                        </article>
                                                </div>

Modified: kylin/site/docs23/howto/howto_ldap_and_sso.html
URL: 
http://svn.apache.org/viewvc/kylin/site/docs23/howto/howto_ldap_and_sso.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/docs23/howto/howto_ldap_and_sso.html (original)
+++ kylin/site/docs23/howto/howto_ldap_and_sso.html Fri Aug 31 01:53:20 2018
@@ -5611,7 +5611,7 @@ var _hmt = _hmt || [];
 <p>Firstly, provide LDAP URL, and username/password if the LDAP server is 
secured; The password in kylin.properties need be encrypted; You can run the 
following command to get the encrypted value:</p>
 
 <div class="highlighter-rouge"><pre class="highlight"><code>cd 
$KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/lib
-java -classpath 
kylin-server-base-\&lt;versioin\&gt;.jar:spring-beans-3.2.17.RELEASE.jar:spring-core-3.2.17.RELEASE.jar:commons-codec-1.7.jar
 org.apache.kylin.rest.security.PasswordPlaceholderConfigurer AES 
&lt;your_password&gt;
+java -classpath 
kylin-server-base-\&lt;versioin\&gt;.jar:kylin-core-common-\&lt;versioin\&gt;.jar:spring-beans-4.3.10.RELEASE.jar:spring-core-4.3.10.RELEASE.jar:commons-codec-1.7.jar
 org.apache.kylin.rest.security.PasswordPlaceholderConfigurer AES 
&lt;your_password&gt;
 </code></pre>
 </div>
 

Modified: kylin/site/docs23/install/advance_settings.html
URL: 
http://svn.apache.org/viewvc/kylin/site/docs23/install/advance_settings.html?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/docs23/install/advance_settings.html (original)
+++ kylin/site/docs23/install/advance_settings.html Fri Aug 31 01:53:20 2018
@@ -5695,7 +5695,7 @@ The compression settings define in <code
 <p>To enable the distributed job scheduler, you need to set or update the 
configs in the kylin.properties:</p>
 
 <div class="highlighter-rouge"><pre 
class="highlight"><code>kylin.job.scheduler.default=2
-kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperDistributedJobLock
+kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperJobLock
 </code></pre>
 </div>
 <p>Please add all job servers and query servers to the <code 
class="highlighter-rouge">kylin.server.cluster-servers</code>.</p>

Modified: kylin/site/feed.xml
URL: 
http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1839718&r1=1839717&r2=1839718&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Fri Aug 31 01:53:20 2018
@@ -19,8 +19,8 @@
     <description>Apache Kylin Home</description>
     <link>http://kylin.apache.org/</link>
     <atom:link href="http://kylin.apache.org/feed.xml"; rel="self" 
type="application/rss+xml"/>
-    <pubDate>Thu, 23 Aug 2018 06:59:42 -0700</pubDate>
-    <lastBuildDate>Thu, 23 Aug 2018 06:59:42 -0700</lastBuildDate>
+    <pubDate>Thu, 30 Aug 2018 18:45:34 -0700</pubDate>
+    <lastBuildDate>Thu, 30 Aug 2018 18:45:34 -0700</lastBuildDate>
     <generator>Jekyll v2.5.3</generator>
     
       <item>