Author: lidong
Date: Sat Jul 28 14:09:14 2018
New Revision: 1836910
URL: http://svn.apache.org/viewvc?rev=1836910&view=rev
Log:
Update configurations for spark engine
Modified:
kylin/site/cn/docs/tutorial/cube_spark.html
kylin/site/docs/tutorial/cube_spark.html
kylin/site/feed.xml
Modified: kylin/site/cn/docs/tutorial/cube_spark.html
URL:
http://svn.apache.org/viewvc/kylin/site/cn/docs/tutorial/cube_spark.html?rev=1836910&r1=1836909&r2=1836910&view=diff
==============================================================================
--- kylin/site/cn/docs/tutorial/cube_spark.html (original)
+++ kylin/site/cn/docs/tutorial/cube_spark.html Sat Jul 28 14:09:14 2018
@@ -193,21 +193,30 @@ export KYLIN_HOME=/usr/local/apache-kyli
<p>Kylin å¨ $KYLIN_HOME/spark ä¸åµå
¥ä¸ä¸ª Spark binary
(v2.1.2)ï¼ææä½¿ç¨ <em>âkylin.engine.spark-conf.â</em> ä½ä¸ºåç¼ç
Spark é
ç½®å±æ§é½è½å¨ $KYLIN_HOME/conf/kylin.properties
ä¸è¿è¡ç®¡çãè¿äºå±æ§å½è¿è¡æ交 Spark job
æ¶ä¼è¢«æå并åºç¨ï¼ä¾å¦ï¼å¦ææ¨é
ç½®
âkylin.engine.spark-conf.spark.executor.memory=4Gâï¼Kylin å°ä¼å¨æ§è¡
âspark-submitâ æä½æ¶ä½¿ç¨ ââconf spark.executor.memory=4Gâ
ä½ä¸ºåæ°ã</p>
-<p>è¿è¡ Spark cubing åï¼å»ºè®®æ¥çä¸ä¸è¿äºé
置并æ
¹æ®æ¨é群çæ
åµè¿è¡èªå®ä¹ãä¸é¢æ¯é»è®¤é
ç½®ï¼ä¹æ¯ sandbox
æä½è¦æ±çé
ç½® (1 个 1GB memory ç
executor)ï¼é常ä¸ä¸ªé群ï¼éè¦æ´å¤ç executors
ä¸æ¯ä¸ä¸ªè³å°æ 4GB memory å 2 cores:</p>
+<p>è¿è¡ Spark cubing åï¼å»ºè®®æ¥çä¸ä¸è¿äºé
置并æ
¹æ®æ¨é群çæ
åµè¿è¡èªå®ä¹ãä¸é¢æ¯å»ºè®®é
ç½®ï¼å¼å¯äº Spark
å¨æèµæºåé
:</p>
<div class="highlight"><pre><code class="language-groff"
data-lang="groff">kylin.engine.spark-conf.spark.master=yarn
kylin.engine.spark-conf.spark.submit.deployMode=cluster
+kylin.engine.spark-conf.spark.dynamicAllocation.enabled=true
+kylin.engine.spark-conf.spark.dynamicAllocation.minExecutors=1
+kylin.engine.spark-conf.spark.dynamicAllocation.maxExecutors=1000
+kylin.engine.spark-conf.spark.dynamicAllocation.executorIdleTimeout=300
kylin.engine.spark-conf.spark.yarn.queue=default
+kylin.engine.spark-conf.spark.driver.memory=2G
kylin.engine.spark-conf.spark.executor.memory=4G
kylin.engine.spark-conf.spark.yarn.executor.memoryOverhead=1024
-kylin.engine.spark-conf.spark.executor.cores=2
-kylin.engine.spark-conf.spark.executor.instances=40
+kylin.engine.spark-conf.spark.executor.cores=1
+kylin.engine.spark-conf.spark.network.timeout=600
kylin.engine.spark-conf.spark.shuffle.service.enabled=true
+#kylin.engine.spark-conf.spark.executor.instances=1
kylin.engine.spark-conf.spark.eventLog.enabled=true
+kylin.engine.spark-conf.spark.hadoop.dfs.replication=2
+kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress=true
+kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec
+kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
kylin.engine.spark-conf.spark.eventLog.dir=hdfs\:///kylin/spark-history
kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs\:///kylin/spark-history
-#kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
## uncomment for HDP
#kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current
Modified: kylin/site/docs/tutorial/cube_spark.html
URL:
http://svn.apache.org/viewvc/kylin/site/docs/tutorial/cube_spark.html?rev=1836910&r1=1836909&r2=1836910&view=diff
==============================================================================
--- kylin/site/docs/tutorial/cube_spark.html (original)
+++ kylin/site/docs/tutorial/cube_spark.html Sat Jul 28 14:09:14 2018
@@ -5624,20 +5624,30 @@ export KYLIN_HOME=/usr/local/apache-kyli
<p>Kylin embedes a Spark binary (v2.1.0) in $KYLIN_HOME/spark, all the Spark
configurations can be managed in $KYLIN_HOME/conf/kylin.properties with prefix
<em>âkylin.engine.spark-conf.â</em>. These properties will be extracted and
applied when runs submit Spark job; E.g, if you configure
âkylin.engine.spark-conf.spark.executor.memory=4Gâ, Kylin will use
ââconf spark.executor.memory=4Gâ as parameter when execute
âspark-submitâ.</p>
-<p>Before you run Spark cubing, suggest take a look on these configurations
and do customization according to your cluster. Below is the default
configurations, which is also the minimal config for a sandbox (1 executor with
1GB memory); usually in a normal cluster, need much more executors and each has
at least 4GB memory and 2 cores:</p>
+<p>Before you run Spark cubing, suggest take a look on these configurations
and do customization according to your cluster. Below is the recommended
configurations:</p>
<div class="highlight"><pre><code class="language-groff"
data-lang="groff">kylin.engine.spark-conf.spark.master=yarn
kylin.engine.spark-conf.spark.submit.deployMode=cluster
+kylin.engine.spark-conf.spark.dynamicAllocation.enabled=true
+kylin.engine.spark-conf.spark.dynamicAllocation.minExecutors=1
+kylin.engine.spark-conf.spark.dynamicAllocation.maxExecutors=1000
+kylin.engine.spark-conf.spark.dynamicAllocation.executorIdleTimeout=300
kylin.engine.spark-conf.spark.yarn.queue=default
-kylin.engine.spark-conf.spark.executor.memory=1G
-kylin.engine.spark-conf.spark.executor.cores=2
-kylin.engine.spark-conf.spark.executor.instances=1
+kylin.engine.spark-conf.spark.driver.memory=2G
+kylin.engine.spark-conf.spark.executor.memory=4G
+kylin.engine.spark-conf.spark.yarn.executor.memoryOverhead=1024
+kylin.engine.spark-conf.spark.executor.cores=1
+kylin.engine.spark-conf.spark.network.timeout=600
+kylin.engine.spark-conf.spark.shuffle.service.enabled=true
+#kylin.engine.spark-conf.spark.executor.instances=1
kylin.engine.spark-conf.spark.eventLog.enabled=true
+kylin.engine.spark-conf.spark.hadoop.dfs.replication=2
+kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress=true
+kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec
+kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
kylin.engine.spark-conf.spark.eventLog.dir=hdfs\:///kylin/spark-history
kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs\:///kylin/spark-history
-#kylin.engine.spark-conf.spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
-
## uncomment for HDP
#kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current
#kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=-Dhdp.version=current
Modified: kylin/site/feed.xml
URL:
http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1836910&r1=1836909&r2=1836910&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Sat Jul 28 14:09:14 2018
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml"; rel="self"
type="application/rss+xml"/>
- <pubDate>Wed, 25 Jul 2018 06:59:25 -0700</pubDate>
- <lastBuildDate>Wed, 25 Jul 2018 06:59:25 -0700</lastBuildDate>
+ <pubDate>Sat, 28 Jul 2018 06:59:23 -0700</pubDate>
+ <lastBuildDate>Sat, 28 Jul 2018 06:59:23 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
