Author: lidong
Date: Wed Mar 2 06:20:13 2016
New Revision: 1733218
URL: http://svn.apache.org/viewvc?rev=1733218&view=rev
Log:
Update upgrade document
Modified:
kylin/site/docs2/howto/howto_upgrade.html
kylin/site/feed.xml
Modified: kylin/site/docs2/howto/howto_upgrade.html
URL:
http://svn.apache.org/viewvc/kylin/site/docs2/howto/howto_upgrade.html?rev=1733218&r1=1733217&r2=1733218&view=diff
==============================================================================
--- kylin/site/docs2/howto/howto_upgrade.html (original)
+++ kylin/site/docs2/howto/howto_upgrade.html Wed Mar 2 06:20:13 2016
@@ -1658,106 +1658,94 @@
<div id="pjax">
<h1
class="post-title">How to Upgrade</h1>
- <p>version:
v1.2, since: v0.7.1</p>
+ <p>version:
v2.0, since: v2.0</p>
<article
class="post-content" >
- <h2
id="upgrade-among-v07x-and-v1x">Upgrade among v0.7.x and v1.x</h2>
+ <h2
id="upgrade-from-v1x-to-v20">Upgrade from v1.x to v2.0</h2>
-<p>From v0.7.1 to latest v1.2, Kylinâs metadata is backward compatible, the
upgrade can be finished in couple of minutes:</p>
+<p>From v1.x to v2.0, Kylinâs cube data is backward compatible, but metadata
has been refactored as new schema, to support new features on cubing and query
enhancement. So if you want to deploy v2.0 on your v1.x base, you need to
upgrade the metadata as following steps:</p>
-<h4 id="backup-metadata">1. Backup metadata</h4>
-<p>Backup the Kylin metadata peridically is a good practice, and is highly
suggested before upgrade;</p>
+<h4 id="backup-metadata-on-v1x">1. Backup metadata on v1.x</h4>
+<p>To avoid data loss during the upgrade, a backup at the very beginning is
always suggested. In case of upgrade failure, you can roll back to original
state with the backup.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>cd $KYLIN_HOME
-./bin/metastore.sh backup
+<div class="highlighter-rouge"><pre
class="highlight"><code>$KYLIN_HOME/bin/metastore.sh backup
</code></pre>
</div>
-<p>It will print the backup folder, take it down and make sure it will not be
deleted before the upgrade finished. If there is no âmetastore.shâ, use
HBaseâs snapshot command to do backup:</p>
+<p>It will print the backup folder, take it down and make sure it will not be
deleted before the upgrade finished. If there is no âmetastore.shâ, you can
use HBaseâs snapshot command to do backup:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>hbase shell
-snapshot 'kylin_metadata', 'kylin_metadata_backup20150610'
+snapshot 'kylin_metadata', 'kylin_metadata_backup20160101'
</code></pre>
</div>
-<p>Here âkylin_metadataâ is the default kylin metadata table name, replace
it with the right table name of your Kylin;</p>
+<p>Here âkylin_metadataâ is the default kylin metadata table name, replace
it with the right table name of your Kylin metastore.</p>
-<h4 id="install-new-kylin-and-copy-back-conf">2. Install new Kylin and copy
back âconfâ</h4>
-<p>Download the new Kylin binary package from Kylinâs download page; Extract
it to a different folder other than current KYLIN_HOME; Before copy back the
âconfâ folder, do a compare and merge between the old and new
kylin.properties to ensure newly introduced property will be kept.</p>
+<h4 id="stop-kylin-v1x-instance">2. Stop Kylin v1.x instance</h4>
+<p>Before deploying Kylin v2.0 instance, you need to stop the old instance.
Note that end users cannot access kylin service from this point.</p>
-<h4 id="stop-old-and-start-new-kylin-instance">3. Stop old and start new Kylin
instance</h4>
-<div class="highlighter-rouge"><pre class="highlight"><code>cd $KYLIN_HOME
-./bin/kylin.sh stop
-export KYLIN_HOME="<path_of_new_installation>"
-cd $KYLIN_HOME
-./bin/kylin.sh start
+<div class="highlighter-rouge"><pre
class="highlight"><code>$KYLIN_HOME/bin/kylin.sh stop
</code></pre>
</div>
+<p>#### 3. Install Kylin v2.0 and copy back âconfâ<br />
+Download the new Kylin v2.0 binary package from Kylinâs download page;
Extract it to a different folder other than current KYLIN_HOME; Before copy
back the âconfâ folder, do a compare and merge between the old and new
kylin.properties to ensure newly introduced property will be kept.</p>
-<h4 id="back-port-if-the-upgrade-is-failed">4. Back-port if the upgrade is
failed</h4>
-<p>If the new version couldnât startup and need back-port, shutdown it and
then switch to the old KYLIN_HOME to start. Idealy that would return to the
origin state. If the metadata is broken, restore it from the backup folder.</p>
+<h4
id="optional-4-upgrading-metadata-will-not-bring-new-features-of-v20-to-existing-cube-built-with-v1x-engine-if-you-want-to-leverage-those-features-please-refer-to-highlight-part">(Optional)
4. Upgrading metadata will not bring new features of v2.0 to existing cube
built with v1.x engine. If you want to leverage those features, please refer to
<a href="">Highlight</a> part.</h4>
-<div class="highlighter-rouge"><pre class="highlight"><code>./bin/metastore.sh
restore <path_of_metadata_backup>
+<h4 id="automaticly-upgrade-metadata">5. Automaticly upgrade metadata</h4>
+<p>Kylin v2.0 package provides a script for metadata automaticly upgrade. In
this upgrade, empty cubes will be updated to v2.0 version and all new features
are enabled for them. But those non-empty cubes are not able to use those new
features.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>export
KYLIN_HOME="<path_of_new_installation>"
+$KYLIN_HOME/bin/upgrade_v2.sh
</code></pre>
</div>
+<p>After this, the metadata in hbase table has been applied with new metadata
schema.</p>
-<h2 id="upgrade-from-v06x-to-v07x">Upgrade from v0.6.x to v0.7.x</h2>
-
-<p>In v0.7, Kylin refactored the metadata structure, for the new features like
inverted-index and streaming; If you have cube created with v0.6 and want to
keep in v0.7, a migration is needed; (Please skip v0.7.1 as<br />
-it has several compatible issues and the fix will be included in v0.7.2) Below
is the steps;</p>
-
-<h4 id="backup-v06-metadata">1. Backup v0.6 metadata</h4>
-<p>To avoid data loss in the migration, a backup at the very beginning is
always suggested; You can use HBaseâs backup or snapshot command to achieve
this; Here is a sample with snapshot:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>hbase shell
-snapshot 'kylin_metadata', 'kylin_metadata_backup20150610'
+<h4 id="start-kylin-v20-instance">6. Start Kylin v2.0 instance</h4>
+<div class="highlighter-rouge"><pre
class="highlight"><code>$KYLIN_HOME/bin/kylin.sh start
</code></pre>
</div>
+<p>Check the log and open web UI to see if the upgrade succeeded.</p>
-<p>âkylin_metadataâ is the default kylin metadata table name, replace it
with the right table name of your Kylin;</p>
+<h2 id="rollback-if-the-upgrade-is-failed">Rollback if the upgrade is
failed</h2>
+<p>If the new version couldnât startup normally, you need to roll back to
orignal v1.x version. The steps are as followed:</p>
-<h4 id="dump-v06-metadata-to-local-file">2. Dump v0.6 metadata to local
file</h4>
-<p>This is also a backup method; As the migration tool is only tested with
local file system, this step is must; All metadata need be downloaded,
including snapshot, dictionary, etc;</p>
+<h4 id="stop-kylin-v20-instance">1. Stop Kylin v2.0 instance</h4>
-<div class="highlighter-rouge"><pre class="highlight"><code>hbase
org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar
org.apache.kylin.common.persistence.ResourceTool download ./meta_dump
+<div class="highlighter-rouge"><pre
class="highlight"><code>$KYLIN_HOME/bin/kylin.sh stop
</code></pre>
</div>
+<p>#### 2. Rstore 1.x metadata from backup folder</p>
-<p>(./meta_dump is the local folder that the metadata will be downloaded,
change to name you preferred)</p>
-
-<h4 id="run-cubemetadataupgrade-to-migrate-the-metadata">3. Run
CubeMetadataUpgrade to migrate the metadata</h4>
-<p>This step is to run the migration tool to parse the v0.6 metadata and then
convert to v0.7 format; A verification will be performed in the last, and
report error if some cube couldnât be migrated;</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>hbase
org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar
org.apache.kylin.job.CubeMetadataUpgrade ./meta_dump
+<div class="highlighter-rouge"><pre class="highlight"><code>export
KYLIN_HOME="<path_of_1.x_installation>"
+$KYLIN_HOME/bin/metastore.sh restore <backup_folder>
</code></pre>
</div>
+<p>#### 3. Deploy coprocessor of v1.x<br />
+Since coprocessor of used HTable are upgraded as v2.0, you need to manually
downgrade them with this command.</p>
-<ol>
- <li>The tool will not overwrite v0.6 metadata; It will create a new folder
with â_v2â suffix in the same folder, in this case the â./meta_dump_v2â
will be created;</li>
- <li>By default this tool will only migrate the job history in last 30 days;
If you want to keep elder job history, please tweak upgradeJobInstance() method
by your own;</li>
- <li>If you see <em>No error or warning messages; The migration is
success</em> , thatâs good; Otherwise please check the error/warning messages
carefully;</li>
- <li>For some problem you may need manually update the JSON file, to check
whether the problem is gone, you can run a verify against the new metadata:</li>
-</ol>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>hbase
org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar
org.apache.kylin.job.CubeMetadataUpgrade ./meta_dump2 verify
+<div class="highlighter-rouge"><pre
class="highlight"><code>$KYLIN_HOME/bin/kylin.sh
org.apache.kylin.job.tools.DeployCoprocessorCLI
$KYLIN_HOME/lib/kylin-coprocessor*.jar -all
</code></pre>
</div>
-<h4 id="upload-the-new-metadata-to-hbase">4. Upload the new metadata to
HBase</h4>
-<p>Now the new format of metadata will be upload to the HBase to replace the
old format; Stop Kylin, and then:</p>
+<h4 id="start-kylin-v1x-instance">4. Start Kylin v1.x instance</h4>
-<div class="highlighter-rouge"><pre class="highlight"><code>hbase
org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar
org.apache.kylin.common.persistence.ResourceTool reset
-hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar
org.apache.kylin.common.persistence.ResourceTool upload ./meta_dump_v2
+<div class="highlighter-rouge"><pre
class="highlight"><code>$KYLIN_HOME/bin/kylin.sh start
</code></pre>
</div>
-<h4 id="update-htables-to-use-new-coprocessor">5. Update HTables to use new
coprocessor</h4>
-<p>Kylin uses HBase coprocessor to do server side aggregation; When Kylin
instance upgrades to V0.7, the HTables that created in V0.6 should also be
updated to use the new coprocessor:</p>
+<h2 id="highlights">Highlights</h2>
+<p>Since old cubes built with v1.x cannot leverage new features of v2.0. But
if you must have them on your cubes, you can choose one of these solutions:<br
/>
+#### 1. Rebuilt cubes<br />
+If the cost of rebuilding is acceptable, if you purge the cube before Step
4(Running Upgrade Scripts). After upgrade done, you need to manually rebuilt
those segments by yourself.<br />
+#### 2. Use hybrid model<br />
+If you donât want to rebuild any cube, but want to leverage new features for
new data. You can use hybrid model, which contains not only your old cube, but
also an empty cube which has same model with the old one. For the empty cube,
you can do incremental building with v2 features. For the old cube, you can
refresh existing segments only.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>hbase
org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar
org.apache.kylin.job.tools.DeployCoprocessorCLI
${KYLIN_HOME}/lib/kylin-coprocessor-x.x.x.jar
+<p>Here is the command to create hybrid model:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>export
KYLIN_HOME="<path_of_v2.0_installation>"
+$KYLIN_HOME/bin/kylin.sh
org.apache.kylin.storage.hbase.util.ExtendCubeToHybridCLI <project_name>
<cube_name>
</code></pre>
</div>
-<p>Done; Update your v0.7 Kylin configure to point to the same metadata HBase
table, then start Kylin server; Check whether all cubes and other information
are kept;</p>
-
</article>
</div>
</div>
Modified: kylin/site/feed.xml
URL:
http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1733218&r1=1733217&r2=1733218&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Wed Mar 2 06:20:13 2016
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml"; rel="self"
type="application/rss+xml"/>
- <pubDate>Tue, 01 Mar 2016 08:37:33 +0000</pubDate>
- <lastBuildDate>Tue, 01 Mar 2016 08:37:33 +0000</lastBuildDate>
+ <pubDate>Tue, 01 Mar 2016 22:13:58 -0800</pubDate>
+ <lastBuildDate>Tue, 01 Mar 2016 22:13:58 -0800</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
@@ -139,7 +139,7 @@ By flexibly using the new aggregation gr
<p>For legacy users you will need to upgrade your metadata store from
Kylin 2.0 to Kylin 2.1. Cube rebuild is not required if youâre upgrading from
Kylin 2.0.</p>
</description>
- <pubDate>Thu, 18 Feb 2016 16:30:00 +0000</pubDate>
+ <pubDate>Thu, 18 Feb 2016 08:30:00 -0800</pubDate>
<link>http://kylin.apache.org/blog/2016/02/18/new-aggregation-group/</link>
<guid
isPermaLink="true">http://kylin.apache.org/blog/2016/02/18/new-aggregation-group/</guid>
@@ -171,7 +171,7 @@ By flexibly using the new aggregation gr
<p>Weâll publish more detailed documents on how to use Kylin Streaming
soon. In latest 2.x branch we are also working on more complicated load
balancing schemes for streaming cubing. Please stay tuned.</p>
</description>
- <pubDate>Wed, 03 Feb 2016 16:30:00 +0000</pubDate>
+ <pubDate>Wed, 03 Feb 2016 08:30:00 -0800</pubDate>
<link>http://kylin.apache.org/blog/2016/02/03/streaming-cubing/</link>
<guid
isPermaLink="true">http://kylin.apache.org/blog/2016/02/03/streaming-cubing/</guid>
@@ -203,7 +203,7 @@ With sub-seconds query latency feature o
<p>Enjoy!</p>
</description>
- <pubDate>Fri, 25 Dec 2015 23:23:00 +0000</pubDate>
+ <pubDate>Fri, 25 Dec 2015 15:23:00 -0800</pubDate>
<link>http://kylin.apache.org/blog/2015/12/25/support-powerbi-tableau9/</link>
<guid
isPermaLink="true">http://kylin.apache.org/blog/2015/12/25/support-powerbi-tableau9/</guid>
@@ -235,7 +235,7 @@ With sub-seconds query latency feature o
<p>Enjoy!</p>
</description>
- <pubDate>Fri, 25 Dec 2015 23:23:00 +0000</pubDate>
+ <pubDate>Fri, 25 Dec 2015 15:23:00 -0800</pubDate>
<link>http://kylin.apache.org/cn/blog/2015/12/25/support-powerbi-tableau9/</link>
<guid
isPermaLink="true">http://kylin.apache.org/cn/blog/2015/12/25/support-powerbi-tableau9/</guid>
@@ -296,7 +296,7 @@ send mail to Apache Kylin dev mailing li
<p><em>Great thanks to everyone who
contributed!</em></p>
</description>
- <pubDate>Wed, 23 Dec 2015 22:28:00 +0000</pubDate>
+ <pubDate>Wed, 23 Dec 2015 14:28:00 -0800</pubDate>
<link>http://kylin.apache.org/blog/2015/12/23/release-v1.2/</link>
<guid
isPermaLink="true">http://kylin.apache.org/blog/2015/12/23/release-v1.2/</guid>
@@ -357,7 +357,7 @@ send mail to Apache Kylin dev mailing li
<p><em>æè°¢æ¯ä¸ä½æåçåä¸åè´¡ç®!</em></p>
</description>
- <pubDate>Wed, 23 Dec 2015 22:28:00 +0000</pubDate>
+ <pubDate>Wed, 23 Dec 2015 14:28:00 -0800</pubDate>
<link>http://kylin.apache.org/cn/blog/2015/12/23/release-v1.2/</link>
<guid
isPermaLink="true">http://kylin.apache.org/cn/blog/2015/12/23/release-v1.2/</guid>
@@ -417,7 +417,7 @@ send mail to Apache Kylin dev mailing li
<p><em>Great thanks to everyone who
contributed!</em></p>
</description>
- <pubDate>Sun, 25 Oct 2015 17:28:00 +0000</pubDate>
+ <pubDate>Sun, 25 Oct 2015 10:28:00 -0700</pubDate>
<link>http://kylin.apache.org/blog/2015/10/25/release-v1.1-incubating/</link>
<guid
isPermaLink="true">http://kylin.apache.org/blog/2015/10/25/release-v1.1-incubating/</guid>
@@ -477,7 +477,7 @@ send mail to Apache Kylin dev mailing li
<p><em>æè°¢åä½çè´¡ç®!</em></p>
</description>
- <pubDate>Sun, 25 Oct 2015 17:28:00 +0000</pubDate>
+ <pubDate>Sun, 25 Oct 2015 10:28:00 -0700</pubDate>
<link>http://kylin.apache.org/cn/blog/2015/10/25/release-v1.1-incubating/</link>
<guid
isPermaLink="true">http://kylin.apache.org/cn/blog/2015/10/25/release-v1.1-incubating/</guid>
@@ -533,7 +533,7 @@ send mail to Apache Kylin dev mailing li
<p class="center"><img
src="/images/blog/meetup_9.jpeg" alt="" /></p>
</description>
- <pubDate>Wed, 14 Oct 2015 17:00:00 +0000</pubDate>
+ <pubDate>Wed, 14 Oct 2015 10:00:00 -0700</pubDate>
<link>http://kylin.apache.org/blog/2015/10/14/Apache-Kylin-Meetup/</link>
<guid
isPermaLink="true">http://kylin.apache.org/blog/2015/10/14/Apache-Kylin-Meetup/</guid>
@@ -674,7 +674,7 @@ No; the purpose of hybrid is to consolid
<p><strong>Question 7</strong>: If a child cube is disabled,
will it be scanned via the hybrid?<br />
No; hybrid instance will check the child realizationâs status before sending
query to it; so if the cube is disabled, it will not be scanned.</p>
</description>
- <pubDate>Fri, 25 Sep 2015 16:00:00 +0000</pubDate>
+ <pubDate>Fri, 25 Sep 2015 09:00:00 -0700</pubDate>
<link>http://kylin.apache.org/blog/2015/09/25/hybrid-model/</link>
<guid
isPermaLink="true">http://kylin.apache.org/blog/2015/09/25/hybrid-model/</guid>
