svn commit: r1890886 [8/8] - in /kylin/site: ./ blog/ blog/2021/ blog/2021/06/ blog/2021/06/17/ blog/2021/06/17/Why-did-Youzan-choose-Kylin4/ cn/blog/ cn/docs40/ cn/docs40/gettingstarted/ cn/docs40/howto/ cn/docs40/install/ cn/docs40/tutorial/ cn_blog/...

[lidong]

Modified: kylin/site/feed.xml
URL: 
http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1890886&r1=1890885&r2=1890886&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Fri Jun 18 02:57:25 2021
@@ -19,11 +19,319 @@
     <description>Apache Kylin Home</description>
     <link>http://kylin.apache.org/</link>
     <atom:link href="http://kylin.apache.org/feed.xml"; rel="self" 
type="application/rss+xml"/>
-    <pubDate>Wed, 02 Jun 2021 20:18:35 -0700</pubDate>
-    <lastBuildDate>Wed, 02 Jun 2021 20:18:35 -0700</lastBuildDate>
+    <pubDate>Thu, 17 Jun 2021 19:32:15 -0700</pubDate>
+    <lastBuildDate>Thu, 17 Jun 2021 19:32:15 -0700</lastBuildDate>
     <generator>Jekyll v2.5.3</generator>
     
       <item>
+        <title>Why did Youzan choose Kylin4</title>
+        <description>&lt;p&gt;At the QCon Global Software Developers 
Conference held on May 29, 2021, Zheng Shengjun, head of Youzan’s data 
infrastructure platform, shared Youzan’s internal use experience and 
optimization practice of Kylin 4.0 on the meeting room of open source big data 
frameworks and applications. &lt;br /&gt;
+For many users of Kylin2/3(Kylin on HBase), this is also a chance to learn how 
and why to upgrade to Kylin 4.&lt;/p&gt;
+
+&lt;p&gt;This sharing is mainly divided into the following parts:&lt;/p&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;The reason for choosing Kylin 4&lt;/li&gt;
+  &lt;li&gt;Introduction to Kylin 4&lt;/li&gt;
+  &lt;li&gt;How to optimize performance of Kylin 4&lt;/li&gt;
+  &lt;li&gt;Practice of Kylin 4 in Youzan&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;h2 id=&quot;the-reason-for-choosing-kylin-4&quot;&gt;01 The reason for 
choosing Kylin 4&lt;/h2&gt;
+
+&lt;h3 id=&quot;introduction-to-youzan&quot;&gt;Introduction to 
Youzan&lt;/h3&gt;
+&lt;p&gt;China Youzan Co., Ltd (stock code 08083.HK). is an enterprise mainly 
engaged in retail technology services.&lt;br /&gt;
+At present, it owns several tools and solutions to provide SaaS software 
products and talent services to help merchants operate mobile social e-commerce 
and new retail channels in an all-round way. &lt;br /&gt;
+Currently Youzan has hundreds of millions of consumers and 6 million existing 
merchants.&lt;/p&gt;
+
+&lt;h3 id=&quot;history-of-kylin-in-youzan&quot;&gt;History of Kylin in 
Youzan&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan/1 
history_of_youzan_OLAP.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;First of all, I would like to share why Youzan chose to upgrade to 
Kylin 4. Here, let me briefly reviewed the history of Youzan OLAP 
infra.&lt;/p&gt;
+
+&lt;p&gt;In the early days of Youzan, in order to iterate develop process 
quickly, we chose the method of pre-computation + MySQL; in 2018, Druid was 
introduced because of query flexibility and development efficiency, but there 
were problems such as low pre-aggregation, not supporting precisely count 
distinct measure. In this situation, Youzan introduced Apache Kylin and 
ClickHouse. Kylin supports high aggregation, precisely count distinct measure 
and the lowest RT, while ClickHouse is quite flexible in usage(ad hoc 
query).&lt;/p&gt;
+
+&lt;p&gt;From the introduction of Kylin in 2018 to now, Youzan has used Kylin 
for more than three years. With the continuous enrichment of business scenarios 
and the continuous accumulation of data volume, Youzan currently has 6 million 
existing merchants, GMV in 2020 is 107.3 billion, and the daily build data 
volume is 10 billion +. At present, Kylin has basically covered all the 
business scenarios of Youzan.&lt;/p&gt;
+
+&lt;h3 id=&quot;the-challenges-of-kylin-3&quot;&gt;The challenges of Kylin 
3&lt;/h3&gt;
+&lt;p&gt;With Youzan’s rapid development and in-depth use of Kylin, we also 
encountered some challenges:&lt;/p&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;First of all, the build performance of Kylin on HBase cannot meet 
the favorable expectations, and the build performance will affect the user’s 
failure recovery time and stability experience;&lt;/li&gt;
+  &lt;li&gt;Secondly, with the access of more large merchants (tens of 
millions of members in a single store, with hundreds of thousands of goods for 
each store), it also brings great challenges to our OLAP system. Kylin on HBase 
is limited by the single-point query of Query Server, and cannot support these 
complex scenarios well;&lt;/li&gt;
+  &lt;li&gt;Finally, because HBase is not a cloud-native system, it is 
difficult to achieve flexible scale up and scale down. With the continuous 
growth of data volume, this system has peaks and valleys for businesses, which 
results in the average resource utilization rate is not high enough.&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;p&gt;Faced with these challenges, Youzan chose to move closer and upgrade 
to the more cloud-native Apache Kylin 4.&lt;/p&gt;
+
+&lt;h2 id=&quot;introduction-to-kylin-4&quot;&gt;02 Introduction to Kylin 
4&lt;/h2&gt;
+&lt;p&gt;First of all, let’s introduce the main advantages of Kylin 4. 
Apache Kylin 4 completely depends on Spark for cubing job and query. It can 
make full use of Spark’s parallelization, quantization(向量化), and global 
dynamic code generation technologies to improve the efficiency of large 
queries.&lt;br /&gt;
+Here is a brief introduction to the principle of Kylin 4, that is storage 
engine, build engine and query engine.&lt;/p&gt;
+
+&lt;h3 id=&quot;storage-engine&quot;&gt;Storage engine&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan/2 kylin4_storage.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;First of all, let’s take a look at the new storage engine, 
comparison between Kylin on HBase and Kylin on Parquet. The cuboid data of 
Kylin on HBase is stored in the table of HBase. Single Segment corresponds to 
one HBase table. Aggregation is pushed down to HBase coprocessor.&lt;/p&gt;
+
+&lt;p&gt;But as we know,  HBase is not a real Columnar Storage and its 
throughput is not enough for OLAP System. Kylin 4 replaces HBase with Parquet, 
all the data is stored in files. Each segment will have a corresponding HDFS 
directory. All queries and cubing jobs read and write files without HBase . 
Although there will be a certain loss of performance for simple queries, the 
improvement brought about by complex queries is more considerable and 
worthwhile.&lt;/p&gt;
+
+&lt;h3 id=&quot;build-engine&quot;&gt;Build engine&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan/3 kylin4_build_engine.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;The second is the new build engine. Based on our test, the build 
speed of Kylin on Parquet has been optimized from 82 minutes to 15 minutes. 
There are several reasons:&lt;/p&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;Kylin 4 removes the encoding of the dimension, eliminating a 
building step of encoding;&lt;/li&gt;
+  &lt;li&gt;Removed the HBase File generation step;&lt;/li&gt;
+  &lt;li&gt;Kylin on Parquet changes the granularity of cubing to cuboid 
level, which is conducive to further improving parallelism of cubing 
job.&lt;/li&gt;
+  &lt;li&gt;Enhanced implementation for global dictionary. In the new 
algorithm, dictionary and source data are hashed into the same buckets, making 
it possible for loading only piece of dictionary bucket to encode source 
data.&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;p&gt;As you can see on the right, after upgradation to Kylin 4, cubing job 
changes from ten steps to two steps, the performance improvement of the 
construction is very obvious.&lt;/p&gt;
+
+&lt;h3 id=&quot;query-engine&quot;&gt;Query engine&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan/4 kylin4_query.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;Next is the new query engine of Kylin 4. As you can see, the 
calculation of Kylin on HBase is completely dependent on the coprocessor of 
HBase and query server process. When the data is read from HBase into query 
server to do aggregation, sorting, etc, the bottleneck will be restricted by 
the single point of query server. But Kylin 4 is converted to a fully 
distributed query mechanism based on Spark, what’s more, it ‘s able to do 
configuration tuning automatically in spark query step !&lt;/p&gt;
+
+&lt;h2 id=&quot;how-to-optimize-performance-of-kylin-4&quot;&gt;03 How to 
optimize performance of Kylin 4&lt;/h2&gt;
+&lt;p&gt;Next, I’d like to share some performance optimizations made by 
Youzan in Kylin 4.&lt;/p&gt;
+
+&lt;h3 id=&quot;optimization-of-query-engine&quot;&gt;Optimization of query 
engine&lt;/h3&gt;
+&lt;p&gt;#### 1.Cache Calcite physical plan&lt;br /&gt;
+&lt;img src=&quot;/images/blog/youzan/5 cache_calcite_plan.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;In Kylin4, SQL will be analyzed, optimized and do code generation in 
calcite. This step takes up about 150ms for some queries. We have supported 
PreparedStatementCache in Kylin4 to cache calcite plan, so that the structured 
SQL don’t have to do the same step again. With this optimization it saved 
about 150ms of time cost.&lt;/p&gt;
+
+&lt;h4 id=&quot;tunning-spark-configuration&quot;&gt;2.Tunning spark 
configuration&lt;/h4&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan/6 
tuning_spark_configuration.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;Kylin4 uses spark as query engine. As spark is a distributed engine 
designed for massive data processing, it’s inevitable to loose some 
performance for small queries. We have tried to do some tuning to catch up with 
the latency in Kylin on HBase for small queries.&lt;/p&gt;
+
+&lt;p&gt;Our first optimization is to make more calculations finish in memory. 
The key is to avoid data spill during aggregation, shuffle and sort. Tuning the 
following configuration is helpful.&lt;/p&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;1.set &lt;code 
class=&quot;highlighter-rouge&quot;&gt;spark.sql.objectHashAggregate.sortBased.fallbackThreshold&lt;/code&gt;
 to larger value to avoid HashAggregate fall back to Sort Based Aggregate, 
which really kills performance when happens.&lt;/li&gt;
+  &lt;li&gt;2.set &lt;code 
class=&quot;highlighter-rouge&quot;&gt;spark.shuffle.spill.initialMemoryThreshold&lt;/code&gt;
 to a large value to avoid to many spills during shuffle.&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;p&gt;Secondly, we route small queries to Query Server which run spark in 
local mode. Because the overhead of task schedule, shuffle read and variable 
broadcast is enlarged for small queries on YARN/Standalone mode.&lt;/p&gt;
+
+&lt;p&gt;Thirdly, we use RAM disk to enhance shuffle performance. Mount RAM 
disk as TMPFS and set spark.local.dir to directory using RAM disk.&lt;/p&gt;
+
+&lt;p&gt;Lastly, we disabled spark’s whole stage code generation for small 
queries, for spark’s whole stage code generation will cost about 100ms~200ms, 
whereas it’s not beneficial to small queries which is a simple 
project.&lt;/p&gt;
+
+&lt;h4 id=&quot;parquet-optimization&quot;&gt;3.Parquet optimization&lt;/h4&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan/7 
parquet_optimization.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;Optimizing parquet is also important for queries.&lt;/p&gt;
+
+&lt;p&gt;The first principal is that we’d better always include shard by 
column in our filter condition, for parquet files are shard by shard-by-column, 
filter using shard by column reduces the data files to read.&lt;/p&gt;
+
+&lt;p&gt;Then look into parquet files, data within files are sorted by rowkey 
columns, that is to say, prefix match in query is as important as Kylin on 
HBase. When a query condition satisfies prefix match, it can filter row groups 
with column’s max/min index. Furthermore, we can reduce row group size to 
make finer index granularity, but be aware that the compression rate will be 
lower if we set row group size smaller.&lt;/p&gt;
+
+&lt;h4 
id=&quot;dynamic-elimination-of-partitioning-dimensions&quot;&gt;4.Dynamic 
elimination of partitioning dimensions&lt;/h4&gt;
+&lt;p&gt;Kylin4 have a new ability that the older version is not capable of, 
which is able to reduce dozens of times of data reading and computing for some 
big queries. It’s offen the case that partition column is used to filter data 
but not used as group dimension. For those cases Kylin would always choose 
cuboid with partition column, but now it is able to use different cuboid in 
that query to reduce IO read and computing.&lt;/p&gt;
+
+&lt;p&gt;The key of this optimization is to split a query into two parts, one 
of the part uses all segment’s data so that partition column doesn’t have 
to be included in cuboid, the other part that uses part of segments data will 
choose cuboid with partition dimension to do the data filter.&lt;/p&gt;
+
+&lt;p&gt;We have tested that in some situations the response time reduced from 
20s to 6s, 10s to 3s.&lt;/p&gt;
+
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan/8 
Dynamic_elimination_of_partitioning_dimensions.png&quot; alt=&quot;&quot; 
/&gt;&lt;/p&gt;
+
+&lt;h3 id=&quot;optimization-of-build-engine&quot;&gt;Optimization of build 
engine&lt;/h3&gt;
+&lt;p&gt;#### 1.cache parent dataset&lt;br /&gt;
+&lt;img src=&quot;/images/blog/youzan/9 cache_parent_dataset.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;Kylin build cube layer by layer. For a parent layer with multi 
cuboids to build, we can choose to cache parent dataset by setting 
kylin.engine.spark.parent-dataset.max.persist.count to a number greater than 0. 
But notice that if you set this value too small, it will affect the parallelism 
of build job, as the build granularity is at cuboid level.&lt;/p&gt;
+
+&lt;h2 id=&quot;practice-of-kylin-4-in-youzan&quot;&gt;04 Practice of Kylin 4 
in Youzan&lt;/h2&gt;
+&lt;p&gt;After introducing Youzan’s experience of performance optimization, 
let’s share the optimization effect. That is, Kylin 4’s practice in Youzan 
includes the upgrade process and the performance of online system.&lt;/p&gt;
+
+&lt;h3 id=&quot;upgrade-metadata-to-adapt-to-kylin-4&quot;&gt;Upgrade metadata 
to adapt to Kylin 4&lt;/h3&gt;
+&lt;p&gt;First of all, for metadata for Kylin 3 which stored on HBase, we have 
developed a tool for seamless upgrading of metadata. First of all, our metadata 
in Kylin on HBase is stored in HBase. We export the metadata in HBase into 
local files, and then use tools to transform and write back the new metadata 
into MySQL. We also updated the operation documents and general principles in 
the official wiki of Apache Kylin. For more details, you can refer to: &lt;a 
href=&quot;https://wiki.apache.org/confluence/display/KYLIN/How+to+migrate+metadata+to+Kylin+4&quot;&gt;How
 to migrate metadata to Kylin 4&lt;/a&gt;.&lt;/p&gt;
+
+&lt;p&gt;Let’s give a general introduction to some compatibility in the 
whole process. The project metadata, tables metadata, permission-related 
metadata, and model metadata do not need be modified. What needs to be modified 
is the cube metadata, including the type of storage and query used by Cube. 
After updating these two fields, you need to recalculate the Cube signature. 
The function of this signature is designed internally by Kylin to avoid some 
problems caused by Cube after Cube is determined.&lt;/p&gt;
+
+&lt;h3 
id=&quot;performance-of-kylin-4-on-youzan-online-system&quot;&gt;Performance of 
Kylin 4 on Youzan online system&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan/10 commodity_insight.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;After the migration of metadata to Kylin4, let’s share the 
qualitative changes and substantial performance improvements brought about by 
some of the promising scenarios. First of all, in a scenario like Commodity 
Insight, there is a large store with several hundred thousand of commodities. 
We have to analyze its transactions and traffic, etc. There are more than a 
dozen precise precisely count distinct measures in single cube. Precisely count 
distinct measure is actually very inefficient if it is not optimized through 
pre-calculation and Bitmap. Kylin currently uses Bitmap to support precisely 
count distinct measure. In a scene that requires complex queries to sort 
hundreds of thousands of commodities in various UV(precisely count distinct 
measure), the RT of Kylin 2 is 27 seconds, while the RT of Kylin 4 is reduced 
from 27 seconds to less than 2 seconds.&lt;/p&gt;
+
+&lt;p&gt;What I find most appealing to me about Kylin 4 is that it’s like a 
manual transmission car, you can control its query concurrency at your will, 
whereas you can’t change query concurrency in Kylin on HBase freely, because 
its concurrency is completely tied to the number of regions.&lt;/p&gt;
+
+&lt;h3 id=&quot;plan-for-kylin-4-in-youzan&quot;&gt;Plan for Kylin 4 in 
Youzan&lt;/h3&gt;
+&lt;p&gt;We have made full test, fixed several bugs and improved apache KYLIN4 
for several months. Now we are migrating cubes from older version to newer 
version. For the cubes already migrated to KYLIN4, its small queries’ 
performance meet our expectations, its complex query and build performance did 
bring us a big surprise. We are planning to migrate all cubes from older 
version to Kylin4.&lt;/p&gt;
+</description>
+        <pubDate>Thu, 17 Jun 2021 08:00:00 -0700</pubDate>
+        
<link>http://kylin.apache.org/blog/2021/06/17/Why-did-Youzan-choose-Kylin4/</link>
+        <guid 
isPermaLink="true">http://kylin.apache.org/blog/2021/06/17/Why-did-Youzan-choose-Kylin4/</guid>
+        
+        
+        <category>blog</category>
+        
+      </item>
+    
+      <item>
+        <title>有赞为什么选择 Kylin4</title>
+        <description>&lt;p&gt;在 2021å¹´5月29日举办的 QCon å…
¨çƒè½¯ä»¶å¼€å‘者大会上，来自有赞的数据基础平台负责人 
郑生俊 在大数据开源框架与应用专题上分享了有赞内部对 
Kylin 4.0 的使用经历和优化实践，对于众多 Kylin 
老用户来说，这也是升级 Kylin 4 的实用攻略。&lt;/p&gt;
+
+&lt;p&gt;本次分享主要分为以下四个部分：&lt;/p&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;有赞选用 Kylin 4 的原因&lt;/li&gt;
+  &lt;li&gt;Kylin 4 原理介绍&lt;/li&gt;
+  &lt;li&gt;Kylin 4 性能优化&lt;/li&gt;
+  &lt;li&gt;Kylin 4 在有赞的实践&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;h2 id=&quot;kylin-4-&quot;&gt;01 有赞选用 Kylin 4 的原因&lt;/h2&gt;
+&lt;p&gt;首先分享有赞为什么会选择升级为 Kylin 4，这里å…
ˆç®€å•å›žé¡¾ä¸€ä¸‹æœ‰èµž OLAP 
的发展历程：有赞初期为了快速迭代，选择了预计算 + MySQL 
的方式；2018年，因为查询灵活和开发效率引入了 
Druid，但是存在预聚合度不高、不支持精确去重和明细 OLAP 
等问题；在这样的背景下，有赞引å…
¥äº†æ»¡è¶³èšåˆåº¦é«˜ã€æ”¯æŒç²¾ç¡®åŽ»é‡å’Œ RT 最低的 Apache Kylin 
和查询非常灵活的 ROLAP ClickHouse。&lt;/p&gt;
+
+&lt;p&gt;从2018年引入 Kylin 到现在，有赞已经使用 Kylin 
三年多了。随着业务场景的不断丰富和数据量的不断积累，有赞目前有
 600 万的存量商家，2020年 GMV 是 1073亿，日构建量为 100 
亿+，目前 Kylin 
已经基本覆盖了有赞所有的业务范围。&lt;/p&gt;
+
+&lt;p&gt;随着有赞自身的迅速发展和不断深入地使用 
Kylin，我们也遇到一些挑战：&lt;br /&gt;
+- 首先 Kylin on HBase 的构建性能无
法满足有赞的预期，构建性能会影响到用户的故
障恢复时间和稳定性的体验；&lt;br /&gt;
+- å…
¶æ¬¡ï¼Œéšç€æ›´å¤šå¤§å•†å®¶ï¼ˆå•åº—千万级别会员、数十万商品）的接å
…¥ï¼Œå¯¹æˆ‘们的查询也带来了很大的挑战。Kylin on HBase 受限于 
QueryServer 单点查询的局限，无
法很好地支持这些复杂的场景；&lt;br /&gt;
+- 最后，因为 HBase 
不是一个云原生系统，很难做到弹性的资源伸缩，随着数据量的不断增长，这个系统对于商家而言，使用时间是存在高峰和低谷的，这就é€
 æˆå¹³å‡çš„资源使用率不够高。&lt;/p&gt;
+
+&lt;p&gt;面对这些挑战，有赞选择去向更云原生的 Apache Kylin 4 
去靠拢和升级。&lt;/p&gt;
+
+&lt;h2 id=&quot;kylin-4--1&quot;&gt;02 Kylin 4 原理介绍&lt;/h2&gt;
+&lt;p&gt;首先介绍一下 Kylin 4 的主要优势。Apache Kylin 4 是完å…
¨åŸºäºŽ Spark 去做构建和查询的，能够充分地利用 
Spark的并行化、向量化和全局动态代ç 
ç”Ÿæˆç­‰æŠ€æœ¯ï¼ŒåŽ»æé«˜å¤§æŸ¥è¯¢çš„效率。&lt;br /&gt;
+这里从存储、构建和查询三个部分简单介绍一下 Kylin 4 
的原理。&lt;/p&gt;
+
+&lt;h3 id=&quot;section&quot;&gt;存储&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/1 kylin4_storage.png&quot; 
alt=&quot;&quot; /&gt;&lt;br /&gt;
+首先来看一下，Kylin on HBase 和 Kylin on Parquet 的对比。Kylin on 
HBase 的 Cuboid 的数据是存放在 HBase 的表里，一个 Segment 
对应了一张 HBase 表，查询下推的工作由 HBase 协理器处理，因
为 HBase 不是真正的列存并且对 OLAP 而言吞吐量不高。Kylin 4 
将 HBase 替换为 Parquet，也就是把所有的数据按ç…
§æ–‡ä»¶å­˜å‚¨ï¼Œæ¯ä¸ª Segment 会存在一个对应的 HDFS 
的目录，所有的查询、构建都是直接通过读写文件的方式，不用再经过
 HBase。虽然对于小查询的性能会有ä
 
¸€å®šæŸå¤±ï¼Œä½†å¯¹äºŽå¤æ‚查询带来的提升是更可观的、更值得的。&lt;/p&gt;
+
+&lt;h3 id=&quot;section-1&quot;&gt;构建引擎&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/2 
kylin4_build_engine.png&quot; alt=&quot;&quot; /&gt;&lt;br /&gt;
+其次是 Kylin 构建引擎，基于有赞的测试，Kylin on Parquet 
的构建速度已经从 82 分钟优化到了 15 分钟，有以下几个原因
：&lt;/p&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;Kylin 4 去掉了维度字典的编码，省去了编ç 
çš„一个构建步骤；&lt;/li&gt;
+  &lt;li&gt;去掉了 HBase File 的生成步骤；&lt;/li&gt;
+  &lt;li&gt;新版本的 Kylin 4 所有的构建步骤都转换为 Spark 
进行构建；&lt;/li&gt;
+  &lt;li&gt;Kylin on Parquet 基于 Cuboid 
去划分构建粒度，有利于进一步地提升并行度。&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;p&gt;可以看到右侧，从十个步骤简化到了两个步骤，构建性能提升的非常明显的。&lt;/p&gt;
+
+&lt;h3 id=&quot;section-2&quot;&gt;查询引擎&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/3 kylin4_query.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;接下来就是 Kylin 4 的查询，大家可以看到，左边这列 
Kylin on HBase 的计算是完全依托于 Calcite 和 HBase 
的协处理器，这就导致当数据从 HBase 
读取后，如果想做聚合、排序等，就会局限于 QueryServer 
单点的瓶颈，而 Kylin 4 则转换为基于 Spark DataFrame 的å…
¨åˆ†å¸ƒå¼çš„查询机制。&lt;/p&gt;
+
+&lt;h2 id=&quot;kylin-4--2&quot;&gt;03 Kylin 4 性能优化&lt;/h2&gt;
+&lt;p&gt;接下来分享有赞在 Kylin 4 
所做的一些性能优化。&lt;/p&gt;
+
+&lt;h3 id=&quot;section-3&quot;&gt;查询性能优化&lt;/h3&gt;
+&lt;p&gt;#### 1.动态消除维度分区&lt;br /&gt;
+&lt;img src=&quot;/images/blog/youzan_cn/4 
dynamic_elimination_dimension_partition.png&quot; alt=&quot;&quot; 
/&gt;&lt;/p&gt;
+
+&lt;p&gt;首å…
ˆæˆ‘们来看一个场景，我们做到了动态消除分区维度，混合使用
 cuboid 来对复杂查询，减少数十倍的计算量。&lt;/p&gt;
+
+&lt;p&gt;这里举一个例子，在一个 Cube 有三个 Segment 的情
况下，Cube 分区字段记作 P，它有三个 Segment 
分别是1月1日到2月1日、2月1日到3月1日，3月1日到3月7日。假设有一个SQL，Select
 count(a) from test where p &amp;gt;= 20200101 and p &amp;lt;= 20200313 group 
by a。&lt;/p&gt;
+
+&lt;p&gt;在这种情况下，因为需要分区过滤，Kylin 它会选择 a 
和 p 预计算维度的组合，转换成执行计划就是最上层的 
Aggregate 然后 Filter，最后会转换成一个 TableScan，这个 TableScan 
就是选择聚合维度为 a 和 p 这样的一个维度组合。实际
上这个查询计划是适合把它优化成右边这种方式的，对于某个
 Segment 完全使用到的数据，我们可以选择一个 Cuboid 为 a 的 
Cuboid 去做查询。对于部分用到的分区或者 
Segment，我们可以选择 a å’
 Œ p 这样的一个维度组合。通过这种方式，在 a 
只有一个可能值的情况下，之前可能要 scan 65 
条数据，优化后只要 scan 8 
条数据。假设时间跨度更长，比如说跨几
个月、半年甚至一年，就会减少数十倍、几十倍的计算量和 
IO。&lt;/p&gt;
+
+&lt;p&gt;在有赞某些场景，RT 可以从 10 秒优化到 3 秒、20s 
提升到 6s，对于更复杂的场景（比如计算密集型的 
HLL），会有更显著的优化效果。这部分优化，有赞也正打算贡献回社区。å›
 ä¸ºæ¶‰åŠåˆ°å¦‚何在多层嵌套和复杂的条件下进行 segment 
分组，以及目前 calcite 和 spark catalyst 
并存，实现上会比较复杂。到时候大家在 Kylin 4.0-GA 
版本可能就可以看到这个优化了。&lt;/p&gt;
+
+&lt;h4 
id=&quot;section-4&quot;&gt;2.复杂过滤条件下的分区裁剪&lt;/h4&gt;
+&lt;p&gt;接下来再介绍一下有赞所做的查询性能优化，就是支持复杂过滤条件下的分区裁剪。目前
 Kylin 4.0 Beta 
版本对于复杂的过滤条件比如多个过滤字段、多层嵌套的 
Filter 等，不支持分区裁剪，导致å…
¨è¡¨æ‰«æã€‚我们做了一个优化，是将复杂的嵌套 Filter 
过滤的语法树转换成基于分区字段 p 
的一个等价表达式，然后再将这个表达式应用到每一个 
Segment 去做过滤，通过这样的方式，去支持它做到一个é
 žå¸¸å¤æ‚的分区过滤裁剪。&lt;/p&gt;
+
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/5 Partition clipping under 
complex filter.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;h4 id=&quot;spark-&quot;&gt;3.Spark 参数调优&lt;/h4&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/6 
tuning_spark_configuration.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;接下来是比较重要的一部分，就是关于 Spark 
的调参。Spark 是一个分布式计算框架，相比 Calcite 
而言，对于小查询是存在一定劣势的。&lt;/p&gt;
+
+&lt;p&gt;首先我们做了一个调整，尽量让 Spark 
所有的计算操作是在内存中完成的。以下两种情况会产生 
spill：&lt;br /&gt;
+- 01 在聚合时，在我们内存不够的时候，Spark 会将 
HashAggregate 转换为 Sort Based Aggregate，实际
上这一步是很耗性能的。我们通过调大阈值的参数，尽量让所有的聚合都在å†
…存中完成。&lt;br /&gt;
+- 02 在 shuffle 的过程中，Spark是不可避免地会进行 
Spill，会落盘，我们能做的尽量在 Shuffle 过程减少 
Spill，只在最后 Shuffle 结束之后进行 Spill。&lt;/p&gt;
+
+&lt;p&gt;第二个我们做的调优是，相比 on YARN/Standalone 
模式下，local 模式大部分都是在进程内
通信的，也不需要产生跨网络的 Shuffle， broadcast 
广播变量也不需要跨网络，所以对于小查询，我们会路由到以
 Local 模式运行的 Spark 
Application，这对于小查询非常有意义。&lt;/p&gt;
+
+&lt;p&gt;第三个优化是 shuffle 使用内存盘。因为内
存盘肯定是最快的，我们将内存盘挂载为 tmpfs 
文件系统，然后将 spark.local.dir 指定为挂载的内存盘去优化 
shuffle 的速度和吞吐。&lt;/p&gt;
+
+&lt;p&gt;第四个优化是我们关闭 Spark 全局动态代码生成。Spark 
的全局动态代码生成是要在运行的时间内去动态拼接代ç 
ï¼Œå†åŽ»åŠ¨æ€ç¼–译代码，这个过程实际
上是很耗时的。对于离线的大数据量下是很有优化意义，但是对于比较小的一些数据场景，我们å
…³æŽ‰è¿™ä¸ªåŠ¨æ€ä»£ç ç”Ÿæˆä¹‹åŽï¼Œèƒ½å¤ŸèŠ‚省大概 100 到 200 
毫秒的耗时。&lt;/p&gt;
+
+&lt;p&gt;目前经过上述一系列的优化，我们能让小查询的 RT 
稳定在大概 300 毫秒左右，尽管 HBase 可能是几十毫秒左右的 
RT，但我们认为目前已经比较接近了，这种为提升大查询提升的
 Tradeoff 我们认为是一个很值得的事情。&lt;/p&gt;
+
+&lt;h4 id=&quot;section-5&quot;&gt;4.小查询优化&lt;/h4&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/8 
small_query_optimization.png&quot; alt=&quot;&quot; /&gt;&lt;br /&gt;
+然后，我来分享一下小查询的优化。Kylin on HBase 依托于 HBase 
能够做到几十毫秒的 RT，因为 HBase 有 bucket cache 缓存。而 
Kylin on Parquet 就完å…
¨åŸºäºŽæ–‡ä»¶çš„读取和计算，缓存依赖于文件系统的 page 
cache，那么它小查询的 RT 会比 HBase 
更高一些，我们能做的就是尽量缩小 Kylin on Parquet 和 Kylin on 
HBase 的 RT 差距。&lt;/p&gt;
+
+&lt;p&gt;经过我们的分析，SQL 会通过 Calcite 解析成 Calcite 
语法树，然后将这个语法树转化为 Spark 
DataFrame，最终再将整个查询交给 Spark 
去执行。在这一步的过程中，SQL 转化成 Calcite 
的过程中，是需要经过语法解析、优化等，这一步大概会消耗
 150 毫秒左右。有赞做的是尽量使用结构化的 SQL，就是 
PreparedStatement，我们在 Kylin 中支持 
PreparedStatementCache，对于固定的 SQL æ 
¼å¼ï¼Œå°†å®ƒçš„执行计划进行缓存，去重用这æ 
·çš„执行计划，降ä½
 Žè¯¥æ­¥éª¤çš„时间消耗，通过这样的优化，可以降低大概 100 
毫秒左右的耗时。&lt;/p&gt;
+
+&lt;h4 id=&quot;parquet-&quot;&gt;5.Parquet 优化&lt;/h4&gt;
+
+&lt;p&gt;关于查询性能的优化，有赞还充分利用了 Parquet 
索引，优化建议包括：&lt;/p&gt;
+
+&lt;ul&gt;
+  &lt;li&gt;
+    &lt;p&gt;Parquet 文件首先根据 Shard By Column 
进行分组，过滤条件尽量包含 Shard By Column；&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;Parquet 中的数据依然按照维度排序，结合 Column 
MetaData 中的 Max、Min 
索引，在命中前缀索引时能够过滤掉大量数据；&lt;/p&gt;
+  &lt;/li&gt;
+  &lt;li&gt;
+    &lt;p&gt;调小 RowGroup Size 增大索引粒度等。&lt;/p&gt;
+  &lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;h3 id=&quot;section-6&quot;&gt;构建性能优化&lt;/h3&gt;
+&lt;p&gt;#### 1.对 parent dataset 做缓存&lt;br /&gt;
+&lt;img src=&quot;/images/blog/youzan_cn/9 cache_parent_dataset.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;h4 
id=&quot;section-7&quot;&gt;2.处理空值导致的数据倾斜&lt;/h4&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/10 Processing data 
skew.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;更多关于构建优化的细节内容大家可以参考 &lt;a 
href=&quot;https://mp.weixin.qq.com/s/T_mK7pTAgk2PXnSJ0lbZ_w&quot;&gt;Kylin 4 
最新功能预览 + 优化实践抢先看&lt;/a&gt;&lt;/p&gt;
+
+&lt;h2 id=&quot;kylin-4--3&quot;&gt;04 Kylin 4 在有赞的实践&lt;/h2&gt;
+&lt;p&gt;介绍有赞的优化之后，我们再来分享一下优化的效果，也就是
 Kylin 4 在有赞的实践包括升级过程以及上线的效果。&lt;/p&gt;
+
+&lt;h3 id=&quot;section-8&quot;&gt;元数据升级&lt;/h3&gt;
+&lt;p&gt;首先是如何升级，我们开发了一个元数据无
缝升级的工具，首先我们在 Kylin on HBase 的元数据是保存在 
HBase 里的，我们将 HBase 里的元数据以文件的æ 
¼å¼å¯¼å‡ºï¼Œå†å°†æ–‡ä»¶æ ¼å¼çš„元数据写入到 MySQL，我们也在 
Apache Kylin 的官方 wiki 
更新了操作文档以及大致的原理，更多详情
大家可以参考：&lt;a 
href=&quot;https://wiki.apache.org/confluence/display/KYLIN/How+to+migrate+metadata+to+Kylin+4&quot;&gt;如何升级å
…ƒæ•°æ®åˆ°kylin4&lt;/a&gt;.&lt;br /&gt;
+&lt;img src=&quot;/images/blog/youzan_cn/11 metadata_upgrade.png&quot; 
alt=&quot;&quot; /&gt;&lt;br /&gt;
+我们大致介绍一下整个过程中的一些å…
¼å®¹æ€§ï¼Œéœ€è¦è¿ç§»çš„数据大概有六个：前三个是 project å…
ƒä¿¡æ¯ï¼Œtables 的元信息，包括一些 Hive 表，还有 model 
模型定义的一些å…
ƒä¿¡æ¯ï¼Œè¿™äº›æ˜¯ä¸éœ€è¦ä¿®æ”¹çš„。需要修改的就是 Cube 的å…
ƒä¿¡æ¯ã€‚这部分需要修改哪些东西呢？首先是 Cube 
所使用的存储和查询的类型，更新完这两个字段之后，需要重新计算一下
 Cube 的签名，这个签名的作用是 Kylin 内部设计的避免 Cube 
确定�
 �¹‹åŽæˆ‘们再去修改 Cube 
导致的一些问题；最后一个是权限相关，这部分也是å…
¼å®¹ï¼Œæ— éœ€ä¿®æ”¹çš„。&lt;/p&gt;
+
+&lt;h3 id=&quot;kylin-4--4&quot;&gt;Kylin 4 
在有赞上线后的表现&lt;/h3&gt;
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/12 
commodity_insight.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;元数据迁移到 Kylin 
之后，我们来分享一下在有赞的一些场景下带来了的质变和大å¹
…度的性能提升。首先像商品洞察这æ 
·ä¸€ä¸ªåœºæ™¯ï¼Œæœ‰ä¸€ä¸ªæ•°åä¸‡å•†å“çš„大店铺，我们要去分析它的交易和流量等，有十å‡
 ä¸ªç²¾ç¡®åŽ»é‡çš„计算。精确去重如果没有通过预计算和 Bitmap 
去做优化实际上效率是很低的，Kylin 目前使用 Bitmap 
去做精确去重的支持。在一个需要对几十万个商品的各种 UV 
去做排序的复杂�
 �Ÿ¥è¯¢çš„场景，Kylin 2 的 RT 是 27 秒，而在 Kylin 4 这个场景的 
RT 从 27 秒降到了 2 秒以内。&lt;/p&gt;
+
+&lt;p&gt;我觉得 Kylin 4 最吸引我的地方是它完å…
¨å˜æˆäº†ä¸€ä¸ªæ‰‹åŠ¨æ¡£ï¼Œè€Œ Kylin on HBase 实际上是一个自动档，因
为它的并发完全和 region 的数量绑定了。&lt;/p&gt;
+
+&lt;p&gt;&lt;img src=&quot;/images/blog/youzan_cn/13 cube_query.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;h3 id=&quot;kylin-4--5&quot;&gt;Kylin 4 在有赞的未来计划&lt;/h3&gt;
+&lt;p&gt;Kylin 4 在有赞的升级大致包含以下几个步骤：&lt;br 
/&gt;
+&lt;img src=&quot;/images/blog/youzan_cn/14 youzan_plan.png&quot; 
alt=&quot;&quot; /&gt;&lt;/p&gt;
+
+&lt;p&gt;第一阶段就是调研和可用性测试，因为 Kylin on Parquet 
实际上是基于 Spark，是有一定的学习
成本的，这个我们也花了一段时间；&lt;/p&gt;
+
+&lt;p&gt;第二阶段就是语法兼容性测试，我们扩展了 Kylin 4 
初期不支持的一些语法，比如说分页查询的语法等；&lt;/p&gt;
+
+&lt;p&gt;第三阶段就是流量重放，逐步地上线 Cube 等；&lt;/p&gt;
+
+&lt;p&gt;我们现在是属于第四阶段，我们已经迁移了一些数据了，未来的话，我们会逐步地下线旧集群，然后将所有的业务往新集群上去迁移。&lt;/p&gt;
+
+&lt;p&gt;关于 Kylin 4 
我们未来计划开发的功能和满足的需求有赞也会在社区去同步。就不在这里做详细介绍了，大家可以å
…³æ³¨æˆ‘们社区的最新动态，以上就是我们的分享。&lt;/p&gt;
+</description>
+        <pubDate>Thu, 17 Jun 2021 08:00:00 -0700</pubDate>
+        
<link>http://kylin.apache.org/cn_blog/2021/06/17/Why-did-Youzan-choose-Kylin4/</link>
+        <guid 
isPermaLink="true">http://kylin.apache.org/cn_blog/2021/06/17/Why-did-Youzan-choose-Kylin4/</guid>
+        
+        
+        <category>cn_blog</category>
+        
+      </item>
+    
+      <item>
         <title>你离可视化酷炫大屏只差一套 Kylin + Davinci</title>
         <description>&lt;p&gt;Kylin 提供与 BI 工具的整合能力，如 
Tableau，PowerBI/Excel，MSTR，QlikSense，Hue 和 
SuperSet。但就可视化工具而言，Davinci 
良好的交互性和个性化的可视化大屏展现效果，使其与 Kylin 
的结合能让大部分用户有更好的可视化分析体验。&lt;/p&gt;
 
@@ -1392,214 +1700,6 @@ Security: (depend on your security setti
         
         
         <category>blog</category>
-        
-      </item>
-    
-      <item>
-        <title>Apache Kylin v3.0.0-alpha 发布</title>
-        <description>&lt;p&gt;近日 Apache Kylin 社区很高å…
´åœ°å®£å¸ƒï¼ŒApache Kylin v3.0.0-alpha 正式发布。&lt;/p&gt;
-
-&lt;p&gt;Apache Kylin 
是一个开源的分布式分析引擎，旨在为极大数据集提供 SQL 
接口和多维分析（OLAP）的能力。&lt;/p&gt;
-
-&lt;p&gt;这是 Kylin 下一代 v3.x 
的第一个发布版本，用于早期预览，主要的功能是实时 
（Real-time） OLAP。完整的改动列表请参见&lt;a 
href=&quot;/docs/release_notes.html&quot;&gt;release 
notes&lt;/a&gt;；这里挑一些主要改进做说明。&lt;/p&gt;
-
-&lt;h1 id=&quot;section&quot;&gt;重要新功能&lt;/h1&gt;
-
-&lt;h3 id=&quot;kylin-3654----olap&quot;&gt;KYLIN-3654 - 实时 OLAP&lt;/h3&gt;
-&lt;p&gt;随着引入新的 real-time receiver 和 coordinator 组件，Kylin 
能够实现毫秒级别的数据准备延迟，数据源来自流式数据如 
Apache Kafka。这意味着，从 v3.0 开始，Kylin 
既能够支持历史批量数据的 
OLAP，也支持对流式数据的准实时（Near real-time）以及完å…
¨å®žæ—¶(real-time)分析。用户可以使用一个 OLAP 
平台来服务不同的使用场景。此方案已经在早期用户如 eBay 
得到部署和验证。关于如何使用此功能，请参考&lt;a 
href=&quot;/docs30/tutorial/realtime_olap.html&quot;&gt;æ­¤æ•
 ™ç¨‹&lt;/a&gt;。&lt;/p&gt;
-
-&lt;h3 id=&quot;kylin-3795----apache-livy--spark-&quot;&gt;KYLIN-3795 - 通过 
Apache Livy 递交 Spark 任务&lt;/h3&gt;
-&lt;p&gt;这个功能允许管理员为 Kylin 配置使用 Apache Livy 
(incubating) 来完成任务的递交。Spark 作业的提交通过 Livy 的 
REST API 来提交，而无需在本地启动 Spark Driver 
进程，从而方便对 Spark 资源的管理监控，同时也降低对 Kylin 
任务进程所在节点的压力。&lt;/p&gt;
-
-&lt;h3 id=&quot;kylin-3820----curator-&quot;&gt;KYLIN-3820 - 基于 Curator 
的任务节点分配和服务发现&lt;/h3&gt;
-&lt;p&gt;新增一种基于Apache Zookeeper 和 
Curator作业调度器，可以自动发现 Kylin 节点，并自动分é…
ä¸€ä¸ªèŠ‚点来进行任务的管理以及故
障恢复。有了这个功能后，管理员可以更加
容易地部署和扩展 Kylin 节点，而不再需要在 &lt;code 
class=&quot;highlighter-rouge&quot;&gt;kylin.properties&lt;/code&gt; 中é…
ç½®æ¯ä¸ª Kylin 节点的地址并重启 Kylin 以使之生效。&lt;/p&gt;
-
-&lt;h1 id=&quot;section-1&quot;&gt;其它改进&lt;/h1&gt;
-
-&lt;h3 id=&quot;kylin-3716---fastthreadlocal--threadlocal&quot;&gt;KYLIN-3716 
- FastThreadLocal 替换 ThreadLocal&lt;/h3&gt;
-&lt;p&gt;使用 Netty 中的 FastThreadLocal 替代 JDK 原生的 
ThreadLocal，可以一定程度上提升 Kylin 
在高并发下的性能。&lt;/p&gt;
-
-&lt;h3 
id=&quot;kylin-3867---enable-jdbc-to-use-key-store--trust-store-for-https-connection&quot;&gt;KYLIN-3867
 - Enable JDBC to use key store &amp;amp; trust store for https 
connection&lt;/h3&gt;
-&lt;p&gt;通过使用HTTPS，保护了JDBC使用的身份验证信息，使得Kylin更åŠ
 å®‰å…¨&lt;/p&gt;
-
-&lt;h3 
id=&quot;kylin-3905---enable-shrunken-dictionary-default&quot;&gt;KYLIN-3905 - 
Enable shrunken dictionary default&lt;/h3&gt;
-&lt;p&gt;默认开启 shrunken 
dictionary，针对高基维进行精确去重的场景，可以显著减少构建用时。&lt;/p&gt;
-
-&lt;h3 
id=&quot;kylin-3839---storage-clean-up-after-the-refreshing-and-deleting-a-segment&quot;&gt;KYLIN-3839
 - Storage clean up after the refreshing and deleting a segment&lt;/h3&gt;
-&lt;p&gt;更加及时地清除不必要的数据文件&lt;/p&gt;
-
-&lt;p&gt;&lt;strong&gt;下载&lt;/strong&gt;&lt;/p&gt;
-
-&lt;p&gt;要下载Apache Kylin 源代码或二进制包，请访问&lt;a 
href=&quot;/download&quot;&gt;下载页面&lt;/a&gt; page.&lt;/p&gt;
-
-&lt;p&gt;&lt;strong&gt;升级&lt;/strong&gt;&lt;/p&gt;
-
-&lt;p&gt;参考&lt;a 
href=&quot;/docs/howto/howto_upgrade.html&quot;&gt;升级指南&lt;/a&gt;.&lt;/p&gt;
-
-&lt;p&gt;&lt;strong&gt;反馈&lt;/strong&gt;&lt;/p&gt;
-
-&lt;p&gt;如果您遇到问题或疑问，请发送邮件至 Apache Kylin dev 
或 user 邮件列表：d...@kylin.apache.org，u...@kylin.apache.org; 
在发送之前，请确保您已通过发送电子邮件至 
dev-subscr...@kylin.apache.org 或 user-subscr...@kylin.apache.org 订阅
了邮件列表。&lt;/p&gt;
-
-&lt;p&gt;&lt;em&gt;非常感谢所有贡献Apache 
Kylin的朋友!&lt;/em&gt;&lt;/p&gt;
-</description>
-        <pubDate>Fri, 19 Apr 2019 13:00:00 -0700</pubDate>
-        
<link>http://kylin.apache.org/cn_blog/2019/04/19/release-v3.0.0-alpha/</link>
-        <guid 
isPermaLink="true">http://kylin.apache.org/cn_blog/2019/04/19/release-v3.0.0-alpha/</guid>
-        
-        
-        <category>cn_blog</category>
-        
-      </item>
-    
-      <item>
-        <title>Real-time Streaming Design in Apache Kylin</title>
-        <description>&lt;h2 
id=&quot;why-build-real-time-streaming-in-kylin&quot;&gt;Why Build Real-time 
Streaming in Kylin&lt;/h2&gt;
-&lt;p&gt;The real-time streaming feature is contributed by eBay big data team 
in Kylin 3.0, the purpose we build real-time streaming is:&lt;/p&gt;
-
-&lt;ul&gt;
-  &lt;li&gt;
-    &lt;p&gt;Milliseconds Data Preparation Delay  &lt;br /&gt;
-Kylin provide sub-second query latency for extremely large dataset, the 
underly magic is precalculation cube. But the cube building often take long 
time(usually hours for large data sets),  in some case, the analyst needs 
real-time data to do analysis, so we want to provide real-time OLAP, which 
means data can be queried immediately when produced to system.&lt;/p&gt;
-  &lt;/li&gt;
-  &lt;li&gt;
-    &lt;p&gt;Support Lambda Architecture  &lt;br /&gt;
-Real-time data often not reliable, that may caused by many reasons, for 
example, the upstream processing system has a bug, or the data need to be 
changed after some time, etc. So we need to support lambda architecture, which 
means the cube can be built from the streaming source(like Kafka), and the 
historical cube data can be refreshed from batch source(like Hive).&lt;/p&gt;
-  &lt;/li&gt;
-  &lt;li&gt;
-    &lt;p&gt;Less MR jobs and HBase Tables   &lt;br /&gt;
-Since Kylin 1.6, community has provided a streaming solution, it uses MR to 
consume Kafka data and then do batch cube building, it can provide minute-level 
data preparation latency, but to ensure the data latency, you need to schedule 
the MR very shortly(5 minutes or even less), that will cause too many hadoop 
jobs and small hbase tables in the system, and dramatically increase the Hadoop 
system’s load.&lt;/p&gt;
-  &lt;/li&gt;
-&lt;/ul&gt;
-
-&lt;h2 id=&quot;architecture&quot;&gt;Architecture&lt;/h2&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_architecture.png&quot; 
alt=&quot;Kylin RT Streaming Architecture&quot; /&gt;&lt;/p&gt;
-
-&lt;p&gt;The blue rectangle is streaming components added in current Kylin’s 
architecture, which is responsible to ingest data from streaming source, and 
provide query for real-time data.&lt;/p&gt;
-
-&lt;p&gt;We divide the unbounded incoming streaming data into 3 stages, the 
data come into different stages are all queryable immediately.&lt;/p&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_stages.png&quot; 
alt=&quot;Kylin RT Streaming stages&quot; /&gt;&lt;/p&gt;
-
-&lt;h3 id=&quot;components&quot;&gt;Components&lt;/h3&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_components.png&quot; 
alt=&quot;Kylin RT Streaming Components&quot; /&gt;&lt;/p&gt;
-
-&lt;p&gt;Streaming Receiver: Responsible to ingest data from stream data 
source, and provide real-time data query.&lt;/p&gt;
-
-&lt;p&gt;Streaming Coordinator: Responsible to do coordination works, for 
example, when new streaming cube is onboard, the coordinator need to decide 
which streaming receivers can be assigned.&lt;/p&gt;
-
-&lt;p&gt;Metadata Store:  Used to store streaming related metadata, for 
example, the cube assignments information, cube build state 
information.&lt;/p&gt;
-
-&lt;p&gt;Query Engine:  Extend the existing query engine, support to query 
real-time data from streaming receiver&lt;/p&gt;
-
-&lt;p&gt;Build Engine: Extend the existing build engine, support to build full 
cube from the real-time data&lt;/p&gt;
-
-&lt;h3 id=&quot;how-streaming-cube-engine-works&quot;&gt;How Streaming Cube 
Engine Works&lt;/h3&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_how_build_work.png&quot; 
alt=&quot;Kylin RT Streaming How Build Works&quot; /&gt;&lt;/p&gt;
-
-&lt;ol&gt;
-  &lt;li&gt;Coordinator ask streaming source for all partitions of the 
cube&lt;/li&gt;
-  &lt;li&gt;Coordinator decide which streaming receivers to assign to consume 
streaming data, and ask streaming receivers to start consuming data.&lt;/li&gt;
-  &lt;li&gt;Streaming receiver start to consume and index streaming 
events&lt;/li&gt;
-  &lt;li&gt;After sometime, streaming receiver copy the immutable segments 
from local files to remote HDFS files&lt;/li&gt;
-  &lt;li&gt;Streaming receiver notify the coordinator that a segment has been 
persisted to HDFS&lt;/li&gt;
-  &lt;li&gt;Coordinator submit a cube build job to Build Engine to triger cube 
full building after all receivers have submitted their segments&lt;/li&gt;
-  &lt;li&gt;Build Engine build all cuboids from the streaming HDFS 
files&lt;/li&gt;
-  &lt;li&gt;Build Engine store cuboid data to Hbase, and then the coordinator 
will ask the streaming receivers to remove the related local real-time 
data.&lt;/li&gt;
-&lt;/ol&gt;
-
-&lt;h3 id=&quot;how-streaming-query-engine-works&quot;&gt;How Streaming Query 
Engine Works&lt;/h3&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_how_query_work.png&quot; 
alt=&quot;Kylin RT Streaming How Query Works&quot; /&gt;&lt;/p&gt;
-
-&lt;ol&gt;
-  &lt;li&gt;If Query hits a streaming cube, Query Engine ask Streaming 
Coordinator what streaming receivers are assigned for the cube&lt;/li&gt;
-  &lt;li&gt;Query Engine send query request to related streaming receivers to 
query realtime segments&lt;/li&gt;
-  &lt;li&gt;Query Engine send query request to Hbase to query historical 
segments&lt;/li&gt;
-  &lt;li&gt;Query Engine aggregate the query results, and send response back 
to client&lt;/li&gt;
-&lt;/ol&gt;
-
-&lt;h2 id=&quot;detail-design&quot;&gt;Detail Design&lt;/h2&gt;
-
-&lt;h3 id=&quot;real-time-segment-store&quot;&gt;Real-time Segment 
Store&lt;/h3&gt;
-&lt;p&gt;Real-time segments are divided by event time, when new event comes, 
it will be calculated which segment it will be located, if the segment 
doesn’t exist, create a new one.&lt;/p&gt;
-
-&lt;p&gt;The new created segment is in ‘Active’ state first, if no further 
events coming into the segment after some preconfigured period, the segment 
state will be changed to ‘Immutable’, and then write to remote 
HDFS.&lt;/p&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_rt_segment_state.png&quot; 
alt=&quot;Kylin RT Streaming Segment State&quot; /&gt;&lt;/p&gt;
-
-&lt;p&gt;Each real-time segment has a memory store, new event will first goes 
into the memory store to do aggregation, when the memory store size reaches the 
configured threshold, it will be then be flushed to local disk as a fragment 
file.&lt;/p&gt;
-
-&lt;p&gt;Not all cuboids are built in the receiver side, only basic cuboid and 
some specified cuboids are built.&lt;/p&gt;
-
-&lt;p&gt;The data is stored as columnar format on disk, and when there are too 
many fragments on disk, the fragment files will be merged by a background 
thread automatically.&lt;/p&gt;
-
-&lt;p&gt;The directory structure in receiver side is like:&lt;/p&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_dir_structure.png&quot; 
alt=&quot;Kylin RT Streaming Segment Directory&quot; /&gt;&lt;/p&gt;
-
-&lt;p&gt;To improve the query performance, the data is stored in columnar 
format, the data format is like:&lt;/p&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_columnar_format.png&quot; 
alt=&quot;Kylin RT Streaming Columnar Format&quot; /&gt;&lt;/p&gt;
-
-&lt;p&gt;Each cuboid data is stored together, and in each cuboid the data is 
stored column by column, the metadata is stored in json format.&lt;/p&gt;
-
-&lt;p&gt;The dimension data is divided into 3 parts:&lt;/p&gt;
-
-&lt;p&gt;The first part is Dictionary part, this part exists when the 
dimension encoding is set to ‘Dict’ in cube design, by default we use &lt;a 
href=&quot;https://kylin.apache.org/blog/2015/08/13/kylin-dictionary/&quot;&gt;tri-tree
 dictionary&lt;/a&gt; to minimize the memory footprints and preserve the 
original order.&lt;/p&gt;
-
-&lt;p&gt;The second part is dictionary encoded values, additional compression 
mechanism can be applied to these values, since the values for the same column 
are usually similar, so the compression rate will be very good.&lt;/p&gt;
-
-&lt;p&gt;The third part is invert-index data, use Roaring Bitmap to store the 
invert-index info, the following picture shows how invert-index data is stored, 
there are two types of format, the first one is dictionary encoding 
dimension’s index data format, the second is other fix-len encoding 
dimension’s index data format.&lt;/p&gt;
-
-&lt;p&gt;&lt;img src=&quot;/images/blog/rt_stream_invertindex_format.png&quot; 
alt=&quot;Kylin RT Streaming InvertIndex Format&quot; /&gt;&lt;/p&gt;
-
-&lt;p&gt;Real-time data is stored in compressed format, currently support two 
type compression: Run Length Encoding and LZ4.&lt;/p&gt;
-
-&lt;ul&gt;
-  &lt;li&gt;Use RLE compression for time-related dim and first dim&lt;/li&gt;
-  &lt;li&gt;Use LZ4 for other dimensions by default&lt;/li&gt;
-  &lt;li&gt;Use LZ4 Compression for simple-type measure(long, 
double)&lt;/li&gt;
-  &lt;li&gt;No compression for complex measure(count distinct, topn, 
etc.)&lt;/li&gt;
-&lt;/ul&gt;
-
-&lt;h3 id=&quot;high-availability&quot;&gt;High Availability&lt;/h3&gt;
-
-&lt;p&gt;Streaming receivers are group into replica-sets, all receivers in the 
same replica-set  share the same assignments, so that when one receiver is 
down, the query and event consuming will not be impacted.&lt;/p&gt;
-
-&lt;p&gt;In each replica-set, there is a lead responsible to upload real-time 
segments to HDFS, and zookeeper is used to do leader election&lt;/p&gt;
-
-&lt;h3 id=&quot;failure-recovery&quot;&gt;Failure Recovery&lt;/h3&gt;
-
-&lt;p&gt;We do checkpoint periodically in receiver side, so that when the 
receiver is restarted, the data can be restored correctly.&lt;/p&gt;
-
-&lt;p&gt;There are two parts in the checkpoint: the first part is the 
streaming source consume info, for Kafka it is {partition:offset} pairs, the 
second part is disk states {segment:framentID} pairs, which means when do the 
checkpoint what’s the max fragmentID for each segment.&lt;/p&gt;
-
-&lt;p&gt;When receiver is restarted, it will check the latest checkpoint, set 
the Kafka consumer to start to consume data from specified partition offsets, 
and remove the fragment files that the fragmentID is larger than the 
checkpointed fragmentID on the disk.&lt;/p&gt;
-
-&lt;p&gt;Besides the local checkpoint, we also have remote checkpoint, to 
restore the state when the disk is crashed, the remote checkpoint is saved to 
Cube Segment metadata after HBase segment build, like:&lt;br /&gt;
-&lt;code class=&quot;highlighter-rouge&quot;&gt;
-    ”segments”:[{…,
-        &quot;stream_source_checkpoint&quot;: {&quot;0&quot;:8946898241, 
“1”: 8193859535, ...}
-                 },
-       ]
-&lt;/code&gt;&lt;br /&gt;
-The checkpoint info is the smallest partition offsets on the streaming 
receiver when real-time segment is sent to full build.&lt;/p&gt;
-
-&lt;h2 id=&quot;future&quot;&gt;Future&lt;/h2&gt;
-&lt;ul&gt;
-  &lt;li&gt;Star Schema Support&lt;/li&gt;
-  &lt;li&gt;Streaming Receiver On Kubernetes/Yarn&lt;/li&gt;
-&lt;/ul&gt;
-</description>
-        <pubDate>Fri, 12 Apr 2019 09:30:00 -0700</pubDate>
-        
<link>http://kylin.apache.org/blog/2019/04/12/rt-streaming-design/</link>
-        <guid 
isPermaLink="true">http://kylin.apache.org/blog/2019/04/12/rt-streaming-design/</guid>
-        
-        
-        <category>blog</category>
         
       </item>
     

Added: kylin/site/images/blog/youzan/1 history_of_youzan_OLAP.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/1%20history_of_youzan_OLAP.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/1 history_of_youzan_OLAP.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/10 commodity_insight.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/10%20commodity_insight.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/10 commodity_insight.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/2 kylin4_storage.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/2%20kylin4_storage.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/2 kylin4_storage.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/3 kylin4_build_engine.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/3%20kylin4_build_engine.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/3 kylin4_build_engine.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/4 kylin4_query.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/4%20kylin4_query.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/4 kylin4_query.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/5 cache_calcite_plan.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/5%20cache_calcite_plan.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/5 cache_calcite_plan.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/6 tuning_spark_configuration.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/6%20tuning_spark_configuration.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/6 tuning_spark_configuration.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/7 parquet_optimization.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/7%20parquet_optimization.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/7 parquet_optimization.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/8 
Dynamic_elimination_of_partitioning_dimensions.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/8%20Dynamic_elimination_of_partitioning_dimensions.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/8 
Dynamic_elimination_of_partitioning_dimensions.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan/9 cache_parent_dataset.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan/9%20cache_parent_dataset.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan/9 cache_parent_dataset.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/1 kylin4_storage.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/1%20kylin4_storage.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/1 kylin4_storage.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/10 Processing data skew.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/10%20Processing%20data%20skew.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/10 Processing data skew.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/11 metadata_upgrade.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/11%20metadata_upgrade.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/11 metadata_upgrade.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/12 commodity_insight.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/12%20commodity_insight.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/12 commodity_insight.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/13 cube_query.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/13%20cube_query.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/13 cube_query.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/14 youzan_plan.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/14%20youzan_plan.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/14 youzan_plan.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/2 kylin4_build_engine.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/2%20kylin4_build_engine.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/2 kylin4_build_engine.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/3 kylin4_query.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/3%20kylin4_query.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/3 kylin4_query.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/4 
dynamic_elimination_dimension_partition.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/4%20dynamic_elimination_dimension_partition.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/4 
dynamic_elimination_dimension_partition.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/5 Partition clipping under complex 
filter.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/5%20Partition%20clipping%20under%20complex%20filter.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/5 Partition clipping under complex 
filter.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/6 tuning_spark_configuration.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/6%20tuning_spark_configuration.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/6 tuning_spark_configuration.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/8 small_query_optimization.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/8%20small_query_optimization.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/8 small_query_optimization.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: kylin/site/images/blog/youzan_cn/9 cache_parent_dataset.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/blog/youzan_cn/9%20cache_parent_dataset.png?rev=1890886&view=auto
==============================================================================
Binary file - no diff available.

Propchange: kylin/site/images/blog/youzan_cn/9 cache_parent_dataset.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Modified: kylin/site/images/docs/quickstart/advance_setting.png
URL: 
http://svn.apache.org/viewvc/kylin/site/images/docs/quickstart/advance_setting.png?rev=1890886&r1=1890885&r2=1890886&view=diff
==============================================================================
Binary files - no diff available.