Modified: kylin/site/feed.xml URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1899440&r1=1899439&r2=1899440&view=diff ============================================================================== --- kylin/site/feed.xml (original) +++ kylin/site/feed.xml Thu Mar 31 14:09:40 2022 @@ -19,11 +19,851 @@ <description>Apache Kylin Home</description> <link>http://kylin.apache.org/</link> <atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/> - <pubDate>Tue, 29 Mar 2022 06:59:11 -0700</pubDate> - <lastBuildDate>Tue, 29 Mar 2022 06:59:11 -0700</lastBuildDate> + <pubDate>Thu, 31 Mar 2022 06:59:26 -0700</pubDate> + <lastBuildDate>Thu, 31 Mar 2022 06:59:26 -0700</lastBuildDate> <generator>Jekyll v2.5.3</generator> <item> + <title>å¦ä½ä½¿ç¨ Excel æ¥è¯¢ Kylinï¼MDX for Kylinï¼</title> + <description><h2 id="kylin--mdx">Kylin 为ä»ä¹éè¦ MDXï¼</h2> + +<h3 id="section">å¤ç»´æ°æ®åºåä¸å¡è¯ä¹å±</h3> + +<p>å¤ç»´æ°æ®åºä¸å ³ç³»åæ°æ®åºçå ³é®åºå«å¨äºä¸å¡è¡¨è¾¾è½åã尽管 SQL 表达è½åå¾å¼ºï¼æ¯æ°æ®åæå¸çåºæ¬æè½ï¼ä½å¦æä»¥ âäººäººé½æ¯åæå¸â ä¸ºç®æ ï¼SQL åå ³ç³»æ°æ®åºå¯¹éææ¯äººåè¿æ¯å¤ªé¾äºãä»éææ¯äººåçè§è§ï¼æ°æ®æ¹åæ°æ®ä»åºå°±å¥½ä¼¼ä¸ä¸ªé»æçæ¿é´ï¼ç¥éå ¶ä¸æå¾å¤æ°æ®ï¼å´å ä¸ºä¸ææ°æ®åºç论å SQLï¼æ æ³çæ¸ ãçè§£ãå使ç¨è¿äºæ°æ®ã</p> + +<p>å¦ä½è®©æ°æ®æ¹(åæ°æ®ä»åº)坹鿿¯äººåä¹ â<strong>æ¸ æ¾è§åº</strong>âï¼è¿å°±éè¦å¼å ¥ä¸ä¸ªå¯¹éææ¯äººåæ´å å好çâ<strong>å ³ç³»æ°æ®æ¨¡å</strong> â <strong>å¤ç»´æ°æ®æ¨¡å</strong>âãå¦æè¯´å ³ç³»æ¨¡åæè¿°äºæ°æ®çææ¯å½¢æï¼é£ä¹å¤ç»´æ¨¡ååæè¿°äºæ°æ®çä¸å¡å½¢æãå¨å¤ç»´æ°æ®åºä¸ï¼åº¦é对åºäºæ¯ä¸ªäººé½æçä¸å¡ææ ï¼ç»´åº¦åæ¯æ¯è¾ãè§å¯è¿äºä¸å¡ææ çè§åº¦ãè¦ä¸ä¸ä¸ªææ¯è¾ KPIï¼è¦å ¨å¹³è¡äºä¸é¨ä¹é´æ¯è¾ç»©æï¼è¿äºæ¯æ¯ä¸ªéææ¯äººåé½çè§£çæ¦å¿µãéè¿å°å ³ç³»æ¨¡åæ å°å°å¤ç»´æ¨¡åï¼æ¬è´¨æ¯å¨ææ¯æ°æ®ä¹ä¸å¢å¼ºäºä¸å¡è¯ä¹ï¼å½¢æä¸å¡è¯ä¹å±ï¼å¸®å©éææ¯äººåä¹è½çæãæ¢ç´¢ãä½¿ç¨æ°æ®ã</p> + +<p>为äºå¢å¼º Kylin ä½ä¸ºå¤ç»´æ°æ®åºçè¯ä¹å±è½åï¼æ¯æå¤ç»´æ¥è¯¢è¯è¨æ¯ Kylin Roadmap ä¸çéç¹å å®¹ï¼æ¯å¦ MDX å DAXãéè¿ MDX å¯ä»¥å° Kylin ä¸çæ°æ®æ¨¡å转æ¢ä¸ºä¸å¡å好çè¯è¨ï¼èµäºæ°æ®ä¸å¡ä»·å¼ï¼æ¹ä¾¿å¯¹æ¥ <strong>Excel</strong>ãTableau ç BI å·¥å ·è¿è¡å¤ç»´åæã</p> + +<h3 id="mdx-">åºäº MDX æé ä¸å¡ææ å¹³å°</h3> + +<p>ä½¿ç¨ MDX æ¥åå»ºå¤æçä¸å¡ææ ï¼ç¸å¯¹ SQL çä¸äºç¹ç¹åä¼å¿å¦ä¸ï¼</p> + +<ol> + <li>æ´å¥½ç<strong>æ¯æå¤æåæåºæ¯</strong>ï¼å¦åç´¯å ãå¤å¯¹å¤ãæ¶é´çªå£åæçï¼</li> + <li>å¹¿æ³æ¯æåç§ BIï¼âKylin + MDXâ ä¸ä» è½å¤éè¿ SQL æ¥å£æ´é²ä¸ºç±»ä¼¼äºå ³ç³»åæ°æ®åºç表ï¼ä¹å¯ä»¥æ´é²ä¸ºå ¼å®¹ XMLA åè®®ç带æè¯ä¹ä¿¡æ¯çæ°æ®æºï¼å¯éè¿ MDX è¯è¨è¿è¡æ¥è¯¢ï¼æ¯æå¯¹æ¥ <strong>Excel</strong> çåªè½éè¿ XMLA åè®®å¯¹æ¥æ°æ®æºç BIï¼</li> + <li>åºäº Kylin ä¸çæ°æ®æ¨¡åçµæ´»å®ä¹ MDX è¯ä¹æ¨¡åï¼å°åºå±æ°æ®ç»æè½¬æ¢ä¸ºä¸å¡å好çè¯è¨ï¼èµäºæ°æ®ä¸å¡ä»·å¼ï¼ä½¿å¾ä¸å¡äººåå¨åææ¶æ éå ³å¿åºå±çææ¯å¤æåº¦åå®ç°ï¼éè¿ MDX 模åå夿´é²ç»ä¸è¯ä¹ï¼å¸®å©ç¨æ·<strong>æå»ºç»ä¸çä¸å¡è¯ä¹å±</strong>ã(详ç»ä¿¡æ¯å¯ä»¥åè龿¥ 1/6/7)</li> +</ol> + +<h2 id="mdx--1">MDX ä»ç»</h2> + +<h3 id="mdx">ä»ä¹æ¯ MDXï¼</h3> + +<p>MDX (Multi Dimensional eXpression) æ¯ä¸ç§ OLAP å¤ç»´æ°æ®éçæ¥è¯¢è¯è¨ï¼æåç± Microsoft äº 1997 å¹´ä½ä¸º OLEDB for OLAP è§èå¼å ¥ï¼éåéæå¨ SSAS ä¸ãç®åï¼MDX å¨ OLAP æ°æ®åºä¸è¢«å¹¿æ³éç¨ã</p> + +<p>MDX å¨å¾å¤æ¹é¢ä¸ç»æåæ¥è¯¢è¯è¨ (SQL) è¯æ³ç¸ä¼¼ï¼ä½å®ä¸æ¯ SQL è¯è¨çæ©å±ï¼äºå®ä¸ï¼MDX ææä¾çä¸äºåè½ä¹å¯ç± SQL æä¾ï¼å°½ç®¡ä¸æ¯é£ä¹æææç´è§ãå¦å SQL æ¥è¯¢ä¸æ ·ï¼MDX æ¥è¯¢å¯ä»¥å æ¬SELECT åå¥)ãFROM åå¥å WHERE åå¥ãè¿äºå ³é®å以åå ¶å®å ³é®åæä¾äºåç§å·¥å ·ï¼ç¨æ¥ä»å¤ç»´æ°æ®éæåæ°æ®çç¹å®é¨åã</p> + +<p>MDX æ¥è¯¢è¯æ³ç¤ºä¾å¦ä¸(详ç»ä¿¡æ¯å¯ä»¥åè龿¥ 3)ï¼</p> + +<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">select</span> <span class="o">&lt;</span><span class="n">axis_specification</span><span class="o">&gt;</span><span class="p">[,</span> <span class="o">&lt;</span><span class="n">axis_specification</span><span class="o">&gt;</span><span class="p">]</span> +<span class="k">from</span> <span class="o">&lt;</span><span class="n">cube_specification</span><span class="o">&gt;</span> +<span class="k">where</span> <span class="o">&lt;</span><span class="n">slicer_specification</span><span class="o">&gt;</span> +</code></pre> +</div> + +<h3 id="mdx--2">MDX åºæ¬æ¦å¿µ</h3> + +<p>å¨äºè§£å¦ä¹ MDX ä¹åï¼è¯·è³å°ææ¡ä»¥ä¸æ¦å¿µãéäºç¸å ³æ¦å¿µå¨ Microsoft ç SSAS å®ç½ææ¯è¾è¯¦ç»çä»ç»ï¼ä¸äºè§£çåå¦å¯ä»¥éè¿æç« æ«å°¾çåè龿¥å¦ä¹ ã(详ç»ä¿¡æ¯å¯ä»¥åè龿¥ 2/3/4)</p> + +<ol> + <li>维度(Dimensions)ã级å«(Levels)ãæå(Members)å度éå¼(Measures)</li> + <li>åå (Cell)ãå ç»(Tuple)åéå(Set)</li> + <li>轴维度(Query Axis)ååçå¨ç»´åº¦(Slicer Axis)</li> +</ol> + +<h3 id="mdx--sql-">MDX å SQL çæ¯è¾</h3> + +<p><strong>æ¥è¯¢å¯¹è±¡</strong>ä¸åï¼MDX çæ¥è¯¢å¯¹è±¡æ¯<strong>å¤ç»´æ°æ®é</strong>(Cube)ï¼æ¯æå Join åèåå¥½çæ°æ®ï¼æ¥è¯¢æ¶ä¸éè¦æå® Join å ³ç³»ãSQL æ¥è¯¢å¯¹è±¡æ¯<strong>å ³ç³»è¡¨</strong>(Table)ï¼æ¯ä¸æ¡æ¡çæç»è®°å½ï¼æ¥è¯¢æ¶éè¦æå®è¡¨ä¹é´ç Join å ³ç³»ã</p> + +<p><strong>æ¥è¯¢ç»æ</strong>ä¸åï¼SQL è¿å<strong>äºç»´æ°æ®åé</strong>ï¼è MDX è¿å<strong>å¤ç»´æ°æ®é</strong>ã(详ç»ä¿¡æ¯å¯ä»¥åè龿¥ 5)</p> + +<h1 id="mdx-for-kylin-">MDX for Kylin ä»ç»</h1> + +<h3 id="mdx-for-kylin--1">ä»ä¹æ¯ MDX for Kylin ?</h3> + +<p><strong>MDX for Kylin</strong> æ¯åºäº <strong>Mondrian</strong> äºæ¬¡å¼åçï¼ç± <strong>Kyligence</strong> è´¡ç®çï¼ä½¿ç¨ <strong>Apache Kylin 4</strong> ä½ä¸ºæ°æ®æºç MDX æ¥è¯¢å¼æ ãMDX for Kylin ç使ç¨ä½éªæ¯è¾æ¥è¿ Microsoft SSASï¼å¯ä»¥éæå¤ç§æ°æ®åæå·¥å ·ï¼å æ¬ Microsoft ExcelãTableau çï¼å¯ä»¥ä¸ºå¤§æ°æ®åæåºæ¯ä¸æä¾æ´æè´çä½éªã</p> + +<h3 id="mdx-for-kylin--2">MDX for Kylin å建ä¸å¡ææ </h3> + +<h4 id="section-1">ååææ åä¸å¡ææ </h4> + +<p>å¨ Kylin Cube æä»¬å建çåç§åº¦éï¼æ¯å¨åç¬çä¸åä¸çè¿è¡çèå计ç®(TopN é¤å¤)ï¼åªå å«äºæéçå ç§èå彿°ï¼å³ Sum/Max/Min/Count/Count Distinctï¼ç¸å¯¹æ¯è¾ç®åï¼æä»¬ç§°ä¹ä¸º<strong>ååææ </strong>ã</p> + +<p>å¨å®é ä¸å¡åºæ¯ä¸ï¼åºäºååææ æä»¬å¯ä»¥å¯¹<strong>ååææ </strong>çåç§å¤æè¿ç®ï¼æ¥å建æä¸å¡å«ä¹çå¤åææ ï¼è¿æ ·çææ æä»¬ç§°ä¹ä¸º<strong>ä¸å¡ææ ã</strong></p> + +<h4 id="section-2">å±çº§ç»æã计ç®åº¦éåå½åé</h4> + +<p><strong>å±çº§ç»æï¼</strong>å±çº§ç»ææ¯åºäºç»´åº¦ç级å«éåï¼å¯ç¨äºæé«æ°æ®åæäººåçåæè½åãä¾å¦ï¼ä½ å¯ä»¥å建ä¸ä¸ªæ¶é´å±çº§ç»æï¼å å«äºå¹´ãå£ãæãå¨åæ¥çº§å«ãè¿æ ·åæäººåå¨å®¢æ·ç«¯ä¸å¯ä»¥å éå¹´åæéå®é¢ï¼å¨éè¦æ¶å¯ä»¥åå«å±å¼âå£åº¦ &gt; æ &gt; å¨ &gt; æ¥âæ¥è¿è¡æ´ç»ç²åº¦çåæã</p> + +<p><strong>计ç®åº¦éï¼</strong>计ç®åº¦éæ¯å¯¹<strong>ååææ </strong>ä½¿ç¨ MDX 表达å¼è¿è¡å¤å计ç®å½¢æçæ°ç度é/ææ ï¼æä»¬ä¸»è¦ä½¿ç¨è®¡ç®åº¦éæ¥å建<strong>ä¸å¡ææ </strong>ã</p> + +<p><strong>å½åéï¼</strong>å¨ MDX for Kylin ç使ç¨ä¸ï¼ç»å¸¸ä¼åºç°éè¦éå¤ä½¿ç¨ä¸ç»æåçéæ±ï¼è¿ç§éæ±å¯ä»¥éè¿å®ä¹å½åéæ¥æ»¡è¶³ãå½åé(NamedSet) æ¯æ ¹æ®æå®çè¡¨è¾¾å¼ è®¡ç®å¾å°çä¸ä¸ªæåSetï¼å³ä¸ç»æåçéåï¼å½åéå¯ä»¥ç´æ¥ç½®äºè½´ä¸ç¨äºå±ç¤ºï¼ä¹å¯ä»¥å¨è®¡ç®åº¦éæå ¶ä»å½åéç表达å¼ä¸ä½¿ç¨ã</p> + +<h4 id="section-3">å建è¯ä¹æ¨¡å</h4> + +<p>å¨ Kylin 4 æ ¹æ®è¡¨ä¸è¡¨ä¹é´çå ³ç³»åå»ºæ°æ®æ¨¡åï¼å¹¶ä¸å¨ Cube ä¸å®ä¹ç»´åº¦å度éï¼è¿äºåº¦éæä»¬å¯ä»¥è®¤ä¸ºæ¯<strong>ååææ </strong>ã</p> + +<p>å¨ MDX for Kylinï¼å°ç¸å ³èç Kylin Cube ç¸å ³èï¼æ¥åå»ºæ°æ®éï¼å¹¶ä¸åºäºååææ ï¼æ¥å建æä¸å¡å«ä¹ç<strong>ä¸å¡ææ ã</strong></p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/1_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h4 id="section-4">æ°æ®åæ</h4> + +<p>ä½¿ç¨æ¶ï¼å®¢æ·ç«¯åé MDX æ¥è¯¢ç» MDX for Kylinï¼MDX for Kylin åè§£æ MDX æ¥è¯¢ç¿»è¯ä¸º SQL å¹¶ä¸åéç» Kylin ï¼ç¶å Kylin éè¿é¢è®¡ç®ç Cuboid åç SQL æ¥è¯¢å¹¶æç»æäº¤è¿ç» MDX for Kylinï¼MDX for Kylin ä¼ååä¸äºè¡çææ ç计ç®ï¼æç»å°å¤ç»´æ°æ®ç»æè¿åç»å®¢æ·ç«¯ã</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/2_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h4 id="section-5">æµç¨æ»ç»</h4> + +<p>æ»çæ¥è¯´ï¼æ¯æ MDX æ¥å£è½å¤å¢å¼º Kylin çè¯ä¹å±è½åï¼ä¸ºç¨æ·å¸¦æ¥ç»ä¸çæ°æ®åæå管çä½éªï¼æ´å¥½å°åæ¥æ°æ®çä»·å¼ãä¸å¾å°±æ¯å°ä»ä¸èä¸ï¼å±ç¤ºä»åå§æ°æ®å å·¥ä¸å¡ææ çè¿ç¨ã</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/3_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h3 id="mdx-for-kylin--3">MDX for Kylin çææ¯ä¼å¿</h3> + +<p>MDX for Kylin ç¸å¯¹å ¶å®å¼æº MDX æ¥è¯¢å¼æï¼å ·æä»¥ä¸ä¼å¿ï¼</p> + +<ul> + <li>æ´å¥½æ¯æBI(Excel/Tableau/Power BIç) 产åï¼éé XMLA åè®®ï¼</li> + <li>é对 BI ç MDX Query è¿è¡äºç¹å®ä¼åéåï¼</li> + <li>éé Kylin æ¥è¯¢ï¼éè¿ Kylin çé¢è®¡ç®è½åå é MDX æ¥è¯¢ï¼</li> + <li>éè¿ç®æ´ææçæä½çé¢ï¼æä¾äºç»ä¸çææ å®ä¹å管çè½åã</li> +</ul> + +<h1 id="docker-">ä» Docker å¿«éå¼å§</h1> + +<h3 id="section-6">æµè¯ç¯å¢</h3> + +<ul> + <li>Macbook Pro ç¬è®°æ¬ + <ul> + <li>Docker Desktop (latest version)</li> + </ul> + </li> + <li>Windows 10 èææº + <ul> + <li>Microsoft Excel (for Windows)</li> + </ul> + </li> +</ul> + +<h3 id="section-7">å¯å¨å®¹å¨</h3> + +<p>è¿ä¸ªå®¹å¨å å«äº YarnãHDFSãMySQLãKylinãMDX for Kylin çè¿ç¨ã</p> + +<pre><code class="language-she">docker run -d \ + -m 8g \ + -p 7070:7070 \ + -p 7080:7080 \ + -p 8088:8088 \ + -p 50070:50070 \ + -p 8032:8032 \ + -p 8042:8042 \ + -p 2181:2181 \ + --name kylin-4.0.1 \ + apachekylin/apache-kylin-standalone:kylin-4.0.1-mondrian +</code></pre> + +<h3 id="section-8">æ£æ¥ç¯å¢</h3> + +<p>çå¾ ä¸æ®µæ¶é´ï¼è¯·ä¾æ¬¡æ£æ¥ HDFS/YARN/Kylin/MDX for Kylin ç Web UI æ¯å¦å¯ä»¥è®¿é®ã</p> + +<table> + <thead> + <tr> + <th><strong>ç»ä»¶</strong></th> + <th><strong>Web UI å°å</strong></th> + </tr> + </thead> + <tbody> + <tr> + <td>HDFS</td> + <td>http://localhost:50070/dfshealth.html</td> + </tr> + <tr> + <td>YARN</td> + <td>http://localhost:8088/cluster</td> + </tr> + <tr> + <td>Kylin</td> + <td>http://localhost:7070/kylin</td> + </tr> + <tr> + <td>MDX for Kylin</td> + <td>http://localhost:7080/overview</td> + </tr> + </tbody> +</table> + +<h3 id="cube">æå»ºæ ·ä¾ Cube</h3> + +<p>è¯·ç´æ¥ä½¿ç¨ Kylin èªå¸¦çæ ·ä¾ Cubeï¼<code class="highlighter-rouge">kylin_sales_cube</code>ã</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/4_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h3 id="mdx--3">å建 MDX æ°æ®é</h3> + +<h5 id="mdx-for-kylin">ç»å½ MDX for Kylin</h5> + +<p>é»è®¤è´¦å·/å¯ç æ¯ ADMIN/KYLINï¼MDX for Kylin çè´¦æ·ä¸ KYLIN ç忥ã</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/5_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h5 id="section-9">åå»ºæ°æ®éåå®ä¹å ³ç³»</h5> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/6_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/7_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h5 id="section-10">å建æ¶é´å±çº§</h5> + +<ul> + <li>ä¿®æ¹<code class="highlighter-rouge">KYLIN_CAL_DT</code>çè¡¨å±æ§</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/8_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<ul> + <li>ä¿®æ¹ <code class="highlighter-rouge">YEAR_BEG_DT</code> çç±»å为âå¹´â</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/9_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/10_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<ul> + <li>åçä¿®æ¹<code class="highlighter-rouge">MONTH_BEG_DT</code>å<code class="highlighter-rouge">WEEK_BEG_DT</code> ï¼å¹¶ä¸éæ©å¯¹åºçå±çº§</li> + <li>å建æ¶é´å±çº§<code class="highlighter-rouge">Calendar</code> ï¼è¯·æ³¨æè®¾ç½®å±çº§ç»æçåå顺åºä¿æä¸ºâå¹´-æ-å¨â</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/11_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h5 id="section-11">ä¿®æ¹ååææ åç§°</h5> + +<ul> + <li>ä¿®æ¹ <code class="highlighter-rouge">GMV_SUM</code> 为 âéå®é¢âï¼ä¿®æ¹<code class="highlighter-rouge">SELLER_CNT_HLL</code>为âåå®¶æ°éâ</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/12_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h5 id="section-12">å建ä¸å¡ææ (计ç®åº¦é)</h5> + +<ul> + <li>å建ä¸å¡ææ âåæ·å¹³åæ¶è´¹é¢â</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/13_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<ul> + <li>便¬¡å建ä¸å¡ææ âéå®é¢å¹´åæå¢é¿çâåâéå®é¢æåæå¢é¿çâ</li> +</ul> + +<table> + <thead> + <tr> + <th><strong>ææ åç§°</strong></th> + <th><strong>MDX 表达å¼</strong></th> + </tr> + </thead> + <tbody> + <tr> + <td>åæ·å¹³åæ¶è´¹é¢</td> + <td>[Measures].[éå®é¢]/[Measures].[åå®¶æ°é]</td> + </tr> + <tr> + <td>éå®é¢å¹´åæå¢é¿ç</td> + <td>[Measures].[éå®é¢] / SUM( ParallelPeriod( [KYLIN_CAL_DT].[Calendar-Hierarchy].[YEAR_BEG_DT], 1, [KYLIN_CAL_DT].[Calendar-Hierarchy].CurrentMember ), [Measures].[éå®é¢] ) - 1</td> + </tr> + <tr> + <td>éå®é¢æåæå¢é¿ç</td> + <td>[Measures].[éå®é¢] / SUM( ParallelPeriod( [KYLIN_CAL_DT].[Calendar-Hierarchy].[MONTH_BEG_DT], 1, [KYLIN_CAL_DT].[Calendar-Hierarchy].CurrentMember ), [Measures].[éå®é¢] ) - 1</td> + </tr> + <tr> + <td>æ»éå®é¢</td> + <td>Fixed([KYLIN_CAL_DT].[YEAR_BEG_DT], [Measures].[éå®é¢])</td> + </tr> + <tr> + <td>å ¨å¹´éå®é¢å æ¯</td> + <td>[Measures].[éå®é¢]/[Measures].[æ»éå®é¢]</td> + </tr> + </tbody> +</table> + +<h3 id="http-api--mdx-">HTTP API æµè¯ MDX æ¥è¯¢</h3> + +<p>å¦æä½ æ²¡æ ä¸ä¸ª Windows ç¯å¢ç Excelï¼å¹¶ä¸ä½ æ³æµè¯ä½ å¨ä¸ä¸æ¥æµè¯å建çä¸å¡ææ ï¼è¯·éè¿ MDX æ´é²çæ¥è¯¢ç¸å ³ç REST API æ¥éªè¯æ¥è¯¢ç»æã妿æ³ä¿®æ¹ MDX æ¥è¯¢è¯å¥ï¼è¯·ä¿®æ¹ä»¥ä¸<code class="highlighter-rouge">&lt;Statement&gt;&lt;/Statement&gt;</code> éé¢ç MDX è¯å¥ï¼å¹¶ä¸è¯·æ ¹æ®æ åµä¿®æ¹<code class="highlighter-rouge">Catalog</code>åæ®µçå¼ã</p> + +<pre><code class="language-she">curl --location --request POST 'http://localhost:7080/mdx/xmla/learn_kylin' \ +--header 'Authorization: Basic QURNSU46S1lMSU4=' \ +--header 'Connection: Keep-Alive' \ +--header 'SOAPAction: "urn:schemas-microsoft-com:xml-analysis:Execute"' \ +--header 'User-Agent: MSOLAP' \ +--header 'Content-Type: text/xml' \ +--header 'Accept: */*' \ +--header 'Cookie: JSESSIONID=22BF2B6D889F183D7F7E898D4D769398; MDXAUTH=ZUt6V1VBRE1JTjoyYTk3Zjg2NTdiNjk0NTE5NzA0NjFiN2ZjYTNkYzg2OToxNjQ2NjMxNDkw' \ +--data-raw '&lt;soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"&gt; + &lt;soap:Header&gt; + &lt;Session xmlns="urn:schemas-microsoft-com:xml-analysis" SessionId="8nblet191q"/&gt; + &lt;/soap:Header&gt; + &lt;soap:Body&gt; + &lt;Execute xmlns="urn:schemas-microsoft-com:xml-analysis"&gt; + &lt;Command&gt; + &lt;Statement&gt; +SELECT {[Measures].[éå®é¢], + [Measures].[åå®¶æ°é], + [Measures].[åæ·å¹³åæ¶è´¹é¢], + [Measures].[å ¨å¹´éå®é¢å æ¯], + [Measures].[éå®é¢å¹´åæå¢é¿ç], + [Measures].[éå®é¢æåæå¢é¿ç]} + DIMENSION PROPERTIES PARENT_UNIQUE_NAME ON COLUMNS , +NON EMPTY Hierarchize(AddCalculatedMembers({DrilldownLevel({[KYLIN_CAL_DT].[Calendar-Hierarchy].[All]})})) + DIMENSION PROPERTIES PARENT_UNIQUE_NAME ON ROWS +FROM [demo0] +CELL PROPERTIES VALUE, FORMAT_STRING, LANGUAGE, BACK_COLOR, FORE_COLOR, FONT_FLAGS + &lt;/Statement&gt; + &lt;/Command&gt; + &lt;Properties&gt; + &lt;PropertyList&gt; + &lt;Catalog&gt;demo0&lt;/Catalog&gt; + &lt;Timeout&gt;0&lt;/Timeout&gt; + &lt;Content&gt;SchemaData&lt;/Content&gt; + &lt;Format&gt;Multidimensional&lt;/Format&gt; + &lt;AxisFormat&gt;TupleFormat&lt;/AxisFormat&gt; + &lt;DbpropMsmdFlattened2&gt;false&lt;/DbpropMsmdFlattened2&gt; + &lt;SafetyOptions&gt;2&lt;/SafetyOptions&gt; + &lt;Dialect&gt;MDX&lt;/Dialect&gt; + &lt;MdxMissingMemberMode&gt;Error&lt;/MdxMissingMemberMode&gt; + &lt;DbpropMsmdOptimizeResponse&gt;9&lt;/DbpropMsmdOptimizeResponse&gt; + &lt;DbpropMsmdActivityID&gt;6C94075F-65AD-4B9E-B3EB-4536A191A6AB&lt;/DbpropMsmdActivityID&gt; + &lt;DbpropMsmdRequestID&gt;9FA20B8A-ACA0-414E-98EA-14649F20CF75&lt;/DbpropMsmdRequestID&gt; + &lt;LocaleIdentifier&gt;1033&lt;/LocaleIdentifier&gt; + &lt;DbpropMsmdMDXCompatibility&gt;1&lt;/DbpropMsmdMDXCompatibility&gt; + &lt;/PropertyList&gt; + &lt;/Properties&gt; + &lt;/Execute&gt; + &lt;/soap:Body&gt; +&lt;/soap:Envelope&gt;' +</code></pre> + +<h3 id="excel-">éè¿ Excel éè§è¡¨è®¿é®ä¸å¡ææ </h3> + +<h5 id="mdx-for-kylin-1">è¿æ¥ MDX for Kylin</h5> + +<ul> + <li>æå¼ Microsoft Excel (for Windows)</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/14_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<ul> + <li>é ç½® MDX for Kylin å°åï¼è¯·æ¿æ¢ IP_Adress ä¸ºä½ ç¬è®°æ¬ç IP å°åï¼ç¨æ·ååå¯ç ä½¿ç¨ Kylin çè´¦å·åå¯ç ã</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/15_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h5 id="section-13">éè¿æ°æ®éè§è¡¨åæéå®é¢</h5> + +<ul> + <li>é ç½®æ°æ®éè§è¡¨</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/16_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<ul> + <li>æ¥çå¹´åæéå®é¢å¢é¿ç</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/17_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<ul> + <li>æ¥çæåæéå®é¢å¢é¿ç</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/18_use_excel_to_query_kylin.cn.png" alt="" /></p> + +<h2 id="section-14">åè龿¥</h2> + +<table> + <thead> + <tr> + <th><strong>ç¼å·</strong></th> + <th><strong>龿¥</strong></th> + <th><strong>注é</strong></th> + <th><strong>产å</strong></th> + </tr> + </thead> + <tbody> + <tr> + <td>1</td> + <td>https://lists.apache.org/thread/4fkhyw1fyf0jg5cb18v7vxyqbn6vm3zv</td> + <td>Kylin 社åºåèµ·å¼åè¯ä¹å±ç讨论</td> + <td>Apache Kylin</td> + </tr> + <tr> + <td>2</td> + <td>https://mondrian.pentaho.com/documentation/mdx.php</td> + <td>Mondrian çå®ç½ææ¡£</td> + <td>Mondrian</td> + </tr> + <tr> + <td>3</td> + <td>https://docs.microsoft.com/en-us/sql/mdx/mdx-syntax-elements-mdx</td> + <td>SSAS å ³äº MDX æ¥è¯¢çè¯æ³è§è</td> + <td>Microsoft SSAS</td> + </tr> + <tr> + <td>4</td> + <td>https://wiki.smartbi.com.cn/pages/viewpage.action?pageId=76692713</td> + <td>SmartBI å ³äº MDX åºæ¬æ¦å¿µçä»ç»</td> + <td>SmartBI</td> + </tr> + <tr> + <td>5</td> + <td>https://dba.stackexchange.com/questions/138311/good-example-of-mdx-vs-sql-for-analytical-queries</td> + <td>å¯¹æ¯ SQL å MDX</td> + <td>N/A</td> + </tr> + <tr> + <td>6</td> + <td>https://kyligence.io/blog/opportunities-for-ssas-in-the-cloud/</td> + <td>Kyligence MDX ææ¯å客</td> + <td>Kyligence</td> + </tr> + <tr> + <td>7</td> + <td>https://kyligence.io/blog/semantic-layer-the-bi-trend-you-dont-want-to-miss-in-2020/</td> + <td>Kyligence MDX ææ¯å客</td> + <td>Kyligence</td> + </tr> + <tr> + <td>8</td> + <td>https://docs.kyligence.io/books/mdx/v1.3/zh-cn/index.html</td> + <td>Kyligence MDX ç¨æ·æå</td> + <td>Kyligence</td> + </tr> + <tr> + <td>9</td> + <td>https://medium.com/airbnb-engineering/how-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70</td> + <td>Airbnb Tech Blog</td> + <td>Airbnb Minerva</td> + </tr> + </tbody> +</table> + +</description> + <pubDate>Thu, 31 Mar 2022 04:00:00 -0700</pubDate> + <link>http://kylin.apache.org/cn_blog/2022/03/31/how-to-use-excel-to-query-kylin/</link> + <guid isPermaLink="true">http://kylin.apache.org/cn_blog/2022/03/31/how-to-use-excel-to-query-kylin/</guid> + + + <category>cn_blog</category> + + </item> + + <item> + <title>How to use Excel to query Kylin? MDX for Kylin!</title> + <description><h2 id="abstract"><strong>Abstract</strong></h2> + +<p>During the <a href="https://lists.apache.org/thread/4fkhyw1fyf0jg5cb18v7vxyqbn6vm3zv">Kylin community discussion</a> at the beginning of this year, we talked about the positioning of multidimensional databases and the idea of building a Kylin-based business semantic layer. After some development efforts, we are delighted to announce the beta release of the <strong>MDX</strong> <strong>for Kylin</strong> <strong>, an MDX query engine for Apache Kylin</strong> to allow Kylin users to use <strong>Excel</strong> for data analysis.</p> + +<h4 id="target-audiences"><strong>Target audiences</strong></h4> + +<ul> + <li> + <p>Kylin users who are not familiar with <strong>MDX</strong></p> + </li> + <li> + <p>Data engineers who are interested in building a <strong>metrics platform</strong> based on Kylin</p> + </li> + <li> + <p>Data analysts who are interested in massive data analysis with <strong>Excel</strong></p> + </li> +</ul> + +<h4 id="what-you-will-learn"><strong>What you will learn</strong></h4> + +<ul> + <li> + <p>Basic concepts of MDX and MDX for Kylin</p> + </li> + <li> + <p>Quickstart tutorial for MDX for Kylin</p> + </li> + <li> + <p>Demonstration of how to use MDX for Kylin to define complex business metrics</p> + </li> +</ul> + +<h2 id="why-kylin-need-mdx"><strong>Why Kylin need MDX?</strong></h2> + +<h3 id="multidimensional-database-and-business-semantic-layer"><strong>Multidimensional database and business semantic layer</strong></h3> + +<p>The primary difference between multidimensional databases and relational databases lies in business semantics. As the must-have skill of data analysts, SQL (Structured Query Language) is extremely expressive, but if we are talking in the context of âevery professional will be an analystâ, it is still too complex for non-technical users. For them, data lakes and data warehouses are like dark rooms that hold a huge amount of data; they cannot see, understand, or use the data for lack of the fundamental knowledge of databases and SQL syntax.</p> + +<p>How to make data lakes and data warehouses <strong>âeasyâ</strong> for a non-technical user to use? One solution is to introduce a more user-friendly <strong>ârelational data model - multidimensional data modelâ</strong>. If relational models are to provide a technique-oriented description of the data, multidimensional models intend to provide a business-oriented description of the data. In multidimensional databases, measures correspond to the business metrics that everyone is familiar with. Measures provide the analytic perspective to check and compare these business metrics. For example, it is like comparing the KPIs between this month and last month, or the performance of different business departments. By mapping the relational model to a multidimensional model, we add a business semantic layer on top of the technical data, thus helping non-technical users understand, explore, and use data.</p> + +<p>In Kylin Roadmap, support to multidimensional query languages (such as MDX and DAX) is an important part, as we aim to enhance the business semantic capability of Kylin as a multi-dimensional database. Users can use MDX to convert the Kylin data model into business-friendly language, so they can perform multidimensional analysis with Excel, Tableau and other BI tools and understand the business values from their data.</p> + +<h3 id="build-a-business-metrics-platform-with-mdx"><strong>Build a business metrics platform with MDX</strong></h3> + +<p>When building complex business metrics, MDX provides the following advantages if compared to SQL:</p> + +<ol> + <li><strong>Better support for complex analysis scenarios</strong>, such as semi-accumulation, many-to-many, and time window analysis;</li> + <li><strong>More BI support</strong>: âKylin + MDXâ can be exposed as relational database tables through the SQL interface, or XMLA-compliant data source with business semantics. It allows MDX queries and integration with Excel and other BI tools through the XMLA protocol;</li> + <li><strong>Flexible defining of MDX semantic model based on Kylin data model</strong>, it will convert the underlying data structure into a business-friendly language and add business value to data. With MDX model, we offer users a unified business semantic layer, they no longer need to worry about the underlying technology or implementation complexity when analyzing data. For more information, see <em><a href="https://lists.apache.org/thread/4fkhyw1fyf0jg5cb18v7vxyqbn6vm3zv">The future of Apache Kylin</a></em>, <em><a href="https://kyligence.io/blog/opportunities-for-ssas-in-the-cloud/">SSAS Disadvantages: Opportunities for SSAS in the Cloud Era</a></em>, and <em><a href="https://kyligence.io/blog/semantic-layer-the-bi-trend-you-dont-want-to-miss-in-2020/">Semantic Layer: The BI Trend You Donât Want to Miss</a>**.</em></li> +</ol> + +<h3 id="mdx-overview"><strong>MDX Overview</strong></h3> + +<h4 id="what-is-mdx"><strong>What is MDX?</strong></h4> + +<p>MDX (Multi Dimensional eXpression) is a query language for OLAP Cube. It was first introduced by Microsoft in 1997 as part of the OLEDB for OLAP specification and later integrated into SSAS. Since then, it has been widely adopted by OLAP databases.</p> + +<p>MDX is similar to SQL in many ways and also offers some SQL features though maybe not as intuitive or effective as SQL. For example, you can include SELECT, FROM, or WHERE clause in your MDX queries. But it is not an extension of SQL. You can use these keywords to dig into specific parts of the Cube.</p> + +<p><a href="https://docs.microsoft.com/en-us/sql/mdx/mdx-syntax-elements-mdx?view=sql-server-ver15">MDX query syntax </a>are as follows:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">select</span> <span class="o">&lt;</span><span class="n">axis_specification</span><span class="o">&gt;</span><span class="p">[,</span> <span class="o">&lt;</span><span class="n">axis_specification</span><span class="o">&gt;</span><span class="p">]</span> + <span class="k">from</span> <span class="o">&lt;</span><span class="n">cube_specification</span><span class="o">&gt;</span> + <span class="k">where</span> <span class="o">&lt;</span><span class="n">slicer_specification</span><span class="o">&gt;</span> +</code></pre> +</div> + +<h4 id="key-concepts-of-mdx"><strong>Key concepts of MDX</strong></h4> + +<p>Please learn some basic MDX concepts before we continue.</p> + +<ol> + <li>Dimensions, Levels, Members, and Measures</li> + <li>Cell, Tuple, and Set</li> + <li>Query Axis and Slicer Axis</li> +</ol> + +<p>For detailed information about these concepts, see <a href="https://docs.microsoft.com/en-us/sql/mdx/mdx-syntax-elements-mdx?view=sql-server-ver15">MDX Syntax Elements (MDX)</a>.</p> + +<h4 id="comparison-of-mdx-and-sql"><strong>Comparison of MDX and SQL</strong></h4> + +<p>The query objects are different. MDX is to query the cube, with data already joined and aggregated, so users neednât specify the join relation when querying. SQL is to query a table with detailed records. Users need to specify the join relation among the tables when querying.</p> + +<p><a href="https://dba.stackexchange.com/questions/138311/good-example-of-mdx-vs-sql-for-analytical-queries">Another difference is the query result</a>. SQL returns a 2d data subset, while MDX returns the cubes.</p> + +<h3 id="mdx-for-kylin-overview"><strong>MDX for Kylin</strong> <strong>Overview</strong></h3> + +<h4 id="what-is-mdx-for-kylin"><strong>What is</strong> <strong>MDX for Kylin</strong>?</h4> + +<p><strong>MDX for Kylin</strong> is an MDX query engine which developed based on <strong>Mondrian</strong>, contributed by <strong>Kyligence,</strong> and with <strong>Apache Kylin</strong> as data source. Like Microsoft SSAS, MDX for Kylin can also integrate many data analysis tools, including Microsoft Excel and Tableau, to provide a better user experience for big data analysis.</p> + +<h4 id="how-to-create-business-metrics"><strong>How</strong> <strong>to create business metrics</strong></h4> + +<h5 id="atomic-metrics-and-business-metrics"><strong>Atomic metrics and business metrics</strong></h5> + +<p>In Kylin Cube, we will perform certain aggregate calculations (such as Sum/Max/Min/Count/Count Distinct, exclude TopN) on a single column when creating measures, and the measures created are called atomic metrics.</p> + +<p>In actual business scenarios, we can run complex calculations based on these atomic metrics to create composite metrics with business implications, and these metrics are called business metrics.</p> + +<h5 id="hierarchy-calculated-measure-and-namedset"><strong>Hierarchy, Calculated Measure, and NamedSet</strong></h5> + +<p><strong>Hierarchy:</strong> Hierarchies are collections of dimension-based hierarchies that can empower data analysts with advanced analytical capabilities. For example, you can create a time hierarchy with year, quarter, month, week, and day as its hierarchy. Then data analysts can do a YOY analysis on the sales volume, or dig into the âQuarter &gt; Month &gt; Week &gt; Dayâ hierarchy for more detailed analysis.</p> + +<p><strong>Calculated Measure:</strong> Calculated Measure are metrics/indexes acquired by running composite computing on the <strong>atomic metrics</strong> with MDX expressions. We mainly use calculated measures to create <strong>business metrics</strong>.</p> + +<p><strong>NamedSet:</strong> Namedset is for the scenario when you need to reuse a set of members in MDX for Kylin. A NamedSet uses specified expressions to get the set members. It can be placed directly on the axis or used in expressions of Calculated Measure for or other Namedset.</p> + +<h4 id="dataset-as-semantic-model"><strong>Dataset as</strong> <strong>semantic model</strong></h4> + +<p>In Kylin 4, we create a data model based on the relationship among tables, and define different dimensions and measures on the Cube. These measures are <strong>atomic metrics</strong>.</p> + +<p>In MDX for Kylin, we join related Kylin Cubes to create datasets and create <strong>business metrics</strong> based on atomic metrics.</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/1_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h4 id="process-of-calculating"><strong>Process of calculating</strong></h4> + +<p>The client(BI/Excel) sends an MDX query to MDX for Kylin, which will then be parsed into SQL and sent to Kylin. After that, Kylin will answer the SQL query based on the pre-computed Cuboid and return the result to MDX for Kylin. Then, MDX for Kylin will do some derived metrics calculation, and return the multidimensional data results to the client.</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/2_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h4 id="summary"><strong>Summary</strong></h4> + +<p><strong>MDX for</strong> <strong>Kylin</strong> supports MDX interface enhancing the semantic capability and creates a unified data analysis and management user experience. Now users can better leverage the value of data. The figure below shows the process of how raw data is processed into business metrics.</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/3_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h3 id="technical-advantages-of-mdx-for-kylin"><strong>Technical advantages of</strong> <strong>MDX for Kylin</strong></h3> + +<p>If compared with other open-source MDX query engines, MDX for Kylin has the following advantages:</p> + +<ul> + <li> + <p>Better support to BIs (Excel/Tableau/Power BI, etc.)and compliance with XMLA protocol</p> + </li> + <li> + <p>Optimize the MDX Query for BIs</p> + </li> + <li> + <p>Accelerate MDX queries with Kylinâs pre-computing capability</p> + </li> + <li> + <p>Easy-to-use interface for metrics definition and management</p> + </li> +</ul> + +<h2 id="quick-start-with-docker"><strong>Quick start with Docker</strong></h2> + +<h4 id="test-environment"><strong>Test environment</strong></h4> + +<ul> + <li>MacBook Pro: Docker Desktop (latest version)</li> + <li>Windows 10 virtual machine: Microsoft Excel (for Windows)</li> +</ul> + +<h4 id="start-the-container"><strong>Start the container</strong></h4> + +<pre><code class="language-she">docker run -d \ + -m 8g \ + -p 7070:7070 \ + -p 7080:7080 \ + -p 8088:8088 \ + -p 50070:50070 \ + -p 8032:8032 \ + -p 8042:8042 \ + -p 2181:2181 \ + --name kylin-4.0.1 \ + apachekylin/apache-kylin-standalone:kylin-4.0.1-mondrian +</code></pre> + +<h4 id="environment-check">Environment Check</h4> + +<p>Wait for a few minutes, then check if you can visit the web UI of HDFS, YARN, Kylin, and MDX for Kylin.</p> + +<table> + <thead> + <tr> + <th><strong>Components</strong></th> + <th><strong>Web UI</strong></th> + </tr> + </thead> + <tbody> + <tr> + <td>HDFS</td> + <td>http://localhost:50070/dfshealth.html</td> + </tr> + <tr> + <td>YARN</td> + <td>http://localhost:8088/cluster</td> + </tr> + <tr> + <td>Kylin</td> + <td>http://localhost:7070/kylin</td> + </tr> + <tr> + <td>MDX for Kylin</td> + <td>http://localhost:7080/overview</td> + </tr> + </tbody> +</table> + +<h4 id="build-a-sample-cube">Build a sample Cube</h4> + +<p>In this tutorial, we will use Kylinâs built-in sample Cube: <code class="highlighter-rouge">kylin_sales_cube</code></p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/4_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h4 id="create-a-mdx-dataset">Create a MDX dataset</h4> + +<h5 id="log-in-to-mdx-for-kylin"><strong>Log in to</strong> <strong>MDX for Kylin</strong></h5> + +<p>Log in to MDX for Kylin through the web UI: http://localhost:7080. The default account/password is <strong>ADMIN/KYLIN</strong>, the same as Kylin.</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/5_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h5 id="define-the-dataset-and-relations">Define the dataset and relations</h5> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/6_use_excel_to_query_kylin.en.png" alt="" /></p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/7_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h5 id="create-a-time-hierarchy"><strong>Create a time hierarchy</strong></h5> + +<ul> + <li>Configure the dimension table <code class="highlighter-rouge">KYLIN_CAL_DT</code>.</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/8_use_excel_to_query_kylin.en.png" alt="" /></p> + +<ul> + <li>Configure the column <code class="highlighter-rouge">YEAR_BEG_DT</code>, and set <strong>Type</strong> to <strong>Year</strong>.</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/9_use_excel_to_query_kylin.en.png" alt="" /></p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/10_use_excel_to_query_kylin.en.png" alt="" /></p> + +<ul> + <li>Configure <code class="highlighter-rouge">MONTH_BEG_DT</code>and <code class="highlighter-rouge">WEEK_BEG_DT</code>, set them to the correspondent hierarchy.</li> + <li>Create a time hierarchy <code class="highlighter-rouge">Calendar</code>. Please be noted the time hierarchy should be in a âyear-month-weekâ order.</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/11_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h5 id="rename-the-atomic-metrics"><strong>Rename the atomic metrics</strong></h5> + +<p>Rename the atomic metric <code class="highlighter-rouge">GMV_SUM</code> to some names with business implications. In this tutorial, we named it as <strong>Sales volume</strong>, and renamed <code class="highlighter-rouge">SELLER_CNT_HLL</code> as <strong>Retailer numbers</strong>.</p> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/12_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h5 id="create-business-metrics-calculated-measures"><strong>Create business metrics (calculated measures)</strong></h5> + +<ul> + <li>Create the business metric <strong>Av</strong><strong>erage sales volume of</strong> <strong>retailers</strong>.</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/13_use_excel_to_query_kylin.en.png" alt="" /></p> + +<ul> + <li>Build âYoY growth rate of sales volumeâ and âMoM growth rate of sales volumeâ sequentially.</li> +</ul> + +<table> + <thead> + <tr> + <th>Metrics</th> + <th><strong>MDX Expression</strong></th> + </tr> + </thead> + <tbody> + <tr> + <td>Average sales volume of retailers</td> + <td>[Measures].[Sales volume]/[Measures].[Retailer number]</td> + </tr> + <tr> + <td>YoY growth rate of sales volume</td> + <td>[Measures].[Sales volume] / SUM( ParallelPeriod( [KYLIN_CAL_DT].[Calendar-Hierarchy].[YEAR_BEG_DT], 1, [KYLIN_CAL_DT].[Calendar-Hierarchy].CurrentMember ), [Measures].[Sales volume] ) - 1</td> + </tr> + <tr> + <td>MoM growth rate of sales volume</td> + <td>[Measures].[Sales volume] / SUM( ParallelPeriod( [KYLIN_CAL_DT].[Calendar-Hierarchy].[MONTH_BEG_DT], 1, [KYLIN_CAL_DT].[Calendar-Hierarchy].CurrentMember ), [Measures].[Sales volume] ) - 1</td> + </tr> + <tr> + <td>Total sales volume</td> + <td>Fixed([KYLIN_CAL_DT].[YEAR_BEG_DT], [Measures].[Sales volume])</td> + </tr> + <tr> + <td>Proportion in total annual sales volume</td> + <td>[Measures].[Sales volume]/[Measures].[Total sales volume]</td> + </tr> + </tbody> +</table> + +<h4 id="access-business-metrics-through-excel-pivot-tables"><strong>Access business metrics through Excel pivot tables</strong></h4> + +<h5 id="connect-mdx-for-kylin"><strong>Connect</strong> <strong>MDX for Kylin</strong><strong>!</strong></h5> + +<ul> + <li>Open Microsoft Excel (for Windows)</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/14_use_excel_to_query_kylin.en.png" alt="" /></p> + +<ul> + <li>Configure the MDX for Kylin server address. Please update the IP_Adress with your IP address, and use Kylinâs account and password(ADMIN/KYLIN in this case).</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/15_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h5 id="check-sales-volume-with-pivot-tables">Check sales volume with pivot tables</h5> + +<ul> + <li>Configure pivot table</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/16_use_excel_to_query_kylin.en.png" alt="" /></p> + +<ul> + <li>Check YoY growth rate of sales volume</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/17_use_excel_to_query_kylin.en.png" alt="" /></p> + +<ul> + <li>Check MoM growth rate of sales volume</li> +</ul> + +<p><img src="/images/blog/how_to_use_excel_to_query_kylin/18_use_excel_to_query_kylin.en.png" alt="" /></p> + +<h4 id="call-api-to-query-mdx-for-kylin">Call API to query MDX for Kylin</h4> + +<p>If you do NOT have a windows version Excel, you can also use REST API to test the business metrics just created. Note: please change the variables in the <code class="highlighter-rouge">&lt;Statement&gt;</code> <code class="highlighter-rouge">&lt;/Statement&gt;</code> section based on your setting and update the value of <code class="highlighter-rouge">Catalog</code> if needed.</p> + +<pre><code class="language-she">curl --location --request POST 'http://localhost:7080/mdx/xmla/learn_kylin' \ +--header 'Authorization: Basic QURNSU46S1lMSU4=' \ +--header 'Connection: Keep-Alive' \ +--header 'SOAPAction: "urn:schemas-microsoft-com:xml-analysis:Execute"' \ +--header 'User-Agent: MSOLAP' \ +--header 'Content-Type: text/xml' \ +--header 'Accept: */*' \ +--header 'Cookie: JSESSIONID=22BF2B6D889F183D7F7E898D4D769398; MDXAUTH=ZUt6V1VBRE1JTjoyYTk3Zjg2NTdiNjk0NTE5NzA0NjFiN2ZjYTNkYzg2OToxNjQ2NjMxNDkw' \ +--data-raw '&lt;soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"&gt; + &lt;soap:Header&gt; + &lt;Session xmlns="urn:schemas-microsoft-com:xml-analysis" SessionId="8nblet191q"/&gt; + &lt;/soap:Header&gt; + &lt;soap:Body&gt; + &lt;Execute xmlns="urn:schemas-microsoft-com:xml-analysis"&gt; + &lt;Command&gt; + &lt;Statement&gt; +SELECT {[Measures].[Sales volume], + [Measures].[Retailer number], + [Measures].[Average sales volume of retailers], + [Measures].[Proportion in total annual sales volume], + [Measures].[YoY growth rate of sales volume], + [Measures].[MoM growth rate of sales volume]} + DIMENSION PROPERTIES PARENT_UNIQUE_NAME ON COLUMNS , +NON EMPTY Hierarchize(AddCalculatedMembers({DrilldownLevel({[KYLIN_CAL_DT].[Calendar-Hierarchy].[All]})})) + DIMENSION PROPERTIES PARENT_UNIQUE_NAME ON ROWS +FROM [demo0] +CELL PROPERTIES VALUE, FORMAT_STRING, LANGUAGE, BACK_COLOR, FORE_COLOR, FONT_FLAGS + &lt;/Statement&gt; + &lt;/Command&gt; + &lt;Properties&gt; + &lt;PropertyList&gt; + &lt;Catalog&gt;demo0&lt;/Catalog&gt; + &lt;Timeout&gt;0&lt;/Timeout&gt; + &lt;Content&gt;SchemaData&lt;/Content&gt; + &lt;Format&gt;Multidimensional&lt;/Format&gt; + &lt;AxisFormat&gt;TupleFormat&lt;/AxisFormat&gt; + &lt;DbpropMsmdFlattened2&gt;false&lt;/DbpropMsmdFlattened2&gt; + &lt;SafetyOptions&gt;2&lt;/SafetyOptions&gt; + &lt;Dialect&gt;MDX&lt;/Dialect&gt; + &lt;MdxMissingMemberMode&gt;Error&lt;/MdxMissingMemberMode&gt; + &lt;DbpropMsmdOptimizeResponse&gt;9&lt;/DbpropMsmdOptimizeResponse&gt; + &lt;DbpropMsmdActivityID&gt;6C94075F-65AD-4B9E-B3EB-4536A191A6AB&lt;/DbpropMsmdActivityID&gt; + &lt;DbpropMsmdRequestID&gt;9FA20B8A-ACA0-414E-98EA-14649F20CF75&lt;/DbpropMsmdRequestID&gt; + &lt;LocaleIdentifier&gt;1033&lt;/LocaleIdentifier&gt; + &lt;DbpropMsmdMDXCompatibility&gt;1&lt;/DbpropMsmdMDXCompatibility&gt; + &lt;/PropertyList&gt; + &lt;/Properties&gt; + &lt;/Execute&gt; + &lt;/soap:Body&gt; +&lt;/soap:Envelope&gt;' +</code></pre> + +<h2 id="contact-us">Contact us</h2> + +<p>If you want to check the official documentation, please check the manual : https://kyligence.github.io/mdx-kylin/en. For developers who want to contribute, please check our Github page : https://github.com/Kyligence/mdx-kylin .</p> + +<p>Feel free to leave your suggestion, ask a question or report a bug by referring https://kyligence.github.io/mdx-kylin/en/contact/ .</p> +</description> + <pubDate>Thu, 31 Mar 2022 04:00:00 -0700</pubDate> + <link>http://kylin.apache.org/blog/2022/03/31/how-to-use-excel-to-query-kylin/</link> + <guid isPermaLink="true">http://kylin.apache.org/blog/2022/03/31/how-to-use-excel-to-query-kylin/</guid> + + + <category>blog</category> + + </item> + + <item> <title>宿ï¼Kylin 4 ç°å·²æ¯æ AWS Glue Catalog</title> <description><h2 id="emr--kylin--glue-">为ä»ä¹å¨ EMR é¨ç½² Kylin éè¦æ¯æ Glue ï¼</h2> @@ -864,75 +1704,6 @@ Kylin 4.0 对æå»ºåæ¥ </item> <item> - <title>Kylin4 äºä¸æ§è½ä¼åï¼æ¬å°ç¼ååè½¯äº²åæ§è°åº¦</title> - <description><h2 id="section">01 èæ¯ä»ç»</h2> -<p>æ¥åï¼Apache Kylin 社åºåå¸äºå ¨æ°æ¶æç Kylin 4.0ãKylin 4.0 çæ¶ææ¯æåå¨å计ç®å离ï¼è¿ä½¿å¾ kylin ç¨æ·å¯ä»¥éåæ´å çµæ´»ã计ç®èµæºå¯ä»¥å¼¹æ§ä¼¸ç¼©çäºä¸é¨ç½²æ¹å¼æ¥è¿è¡ Kylin 4.0ãåå©äºä¸çåºç¡è®¾æ½ï¼ç¨æ·å¯ä»¥éæ©ä½¿ç¨ä¾¿å®ä¸å¯é ç对象å卿¥å¨å cube æ°æ®ï¼æ¯å¦ S3 çãä¸è¿å¨åå¨ä¸è®¡ç®åç¦»çæ¶æä¸ï¼æä»¬éè¦èèå°ï¼è®¡ç®èç¹éè¿ç½ç»ä»è¿ç«¯åå¨è¯»åæ°æ®ä»ç¶æ¯ä¸ä¸ªä»£ä»·è¾å¤§çæä½ï¼å¾å¾ ä¼å¸¦æ¥æ§è½çæèã<br /> -ä¸ºäºæé« Kylin 4.0 å¨ä½¿ç¨äºä¸å¯¹è±¡åå¨ä½ä¸ºå卿¶çæ¥è¯¢æ§è½ï¼æä»¬å°è¯å¨ Kylin 4.0 çæ¥è¯¢å¼æä¸å¼å ¥æ¬å°ç¼åï¼Local Cacheï¼æºå¶ï¼å¨æ§è¡æ¥è¯¢æ¶ï¼å°ç»å¸¸ä½¿ç¨çæ°æ®ç¼å卿¬å°ç£çï¼åå°ä»è¿ç¨å¯¹è±¡åå¨ä¸æåæ°æ®å¸¦æ¥çå»¶è¿ï¼å®ç°æ´å¿«çæ¥è¯¢ååºï¼é¤æ¤ä¹å¤ï¼ä¸ºäºé¿å åæ ·çæ°æ®å¨å¤§é spark executor ä¸åæ¶ç¼å浪费ç£ç空é´ï¼å¹¶ä¸è®¡ç®èç¹å¯ä»¥æ´å¤ç仿¬å°ç¼å读åæéæ°æ®ï¼æä»¬å¼å ¥äº 软äº� �åæ§ï¼Soft Affinity ï¼çè°åº¦çç¥ï¼æè°è½¯äº²åæ§çç¥ï¼å°±æ¯éè¿æç§æ¹æ³å¨ spark executor åæ°æ®æä»¶ä¹é´å»ºç«å¯¹åºå ³ç³»ï¼ä½¿å¾åæ ·çæ°æ®å¨å¤§é¨åæ åµä¸è½å¤æ»æ¯å¨åä¸ä¸ª executor ä¸é¢è¯»åï¼ä»èæé«ç¼åçå½ä¸çã</p> - -<h2 id="section-1">02 å®ç°åç</h2> - -<h4 id="section-2">1.æ¬å°ç¼å</h4> -<p>å¨ Kylin 4.0 æ§è¡æ¥è¯¢æ¶ï¼ä¸»è¦ç»è¿ä»¥ä¸å ä¸ªé¶æ®µï¼å ¶ä¸ç¨èçº¿æ æ³¨åºäºå¯ä»¥ä½¿ç¨æ¬å°ç¼åæ¥æåæ§è½çé¶æ®µï¼</p> - -<p><img src="/images/blog/local-cache/Local_cache_stage.png" alt="" /></p> - -<ul> - <li>File list cacheï¼å¨ spark driver 端对 file status è¿è¡ç¼åã卿§è¡æ¥è¯¢æ¶ï¼spark driver éè¦è¯»åæä»¶å表ï¼è·åä¸äºæä»¶ä¿¡æ¯è¿è¡åç»çè°åº¦æ§è¡ï¼è¿éä¼å° file status ä¿¡æ¯ç¼åå°æ¬å°é¿å é¢ç¹è¯»åè¿ç¨æä»¶ç®å½ã</li> - <li>Data cacheï¼å¨ spark executor ç«¯å¯¹æ°æ®è¿è¡ç¼åãç¨æ·å¯ä»¥è®¾ç½®å°æ°æ®ç¼åå°å åææ¯ç£çï¼è¥è®¾ç½®ä¸ºç¼åå°å åï¼åéè¦éå½è°å¤§ executor memoryï¼ä¿è¯ executor æè¶³å¤çå åå¯ä»¥è¿è¡æ°æ®ç¼åï¼è¥æ¯ç¼åå°ç£çï¼éè¦ç¨æ·è®¾ç½®æ°æ®ç¼åç®å½ï¼æå¥½è®¾ç½®ä¸º SSD ç£çç®å½ã餿¤ä¹å¤ï¼ç¼åæ°æ®çæå¤§å®¹éãå¤ä»½æ°éçåå¯ç±ç¨æ·é ç½®è°æ´ã</li> -</ul> - -<p>åºäºä»¥ä¸è®¾è®¡ï¼å¨ Kylin 4.0 çæ¥è¯¢å¼æ sparder ç driver 端å executor 端åå«åä¸åç±»åçç¼åï¼åºæ¬æ¶æå¦ä¸ï¼</p> - -<p><img src="/images/blog/local-cache/kylin4_local_cache.png" alt="" /></p> - -<h4 id="section-3">2.è½¯äº²åæ§è°åº¦</h4> -<p>å¨ executor 端å data cache æ¶ï¼å¦æå¨ææç executor ä¸é½ç¼åå ¨é¨çæ°æ®ï¼é£ä¹ç¼åæ°æ®ç大å°å°ä¼é常å¯è§ï¼æå¤§ç浪费ç£ç空é´ï¼åæ¶ä¹å®¹æå¯¼è´ç¼åæ°æ®è¢«é¢ç¹æ¸ çãä¸ºäºæå¤§å spark executor çç¼åå½ä¸çï¼spark driver éè¦å°å䏿件ç task å¨èµæºæ¡ä»¶æ»¡è¶³çæ åµä¸å°½å¯è½è°åº¦å°åæ ·ç executorï¼è¿æ ·å¯ä»¥ä¿è¯ç¸åæä»¶çæ°æ®è½å¤ç¼åå¨ç¹å®çæä¸ªæè æå 个 executor ä¸ï¼åæ¬¡è¯»åæ¶ä¾¿å¯ä»¥éè¿ç¼åè¯»åæ°æ ®ã<br /> -为æ¤ï¼æä»¬éåæ ¹æ®æä»¶åè®¡ç® hash ä¹ååä¸ executors num 忍¡çç»ææ¥è®¡ç®ç®æ executor å表ï¼å¨å¤å°ä¸ª executor ä¸é¢åç¼åç±ç¨æ·é ç½®çç¼åå¤ä»½æ°éå³å®ï¼ä¸è¬æ åµä¸ï¼ç¼åå¤ä»½æ°éè¶å¤§ï¼å»ä¸ç¼åçæ¦çè¶é«ãå½ç®æ executor åä¸å¯è¾¾æè 没æèµæºä¾è°åº¦æ¶ï¼è°åº¦ç¨åºå°åéå° spark çéæºè°åº¦æºå¶ä¸ãè¿ç§è°åº¦æ¹å¼ä¾¿ç§°ä¸ºè½¯äº²åæ§è°åº¦çç¥ï¼å®è½ç¶ä¸è½ä¿è¯ 100% å»ä¸ç¼åï¼ä½è½å¤æææé«ç¼åå½ä� �çï¼å¨å°½é䏿失æ§è½çåæä¸é¿å full cache 浪费大éç£ç空é´ã</p> - -<h2 id="section-4">03 ç¸å ³é ç½®</h2> -<p>æ ¹æ®ä»¥ä¸åçï¼æä»¬å¨ Kylin 4.0 ä¸å®ç°äºæ¬å°ç¼å+è½¯äº²åæ§è°åº¦çåºç¡åè½ï¼å¹¶åå«åºäº ssb æ°æ®éå tpch æ°æ®éåäºæ¥è¯¢æ§è½æµè¯ã<br /> -è¿éååºå 个æ¯è¾éè¦çé 置项ä¾ç¨æ·äºè§£ï¼å®é 使ç¨çé ç½®å°å¨ç»å°¾é¾æ¥ä¸ç»åºï¼<br /> -- æ¯å¦å¼å¯è½¯äº²åæ§è°åº¦çç¥ï¼kylin.query.spark-conf.spark.kylin.soft-affinity.enabled<br /> -- æ¯å¦å¼å¯æ¬å°ç¼åï¼kylin.query.spark-conf.spark.hadoop.spark.kylin.local-cache.enabled<br /> -- Data cache çå¤ä»½æ°éï¼å³å¨å¤å°ä¸ª executor ä¸å¯¹å䏿°æ®æä»¶è¿è¡ç¼åï¼kylin.query.spark-conf.spark.kylin.soft-affinity.replications.num<br /> -- ç¼åå°å åä¸è¿æ¯æ¬å°ç®å½ï¼ç¼åå°å å设置为 BUFFï¼ç¼åå°æ¬å°è®¾ç½®ä¸º LOCALï¼kylin.query.spark-conf.spark.hadoop.alluxio.user.client.cache.store.type<br /> -- æå¤§ç¼å容éï¼kylin.query.spark-conf.spark.hadoop.alluxio.user.client.cache.size</p> - -<h2 id="section-5">04 æ§è½å¯¹æ¯</h2> -<p>æä»¬å¨ AWS EMR ç¯å¢ä¸è¿è¡äº 3 ç§åºæ¯çæ§è½æµè¯ï¼å¨ scale factor = 10çæ åµä¸ï¼å¯¹ ssb æ°æ®éè¿è¡åå¹¶åæ¥è¯¢æµè¯ãtpch æ°æ®éè¿è¡åå¹¶åæ¥è¯¢ä»¥å 4 å¹¶åæ¥è¯¢æµè¯ï¼å®éªç»åå¯¹ç §ç»åé ç½® s3 ä½ä¸ºåå¨ï¼å¨å®éªç»ä¸å¼å¯æ¬å°ç¼ååè½¯äº²åæ§è°åº¦ï¼å¯¹ç §ç»åä¸å¼å¯ã餿¤ä¹å¤ï¼æä»¬è¿å°å®éªç»ç»æä¸ç¸åç¯å¢ä¸ hdfs ä½ä¸ºå卿¶çç»æè¿è¡å¯¹æ¯ï¼ä»¥ä¾¿ç¨æ·å¯ä»¥ç´è§çæåå° æ¬å°ç¼å+è½¯äº²åæ§è°åº¦ å¯¹ä ºä¸é¨ç½² Kylin 4.0 使ç¨å¯¹è±¡åå¨ä½ä¸ºåå¨åºæ¯ä¸çä¼åææã</p> - -<p><img src="/images/blog/local-cache/local_cache_benchmark_result_ssb.png" alt="" /></p> - -<p><img src="/images/blog/local-cache/local_cache_benchmark_result_tpch1.png" alt="" /></p> - -<p><img src="/images/blog/local-cache/local_cache_benchmark_result_tpch4.png" alt="" /></p> - -<p>ä»ä»¥ä¸ç»æå¯ä»¥çåºï¼<br /> -1. å¨ ssb 10 æ°æ®éåå¹¶ååºæ¯ä¸ï¼ä½¿ç¨ s3 ä½ä¸ºå卿¶ï¼å¼å¯æ¬å°ç¼ååè½¯äº²åæ§è°åº¦è½å¤è·å¾3åå·¦å³çæ§è½æåï¼å¯ä»¥è¾¾å°ä¸ hdfs ä½ä¸ºå卿¶çç¸åæ§è½çè³è¿æ 5% å·¦å³çæåã<br /> -2. å¨ tpch 10 æ°æ®éä¸ï¼ä½¿ç¨ s3 ä½ä¸ºå卿¶ï¼æ 论æ¯åå¹¶åæ¥è¯¢è¿æ¯å¤å¹¶åæ¥è¯¢ï¼å¼å¯æ¬å°ç¼ååè½¯äº²åæ§è°åº¦åï¼åºæ¬å¨æææ¥è¯¢ä¸é½è½å¤è·å¾å¤§å¹ åº¦çæ§è½æåã</p> - -<p>ä¸è¿å¨ tpch 10 æ°æ®éç 4 å¹¶åæµè¯ä¸ç Q21 ç对æ¯ç»æä¸ï¼æä»¬è§å¯å°ï¼å¼å¯æ¬å°ç¼ååè½¯äº²åæ§è°åº¦çç»æåèæ¯åç¬ä½¿ç¨ s3 ä½ä¸ºå卿¶ææä¸éï¼è¿éå¯è½æ¯ç±äºæç§åå å¯¼è´æ²¡æéè¿ç¼åè¯»åæ°æ®ï¼æ·±å±åå 卿¤æ¬¡æµè¯ä¸æ²¡æè¿è¡è¿ä¸æ¥çåæï¼å¨åç»çä¼åè¿ç¨ä¸æä»¬ä¼éæ¥æ¹è¿ãç±äº tpch çæ¥è¯¢æ¯è¾å¤æä¸ SQL ç±»ååå¼ï¼ä¸ hdfs ä½ä¸ºå卿¶çç»æç¸æ¯ï¼ä»ç¶æé¨å sql çæ§è½ç¥æ� �ä¸è¶³ï¼ä¸è¿æ»ä½æ¥è¯´å·²ç»ä¸ hdfs çç»ææ¯è¾æ¥è¿ã<br /> -æ¬æ¬¡æ§è½æµè¯çç»ææ¯ä¸æ¬¡å¯¹ æ¬å°ç¼å+è½¯äº²åæ§è°åº¦ æ§è½æåææç忥éªè¯ï¼ä»æ»ä½ä¸æ¥çï¼æ¬å°ç¼å+è½¯äº²åæ§è°åº¦ æ 论对äºç®åæ¥è¯¢è¿æ¯å¤ææ¥è¯¢é½è½å¤è·å¾ææ¾çæ§è½æåï¼ä½æ¯å¨é«å¹¶åæ¥è¯¢åºæ¯ä¸åå¨ä¸å®çæ§è½æå¤±ã<br /> -å¦æç¨æ·ä½¿ç¨äºä¸å¯¹è±¡åå¨ä½ä¸º Kylin 4.0 çåå¨ï¼å¨å¼å¯ æ¬å°ç¼å+ è½¯äº²åæ§è°åº¦çæ åµä¸ï¼æ¯å¯ä»¥è·å¾å¾å¥½çæ§è½ä½éªçï¼è¿ä¸º Kylin 4.0 å¨äºä¸ä½¿ç¨è®¡ç®ååå¨åç¦»æ¶ææä¾äºæ§è½ä¿éã</p> - -<h2 id="section-6">05 代ç å®ç°</h2> -<p>ç±äºç®åç代ç å®ç°è¿å¤äºæ¯è¾åºç¡çé¶æ®µï¼è¿æè®¸å¤ç»èéè¦å®åï¼æ¯å¦å®ç°ä¸è´æ§åå¸ãå½ executor æ°éåçååæ¶å¦ä½å¤çå·²æ cache çï¼æä»¥ä½è è¿æªå社åºä»£ç åºæäº¤ PRï¼æ³è¦æåé¢è§çå¼åè å¯ä»¥éè¿ä¸é¢ç龿¥æ¥çæºç ï¼<br /> -<a href="https://github.com/zzcclp/kylin/commit/4e75b7fa4059dd2eaed24061fda7797fecaf2e35">Kylin4.0 æ¬å°ç¼å+è½¯äº²åæ§è°åº¦ä»£ç å®ç°</a></p> - -<h2 id="section-7">06 ç¸å ³é¾æ¥</h2> -<p>éè¿é¾æ¥å¯æ¥é æ§è½æµè¯ç»ææ°æ®åå ·ä½é ç½®ï¼<br /> -<a href="https://github.com/Kyligence/kylin-tpch/issues/9">Kylin4.0 æ¬å°ç¼å+è½¯äº²åæ§è°åº¦æµè¯</a></p> -</description> - <pubDate>Thu, 21 Oct 2021 04:00:00 -0700</pubDate> - <link>http://kylin.apache.org/cn_blog/2021/10/21/Local-Cache-and-Soft-Affinity-Scheduling/</link> - <guid isPermaLink="true">http://kylin.apache.org/cn_blog/2021/10/21/Local-Cache-and-Soft-Affinity-Scheduling/</guid> - - - <category>cn_blog</category> - - </item> - - <item> <title>Performance optimization of Kylin 4.0 in cloud -- local cache and soft affinity scheduling</title> <description><h2 id="background-introduction">01 Background Introduction</h2> <p>Recently, the Apache Kylin community released Kylin 4.0.0 with a new architecture. The architecture of Kylin 4.0 supports the separation of storage and computing, which enables kylin users to run Kylin 4.0 in a more flexible cloud deployment mode with flexible computing resources. With the cloud infrastructure, users can choose to use cheap and reliable object storage to store cube data, such as S3. However, in the architecture of separation of storage and computing, we need to consider that reading data from remote storage by computing nodes through the network is still a costly operation, which often leads to performance loss.<br /> @@ -1015,6 +1786,75 @@ If users use cloud object storage as Kyl </item> <item> + <title>Kylin4 äºä¸æ§è½ä¼åï¼æ¬å°ç¼ååè½¯äº²åæ§è°åº¦</title> + <description><h2 id="section">01 èæ¯ä»ç»</h2> +<p>æ¥åï¼Apache Kylin 社åºåå¸äºå ¨æ°æ¶æç Kylin 4.0ãKylin 4.0 çæ¶ææ¯æåå¨å计ç®å离ï¼è¿ä½¿å¾ kylin ç¨æ·å¯ä»¥éåæ´å çµæ´»ã计ç®èµæºå¯ä»¥å¼¹æ§ä¼¸ç¼©çäºä¸é¨ç½²æ¹å¼æ¥è¿è¡ Kylin 4.0ãåå©äºä¸çåºç¡è®¾æ½ï¼ç¨æ·å¯ä»¥éæ©ä½¿ç¨ä¾¿å®ä¸å¯é ç对象å卿¥å¨å cube æ°æ®ï¼æ¯å¦ S3 çãä¸è¿å¨åå¨ä¸è®¡ç®åç¦»çæ¶æä¸ï¼æä»¬éè¦èèå°ï¼è®¡ç®èç¹éè¿ç½ç»ä»è¿ç«¯åå¨è¯»åæ°æ®ä»ç¶æ¯ä¸ä¸ªä»£ä»·è¾å¤§çæä½ï¼å¾å¾ ä¼å¸¦æ¥æ§è½çæèã<br /> +ä¸ºäºæé« Kylin 4.0 å¨ä½¿ç¨äºä¸å¯¹è±¡åå¨ä½ä¸ºå卿¶çæ¥è¯¢æ§è½ï¼æä»¬å°è¯å¨ Kylin 4.0 çæ¥è¯¢å¼æä¸å¼å ¥æ¬å°ç¼åï¼Local Cacheï¼æºå¶ï¼å¨æ§è¡æ¥è¯¢æ¶ï¼å°ç»å¸¸ä½¿ç¨çæ°æ®ç¼å卿¬å°ç£çï¼åå°ä»è¿ç¨å¯¹è±¡åå¨ä¸æåæ°æ®å¸¦æ¥çå»¶è¿ï¼å®ç°æ´å¿«çæ¥è¯¢ååºï¼é¤æ¤ä¹å¤ï¼ä¸ºäºé¿å åæ ·çæ°æ®å¨å¤§é spark executor ä¸åæ¶ç¼å浪费ç£ç空é´ï¼å¹¶ä¸è®¡ç®èç¹å¯ä»¥æ´å¤ç仿¬å°ç¼å读åæéæ°æ®ï¼æä»¬å¼å ¥äº 软äº� �åæ§ï¼Soft Affinity ï¼çè°åº¦çç¥ï¼æè°è½¯äº²åæ§çç¥ï¼å°±æ¯éè¿æç§æ¹æ³å¨ spark executor åæ°æ®æä»¶ä¹é´å»ºç«å¯¹åºå ³ç³»ï¼ä½¿å¾åæ ·çæ°æ®å¨å¤§é¨åæ åµä¸è½å¤æ»æ¯å¨åä¸ä¸ª executor ä¸é¢è¯»åï¼ä»èæé«ç¼åçå½ä¸çã</p> + +<h2 id="section-1">02 å®ç°åç</h2> + +<h4 id="section-2">1.æ¬å°ç¼å</h4> +<p>å¨ Kylin 4.0 æ§è¡æ¥è¯¢æ¶ï¼ä¸»è¦ç»è¿ä»¥ä¸å ä¸ªé¶æ®µï¼å ¶ä¸ç¨èçº¿æ æ³¨åºäºå¯ä»¥ä½¿ç¨æ¬å°ç¼åæ¥æåæ§è½çé¶æ®µï¼</p> + +<p><img src="/images/blog/local-cache/Local_cache_stage.png" alt="" /></p> + +<ul> + <li>File list cacheï¼å¨ spark driver 端对 file status è¿è¡ç¼åã卿§è¡æ¥è¯¢æ¶ï¼spark driver éè¦è¯»åæä»¶å表ï¼è·åä¸äºæä»¶ä¿¡æ¯è¿è¡åç»çè°åº¦æ§è¡ï¼è¿éä¼å° file status ä¿¡æ¯ç¼åå°æ¬å°é¿å é¢ç¹è¯»åè¿ç¨æä»¶ç®å½ã</li> + <li>Data cacheï¼å¨ spark executor ç«¯å¯¹æ°æ®è¿è¡ç¼åãç¨æ·å¯ä»¥è®¾ç½®å°æ°æ®ç¼åå°å åææ¯ç£çï¼è¥è®¾ç½®ä¸ºç¼åå°å åï¼åéè¦éå½è°å¤§ executor memoryï¼ä¿è¯ executor æè¶³å¤çå åå¯ä»¥è¿è¡æ°æ®ç¼åï¼è¥æ¯ç¼åå°ç£çï¼éè¦ç¨æ·è®¾ç½®æ°æ®ç¼åç®å½ï¼æå¥½è®¾ç½®ä¸º SSD ç£çç®å½ã餿¤ä¹å¤ï¼ç¼åæ°æ®çæå¤§å®¹éãå¤ä»½æ°éçåå¯ç±ç¨æ·é ç½®è°æ´ã</li> +</ul> + +<p>åºäºä»¥ä¸è®¾è®¡ï¼å¨ Kylin 4.0 çæ¥è¯¢å¼æ sparder ç driver 端å executor 端åå«åä¸åç±»åçç¼åï¼åºæ¬æ¶æå¦ä¸ï¼</p> + +<p><img src="/images/blog/local-cache/kylin4_local_cache.png" alt="" /></p> + +<h4 id="section-3">2.è½¯äº²åæ§è°åº¦</h4> +<p>å¨ executor 端å data cache æ¶ï¼å¦æå¨ææç executor ä¸é½ç¼åå ¨é¨çæ°æ®ï¼é£ä¹ç¼åæ°æ®ç大å°å°ä¼é常å¯è§ï¼æå¤§ç浪费ç£ç空é´ï¼åæ¶ä¹å®¹æå¯¼è´ç¼åæ°æ®è¢«é¢ç¹æ¸ çãä¸ºäºæå¤§å spark executor çç¼åå½ä¸çï¼spark driver éè¦å°å䏿件ç task å¨èµæºæ¡ä»¶æ»¡è¶³çæ åµä¸å°½å¯è½è°åº¦å°åæ ·ç executorï¼è¿æ ·å¯ä»¥ä¿è¯ç¸åæä»¶çæ°æ®è½å¤ç¼åå¨ç¹å®çæä¸ªæè æå 个 executor ä¸ï¼åæ¬¡è¯»åæ¶ä¾¿å¯ä»¥éè¿ç¼åè¯»åæ°æ ®ã<br /> +为æ¤ï¼æä»¬éåæ ¹æ®æä»¶åè®¡ç® hash ä¹ååä¸ executors num 忍¡çç»ææ¥è®¡ç®ç®æ executor å表ï¼å¨å¤å°ä¸ª executor ä¸é¢åç¼åç±ç¨æ·é ç½®çç¼åå¤ä»½æ°éå³å®ï¼ä¸è¬æ åµä¸ï¼ç¼åå¤ä»½æ°éè¶å¤§ï¼å»ä¸ç¼åçæ¦çè¶é«ãå½ç®æ executor åä¸å¯è¾¾æè 没æèµæºä¾è°åº¦æ¶ï¼è°åº¦ç¨åºå°åéå° spark çéæºè°åº¦æºå¶ä¸ãè¿ç§è°åº¦æ¹å¼ä¾¿ç§°ä¸ºè½¯äº²åæ§è°åº¦çç¥ï¼å®è½ç¶ä¸è½ä¿è¯ 100% å»ä¸ç¼åï¼ä½è½å¤æææé«ç¼åå½ä� �çï¼å¨å°½é䏿失æ§è½çåæä¸é¿å full cache 浪费大éç£ç空é´ã</p> + +<h2 id="section-4">03 ç¸å ³é ç½®</h2> +<p>æ ¹æ®ä»¥ä¸åçï¼æä»¬å¨ Kylin 4.0 ä¸å®ç°äºæ¬å°ç¼å+è½¯äº²åæ§è°åº¦çåºç¡åè½ï¼å¹¶åå«åºäº ssb æ°æ®éå tpch æ°æ®éåäºæ¥è¯¢æ§è½æµè¯ã<br /> +è¿éååºå 个æ¯è¾éè¦çé 置项ä¾ç¨æ·äºè§£ï¼å®é 使ç¨çé ç½®å°å¨ç»å°¾é¾æ¥ä¸ç»åºï¼<br /> +- æ¯å¦å¼å¯è½¯äº²åæ§è°åº¦çç¥ï¼kylin.query.spark-conf.spark.kylin.soft-affinity.enabled<br /> +- æ¯å¦å¼å¯æ¬å°ç¼åï¼kylin.query.spark-conf.spark.hadoop.spark.kylin.local-cache.enabled<br /> +- Data cache çå¤ä»½æ°éï¼å³å¨å¤å°ä¸ª executor ä¸å¯¹å䏿°æ®æä»¶è¿è¡ç¼åï¼kylin.query.spark-conf.spark.kylin.soft-affinity.replications.num<br /> +- ç¼åå°å åä¸è¿æ¯æ¬å°ç®å½ï¼ç¼åå°å å设置为 BUFFï¼ç¼åå°æ¬å°è®¾ç½®ä¸º LOCALï¼kylin.query.spark-conf.spark.hadoop.alluxio.user.client.cache.store.type<br /> +- æå¤§ç¼å容éï¼kylin.query.spark-conf.spark.hadoop.alluxio.user.client.cache.size</p> +
[... 238 lines stripped ...]