This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris-website.git
The following commit(s) were added to refs/heads/master by this push: new 546e678 Add weekly 20220306 and doris-docker-cluster-install 546e678 is described below commit 546e678636fd4d129169cc2595158e949524aa0d Author: morningman <morning...@163.com> AuthorDate: Sat Mar 12 21:59:23 2022 +0800 Add weekly 20220306 and doris-docker-cluster-install --- blogs/en/doris-docker-cluster-install.md | 37 + blogs/en/weekly-20220306.md | 133 +++ .../2022-03-03-123903.png | Bin 0 -> 610456 bytes .../2022-03-04-041009.png | Bin 0 -> 840158 bytes blogs/zh-CN/doris-docker-cluster-install.md | 932 +++++++++++++++++++++ blogs/zh-CN/weekly-20220306.md | 132 +++ 6 files changed, 1234 insertions(+) diff --git a/blogs/en/doris-docker-cluster-install.md b/blogs/en/doris-docker-cluster-install.md new file mode 100644 index 0000000..d38df7c --- /dev/null +++ b/blogs/en/doris-docker-cluster-install.md @@ -0,0 +1,37 @@ +--- +{ + "title": "Apache Doris single-node (can be multi-node) Docker cluster production tutorial", + "description": "Apache Doris is a very hot and popular MPP architecture OLAP database, many students want to learn/test the use and ability of Doris, but suffer from the lack of environment or fear of the long compilation + build process, the whole process has greatly discouraged many students who are willing to try, but due to a variety of objective factors can not afford the high cost of learning, resulting in failed attempts. Therefore, three community members of Apache Doris comm [...] + "date": "2022-03-07", + "metaTitle": "Apache Doris single-node (can be multi-node) Docker cluster production tutorial", + "isArticle": true, + "language": "en", + "author": "苏奕嘉,种益,杨春东", + "layout": "Article", + "sidebar": false +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +(TODO) + +This artical only has Chinese version. If you are willing to translate it into English, please contact us by sending email to d...@doris.apache.org. We are looking forward to your contribution! + diff --git a/blogs/en/weekly-20220306.md b/blogs/en/weekly-20220306.md new file mode 100644 index 0000000..e526d12 --- /dev/null +++ b/blogs/en/weekly-20220306.md @@ -0,0 +1,133 @@ +--- +{ + "title": "[Doris Weekly] 20220221-20220306", + "description": "Doris Weekly 20220221-20220306", + "date": "2022-03-06", + "metaTitle": "[Doris Weekly] 20220221-20220306", + "isArticle": true, + "language": "en", + "author": "morningman", + "layout": "Article", + "sidebar": false +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Statistics + +A total of 44 authors have submitted 101 Commits. Thanks to the following authors for their contributions (Github IDs, in alphabetical order). + +@adonis0147, @amosbird, @aopangzi, @arthuryangcs, @awakeljw, @BiteTheDDDDt, @bridgeDream, @caiconghui, @CalvinKirs, @cambyzju, @ChPi, @daikon12, @DarvenDuan, @dataalive, @dataroaring, @GoGoWen, @HappenLee, @Henry2SS, @hf200012, @jacktengg, @kylinmac, @liaoxin01, @liutang123, @lonre, @madongz, @morningman, @mrhhsg, @Nivane, @qidaye, @steadyBoy, @SWJTU-ZhangLei, @wangbo, @wangshuo128, @wangyf0555, @weizuo93, @wunan1210, @xingtanzjr, @yangzhg, @yiguolei, @zbtzbtzbt, @zhangstar333, @zhangyif [...] + +In the last 2 weeks, 8654 new lines of code were added and 8608 lines of code were deleted. + +## Main progress + +### New features + +* https://github.com/apache/incubator-doris/pull/8188 + + The show routine load statement supports fuzzy matching of job names by "like". + +* https://github.com/apache/incubator-doris/pull/8041 + https://github.com/apache/incubator-doris/pull/8259 + + Support for creating random distribution tables. And support for converting a hash distribution table to random distribution.. + +* https://github.com/apache/incubator-doris/pull/8179 + + Support for querying iceberg external tables. + +* https://github.com/apache/incubator-doris-flink-connector/pull/12 + + Flink connector supports exactly once semantics + +### Work-in-Progess + +* https://github.com/apache/incubator-doris/pull/8212 + + Supports PB structure definition of array type. + +* https://github.com/apache/incubator-doris/pull/8279 + + Refactor typo info to support nested data types + +### Feature improvements + +* https://github.com/apache/incubator-doris-flink-connector/pull/11 + + flink connector supports BE ip fetching via multiple FEs. + +* https://github.com/apache/incubator-doris/pull/8146 + + Added FE config to support ignoring replica of slow compaction when querying. + +* https://github.com/apache/incubator-doris/pull/8229 + + Optimize performance of string comparison on BE side. + +* https://github.com/apache/incubator-doris/pull/8245 + + Support for partitions of a table to be queried normally while the restore job is in progress. + +* https://github.com/apache/incubator-doris/pull/8119 + + Optimized the performance of HashJoin in the vectorized execution engine. + +* https://github.com/apache/incubator-doris/pull/8158 + https://github.com/apache/incubator-doris/pull/8138 + + Support for more vectorized implementations of functions. + + ``` + json_array + json_object + json_quote + bitmap_and_not_count + bitmap_or + bitmap_xor + bitmap_xor_count + bitmap_and + bitmap_and_not + bitmap_subset_in_range + bitmap_subset_limit + bitmap_and_count + sub_bitmap + bitmap_has_all + bitmap_or_count + ``` + +### Bug Fixes + +* https://github.com/apache/incubator-doris/pull/8152 + + Fix `explode_json_array_string` that returns incorrect results in some cases. + +* https://github.com/apache/incubator-doris/pull/8142 + + Fix a BE crash caused by converting int columns to string columns. + +### Others + +* https://github.com/apache/incubator-doris/pull/8249 + + Support for source code compilation on ARM. + diff --git a/blogs/images/blogs/doris-docker-cluster-install/2022-03-03-123903.png b/blogs/images/blogs/doris-docker-cluster-install/2022-03-03-123903.png new file mode 100644 index 0000000..ad4ab05 Binary files /dev/null and b/blogs/images/blogs/doris-docker-cluster-install/2022-03-03-123903.png differ diff --git a/blogs/images/blogs/doris-docker-cluster-install/2022-03-04-041009.png b/blogs/images/blogs/doris-docker-cluster-install/2022-03-04-041009.png new file mode 100644 index 0000000..31ad82b Binary files /dev/null and b/blogs/images/blogs/doris-docker-cluster-install/2022-03-04-041009.png differ diff --git a/blogs/zh-CN/doris-docker-cluster-install.md b/blogs/zh-CN/doris-docker-cluster-install.md new file mode 100644 index 0000000..b974fd5 --- /dev/null +++ b/blogs/zh-CN/doris-docker-cluster-install.md @@ -0,0 +1,932 @@ +--- +{ + "title": "Apache Doris 单节点(可多节点)Docker集群制作教程", + "description": "Apache Doris是当下非常火热和流行的MPP架构OLAP数据库,很多同学想自学/测试Doris的使用和能力,但是又苦于没有环境或者畏惧冗长的编译+搭建过程,整个过程极大的劝退了很多有些尝试意愿、但又由于各种客观因素无法承担过高学习成本导致尝试失败的同学,故此Apache Doris社区三位社区同学苏奕嘉、种益、杨春东制作了三个不同设计的版本安装方式并提供下载,以此降低大家的学习门槛和提升学习/测试效率.", + "date": "2022-03-07", + "metaTitle": "Apache Doris 单节点(可多节点)Docker集群制作教程", + "isArticle": true, + "language": "zh-CN", + "author": "苏奕嘉,种益,杨春东", + "layout": "Article", + "sidebar": false +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 前言 + +Apache Doris是当下非常火热和流行的MPP架构OLAP数据库,很多同学想自学/测试Doris的使用和能力,但是又苦于没有环境或者畏惧冗长的编译+搭建过程,整个过程极大的劝退了很多有些尝试意愿、但又由于各种客观因素无法承担过高学习成本导致尝试失败的同学,故此Apache Doris社区三位社区同学苏奕嘉、种益、杨春东制作了三个不同设计的版本安装方式并提供下载,以此降低大家的学习门槛和提升学习/测试效率。 + +**重要说明:该教程提供的编译方式及运行环境都以单节点部署伪集群为目标,故性能会【大打折扣】,如想体验完整Apache Doris数据库的能力,请以完整集群部署,单节点伪集群【仅适用于学习、功能测试】所用!** + +## 版本说明 + +### 1. 极速体验版 + +#### 1.1. 优点 + +1. 超快速的部署体验(网速OK的话十五分钟内部署完毕) +2. 单节点部署 +3. 支持多环境运行:虚拟机/云服务器/支持Docker的物理机(Mac/Win/Linux) + +#### 1.2. 缺点 + +1. 数据存储是在Docker容器中,如容器如损坏,会导致数据丢失 +2. 若非干净纯净的系统环境,可能需要手动执行部分BE注册FE的命令 + +#### 1.3. 适用人群 + +学生、培训机构、体验/测试人员 + +#### 1.4. 安装建议 + +系统为纯净新系统最佳,无需任何修改即可开箱即用 + +**** + +### 2. 完全部署版 + +#### 2.1. 优点 + +1. 完整的环境部署(MySQL-Client等组件) +2. 自由的部署安排(有众多可选安装参数) +3. 无惧Docker容器损坏(最小降低损失,可极速恢复) +4. 单节点部署 +5. 支持多环境运行:虚拟机/云服务器,暂未适配物理机(后续升级版本会支持) + +#### 2.2. 缺点 + +1. 安装过程时间较长(视网速和机器性能而定) +2. 安装步骤多,代表可能故障率较高 + +#### 2.3. 适用人群 + +学生、培训机构、体验/测试人员中的持续性教学受众(数据不易丢失) + +#### 2.4. 安装建议 + +该版本建议完完全全的纯净新系统,以此降低安装故障率 + +**** + +### 3. 存算分离版 + +该版本还在制作过程中,教程及相关文档后续推出,可视为完全部署版的Plus版本。 + +**** + +## 目的 + +该教程最后成果模块提供了`各个版本`下载地址,只需在服务器拉取`不同版本shell脚本`运行即可,在`/opt/docker/doris/sbin`目录下会有`start_doris_docker.sh`和`stop_doris_docker.sh`脚本支持一键启停,同时会在一键部署的过程中将两个脚本添加至环境变量,最大程度简化单节点测试部署和启停操作。 + +**步骤过程可以忽略**,除非有定制化的一键部署Docker集群的镜像集群制作需求,大可不必照着教程再来一遍,官方已提供了下载地址,无需重复劳动。 + +## 环境 + +- 服务器:腾讯云 2C 4G 6M 一台 +- OS:CentOS 7.6 +- Docker-V:20.10.12 +- Doris-V:0.15 +- MySQL-Client-V:5.7 +- FE-Num:1 +- BE-Num:3 + +## 步骤 + +### 1. 安装Docker环境 + +1. Docker 要求 CentOS 系统的内核版本高于`3.10` ,首先查看系统内核版本是否满足 + + ```shell + uname -r + ``` + +2. 使用 `root` 权限登录系统,确保 yum 包更新到最新 + + ```shell + sudo yum update -y + ``` + +3. 假如安装过旧版本,先卸载旧版本 + + ```shell + sudo yum remove docker docker-common docker-selinux docker-engine + ``` + +4. 安装需要的软件包, yum-util 提供yum-config-manager功能,另外两个是devicemapper驱动依赖的 + + ```shell + sudo yum install -y yum-utils device-mapper-persistent-data lvm2 + ``` + +5. 设置yum源(加速yum下载速度) + + ```shell + sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo + ``` + +6. 查看所有仓库中所有docker版本,并选择特定版本安装,一般可直接安装最新版 + + ```shell + yum list docker-ce --showduplicates | sort -r + ``` + +7. 安装docker + + - 安装最新稳定版本 + + ```shell + sudo yum install docker-ce -y #安装的是最新稳定版本,因为repo中默认只开启stable仓库 + ``` + + - 安装指定版本 + + ```shell + sudo yum install <FQPN> -y + # 例如: + sudo yum install docker-ce-20.10.11.ce -y + ``` + +8. 启动并加入开机启动 + + ```shell + sudo systemctl start docker #启动docker + sudo systemctl enable docker #加入开机自启动 + ``` + +9. 查看Version,验证是否安装成功 + + ```shell + docker version + ``` + + 若出现Client和Server两部分内容,则证明安装成功 + +### 2. 容器创建及测试 + +**在创建之前,请准备好已完成编译的FE/BE文件,此教程不再赘述编译过程。** + +1. 拉取Doris编译镜像做测试 + + ```shell + # 拉取 + docker pull apache/incubator-doris:build-env-ldb-toolchain-latest + ``` + +2. 创建Doris-Docker的文件(包括元数据文件夹) + + ```shell + mkdir -p /opt/docker/doris + ``` + +3. 将编译好的FE和BE拷贝至Docker文件群内 + + ```shell + cp -r 编译好的Doris根目录/fe/ /opt/docker/doris/ + cp -r 编译好的Doris根目录/be/ /opt/docker/doris/be-01 + cp -r 编译好的Doris根目录/be/ /opt/docker/doris/be-02 + cp -r 编译好的Doris根目录/be/ /opt/docker/doris/be-03 + ``` + +4. 启动FE-Docker + + ```shell + docker run -it -p 8030:8030 -p 9030:9030 -d --name=doris-fe -v /opt/docker/doris/fe:/opt/doris/fe -v /opt/docker/doris/doris-meta:/opt/doris/doris-meta apache/incubator-doris:build-env-ldb-toolchain-latest + ``` + +5. 进入FE-Docker以及安装组件 + + ```shell + # 进入fe-docker + docker exec -ti doris-fe /bin/bash + # 安装net-tools用于查看IP + yum install net-tools -y + ``` + +6. 修改FE配置 + + ```shell + # 查看fe-docker的IPv4地址 + ifconfig + # 修改配置文件 + vim /opt/doris/fe/conf/fe.conf + # 取消priority_networks的注解,并根据Docker的网段进行配置 + priority_networks = 172.17.0.0/16 #这里要根据你Docker的IP确定 + ``` + +7. 切换Docker-JDK版本 + + ```shell + # 切换Java版本为JDK1.8,该镜像默认为JDK11 + alternatives --set java java-1.8.0-openjdk.x86_64 + alternatives --set javac java-1.8.0-openjdk.x86_64 + export JAVA_HOME=/usr/lib/jvm/java-1.8.0 + # 校验是否切换版本成功 + java -version + ``` + +8. 配置FE-Docker的环境变量 + + ```shell + # 配置环境变量 + vim /etc/profile.d/doris.sh + export DORIS_HOME=/opt/doris/fe/ + export PATH=$PATH:$DORIS_HOME/bin + # 保存并source + source /etc/profile.d/doris.sh + ``` + +9. 启动Doris-FE + + ```shell + start_fe.sh --daemon + ``` + +10. 检查FE是否启动成功 + + > 1. 检查是否启动成功,JPS命令下有没有PaloFe进程 + > + > 2. FE 进程启动后,会首先加载元数据,根据 FE 角色的不同,在日志中会看到 `transfer from UNKNOWN to MASTER/FOLLOWER/OBSERVER`。最终会看到 `thrift server started` 日志,并且可以通过 mysql 客户端连接到 FE,则表示 FE 启动成功。 + > + > 3. 也可以通过如下连接查看是否启动成功: + > `http://fe_host:fe_http_port/api/bootstrap` + > + > 如果返回: + > `{"status":"OK","msg":"Success"}` + > + > 则表示启动成功,其余情况,则可能存在问题。 + > + > 4. 外网环境访问`http://fe_host:fe_http_port` 查看是否可以访问WebUI界面,登录账号默认为root,密码为空 + > + > 注:如果在 fe.log 中查看不到启动失败的信息,也许在 fe.out 中可以看到。 + +11. 宿主机安装MySQL客户端 + + ```shell + wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.37-1.el7.x86_64.rpm-bundle.tar + tar -xvf mysql-5.7.37-1.el7.x86_64.rpm-bundle.tar + rpm -ivh mysql-community-common-5.7.37-1.el7.x86_64.rpm + rpm -ivh mysql-community-libs-5.7.37-1.el7.x86_64.rpm + rpm -ivh mysql-community-client-5.7.37-1.el7.x86_64.rpm + ``` + +12. 连接FE并修改密码 + + ```shell + mysql -h FE-Docer的IP -P 9030 -uroot + SET PASSWORD FOR 'root' = PASSWORD('your_password'); + # 也可以创建新用户 + CREATE USER 'test' IDENTIFIED BY 'test_passwd'; + ``` + + 后续链接时需要使用如下格式 + + ```shell + mysql -h FE_HOST -P9030 -uusername -ppassword + ``` + + 添加BE节点注册信息 + + ```shell + ALTER SYSTEM ADD BACKEND "host:port"; + # 其中 host 为 BE 所在节点 ip;port 为 be/conf/be.conf 中的 heartbeat_service_port,默认9050。 + ``` + +13. 启动BE-Docker + + ```shell + docker run -it -p 9061:9060 -d --name=doris-be-01 -v /opt/docker/doris/be-01:/opt/doris/be apache/incubator-doris:build-env-ldb-toolchain-latest + docker run -it -p 9062:9060 -d --name=doris-be-02 -v /opt/docker/doris/be-02:/opt/doris/be apache/incubator-doris:build-env-ldb-toolchain-latest + docker run -it -p 9063:9060 -d --name=doris-be-03 -v /opt/docker/doris/be-03:/opt/doris/be apache/incubator-doris:build-env-ldb-toolchain-latest + ``` + +14. 进入BE-Docker以及安装组件 + + ```shell + # 进入fe-docker,以01为例 + docker exec -ti doris-be-01 /bin/bash + # 安装net-tools用于查看IP + yum install net-tools -y + ``` + +15. 修改BE配置 + + ```shell + # 查看fe-docker的IPv4地址 + ifconfig + # 修改配置文件 + vim /opt/doris/be/conf/be.conf + # 取消priority_networks的注解,并根据Docker的网段进行配置 + priority_networks = 172.17.0.0/16 #这里要根据你Docker的IP确定 + ``` + +16. 配置BE-Docker的环境变量 + + ```shell + # 配置环境变量 + vim /etc/profile.d/doris.sh + export DORIS_HOME=/opt/doris/be/ + export PATH=$PATH:$DORIS_HOME/bin + # 保存并source + source /etc/profile.d/doris.sh + ``` + +17. 启动Doris-BE + + ```shell + start_be.sh --daemon + ``` + +18. 检查BE是否启动成功 + + > 1. BE 进程启动后,如果之前有数据,则可能有数分钟不等的数据索引加载时间。 + > + > 2. 如果是 BE 的第一次启动,或者该 BE 尚未加入任何集群,则 BE 日志会定期滚动 `waiting to receive first heartbeat from frontend` 字样。表示 BE 还未通过 FE 的心跳收到 Master 的地址,正在被动等待。这种错误日志,在 FE 中 ADD BACKEND 并发送心跳后,就会消失。如果在接到心跳后,又重复出现 `master client, get client from cache failed.host: , port: 0, code: 7` 字样,说明 FE 成功连接了 BE,但 BE 无法主动连接 FE。可能需要检查 BE 到 FE 的 rpc_port 的连通性。 + > + > 3. 如果 BE 已经被加入集群,日志中应该每隔 5 秒滚动来自 FE 的心跳日志:`get heartbeat, host: xx.xx.xx.xx, port: 9020, cluster id: xxxxxx`,表示心跳正常。 + > + > 4. 其次,日志中应该每隔 10 秒滚动 `finish report task success. return code: 0` 的字样,表示 BE 向 FE 的通信正常。 + > + > 5. 同时,如果有数据查询,应该能看到不停滚动的日志,并且有 `execute time is xxx` 日志,表示 BE 启动成功,并且查询正常。 + > + > 6. 也可以通过如下连接查看是否启动成功: + > `http://be_host:be_http_port/api/health` + > + > 如果返回: + > `{"status": "OK","msg": "To Be Added"}` + > + > 则表示启动成功,其余情况,则可能存在问题。 + > + > 注:如果在 be.INFO 中查看不到启动失败的信息,也许在 be.out 中可以看到。 + +19. 测试连通性 + + ```shell + # 登录FE-MySQL + mysql -h FE_HOST -P9030 -uusername -ppassword + # 执行命令查看BE运行情况。如一切正常,isAlive 列应为 true。 + SHOW PROC '/backends'; + ``` + +20. 若连通性测试成功,则循环完成其他BE节点的部署即可 + +### 3. 安装ETCD环境(若多节点Dokcer需配置|单节点可忽略) + +1. 配置Hosts文件映射 + + ```shell + vim /etc/hosts + 你本机内网IP地址 master + ``` + +2. 安装ETCD + + ```SHELL + # 安装ETCD + yum install -y etcd + # 重启ETCD + systemctl restart etcd + ``` + +3. 设置开机启动 + + ```shell + systemctl enable etcd + ``` + +4. 修改ETCD配置 + + ```shell + # 先查找本机的IP地址 + ifconfig + # 备份原始配置文件 + cp /etc/etcd/etcd.conf /etc/etcd/etcd.conf.bak + # 编辑ETCD的conf文件 + vim /etc/etcd/etcd.conf + # 修改监听客户端地址为 + ETCD_LISTEN_CLIENT_URLS="http://master:2379,http://127.0.0.1:2379,http://master:4001,http://127.0.0.1:4001" + # 修改通知客户端地址为 + ETCD_ADVERTISE_CLIENT_URLS="http://master:2379,http://master:4001" + # 保存退出 + ``` + +5. 设置ETCD网段 + + ```shell + # Flannel使用Etcd进行配置,来保证多个Flannel实例之间的配置一致性,所以需要在etcd上进行如下配置('/atomic.io/network/config'这个key与上文/etc/sysconfig/flannel中的配置项FLANNEL_ETCD_PREFIX是相对应的,错误的话启动就会出错) + etcdctl mk /atomic.io/network/config '{"Network":"172.20.0.0/16","SubnetMin":"172.20.1.0","SubnetMax":"172.20.254.0"}' + ``` + +6. 重启ETCD + + ```shell + systemctl restart etcd + ``` + +7. 测试 + + ```shell + # 查看ETCD进程是否存在 + ps -ef|grep etcd + # 查看端口使用情况,因为ETCD默认TCP:2379端口通讯 + lsof -i:2379 + # 使用get命令查看是否设置成功 + etcdctl get /atomic.io/network/config + # 若出现以下信息,则代表设置成功 + {"Network":"172.20.0.0/16","SubnetMin":"172.20.1.0","SubnetMax":"172.20.254.0"} + # 查看cluster-health + etcdctl -C http://master:4001 cluster-health + etcdctl -C http://master:2379 cluster-health + # 若出现如下信息,则代表成功 + member 8e9e05c52164694d is healthy: got healthy result from http://你IP地址:2379(和4001) + ``` + +### 4. 安装Flannel环境(若多节点Dokcer需配置|单节点可忽略) + +1. Yum安装Flannel + + ```shell + yum install -y flannel + ``` + +2. 配置Flannel + + ```shell + # 备份原始配置文件 + cp /etc/sysconfig/flanneld /etc/sysconfig/flanneld.bak + # 编辑配置文件 + vim /etc/sysconfig/flanneld + # 修改以下配置项 + FLANNEL_ETCD_ENDPOINTS="http://master:2379" + ``` + +3. 设置开机自启 + + ```shell + systemctl enable flanneld.service + ``` + +4. 启动Flannel + + ```shell + systemctl start flanneld.service + ``` + +5. 重启Docker + + ```shell + systemctl restart docker + ``` + +6. 测试 + + ```shell + # 查看Flannel进程 + ps -ef | grep flannel + ``` + +### 5. 测试及远程连接 + +可使用Navicat等远端工具连接FE,地址为部署了FE服务的单机外网IP,端口为9030,如图所示 + + + +### 6. SHELL脚本设计及开发 + +#### 6.1. 完整部署版整体设计示意图 + + + +#### 6.2. 思路梳理 + +##### 6.2.1 极速体验版(极速体验免除安装) + +1. 默认1FE 3BE安装 + +2. Docker安装(可参照**步骤1**) + +3. 拉取Docker镜像群 + + ```shell + docker pull freeoneplus/doris-fe:1.0 + docker pull freeoneplus/doris-be:1.0 + ``` + +4. 创建FE-Docker容器 + + ```shell + docker run -it -p 8030:8030 -p 9030:9030 -d --name=doris-fe freeoneplus/doris-fe:1.0 + ``` + +5. 进入FE-Docker并获取IPv4地址 + + ```shell + docker exec -it doris-fe /bin/bash + ifconfig + exit + docker exec -d doris-fe /bin/bash /opt/doris/fe/start_fe.sh --daemon + ``` + +6. 循环创建BE-Docker容器并启动BE + + ```shell + docker run -it -p 9061:9060 -d --name=doris-be-01 freeoneplus/doris-be:1.0 + docker exec -d doris-be-01 /bin/bash /opt/doris/be/start_be.sh --daemon + docker run -it -p 9062:9060 -d --name=doris-be-02 freeoneplus/doris-be:1.0 + docker exec -d doris-be-02 /bin/bash /opt/doris/be/start_be.sh --daemon + docker run -it -p 9063:9060 -d --name=doris-be-03 freeoneplus/doris-be:1.0 + docker exec -d doris-be-03 /bin/bash /opt/doris/be/start_be.sh --daemon + ``` + +7. 提示用户进行BE注册 + + ```shell + 亲爱的用户,欢迎使用 Apache Doris-极简版-Docker集群! + 接下来的文字请认真阅读: + 1. 此版本集群为极简版单节点docker集群,所有数据均挂载在Docker集群内,请谨慎修改或删除容器! + 2. 此版本预制注册三个BE节点至FE,但可能由于不同环境影响,预先注册的IP地址可能会出现错误,所以请仔细观察FE的预制IP地址:${FE-IP地址},若以上地址为172.17.0.2,则无需做任何修改即可直接使用,如果是其他数值,则需要进行链接FE进行BE注册 + 3. 您可以使用任意MySQL-Client或者MySQL工具连接FE-MySQL-Server + 若宿主机(您的虚拟机/云服务器)有MySQL-Client,则需要执行以下命令链接FE-MySQL-Server + mysql -h ${FE-IP地址} -P 9030 -uroot -p123456 + 若您使用外网机器链接FE-MySQL-Server,则需要填入以下参数,您需要提前打开9030外网端口 + url:您的服务器外网IP(虚拟机则视网络桥接方式) + port:9030 + username:root + password:123456 + 然后执行以下命令清除已注册至FE的BE节点信息 + + `以下需要逻辑处理` + 预设的三个BE地址为[172.17.0.3,172.17.0.4,172.17.0.5] + 该地址应为FE-IP地址最后一位自增3,所以如果预设错误,需要给出删除语句和增添语句 + 比如FE-IP为 172.17.0.4,则需要给出删除[172.17.0.3,172.17.0.4]两个BE节点的语句 + ALTER SYSTEM DECOMMISSION BACKEND "${FE-IP地址}:9050"; + 然后再给出新增的两个节点的IP[172.17.0.6,172.17.0.7]注册语句 + ALTER SYSTEM ADD BACKEND "${FE-IP地址}:9050"; + `以上需要逻辑处理` + + 感谢您的安装和使用Apache Doris! + 感谢您为开源世界作出的一份贡献! + 如有问题请打开地址:doris.freeoneplus.com + 扫描二维码添加Apache Doris社区微信群获取答疑~ + ``` + +##### 6.2.2 完整部署版(数据落盘无惧丢失) + +1. 校验脚本执行口令,防止误操作 + + - 输出一段文字说明 + - 等待接收`Doris`这五个字母,成功则继续,未成功则终止 + +2. 依次询问参数配置设置,接收参数,可参考的有: + + > 1. 是否默认配置安装(Y/N) + > 2. BE数量(默认为3) + > 3. root密码(默认为空) + > 4. 操作员账户名称(默认无) + > 5. 操作员账户密码(默认无) + > 6. FE-Http-Port端口(默认8030) + > 7. FE-MySQL-Cli-Port端口(默认9030) + +3. 创建宿主机资源目录并进入 + + ```shell + mkdir -p /opt/docker/doris/ + cd /opt/docker/doris/ + ``` + +4. 拉取编译好的文件包至上述目录(当前版本为Apache Doris-1.0.0 bate测试版) + + ```shell + wget https://jiafeng2022.oss-cn-beijing.aliyuncs.com/doris-1.0.0-jdk8-20220301.tar.gz + ``` + +5. 解压文件包 + + ``` + tar -zxvf /opt/docker/doris/apache-doris-install.tar.gz + ``` + +6. 根据传参的BE数量循环复制BE目录,以默认数量为样例,命令执行为 + + ```shell + cp -r /opt/docker/doris/be /opt/docker/doris/be-01 + cp -r /opt/docker/doris/be /opt/docker/doris/be-02 + cp -r /opt/docker/doris/be /opt/docker/doris/be-03 + ``` + +7. 监测Docker是否安装 + + ```shell + docker version + ``` + +8. 如果已安装则跳过,未安装则安装Docker + + ```shell + # 监测内核版本,若小于3.10则终止安装并通知失败,告知失败原因 + uname -r + # 如果大于3.10则开始安装,依次执行以下命令 + sudo yum update -y + sudo yum install -y yum-utils device-mapper-persistent-data lvm2 + sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo + sudo yum install docker-ce -y + sudo systemctl start docker + sudo systemctl enable docker + # 执行结束,监测执行是否都已成功 + docker version + ``` + +9. 监测MySQL-Client是否已安装 + + ```shell + mysql --version + ``` + +10. 如果已安装则跳过,未安装则安装MySQL-Client + + ```shell + mkdir -p /opt/software + cd /opt/software + wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.37-1.el7.x86_64.rpm-bundle.tar + tar -xvf mysql-5.7.37-1.el7.x86_64.rpm-bundle.tar + rpm -ivh mysql-community-common-5.7.37-1.el7.x86_64.rpm + rpm -ivh mysql-community-libs-5.7.37-1.el7.x86_64.rpm + rpm -ivh mysql-community-client-5.7.37-1.el7.x86_64.rpm + ``` + +11. 拉取Doris编译镜像为基础环境镜像 + + ```shell + docker pull apache/incubator-doris:build-env-ldb-toolchain-latest + ``` + +12. 制作FE容器 + + - 构建FE容器 + + ```shell + docker run -it -p 8030:8030 -p 9030:9030 -d --name=doris-fe -v /opt/docker/doris/fe:/opt/doris/fe -v /opt/docker/doris/doris-meta:/opt/doris/doris-meta apache/incubator-doris:build-env-ldb-toolchain-latest + ``` + + - 进入容器 + + ```shell + docker exec -ti doris-fe /bin/bash + ``` + + - 修改FE配置文件 + + ```shell + vim /opt/doris/fe/conf/fe.conf + # 如FE两个对外端口都是默认值,则无需修改,若有改变,则改变该值 + http_port = 8030 + query_port = 9030 + # 修改网段 + priority_networks = 172.17.0.0/16 + ``` + + - 切换JDK版本 + + ```shell + # 切换Java版本为JDK1.8,该镜像默认为JDK11 + alternatives --set java java-1.8.0-openjdk.x86_64 + alternatives --set javac java-1.8.0-openjdk.x86_64 + export JAVA_HOME=/usr/lib/jvm/java-1.8.0 + ``` + + - 配置Doris环境变量 + + ```shell + # 配置环境变量 + vim /etc/profile.d/doris.sh + export DORIS_HOME=/opt/doris/fe/ + export PATH=$PATH:$DORIS_HOME/bin + # 保存并source + source /etc/profile.d/doris.sh + ``` + + - 安装Net-Tools工具以便于查看IP地址 + + ```shell + yum install net-tools -y + ``` + + - 使用命令查看该Docker的IPv4地址,并记录下来 + + ```shell + ifconfig + ``` + + - 启动FE + + ```shell + start_fe.sh + # 最好执行命令以后再等待10秒左右 + ``` + + - 退出该容器,返回宿主机 + + ```shell + exit + ``` + +13. 用MySQL-Client连接Doris + + ```shell + mysql -h ${记录下的FE-Docker的IPv4地址} -P ${默认9030,如有改变则使用改变后的query-port} -uroot + ``` + +14. 注册BE至FE + + ```shell + ALTER SYSTEM ADD BACKEND "${FE-Docker的IPv4地址的第四位自增1}:9050"; + # 这里需要说明的是,这命令执行时应该是根据BE的数量来循环的,比如BE为默认值3,记录下FE-Docker的地址为172.17.0.3,那么就应该循环添加 172.17.0.4:9050、172.17.0.5:9050、172.17.0.6:9050三条注册信息,以此类推 + ``` + +15. 若有用户修改密码和注册了操作员账户,则执行以下命令 + + ```shell + # 修改密码 + SET PASSWORD FOR 'root' = PASSWORD('${填写的root密码}'); + # 也可以创建新用户 + CREATE USER '${填写的操作员账户}' IDENTIFIED BY '${填写的操作员密码}'; + ``` + +16. 退出MySQL-Client + + ```shell + exit + ``` + +17. 制作BE容器,该处应该进入以BE数量为最大数值从1开始的循环中(以BE-01为例) + + **假设BE的节点数量从1自增的变量为n,在以下示例中取值方式为`${n}`** + + - 构建BE容器 + + ```shell + # 标准格式为如下所示,其中三处被替换为${n} + docker run -it -p 906${n}:9060 -d --name=doris-be-0${n} -v /opt/docker/doris/be-0${n}:/opt/doris/be apache/incubator-doris:build-env-ldb-toolchain-latest + # 示例 + docker run -it -p 9061:9060 -d --name=doris-be-01 -v /opt/docker/doris/be-01:/opt/doris/be apache/incubator-doris:build-env-ldb-toolchain-latest + ``` + + - 进入容器 + + ```shell + # 这里需要注意,也是要根据循环进行取值 + docker exec -ti doris-be-0${n} /bin/bash + ``` + + - 修改BE配置文件 + + ```shell + vim /opt/doris/be/conf/be.conf + # 取消priority_networks的注解,并根据Docker的网段进行配置 + priority_networks = 172.17.0.0/16 #这里要根据你Docker的IP确定 + ``` + + - 配置BE环境变量 + + ```shell + # 配置环境变量 + vim /etc/profile.d/doris.sh + export DORIS_HOME=/opt/doris/be/ + export PATH=$PATH:$DORIS_HOME/bin + # 保存并source + source /etc/profile.d/doris.sh + ``` + + - 启动BE + + ``` + start_be.sh + ``` + + - 退出容器,开始下一次循环 + + ```shell + exit + ``` + +18. 循环结束,清除临时解压缩及部分下载文件 + + ```shell + rm -rf /opt/software/*.rpm + rm -rf /opt/docker/doris/apache-doris-install.tar.gz + ``` + +19. 制作启动、停止脚本(前提Docker容器是启动的,若未启动则报错) + + 启动脚本需以`start_doris_docker.sh`命名,停止脚本以`stop_doris_docker.sh`命名 + + 两个脚本均写在`/opt/docker/doris/sbin/`目录下 + + - 创建目录 + + ```shell + mkdir -p /opt/docker/doris/sbin/ + ``` + + - 启动脚本内容 + + - 启动FE + + ```shell + docker exec -d doris-fe /bin/bash /opt/doris/fe/start_fe.sh --daemon + ``` + + - 循环启动BE + + ```shell + docker exec -d doris-be-0${n} /bin/bash /opt/doris/be/start_be.sh --daemon + ``` + + - 停止脚本内容 + + - 循环停止BE + + ```shell + docker exec -d doris-be-0${n} /bin/bash /opt/doris/be/stop_be.sh --daemon + ``` + + - 停止FE + + ```shell + docker exec -d doris-fe /bin/bash /opt/doris/fe/stop_fe.sh --daemon + ``` + + - 配置环境变量 + + ```shell + vim /etc/profile.d/doris-docker.sh + export DORIS_DOCKER_HOME=/opt/docker/doris/sbin + export PATH=$PATH:$DORIS_DOCKER_HOME + ``` + + - 刷新环境变量 + + ```shell + source /etc/profile.d/doris-docker.sh + ``` + +## 成果 + +**极速体验版部署流程**(此脚本部署将部署最新版本Apache Doris) + +```shell +wget http://download.freeoneplus.com/doris/doris_docker_fast_install.sh +./doris_docker_fast_install.sh +``` + +**完全部署版部署流程**(此脚本部署将部署最新版本Apache Doris) + +```shell +wget http://download.freeoneplus.com/doris/doris_docker_whole_install.sh +./doris_docker_whole_install.sh +``` + +**假设需要指定版本的部署,请使用以下部署流程** + +```shell +# 极速体验版部署流程 +wget http://download.freeoneplus.com/doris/doris_docker_fast_install_${指定版本号}.sh +./doris_docker_fast_install_${指定版本号}.sh +# 案例:极速体验版 Apache Doris 0.15版本 +wget http://download.freeoneplus.com/doris/doris_docker_fast_install_0.15.sh +./doris_docker_fast_install_0.15.sh + +# 完全部署版部署流程 +wget http://download.freeoneplus.com/doris/doris_docker_whole_install_${指定版本号}.sh +./doris_docker_whole_install_${指定版本号}.sh +# 案例:完全部署版 Apache Doris 0.15版本 +wget http://download.freeoneplus.com/doris/doris_docker_whole_install_0.15.sh +./doris_docker_whole_install_0.15.sh +``` + +**当前支持版本对照表** + +| Apache Doris version | 是否支持 | +| -------------------- | ------------ | +| 1.0.0-beta | 支持 | +| 0.15 | 3月8日起支持 | +| 0.14及以下 | 不支持 | + diff --git a/blogs/zh-CN/weekly-20220306.md b/blogs/zh-CN/weekly-20220306.md new file mode 100644 index 0000000..5d9917c --- /dev/null +++ b/blogs/zh-CN/weekly-20220306.md @@ -0,0 +1,132 @@ +--- +{ + "title": "[Doris 周报] 20220221-20220306", + "description": "Doris 周报 20220221-20220306", + "date": "2022-03-06", + "metaTitle": "[Doris 周报] 20220221-20220306", + "isArticle": true, + "language": "zh-CN", + "author": "morningman", + "layout": "Article", + "sidebar": false +} +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## 统计数据 + +共 44 位作者提交了 101 个 Commit 。感谢以下作者的贡献(Github ID,按字母序): + +@adonis0147, @amosbird, @aopangzi, @arthuryangcs, @awakeljw, @BiteTheDDDDt, @bridgeDream, @caiconghui, @CalvinKirs, @cambyzju, @ChPi, @daikon12, @DarvenDuan, @dataalive, @dataroaring, @GoGoWen, @HappenLee, @Henry2SS, @hf200012, @jacktengg, @kylinmac, @liaoxin01, @liutang123, @lonre, @madongz, @morningman, @mrhhsg, @Nivane, @qidaye, @steadyBoy, @SWJTU-ZhangLei, @wangbo, @wangshuo128, @wangyf0555, @weizuo93, @wunan1210, @xingtanzjr, @yangzhg, @yiguolei, @zbtzbtzbt, @zhangstar333, @zhangyif [...] + +最近 2 周,共修改新增代码行 8654 ,删除代码行 8608。 + +## 主要进展 + +### 新增功能 + +* https://github.com/apache/incubator-doris/pull/8188 + + show routine load 语句支持通过 like 模糊匹配作业名称。 + +* https://github.com/apache/incubator-doris/pull/8041 + https://github.com/apache/incubator-doris/pull/8259 + + 支持创建 random 分桶的表。并支持将hash 分桶的表转换成 random 分桶的表。 + +* https://github.com/apache/incubator-doris/pull/8179 + + 支持 iceberg 外表的查询。 + +* https://github.com/apache/incubator-doris-flink-connector/pull/12 + + Flink connector 支持 exactly once 语义 + +### Work-in-Progess + +* https://github.com/apache/incubator-doris/pull/8212 + + 支持 array 类型的 PB 结构定义。 + +* https://github.com/apache/incubator-doris/pull/8279 + + 重构 typo info 以支持复合数据类型 + +### 功能改进 + +* https://github.com/apache/incubator-doris-flink-connector/pull/11 + + flink connector 支持通过多个 FE 进行 BE ip 的获取。 + +* https://github.com/apache/incubator-doris/pull/8146 + + 新增 FE 参数以支持查询时忽略 compaction 较慢的副本。 + +* https://github.com/apache/incubator-doris/pull/8229 + + 优化 BE 端字符串比较的性能。 + +* https://github.com/apache/incubator-doris/pull/8245 + + 支持当表的部分分区在进行 restore 任务时,其他分区依然可以正常查询。 + +* https://github.com/apache/incubator-doris/pull/8119 + + 优化了向量化执行引擎中,HashJoin 的性能。 + +* https://github.com/apache/incubator-doris/pull/8158 + https://github.com/apache/incubator-doris/pull/8138 + + 支持更多函数的向量化实现。 + + ``` + json_array + json_object + json_quote + bitmap_and_not_count + bitmap_or + bitmap_xor + bitmap_xor_count + bitmap_and + bitmap_and_not + bitmap_subset_in_range + bitmap_subset_limit + bitmap_and_count + sub_bitmap + bitmap_has_all + bitmap_or_count + ``` + +### Bug 修复 + +* https://github.com/apache/incubator-doris/pull/8152 + + 修复 `explode_json_array_string` 某些情况下返回结果不对的问题。 + +* https://github.com/apache/incubator-doris/pull/8142 + + 修复 int 列转换成字符串列导致 BE crash 的问题。 + +### 其他 + +* https://github.com/apache/incubator-doris/pull/8249 + + 支持在 arm 服务器上的源码编译。 \ No newline at end of file --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org