This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d47198dd9df [docker](readme) add readme for how to build doris image
(#39241)
d47198dd9df is described below
commit d47198dd9df5ef7fecf273db80b4dc4a381ed9e4
Author: catpineapple <catpineapple1...@gmail.com>
AuthorDate: Mon Aug 19 14:36:57 2024 +0800
[docker](readme) add readme for how to build doris image (#39241)
1. add a readme for how to build doris image.
2. delete the docker/runtime/k8s directory, and no longer need native
k8s resources to build a doris cluster.
---
docker/runtime/README.md | 111 ++++++++++++++++++++++++++++
docker/runtime/k8s/doris_be.yml | 105 ---------------------------
docker/runtime/k8s/doris_cn.yml | 106 ---------------------------
docker/runtime/k8s/doris_follower.yml | 131 ----------------------------------
4 files changed, 111 insertions(+), 342 deletions(-)
diff --git a/docker/runtime/README.md b/docker/runtime/README.md
new file mode 100644
index 00000000000..71818b0823c
--- /dev/null
+++ b/docker/runtime/README.md
@@ -0,0 +1,111 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Building the Docker image
+
+### Download and copy the Doris code repo
+
+clone [Doris code repo](https://github.com/apache/doris.git) and enter the
doris/docker/runtime path.
+your workspace should like this:
+
+```
+.
+├── runtime
+│ ├── README.md
+│ ├── all-in-one
+│ ├── base-image
+│ ├── be
+│ ├── broker
+│ ├── docker-compose-demo
+│ ├── doris-compose
+│ ├── fe
+│ ├── ms
+```
+
+### Download, extract archive and copy into resource directories
+
+1. Go to the doris [official website](https://doris.apache.org/download) to
download the binary package you need(pay attention to selecting the doris
version and architecture you need), and extract archive binary
+
+Here we take the build of x64 (avx2) and arm64 platforms of 2.1.5 version as
an example.
+
+```shell
+$ wget
https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-2.1.5-bin-x64.tar.gz
&& tar -zxvf apache-doris-2.1.5-bin-x64.tar.gz
+# or
+$ wget
https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-2.1.5-bin-arm64.tar.gz
&& tar -zxvf apache-doris-2.1.5-bin-arm64.tar.gz
+```
+
+2. You need to copy the corresponding directories to different directories in
sequence as below in the table:
+
+amd64(avx2) platform:
+
+| doris type | doris package |
docker file path
|
+|:----------:|:--------------------------------------------------------:|:--------------------------------------------------------------------------------------:|
+| fe | apache-doris-2.1.5-bin-x64/fe |
runtime/fe/resource/amd64/apache-doris-2.1.5-bin-x64/fe
|
+| be | apache-doris-2.1.5-bin-x64/be |
runtime/be/resource/amd64/apache-doris-2.1.5-bin-x64/be
|
+| ms | apache-doris-2.1.5-bin-x64/ms |
runtime/ms/resource/amd64/apache-doris-2.1.5-bin-x64/ms
|
+| broker | apache-doris-2.1.5-bin-x64/extensions/apache_hdfs_broker |
runtime/broker/resource/amd64/apache-doris-2.1.5-bin-x64/extensions/apache_hdfs_broker
|
+
+arm64 platform:
+
+| doris type | doris package |
docker file path
|
+|:----------:|:----------------------------------------------------------:|:----------------------------------------------------------------------------------------:|
+| fe | apache-doris-2.1.5-bin-arm64/fe |
runtime/fe/resource/arm64/apache-doris-2.1.5-bin-arm64/fe
|
+| be | apache-doris-2.1.5-bin-arm64/be |
runtime/be/resource/arm64/apache-doris-2.1.5-bin-arm64/be
|
+| ms | apache-doris-2.1.5-bin-arm64/ms |
runtime/ms/resource/arm64/apache-doris-2.1.5-bin-arm64/ms
|
+| broker | apache-doris-2.1.5-bin-arm64/extensions/apache_hdfs_broker |
runtime/broker/resource/arm64/apache-doris-2.1.5-bin-arm64/extensions/apache_hdfs_broker
|
+
+### Build base image
+
+**NOTICE**
+
+The below images depend on the base image selectdb/base. If your environment
cannot access it, you can pre-build the base image. The base image contains the
basic environment for doris to run, including JDK, openssl, etc.
+
+1. As mentioned in the preparation steps above, the Dockerfile of the Base
image is under the runtime/base-image path. To build it, make sure you have
pulled the Doris code repo and execute the following command.
+
+```shell
+$ cd doris/runtime/base-image && docker build . -t doris-base:latest -f
Dockerfile_base
+```
+
+2. Adjust the base image name used by the Dockerfile of the doris component
and replace the base image you built yourself, As shown in the following
example:
+
+```dockerfile
+...
+# Adjust the base image here
+FROM doris-base:latest:latest
+
+ARG TARGETARCH
+
+ARG DORIS_VERSION="x.x.x"
+...
+```
+
+### Build Doris docker image
+
+as the following commands, Docker will automatically confirm the architecture
+
+```shell
+$ cd doris/runtime/fe && docker build . -t doris.fe:2.1.5 -f Dockerfile
--build-arg DORIS_VERSION=2.1.5
+$ cd doris/runtime/be && docker build . -t doris.be:2.1.5 -f Dockerfile
--build-arg DORIS_VERSION=2.1.5
+$ cd doris/runtime/ms && docker build . -t doris.ms:2.1.5 -f Dockerfile
--build-arg DORIS_VERSION=2.1.5
+$ cd doris/runtime/broker && docker build . -t doris.broker:2.1.5 -f
Dockerfile --build-arg DORIS_VERSION=2.1.5
+```
+
+### Latest update time
+
+2024-8-12
diff --git a/docker/runtime/k8s/doris_be.yml b/docker/runtime/k8s/doris_be.yml
deleted file mode 100644
index 84c7f2a0da8..00000000000
--- a/docker/runtime/k8s/doris_be.yml
+++ /dev/null
@@ -1,105 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-apiVersion: v1
-kind: Service
-metadata:
- name: doris-be-cluster1
- labels:
- app: doris-be-cluster1
-spec:
- ports:
- - port: 9060
- name: be-port
- - port: 8040
- name: webserver-port
- - port: 9050
- name: heartbeat-port #This name should be fixed. Doris will get the port
information through this name
- - port: 8060
- name: brpc-port
- clusterIP: None
- selector:
- app: doris-be-cluster1
----
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
- name: doris-be-cluster1
- labels:
- app: doris-be-cluster1
-spec:
- selector:
- matchLabels:
- app: doris-be-cluster1
- serviceName: doris-be-cluster1
- replicas: 3
- template:
- metadata:
- name: doris-be-cluster1
- labels:
- app: doris-be-cluster1
- spec:
- containers:
- - name: doris-be-cluster1
- #Need to change to real mirror information
- image: apache-doris-be:test
- imagePullPolicy: IfNotPresent
- env:
- #Specify the startup type as k8s to bypass some restrictions of
the official image initialization script
- - name: BUILD_TYPE
- value: "k8s"
- ports:
- - containerPort: 9060
- name: be-port
- - containerPort: 8040
- name: webserver-port
- - containerPort: 9050
- name: heartbeat-port
- - containerPort: 8060
- name: brpc-port
- volumeMounts:
- #Mount the configuration file in the way of configmap
- - name: conf
- mountPath: /opt/apache-doris/be/conf
- #Ifnot mounted, when enable_profile, error will be reported when
querying the data from jdbc catalog
- #Error message: error setting certificate verify locations:
CAfile:/etc/pki/tls/certs/ca-bundle.crt CApath: none
- - name: sys
- mountPath: /etc/pki
- readOnly: true
- volumes:
- - name: conf
- configMap:
- name: be-conf
- - name: sys
- hostPath:
- path: /etc/pki
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
- name: be-conf
-data:
- be.conf: |
- PPROF_TMPDIR="$DORIS_HOME/log/"
- sys_log_level = INFO
-
- be_port = 9060
- webserver_port = 8040
- heartbeat_service_port = 9050
- brpc_port = 8060
-
- priority_networks = 172.16.0.0/24
\ No newline at end of file
diff --git a/docker/runtime/k8s/doris_cn.yml b/docker/runtime/k8s/doris_cn.yml
deleted file mode 100644
index 391df11162b..00000000000
--- a/docker/runtime/k8s/doris_cn.yml
+++ /dev/null
@@ -1,106 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-apiVersion: v1
-kind: Service
-metadata:
- name: doris-cn-cluster1
- labels:
- app: doris-cn-cluster1
-spec:
- ports:
- - port: 9060
- name: be-port
- - port: 8040
- name: webserver-port
- - port: 9050
- name: heartbeat-port #This name should be fixed. Doris will get the port
information through this name
- - port: 8060
- name: brpc-port
- clusterIP: None
- selector:
- app: doris-cn-cluster1
----
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
- name: doris-cn-cluster1
- labels:
- app: doris-cn-cluster1
-spec:
- selector:
- matchLabels:
- app: doris-cn-cluster1
- serviceName: doris-cn-cluster1
- replicas: 3
- template:
- metadata:
- name: doris-cn-cluster1
- labels:
- app: doris-cn-cluster1
- spec:
- containers:
- - name: doris-cn-cluster1
- #Need to change to real mirror information
- image: apache-doris-be:test
- imagePullPolicy: IfNotPresent
- env:
- #Specify the startup type as k8s to bypass some restrictions of
the official image initialization script
- - name: BUILD_TYPE
- value: "k8s"
- ports:
- - containerPort: 9060
- name: be-port
- - containerPort: 8040
- name: webserver-port
- - containerPort: 9050
- name: heartbeat-port
- - containerPort: 8060
- name: brpc-port
- volumeMounts:
- #Mount the configuration file in the way of configmap
- - name: conf
- mountPath: /opt/apache-doris/be/conf
- #Ifnot mounted, when enable_profile, error will be reported when
querying the data from jdbc catalog
- #Error message: error setting certificate verify locations:
CAfile:/etc/pki/tls/certs/ca-bundle.crt CApath: none
- - name: sys
- mountPath: /etc/pki
- readOnly: true
- volumes:
- - name: conf
- configMap:
- name: cn-conf
- - name: sys
- hostPath:
- path: /etc/pki
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
- name: cn-conf
-data:
- be.conf: |
- PPROF_TMPDIR="$DORIS_HOME/log/"
- sys_log_level = INFO
-
- be_port = 9060
- webserver_port = 8040
- heartbeat_service_port = 9050
- brpc_port = 8060
- #Specify node type as calculation node
- be_node_role = computation
- priority_networks = 172.16.0.0/24
\ No newline at end of file
diff --git a/docker/runtime/k8s/doris_follower.yml
b/docker/runtime/k8s/doris_follower.yml
deleted file mode 100644
index b22915fe249..00000000000
--- a/docker/runtime/k8s/doris_follower.yml
+++ /dev/null
@@ -1,131 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-apiVersion: v1
-kind: Service
-metadata:
- name: doris-follower-cluster1
- labels:
- app: doris-follower-cluster1
-spec:
- ports:
- - port: 8030
- name: http-port
- - port: 9020
- name: rpc-port
- - port: 9030
- name: query-port
- - port: 9010
- name: edit-log-port #This name should be fixed. Doris will get the port
information through this name
- clusterIP: None
- selector:
- app: doris-follower-cluster1
----
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
- name: doris-follower-cluster1
- labels:
- app: doris-follower-cluster1
-spec:
- selector:
- matchLabels:
- app: doris-follower-cluster1
- serviceName: doris-follower-cluster1
- replicas: 3
- template:
- metadata:
- name: doris-follower-cluster1
- labels:
- app: doris-follower-cluster1
- spec:
- containers:
- - name: doris-follower-cluster1
- #Need to change to real mirror information
- image: apache-doris-fe:test
- imagePullPolicy: IfNotPresent
- env:
- #Specify the startup type as k8s to bypass some restrictions of
the official image initialization script
- - name: BUILD_TYPE
- value: "k8s"
- #Initialize the fe of three nodes
- - name: FE_INIT_NUMBER
- value: "3"
- #ServiceName of bakend_cn node,（if do not have bakend_cn node,do
not configure this environment variable）
- - name: CN_SERVICE
- value: "doris-cn-cluster1"
- #StatefulSetName of bakend_cn node,（if do not have bakend_cn
node,do not configure this environment variable)
- - name: CN_STATEFULSET
- value: "doris-cn-cluster1"
- #ServiceName of bakend node,（if do not have bakend node,do not
configure this environment variable）
- - name: BE_SERVICE
- value: "doris-be-cluster1"
- #StatefulSetName of bakend node,（if do not have bakend node,do not
configure this environment variable)
- - name: BE_STATEFULSET
- value: "doris-be-cluster1"
- #ServiceName of follower node,（if do not have follower node,do not
configure this environment variable）
- - name: FE_SERVICE
- value: "doris-follower-cluster1"
- ##StatefulSetName of follower node,（if do not have follower
node,do not configure this environment variable)
- - name: FE_STATEFULSET
- value: "doris-follower-cluster1"
- ports:
- - containerPort: 8030
- name: http-port
- - containerPort: 9020
- name: rpc-port
- - containerPort: 9030
- name: query-port
- - containerPort: 9010
- name: edit-log-port
- volumeMounts:
- #Mount the configuration file in the way of configmap
- - name: conf
- mountPath: /opt/apache-doris/fe/conf
- #In order to call the api of k8s
- - name: kube
- mountPath: /root/.kube/config
- readOnly: true
- volumes:
- - name: conf
- configMap:
- name: follower-conf
- - name: kube
- hostPath:
- path: /root/.kube/config
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
- name: follower-conf
-data:
- fe.conf: |
- priority_networks = 172.16.0.0/24
- #It can automatically maintain node information by getting the number of
replicas of StatefulSet, similar to alter system add/drop back
- enable_deploy_manager = k8s
- #Automatically adjust the IP of the node according to the domain name (for
example, after the pod is restarted, the domain name is still
doris-be-cluster1-0-doris-be-cluster1.default.svc.cluster.local, but the IP may
change from 172.16.0.9 to 172.16.0.10)
- enable_fqdn_mode = true
- LOG_DIR = ${DORIS_HOME}/log
- sys_log_level = INFO
- http_port = 8030
- rpc_port = 9020
- query_port = 9030
- edit_log_port = 9010
- #Doris needs to generate the log4j configuration file according to the
fe.yml configuration information, which is written in the same directory as
fe.yml by default, but the config we mount is readonly, so specify this
configuration to write the log4j file to another location
- custom_config_dir = /opt/apache-doris/
- #when set to false, the backend will not be dropped and remaining in
DECOMMISSION state
- drop_backend_after_decommission = false
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
