This is an automated email from the ASF dual-hosted git repository.
yaqian pushed a commit to branch kylin4_on_cloud
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/kylin4_on_cloud by this push:
new d122460 Modify docs (#1838)
d122460 is described below
commit d122460ee5e18b47306bf4dd808b9d36419388ee
Author: Yaqian Zhang <598593...@qq.com>
AuthorDate: Wed Mar 30 09:49:18 2022 +0800
Modify docs (#1838)
---
images/check_role_permission.png | Bin 0 -> 134746 bytes
images/check_user_permission.png | Bin 0 -> 92931 bytes
images/create_role.png | Bin 0 -> 119270 bytes
readme/commands.md | 2 +-
readme/configs.md | 48 ++++++
readme/prepare_files_manually.md | 74 +++++++++
readme/prerequisites.md | 233 ++++++----------------------
readme/quick_start.md | 25 +--
readme/quick_start_for_multiple_clusters.md | 8 +-
9 files changed, 187 insertions(+), 203 deletions(-)
diff --git a/images/check_role_permission.png b/images/check_role_permission.png
new file mode 100644
index 0000000..4e6dc72
Binary files /dev/null and b/images/check_role_permission.png differ
diff --git a/images/check_user_permission.png b/images/check_user_permission.png
new file mode 100644
index 0000000..ddff924
Binary files /dev/null and b/images/check_user_permission.png differ
diff --git a/images/create_role.png b/images/create_role.png
new file mode 100644
index 0000000..c838a79
Binary files /dev/null and b/images/create_role.png differ
diff --git a/readme/commands.md b/readme/commands.md
index 49b80dc..7bca88f 100644
--- a/readme/commands.md
+++ b/readme/commands.md
@@ -19,7 +19,7 @@ python deploy.py --type [deploy|destroy|list|scale]
--scale-type [up|down] --nod
> 1. Current support to scale up/down `kylin` or `spark_worker` for a
specific cluster.
> 2. Before scaling up/down `kylin` or `spark_worker` nodes, Cluster
services must be ready.
> 3. If you want to scale a `kylin` or `spark_worker` node to a specified
cluster, please add the `--cluster ${cluster ID}` to specify the expected node
add to the cluster `${cluster ID}`.
- > 4. For details about the index of the cluster, please check [Indexes of
clusters](./prerequisites.md#indexofcluster).
+ > 4. For details about the index of the cluster, please check [Indexes of
clusters](./configs.md#indexofcluster).
### Command for deploy
diff --git a/readme/configs.md b/readme/configs.md
new file mode 100644
index 0000000..b3281f7
--- /dev/null
+++ b/readme/configs.md
@@ -0,0 +1,48 @@
+## Configs
+
+#### I. Configure the `kylin_configs.yaml`
+
+**Required parameters**:
+
+- `AWS_REGION`: Current region for EC2 instances.
+- `IAMRole`: IAM role which has the access to aws authority. This parameter
will be set to the created **name** of the IAM role.
+- `S3_URI`: the prefix path of storing `jars/scripts/tar`. For example, this
parameter will be set to `s3://.../kylin4-aws-test`.
+- `KeyName`: Security key name is a set of security credentials that you use
to prove your identity when connecting to an instance. This parameter will be
set to the created **name** of key pair`.
+- `CIDR_IP`: An inbound rule permits instances to receive traffic from the
specified IPv4 or IPv6 CIDR address range, or the instances associated with the
specified security group.
+- `DB_IDENTIFIER`: this param should be only one in the `RDS -> Databases`.
And it will be the name of created RDS database.
+- `DB_PORT`: this param will be the port of created RDS database, default is
`3306`.
+- `DB_USER`: this param will be a login ID for the master user of your DB
instance, the default is `root`.
+- `DB_PASSWORD`: this param will be the password of `DB_USER` to access the DB
instance. default is `123456test`, it's strongly suggested you change it.
+
+#### II. Configure the `kylin.properties` in `backup/properties`
directories.<a name="cluster"></a>
+
+> About the indexes of clusters:<a name="indexofcluster"></a>
+>
+> 1. **Tool will deploy a cluster which will mark to `default` if a user does
not specify a cluster to deploy.**
+> 2. The range of cluster indexes must be in `CLUSTER_INDEXES` which is
configured in the `kylin_configs.yml`.
+> 3. **`CLUSTER_INDEXES` is for users to create multiple clusters as
READ-WRITE separation mode**, such as a user may want to create 2 query nodes
of Kylin and 2 job nodes of Kylin.
+> 1. User can modify the `CLUSTER_INDEXES` to be `(1, 3)`, then as
following steps of this module to modify the `kylin.properties` file for
clusters.
+> 2. User can mark the Kylin node of the `default` cluster to be `query`
mode and the Kylin node of the cluster whose index is `1` to be `query` mode.
User can mark the Kylin node of the cluster which index is `2` to be `job` mode
and the Kylin node of the cluster which index is `3` to job mode.
+> 3. User can also modify the `CLUSTER_INDEXES` to be `(1, 4)`, and mark
the Kylin node of clusters whose index is `1` and `2` to be `query` mode and
the Kylin node of clusters whose index is `3` and `4` to be `job` mode. Just
don't use `default` to mark a cluster and execute exactly deploy cluster
commands. For details about commands, please check to [deploy multiple
clusters](./Commands.md#deploycluster).
+> 4. The list of mark name for clusters will be [`default`, `{cluster ID}`
...] and `{cluster ID}` is in the range of `CLUSTER_INDEXES`.
+> 5. For example, if `CLUSTER_INDEXES` is (1, 3) means that tool can deploy a
cluster and it can be marked as `default` or `1` or `2` or `3`. And tool can
execute to deploy total 1(`default`) + 3(clusters which mark name can be `1`,
`2`, `3`) = 4 clusters.
+> 6. Every cluster contains 3 `Zookeepers Node`, 1 `Kylin Node`, 1 `Spark
Master Node,` and 3 `Spark Slaves Node` after deployed and it can scale needed
nodes of Kylin and Spark workers.
+> 7. **The difference between clusters only can be the index or customized
configs of EC2 instances or customized properties of Kylin (and spark and
zookeeper).**
+
+
+
+1. The `kylin.properties` is for starting kylin instance in the cluster.
+2. The default cluster will check the `kylin.properties` in the
`backup/properties/default`, and other specific clusters will check the related
num directory such as `1`, `2,` and `3`.
+3. User needs to create a new dir for the cluster ID in `backup/properties`,
and name it to the `${cluster ID}`, such as `1`, `2` ,`3` and so on.
+4. Following the step `2`, copy the `kylin.properties.template` which is in
`backup/properties/templates` to the related `${cluster ID} ` directories， and
rename the template to `kylin.properties`.
+5. The range of cluster IDs must match the config `CLUSTER_INDEXES`, such as
`CLUSTER_INDEXES: (1, 3)` then the directories must be `1`, `2`,`3` in the
`backup/properties`.
+
+
+
+6. Customize the config which you want to change except the config which
contains `{{ }}` mark.
+
+7. If the related `kylin.properties` was filled with cluster information, a
file named `.rendered` will create in the directory.
+
+
+
+8. **(Important !!!)** If you want to re-fill the `kylin.properties` for a
`kylin` node in a specific cluster, you need to remove the `.rendered` file
and re-copy the `kylin.propertie.template`. Redo steps from `3` to `6`.
\ No newline at end of file
diff --git a/readme/prepare_files_manually.md b/readme/prepare_files_manually.md
new file mode 100644
index 0000000..8a87b71
--- /dev/null
+++ b/readme/prepare_files_manually.md
@@ -0,0 +1,74 @@
+## Prepare files manually
+
+#### **(Optional)** II. Manually Download packages & Upload them to the S3
path which suffix is `*/tar`
+
+> Note:
+>
+> This step will automatically do by tools. So you can skip this step, or
you want to check packages by yourself.
+
+
+
+> Create the directory named `tar` **in the path which was created by
yourself**. For example, the full path would be `s3://.../kylin4-aws-test/tar`.
+>
+
+1. Download the Kylin4 package on the [official
website](https://kylin.apache.org/download/).
+2. Download Hadoop, [version
3.2.0](https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz).
+3. Download Spark with hadoop3.2, [version
3.1.1](https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz).
+4. Download Hive, [version
2.3.9](https://archive.apache.org/dist/hive/hive-2.3.9/apache-hive-2.3.9-bin.tar.gz).
+5. Download Zookeeper, [version
3.4.9.](https://archive.apache.org/dist/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz)
+6. Download JDK, [version
1.8_301](https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html).
+7. Download Node Exporter, [version
1.3.1.](https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz)
+8. Download Prometheus Server, [version
2.31.1](https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-amd64.tar.gz).
+9. Download the Kylin4 package with local cache + soft affinity feature by the
[public
website.](https://s3.cn-north-1.amazonaws.com.cn/public.kyligence.io/kylin/tar/apache-kylin-4.0.0-bin-spark3-soft.tar.gz)
+
+
+
+> Note:
+> If you want to use Kylin4 with a local cache + soft affinity feature,
please download the `experimental` package above.
+
+
+
+
+
+#### (Optional) III. Upload `backup/jars/*` to the S3 Path which suffix is
`*/jars`
+
+> Note:
+>
+> This step will automatically do by tools. So you can skip this step, or
you want to check jars by yourself.
+
+
+
+> Create the directory named `jars` **in the path which was created by
yourself**. For example, the full path would be
`s3://.../kylin4-aws-test/jars`.
+>
+
+Kylin4 needed extra jars
+
+- Basic jars
+ - commons-configuration-1.3.jar
+ - mysql-connector-java-5.1.40.jar
+- Local Cache + Soft Affinity feature needed jars
+ - alluxio-2.6.1-client.jar
+ - kylin-soft-affinity-cache-4.0.0-SNAPSHOT.jar
+
+
+
+#### (Optional) IV. Upload `backup/scripts/*` to the S3 Path which suffix is
`*/scripts`
+
+> Note:
+>
+> This step will automatically do by tools. So you can skip this step, or
you want to check scripts by yourself.
+
+
+
+> Create the directory named `scripts` **in the path which was created by
yourself**. For example, the full path would be
`s3://.../kylin4-aws-test/scripts`.
+>
+
+Scripts:
+
+- prepare-ec2-env-for-kylin4.sh
+- prepare-ec2-env-for-spark-master.sh
+- prepare-ec2-env-for-spark-slave.sh
+- prepare-ec2-env-for-static-services.sh
+- prepare-ec2-env-for-zk.sh
+
+
diff --git a/readme/prerequisites.md b/readme/prerequisites.md
index 2a6497d..400f62d 100644
--- a/readme/prerequisites.md
+++ b/readme/prerequisites.md
@@ -1,41 +1,12 @@
-## Prerequisites
+## Prerequisites
-### Download source code & checkout to a branch of `kylin4_on_cloud`
+### I. Check user permission
-commands:
+Login AWS account and check whether the current user has sufficient
permissions in AWS IAM service:
-```shell
-git clone https://github.com/apache/kylin.git && cd kylin && git checkout
kylin4_on_cloud
-```
-
-### Initiliaze an AWS Account
-
-#### I. Create an `IAM` role<a name="IAM"></a>
+
-> Note:
->
-> `IAM` role must have the access which contains `AmazonEC2RoleforSSM`,
`AmazonSSMFullAccess,` and `AmazonSSMManagedInstanceCore`, and
`AWSGlueConsoleFullAccess` (this role is needed if you want to use glue).
->
-> This `IAM` Role will be used to initialize every ec2 instances which is
for creating a kylin4 cluster on AWS. And it will configure in `Initialize Env
of Local Machine` part.
-
-
-
-> Q: Why should need an `IAM` role?
->
-> A:
->
-> 1. At first, the `IAM` role is an IAM identity that you can create in
your account that has specific permissions.
-> 1. An IAM role does not have standard long-term credentials such as a
password or access keys associated with it. Instead, when you assume a role,
it provides you with temporary security credentials for your role session.
-> 1. The access will not cost money. The cost will depend on what you
using on S3 and EC2 instances.
-> 1. For more details about `IAM`, please visit the [official
website](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) by aws.
-
-
-
-#### II. Create a `User`
-
-> Note:
->
-> The `User` who is created in aws will be used to execute some
operations on an EC2 instance. So the `User` must have some authority as below.
+If the current user lacks some permissions listed below, you need to add the
permissions to ensure that the current user can carry out subsequent operations
smoothly:
| Services | Access level
| Resources | Request condition |
| ------------------- | ------------------------------------------------------
| ------------- | ----------------- |
@@ -48,145 +19,90 @@ git clone https://github.com/apache/kylin.git && cd kylin
&& git checkout kylin4
| **STS** | **Limited**: Write
| All Resources | None |
| **Systems Manager** | **Limited**: Write
| All Resources | None |
-
-
-> Q: What's the relationship between the `User` and `IAM` role?
->
-> A:
->
-> 1. With `IAM` roles, you can establish trust relationships between your
***trusting*** account and other AWS ***trusted*** accounts.
-> 1. A `User` is an account.
-> 1. The establishment of the relationship will cost nothing.
-> 1. The cost majorly depends on what you are using on S3 and EC2
instances.
-
-
-
-#### III. Login the created `User` to create key pair and Access Key<a
name="keypair"></a>
+### II. Create key pair and Access Key<a name="keypair"></a>
> Note:
>
> To deploy EC2 instances for Kylin4 Cluster need key pair and access key.
-##### 1. Create a new `Key Pair` in the `EC2 -> Key pairs`
+#### 1. Create a new `Key Pair` in the `EC2 -> Key pairs`
-##### 2. Generate a new `Access Key` in the `My Security Credentials`
+#### 2. Generate a new `Access Key` in the `My Security Credentials`
-> Note:
+> Note:
>
> Please download the generated CSV file of `Access Key` immediately. Get
> the `Access Key` and `Secret Key` to initialize local machine to access aws.
-
-
-##### (!!!Important)IV. Make sure current account has 32 vCpu at least
-
-The current tool will deploy a cluster by EC2 instances, and `2 cores * 3(for
zookeeper) + 2 cores(for monitor services) + 4 cores * 3(for spark slaves) + 4
cores(for spark master) + 4 cores (for kylin)=28 ` cpus to create instances in
default.
-
-> Note: **32 vCPUS is the limit of an default aws account.**
-
-**User can check this quota limit in the limit of EC2 dashborad.**
+### III. Create an `IAM` role<a name="IAM"></a>
-
+Create a role in AWS IAM service to provide temporary access credentials with
specific permissions in EC2 nodes created later:
+
+Add permissions policies `AmazonEC2RoleforSSM`, `AmazonSSMFullAccess,` and
`AmazonS3FullAccess`, and `AWSGlueConsoleFullAccess` for this role:
-For more details about user current limit, please check [View your current
limits](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html).
+
-### Initialize a S3 direcotry<a name="S3"></a>
-
-#### I. Create a directory on S3
-
-> Note:
>
-> This directory will store tars, scripts, and jars below. And it will
also be the root path of working dir for kylin4.
-
-Example: make a directory named `kylin4-aws-test`. You can also create a
directory named what you like.
-
-
-
-
+> This `IAM` Role will be used to initialize every ec2 instances which is
for creating a kylin4 cluster on AWS. And it will configure in `Initialize Env
of Local Machine` part.
-#### **(Optional)** II. Manually Download packages & Upload them to the S3
path which suffix is `*/tar`
-> Note:
+> Q: Why should need an `IAM` role?
>
-> This step will automatically do by tools. So you can skip this step, or
you want to check packages by yourself.
-
-
-
-> Create the directory named `tar` **in the path which was created by
yourself**. For example, the full path would be `s3://.../kylin4-aws-test/tar`.
+> A:
>
+> 1. At first, the `IAM` role is an IAM identity that you can create in
your account that has specific permissions.
+> 2. An IAM role does not have standard long-term credentials such as a
password or access keys associated with it. Instead, when you assume a role,
it provides you with temporary security credentials for your role session.
+> 3. The access will not cost money. The cost will depend on what you
using on S3 and EC2 instances.
+> 4. For more details about `IAM`, please visit the [official
website](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) by aws.
-1. Download the Kylin4 package on the [official
website](https://kylin.apache.org/download/).
-2. Download Hadoop, [version
3.2.0](https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz).
-3. Download Spark with hadoop3.2, [version
3.1.1](https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz).
-4. Download Hive, [version
2.3.9](https://archive.apache.org/dist/hive/hive-2.3.9/apache-hive-2.3.9-bin.tar.gz).
-5. Download Zookeeper, [version
3.4.9.](https://archive.apache.org/dist/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz)
-6. Download JDK, [version
1.8_301](https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html).
-7. Download Node Exporter, [version
1.3.1.](https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz)
-8. Download Prometheus Server, [version
2.31.1](https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-amd64.tar.gz).
-9. Download the Kylin4 package with local cache + soft affinity feature by the
[public
website.](https://s3.cn-north-1.amazonaws.com.cn/public.kyligence.io/kylin/tar/apache-kylin-4.0.0-bin-spark3-soft.tar.gz)
-
-
-
-> Note:
-> If you want to use Kylin4 with a local cache + soft affinity feature,
please download the `experimental` package above.
-
-
-
-
-
-#### (Optional) III. Upload `backup/jars/*` to the S3 Path which suffix is
`*/jars`
-> Note:
+> Q: What's the relationship between the `User` and `IAM` role?
+>
+> A:
>
-> This step will automatically do by tools. So you can skip this step, or
you want to check jars by yourself.
+> 1. With `IAM` roles, you can establish trust relationships between your
***trusting*** account and other AWS ***trusted*** accounts.
+> 1. A `User` is an account.
+> 1. The establishment of the relationship will cost nothing.
+> 1. The cost majorly depends on what you are using on S3 and EC2
instances.
+### (!!!Important)IV. Make sure current account has 32 vCpu at least
-> Create the directory named `jars` **in the path which was created by
yourself**. For example, the full path would be
`s3://.../kylin4-aws-test/jars`.
->
+The current tool will deploy a cluster by EC2 instances, and `2 cores * 3(for
zookeeper) + 2 cores(for monitor services) + 4 cores * 3(for spark slaves) + 4
cores(for spark master) + 4 cores (for kylin)=28 ` cpus to create instances in
default.
-Kylin4 needed extra jars
+> Note: **32 vCPUS is the limit of an default aws account.**
-- Basic jars
- - commons-configuration-1.3.jar
- - mysql-connector-java-5.1.40.jar
-- Local Cache + Soft Affinity feature needed jars
- - alluxio-2.6.1-client.jar
- - kylin-soft-affinity-cache-4.0.0-SNAPSHOT.jar
+**User can check this quota limit in the limit of EC2 dashborad.**
-
+
-#### (Optional) IV. Upload `backup/scripts/*` to the S3 Path which suffix is
`*/scripts`
-> Note:
->
-> This step will automatically do by tools. So you can skip this step, or
you want to check scripts by yourself.
+For more details about user current limit, please check [View your current
limits](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html).
+### V. Create a S3 direcotry<a name="S3"></a>
+Create a S3 directory as the working directory for deploying kylin cluster:
-> Create the directory named `scripts` **in the path which was created by
yourself**. For example, the full path would be
`s3://.../kylin4-aws-test/scripts`.
+> Note:
>
+> This directory will store tars, scripts, and jars below. And it will
also be the root path of working dir for kylin4.
-Scripts:
+Example: create a directory named `kylin4-aws-test`. You can also create a
directory named what you like.
-- prepare-ec2-env-for-kylin4.sh
-- prepare-ec2-env-for-spark-master.sh
-- prepare-ec2-env-for-spark-slave.sh
-- prepare-ec2-env-for-static-services.sh
-- prepare-ec2-env-for-zk.sh
+
-
+During deployment, the required jar packages and other files will be
automatically downloaded to this S3 directory, and users do not need to operate
by themselves.
+If you want to prepare files in S3 directory manually, please check [Prepare
files manually](./prepare_files_manually.md)
-### Initialize Env Of Local Machine
-#### I. Initilize an aws account on local machine to access AWS<a
name="localaws"></a>
+### VI. Initilize an aws account on local machine to access AWS<a
name="localaws"></a>
> Use `Access Key` and `Secret Key` above to Initialize an AWS account on a
> local machine.
@@ -194,7 +110,7 @@ Scripts:
$ aws configure
AWS Access Key ID : *************
AWS Secret Access Key : *************
-Default region name : cn-northwest-1
+Default region name : ${region_name}
Default output format : json
```
@@ -202,64 +118,3 @@ Default output format : json
>
> 1. If this command got the response `-bash: aws: command not found`, please
> check in [Getting started with the AWS
> CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html).
> 2. Region name can be in [Available
> Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions).
> The default output format is `json`.
-
-#### II. Init a Python virtual env
-
-> Note:
->
-> Make sure that your machine already has a Python which version is 3.6.6
or later.
-
-commands:
-
-```shell
-$ bin/init.sh
-```
-
-> Note: Follow the information after `bin/init.sh` to activate the python
virtual env.
-
-#### III. Configure the `kylin_configs.yaml`
-
-**Required parameters**:
-
-- `AWS_REGION`: Current region for EC2 instances.
-- `IAMRole`: IAM role which has the access to aws authority. This parameter
will be set to the created **name** of the IAM role.
-- `S3_URI`: the prefix path of storing `jars/scripts/tar`. For example, this
parameter will be set to `s3://.../kylin4-aws-test`.
-- `KeyName`: Security key name is a set of security credentials that you use
to prove your identity when connecting to an instance. This parameter will be
set to the created **name** of key pair`.
-- `CIDR_IP`: An inbound rule permits instances to receive traffic from the
specified IPv4 or IPv6 CIDR address range, or the instances associated with the
specified security group.
-- `DB_IDENTIFIER`: this param should be only one in the `RDS -> Databases`.
And it will be the name of created RDS database.
-- `DB_PORT`: this param will be the port of created RDS database, default is
`3306`.
-- `DB_USER`: this param will be a login ID for the master user of your DB
instance, the default is `root`.
-- `DB_PASSWORD`: this param will be the password of `DB_USER` to access the DB
instance. default is `123456test`, it's strongly suggested you change it.
-
-#### IV. Configure the `kylin.properties` in `backup/properties`
directories.<a name="cluster"></a>
-
-> About the indexes of clusters:<a name="indexofcluster"></a>
->
-> 1. **Tool will deploy a cluster which will mark to `default` if a user does
not specify a cluster to deploy.**
-> 2. The range of cluster indexes must be in `CLUSTER_INDEXES` which is
configured in the `kylin_configs.yml`.
-> 3. **`CLUSTER_INDEXES` is for users to create multiple clusters as
READ-WRITE separation mode**, such as a user may want to create 2 query nodes
of Kylin and 2 job nodes of Kylin.
-> 1. User can modify the `CLUSTER_INDEXES` to be `(1, 3)`, then as
following steps of this module to modify the `kylin.properties` file for
clusters.
-> 2. User can mark the Kylin node of the `default` cluster to be `query`
mode and the Kylin node of the cluster whose index is `1` to be `query` mode.
User can mark the Kylin node of the cluster which index is `2` to be `job` mode
and the Kylin node of the cluster which index is `3` to job mode.
-> 3. User can also modify the `CLUSTER_INDEXES` to be `(1, 4)`, and mark
the Kylin node of clusters whose index is `1` and `2` to be `query` mode and
the Kylin node of clusters whose index is `3` and `4` to be `job` mode. Just
don't use `default` to mark a cluster and execute exactly deploy cluster
commands. For details about commands, please check to [deploy multiple
clusters](./Commands.md#deploycluster).
-> 4. The list of mark name for clusters will be [`default`, `{cluster ID}`
...] and `{cluster ID}` is in the range of `CLUSTER_INDEXES`.
-> 5. For example, if `CLUSTER_INDEXES` is (1, 3) means that tool can deploy a
cluster and it can be marked as `default` or `1` or `2` or `3`. And tool can
execute to deploy total 1(`default`) + 3(clusters which mark name can be `1`,
`2`, `3`) = 4 clusters.
-> 6. Every cluster contains 3 `Zookeepers Node`, 1 `Kylin Node`, 1 `Spark
Master Node,` and 3 `Spark Slaves Node` after deployed and it can scale needed
nodes of Kylin and Spark workers.
-> 7. **The difference between clusters only can be the index or customized
configs of EC2 instances or customized properties of Kylin (and spark and
zookeeper).**
-
-
-
-1. The `kylin.properties` is for starting kylin instance in the cluster.
-2. The default cluster will check the `kylin.properties` in the
`backup/properties/default`, and other specific clusters will check the related
num directory such as `1`, `2,` and `3`.
-3. User needs to create a new dir for the cluster ID in `backup/properties`,
and name it to the `${cluster ID}`, such as `1`, `2` ,`3` and so on.
-4. Following the step `2`, copy the `kylin.properties.template` which is in
`backup/properties/templates` to the related `${cluster ID} ` directories， and
rename the template to `kylin.properties`.
-5. The range of cluster IDs must match the config `CLUSTER_INDEXES`, such as
`CLUSTER_INDEXES: (1, 3)` then the directories must be `1`, `2`,`3` in the
`backup/properties`.
-
-
-
-6. Customize the config which you want to change except the config which
contains `{{ }}` mark.
-
-7. If the related `kylin.properties` was filled with cluster information, a
file named `.rendered` will create in the directory.
-
-
-
-8. **(Important !!!)** If you want to re-fill the `kylin.properties` for a
`kylin` node in a specific cluster, you need to remove the `.rendered` file
and re-copy the `kylin.propertie.template`. Redo steps from `3` to `6`.
\ No newline at end of file
diff --git a/readme/quick_start.md b/readme/quick_start.md
index 4f88b6a..7311c2b 100644
--- a/readme/quick_start.md
+++ b/readme/quick_start.md
@@ -7,7 +7,7 @@
## Quick Start
-1. Initialize aws account credential on the local machine, please check
[details](./prerequisites.md#localaws).
+1. Initialize aws account credential on the local machine according to
[prerequisites](./prerequisites.md).
2. Download the source code:
@@ -17,13 +17,13 @@
3. Modify the `kylin_config.yml`.
- 1. Set the `AWS_REGION`.
+ 1. Set the `AWS_REGION`, such as us-east-1.
- 2. Set the `IAMRole`, please check the [details](./prerequisites.md#IAM).
+ 2. Set the `IAMRole`, please check [Create an IAM
role](./prerequisites.md#IAM).
- 3. Set the `S3_URI`, please check [details](./prerequisites.md#S3).
+ 3. Set the `S3_URI`, please check [Create a S3
direcotry](./prerequisites.md#S3).
- 4. Set the `KeyName`, please check [details](./prerequisites.md#keypair).
+ 4. Set the `KeyName`, please check [Create a
keypair](./prerequisites.md#keypair).
5. Set the `CIDR_IP`, make sure that the `CIDR_IP` match the pattern
`xxx.xxx.xxx.xxx/16[|24|32]`.
@@ -33,13 +33,20 @@
>
> 2. In one word, it will let your mac which IP is in the `CIDR_IP` to
access instances.
-4. Init local env.
+4. Init python env.
```shell
$ bin/init.sh
+$ source venv/bin/activate
```
-> Note: Following the information into a python virtual env and get the help
messages.
+Check the python version:
+
+```shell
+$ python --version
+```
+
+> Note: If Python is already installed locally, you need to ensure that the
python version is 3.6.6 or later.
5. Execute commands to deploy a cluster quickly.
@@ -51,8 +58,8 @@ After this cluster is ready, you will see the message `Kylin
Cluster already sta
> Note:
>
-> 1. For more details about the properties of kylin4 in a cluster, please
check [configure kylin.properties](./prerequisites.md#cluster).
-> 2. For more details about the index of the clusters, please check [Indexes
of clusters](./prerequisites.md#indexofcluster).
+> 1. For more details about the properties of kylin4 in a cluster, please
check [configure kylin.properties](./configs.md#cluster).
+> 2. For more details about the index of the clusters, please check [Indexes
of clusters](./configs.md#indexofcluster).
6. Execute commands to list nodes of the cluster.
diff --git a/readme/quick_start_for_multiple_clusters.md
b/readme/quick_start_for_multiple_clusters.md
index 14ee9a7..fd18836 100644
--- a/readme/quick_start_for_multiple_clusters.md
+++ b/readme/quick_start_for_multiple_clusters.md
@@ -8,17 +8,17 @@
>
> 1. `CLUSTER_INDEXES` means that cluster index is in the range of
`CLUSTER_INDEXES`.
> 2. Configs for multiple clusters are also from `kylin_configs.yaml`.
- > 3. For more details about the index of the clusters, please check
[Indexes of clusters](./prerequisites.md#indexofcluster).
+ > 3. For more details about the index of the clusters, please check
[Indexes of clusters](./configs.md#indexofcluster).
-2. Copy `kylin.properties.template` for expecting clusters to deploy, please
check the [details](./prerequisites.md#cluster).
+2. Copy `kylin.properties.template` for expecting clusters to deploy, please
check the [details](./configs.md#cluster).
-3. Execute commands to deploy `all` clusters.
+3. Execute commands to deploy all of the clusters.
```shell
python deploy.py --type deploy --cluster all
```
-4. Destroy all clusters.
+4. Destroy all of the clusters.
```shell
python deploy.py --type destroy --cluster all
