This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0bf6d1fd79 [typo](doc)Datax doris writer doc update (#14328)
0bf6d1fd79 is described below
commit 0bf6d1fd79ec040b48039cae161dc18c214375ea
Author: jiafeng.zhang <zhang...@gmail.com>
AuthorDate: Thu Nov 17 08:53:55 2022 +0800
[typo](doc)Datax doris writer doc update (#14328)
---
docs/en/docs/ecosystem/datax.md | 207 ++++++++++++++++++++++++++++++++++--
docs/zh-CN/docs/ecosystem/datax.md | 208 +++++++++++++++++++++++++++++++++++--
2 files changed, 398 insertions(+), 17 deletions(-)
diff --git a/docs/en/docs/ecosystem/datax.md b/docs/en/docs/ecosystem/datax.md
index e3f661fe74..4322f75b42 100644
--- a/docs/en/docs/ecosystem/datax.md
+++ b/docs/en/docs/ecosystem/datax.md
@@ -67,6 +67,8 @@ Because the doriswriter plug-in depends on some modules in
the DataX code base,
### How to build
+#### Doris code base compilation
+
1. Run `init-env.sh`
2. Modify code of doriswriter in `DataX/doriswriter` if you need.
3. Build doriswriter
@@ -83,7 +85,7 @@ Because the doriswriter plug-in depends on some modules in
the DataX code base,
> hdfsreader, hdfswriter and oscarwriter needs some extra jar
packages. If you don't need to use these components, you can comment out the
corresponding module in DataX/pom.xml.
- 3. Compilation error
+ 3. Compilation error
If you encounter the following compilation errors:
@@ -99,6 +101,101 @@ Because the doriswriter plug-in depends on some modules in
the DataX code base,
4. Commit code of doriswriter in `doriswriter` if you need.
+#### Datax code base compilation
+
+Pull the code from the datax code library and execute the compilation
+
+```
+git clone https://github.com/alibaba/DataX.git
+cd datax
+mvn package assembly:assembly -Dmaven.test.skip=true
+```
+
+After compiling, you can see the datax.tar.gz package under
`datax/target/Datax`
+
+### Datax DorisWriter parameter introduction:
+
+* **jdbcUrl**
+
+ - Description: Doris's JDBC connection string, the user executes preSql or
postSQL.
+ - Mandatory: Yes
+ - Default: None
+* **loadUrl**
+
+ - Description: As a connection target for Stream Load. The format is
"ip:port". Where IP is the FE node IP, port is the http_port of the FE node.
You can fill in more than one, separated by a semicolon in English: `;`,
doriswriter will visit in a polling manner.
+ - Mandatory: Yes
+ - Default: None
+* **username**
+
+ - Description: The username to access the Doris database
+ - Mandatory: Yes
+ - Default: None
+* **password**
+
+ - Description: Password to access Doris database
+ - Mandatory: No
+ - Default: empty
+* **connection.selectedDatabase**
+ - Description: The name of the Doris database that needs to be written.
+ - Mandatory: Yes
+ - Default: None
+* **connection. table**
+ - Description: The name of the Doris table that needs to be written.
+ - Mandatory: Yes
+ - Default: None
+* **flushInterval**
+ - Description: The time interval at which data is written in batches. If
this time interval is set too small, it will cause Doris write blocking
problem, error code -235, and if you set this time interval too small,
`maxBatchRows` and `batchSize` parameters are set too large, then it may not be
able to reach you The data size set by this will also be imported.
+ - Mandatory: No
+ - Default: 30000 (ms)
+* **column**
+ - Description: The fields that the destination table needs to write data
into, these fields will be used as the field names of the generated Json data.
Fields are separated by commas. For example: "column": ["id","name","age"].
+ - Mandatory: Yes
+ - Default: No
+* **preSql**
+
+ - Description: Before writing data to the destination table, the standard
statement here will be executed first.
+ - Mandatory: No
+ - Default: None
+* **postSql**
+
+ - Description: After writing data to the destination table, the standard
statement here will be executed.
+ - Mandatory: No
+ - Default: None
+
+
+* **maxBatchRows**
+ - Description: The maximum number of rows for each batch of imported data.
Together with **maxBatchSize**, it controls the number of imported record rows
per batch. When each batch of data reaches one of the two thresholds, the data
of this batch will start to be imported.
+ - Mandatory: No
+ - Default: 500000
+
+* **batchSize**
+ - Description: The maximum amount of data imported in each batch. Works with
**maxBatchRows** to control the number of imports per batch. When each batch of
data reaches one of the two thresholds, the data of this batch will start to be
imported.
+ - Mandatory: No
+ - Default: 104857600
+
+* **maxRetries**
+
+ - Description: The number of retries after each batch of failed data imports.
+ - Mandatory: No
+ - Default: 3
+
+
+* **labelPrefix**
+
+ - Description: The label prefix for each batch of imported tasks. The final
label will have `labelPrefix + UUID` to form a globally unique label to ensure
that data will not be imported repeatedly
+ - Mandatory: No
+ - Default: `datax_doris_writer_`
+
+* **loadProps**
+
+ - Description: The request parameter of StreamLoad. For details, refer to
the StreamLoad introduction page. [Stream load - Apache
Doris](https://doris.apache.org/zh-CN/docs/data-operate/import/import-way/stream-load-manual)
+
+ This includes the imported data format: format, etc. The imported data
format defaults to csv, which supports JSON. For details, please refer to the
type conversion section below, or refer to the official information of Stream
load above.
+
+ - Mandatory: No
+
+ - Default: None
+
### Example
#### 1. Stream reads the data and imports it to Doris
@@ -133,8 +230,8 @@ CREATE TABLE `ods_t_test` (
`unit_no` varchar(30) DEFAULT NULL COMMENT '',
`unit_name` varchar(30) DEFAULT NULL COMMENT '',
`price` decimal(12,2) DEFAULT NULL COMMENT ''
-)ENGINE=OLAP
-UNIQUE KEY(`id`, `order_code`)
+）ENGINE=OLAP
+UNIQUE KEY(id`, `order_code`)
DISTRIBUTED BY HASH(`order_code`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
@@ -145,6 +242,8 @@ PROPERTIES (
3.Create datax script
+my_import.json
+
```json
{
"job": {
@@ -153,7 +252,7 @@ PROPERTIES (
"reader": {
"name": "mysqlreader",
"parameter": {
- "column": ["emp_no", "birth_date",
"first_name","last_name","gender","hire_date"],
+ "column":
["id","order_code","line_code","remark","unit_no","unit_name","price"],
"connection": [
{
"jdbcUrl":
["jdbc:mysql://localhost:3306/demo"],
@@ -168,10 +267,10 @@ PROPERTIES (
"writer": {
"name": "doriswriter",
"parameter": {
- "loadUrl": ["172.16.0.13:8030"],
+ "loadUrl": ["127.0.0.1:8030"],
"loadProps": {
},
- "column": ["emp_no", "birth_date",
"first_name","last_name","gender","hire_date"],
+ "column":
["id","order_code","line_code","remark","unit_no","unit_name","price"],
"username": "root",
"password": "xxxxxx",
"postSql": ["select count(1) from all_employees_info"],
@@ -179,14 +278,15 @@ PROPERTIES (
"flushInterval":30000,
"connection": [
{
- "jdbcUrl": "jdbc:mysql://172.16.0.13:9030/demo",
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:9030/demo",
"selectedDatabase": "demo",
"table": ["all_employees_info"]
}
],
"loadProps": {
"format": "json",
- "strip_outer_array": true
+ "strip_outer_array":"true",
+ "line_delimiter": "\\x02"
}
}
}
@@ -201,4 +301,95 @@ PROPERTIES (
}
```
+>Remark:
+>
+>```json
+>"loadProps": {
+> "format": "json",
+> "strip_outer_array": "true",
+> "line_delimiter": "\\x02"
+>}
+>```
+>
+>1. Here we use JSON format to import data
+>2. `line_delimiter` defaults to a newline character, which may conflict with
the value in the data, we can use some special characters or invisible
characters to avoid import errors
+>3. strip_outer_array : Represents multiple rows of data in a batch of
imported data. Doris will expand the array when parsing, and then parse each
Object in it as a row of data in turn.
+>4. For more parameters of Stream load, please refer to [Stream load
document]([Stream load - Apache
Doris](https://doris.apache.org/zh-CN/docs/dev/data-operate/import/import-way
/stream-load-manual))
+>5. If it is in CSV format, we can use it like this
+>
+>```json
+>"loadProps": {
+> "format": "csv",
+> "column_separator": "\\x01",
+> "line_delimiter": "\\x02"
+>}
+>```
+>
+>**CSV format should pay special attention to row and column separators to
avoid conflicts with special characters in the data. Hidden characters are
recommended here. The default column separator is: \t, row separator: \n**
+
4.Execute the datax task, refer to the specific [datax official
website](https://github.com/alibaba/DataX/blob/master/userGuid.md)
+
+```
+python bin/datax.py my_import.json
+```
+
+After execution, we can see the following information
+
+```
+2022-11-16 14:28:54.012 [job-0] INFO JobContainer - jobContainer starts to do
prepare ...
+2022-11-16 14:28:54.012 [job-0] INFO JobContainer - DataX Reader.Job
[mysqlreader] do prepare work .
+2022-11-16 14:28:54.013 [job-0] INFO JobContainer - DataX Writer.Job
[doriswriter] do prepare work .
+2022-11-16 14:28:54.020 [job-0] INFO JobContainer - jobContainer starts to do
split ...
+2022-11-16 14:28:54.020 [job-0] INFO JobContainer - Job set Channel-Number to
1 channels.
+2022-11-16 14:28:54.023 [job-0] INFO JobContainer - DataX Reader.Job
[mysqlreader] splits to [1] tasks.
+2022-11-16 14:28:54.023 [job-0] INFO JobContainer - DataX Writer.Job
[doriswriter] splits to [1] tasks.
+2022-11-16 14:28:54.033 [job-0] INFO JobContainer - jobContainer starts to do
schedule ...
+2022-11-16 14:28:54.036 [job-0] INFO JobContainer - Scheduler starts [1]
taskGroups.
+2022-11-16 14:28:54.037 [job-0] INFO JobContainer - Running by standalone
Mode.
+2022-11-16 14:28:54.041 [taskGroup-0] INFO TaskGroupContainer -
taskGroupId=[0] start [1] channels for [1] tasks.
+2022-11-16 14:28:54.043 [taskGroup-0] INFO Channel - Channel set
byte_speed_limit to -1, No bps activated.
+2022-11-16 14:28:54.043 [taskGroup-0] INFO Channel - Channel set
record_speed_limit to -1, No tps activated.
+2022-11-16 14:28:54.049 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0]
taskId[0] attemptCount[1] is started
+2022-11-16 14:28:54.052 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to
read record by Sql: [select
taskid,projectid,taskflowid,templateid,template_name,status_task from
dwd_universal_tb_task
+]
jdbcUrl:[jdbc:mysql://localhost:3306/demo?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
+Wed Nov 16 14:28:54 GMT+08:00 2022 WARN: Establishing SSL connection without
server's identity verification is not recommended. According to MySQL 5.5.45+,
5.6.26+ and 5.7.6+ requirements SSL connection must be established by default
if explicit option isn't set. For compliance with existing applications not
using SSL the verifyServerCertificate property is set to 'false'. You need
either to explicitly disable SSL by setting useSSL=false, or set useSSL=true
and provide truststore for ser [...]
+2022-11-16 14:28:54.071 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished
read record by Sql: [select
taskid,projectid,taskflowid,templateid,template_name,status_task from
dwd_universal_tb_task
+]
jdbcUrl:[jdbc:mysql://localhost:3306/demo?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
+2022-11-16 14:28:54.104 [Thread-1] INFO DorisStreamLoadObserver - Start to
join batch data: rows[2] bytes[438]
label[datax_doris_writer_c4e08cb9-c157-4689-932f-db34acc45b6f].
+2022-11-16 14:28:54.104 [Thread-1] INFO DorisStreamLoadObserver - Executing
stream load to:
'http://127.0.0.1:8030/api/demo/dwd_universal_tb_task/_stream_load', size: '441'
+2022-11-16 14:28:54.224 [Thread-1] INFO DorisStreamLoadObserver - StreamLoad
response
:{"Status":"Success","BeginTxnTimeMs":0,"Message":"OK","NumberUnselectedRows":0,"CommitAndPublishTimeMs":17,"Label":"datax_doris_writer_c4e08cb9-c157-4689-932f-db34acc45b6f","LoadBytes":441,"StreamLoadPutTimeMs":1,"NumberTotalRows":2,"WriteDataTimeMs":11,"TxnId":217056,"LoadTimeMs":31,"TwoPhaseCommit":"false","ReadDataTimeMs":0,"NumberLoadedRows":2,"NumberFilteredRows":0}
+2022-11-16 14:28:54.225 [Thread-1] INFO DorisWriterManager - Async stream
load finished: label[datax_doris_writer_c4e08cb9-c157-4689-932f-db34acc45b6f].
+2022-11-16 14:28:54.249 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0]
taskId[0] is successed, used[201]ms
+2022-11-16 14:28:54.250 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0]
completed it's tasks.
+2022-11-16 14:29:04.048 [job-0] INFO StandAloneJobContainerCommunicator -
Total 2 records, 214 bytes | Speed 21B/s, 0 records/s | Error 0 records, 0
bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s |
Percentage 100.00%
+2022-11-16 14:29:04.049 [job-0] INFO AbstractScheduler - Scheduler
accomplished all tasks.
+2022-11-16 14:29:04.049 [job-0] INFO JobContainer - DataX Writer.Job
[doriswriter] do post work.
+Wed Nov 16 14:29:04 GMT+08:00 2022 WARN: Establishing SSL connection without
server's identity verification is not recommended. According to MySQL 5.5.45+,
5.6.26+ and 5.7.6+ requirements SSL connection must be established by default
if explicit option isn't set. For compliance with existing applications not
using SSL the verifyServerCertificate property is set to 'false'. You need
either to explicitly disable SSL by setting useSSL=false, or set useSSL=true
and provide truststore for ser [...]
+2022-11-16 14:29:04.187 [job-0] INFO DorisWriter$Job - Start to execute
preSqls:[select count(1) from dwd_universal_tb_task]. context
info:jdbc:mysql://172.16.0.13:9030/demo.
+2022-11-16 14:29:04.204 [job-0] INFO JobContainer - DataX Reader.Job
[mysqlreader] do post work.
+2022-11-16 14:29:04.204 [job-0] INFO JobContainer - DataX jobId [0] completed
successfully.
+2022-11-16 14:29:04.204 [job-0] INFO HookInvoker - No hook invoked, because
base dir not exists or is a file: /data/datax/hook
+2022-11-16 14:29:04.205 [job-0] INFO JobContainer -
+ [total cpu info] =>
+ averageCpu | maxDeltaCpu
| minDeltaCpu
+ -1.00% | -1.00%
| -1.00%
+
+
+ [total gc info] =>
+ NAME | totalGCCount | maxDeltaGCCount
| minDeltaGCCount | totalGCTime | maxDeltaGCTime |
minDeltaGCTime
+ PS MarkSweep | 1 | 1
| 1 | 0.017s | 0.017s | 0.017s
+ PS Scavenge | 1 | 1
| 1 | 0.007s | 0.007s | 0.007s
+
+2022-11-16 14:29:04.205 [job-0] INFO JobContainer - PerfTrace not enable!
+2022-11-16 14:29:04.206 [job-0] INFO StandAloneJobContainerCommunicator -
Total 2 records, 214 bytes | Speed 21B/s, 0 records/s | Error 0 records, 0
bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s |
Percentage 100.00%
+2022-11-16 14:29:04.206 [job-0] INFO JobContainer -
+任务启动时刻 : 2022-11-16 14:28:53
+任务结束时刻 : 2022-11-16 14:29:04
+任务总计耗时 : 10s
+任务平均流量 : 21B/s
+记录写入速度 : 0rec/s
+读出记录总数 : 2
+读写失败总数 : 0
+
+```
+
diff --git a/docs/zh-CN/docs/ecosystem/datax.md
b/docs/zh-CN/docs/ecosystem/datax.md
index f2978c5ace..2d623827eb 100644
--- a/docs/zh-CN/docs/ecosystem/datax.md
+++ b/docs/zh-CN/docs/ecosystem/datax.md
@@ -67,6 +67,8 @@ doriswriter 插件依赖的 DataX 代码中的一些模块。而这些模块并
### 编译
+#### Doris 代码库编译
+
1. 运行 `init-env.sh`
2. 按需修改 `DataX/doriswriter` 中的代码。
3. 编译 doriswriter：
@@ -99,6 +101,101 @@ doriswriter 插件依赖的 DataX 代码中的一些模块。而这些模块并
4. 按需提交修改。
+#### Datax 代码库编译
+
+从datax 代码库拉取代码，执行编译
+
+```
+git clone https://github.com/alibaba/DataX.git
+cd datax
+mvn package assembly:assembly -Dmaven.test.skip=true
+```
+
+编译完成后可以在 `datax/target/Datax` 下看到datax.tar.gz 包
+
+### Datax DorisWriter 参数介绍：
+
+* **jdbcUrl**
+
+ - 描述：Doris 的 JDBC 连接串，用户执行 preSql 或 postSQL。
+ - 必选：是
+ - 默认值：无
+* **loadUrl**
+
+ - 描述：作为 Stream Load 的连接目标。格式为 "ip:port"。其中 IP 是 FE 节点 IP，port 是 FE 节点的
http_port。可以填写多个，多个之间使用英文状态的分号隔开:`;`，doriswriter 将以轮询的方式访问。
+ - 必选：是
+ - 默认值：无
+* **username**
+
+ - 描述：访问Doris数据库的用户名
+ - 必选：是
+ - 默认值：无
+* **password**
+
+ - 描述：访问Doris数据库的密码
+ - 必选：否
+ - 默认值：空
+* **connection.selectedDatabase**
+ - 描述：需要写入的Doris数据库名称。
+ - 必选：是
+ - 默认值：无
+* **connection.table**
+ - 描述：需要写入的Doris表名称。
+ - 必选：是
+ - 默认值：无
+* **flushInterval**
+ - 描述：数据写入批次的时间间隔。如果这个时间间隔设置的太小会造成 Doris 写阻塞问题，错误代码
-235，同时如果你这个时间设置太小，`maxBatchRows` 和 `batchSize`
参数设置的有很大，那么很可能达不到你这设置的数据量大小，也会执行导入。
+ - 必选：否
+ - 默认值：30000（ms）
+* **column**
+ - 描述：目的表需要写入数据的字段，这些字段将作为生成的 Json 数据的字段名。字段之间用英文逗号分隔。例如: "column":
["id","name","age"]。
+ - 必选：是
+ - 默认值：否
+* **preSql**
+
+ - 描述：写入数据到目的表前，会先执行这里的标准语句。
+ - 必选：否
+ - 默认值：无
+* **postSql**
+
+ - 描述：写入数据到目的表后，会执行这里的标准语句。
+ - 必选：否
+ - 默认值：无
+
+
+* **maxBatchRows**
+ - 描述：每批次导入数据的最大行数。和 **maxBatchSize**
共同控制每批次的导入记录行数。每批次数据达到两个阈值之一，即开始导入这一批次的数据。
+ - 必选：否
+ - 默认值：500000
+
+* **batchSize**
+ - 描述：每批次导入数据的最大数据量。和 **maxBatchRows**
共同控制每批次的导入数量。每批次数据达到两个阈值之一，即开始导入这一批次的数据。
+ - 必选：否
+ - 默认值：104857600
+
+* **maxRetries**
+
+ - 描述：每批次导入数据失败后的重试次数。
+ - 必选：否
+ - 默认值：3
+
+
+* **labelPrefix**
+
+ - 描述：每批次导入任务的 label 前缀。最终的 label 将有 `labelPrefix + UUID` 组成全局唯一的
label，确保数据不会重复导入
+ - 必选：否
+ - 默认值：`datax_doris_writer_`
+
+* **loadProps**
+
+ - 描述：StreamLoad 的请求参数，详情参照StreamLoad介绍页面。[Stream load - Apache
Doris](https://doris.apache.org/zh-CN/docs/data-operate/import/import-way/stream-load-manual)
+
+ 这里包括导入的数据格式：format等，导入数据格式默认我们使用csv，支持JSON，具体可以参照下面类型转换部分，也可以参照上面Stream
load 官方信息
+
+ - 必选：否
+
+ - 默认值：无
+
### 示例
#### 1.Stream读取数据后导入至Doris
@@ -133,8 +230,8 @@ CREATE TABLE `ods_t_test` (
`unit_no` varchar(30) DEFAULT NULL COMMENT '',
`unit_name` varchar(30) DEFAULT NULL COMMENT '',
`price` decimal(12,2) DEFAULT NULL COMMENT ''
-)ENGINE=OLAP
-UNIQUE KEY(`id`, `order_code`)
+）ENGINE=OLAP
+UNIQUE KEY(id`, `order_code`)
DISTRIBUTED BY HASH(`order_code`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
@@ -143,7 +240,9 @@ PROPERTIES (
);
```
-3.创建datax脚本
+3.创建datax脚本
+
+my_import.json
```json
{
@@ -153,7 +252,7 @@ PROPERTIES (
"reader": {
"name": "mysqlreader",
"parameter": {
- "column": ["emp_no", "birth_date",
"first_name","last_name","gender","hire_date"],
+ "column":
["id","order_code","line_code","remark","unit_no","unit_name","price"],
"connection": [
{
"jdbcUrl":
["jdbc:mysql://localhost:3306/demo"],
@@ -168,10 +267,10 @@ PROPERTIES (
"writer": {
"name": "doriswriter",
"parameter": {
- "loadUrl": ["172.16.0.13:8030"],
+ "loadUrl": ["127.0.0.1:8030"],
"loadProps": {
},
- "column": ["emp_no", "birth_date",
"first_name","last_name","gender","hire_date"],
+ "column":
["id","order_code","line_code","remark","unit_no","unit_name","price"],
"username": "root",
"password": "xxxxxx",
"postSql": ["select count(1) from all_employees_info"],
@@ -179,14 +278,15 @@ PROPERTIES (
"flushInterval":30000,
"connection": [
{
- "jdbcUrl": "jdbc:mysql://172.16.0.13:9030/demo",
+ "jdbcUrl": "jdbc:mysql://127.0.0.1:9030/demo",
"selectedDatabase": "demo",
"table": ["all_employees_info"]
}
],
"loadProps": {
"format": "json",
- "strip_outer_array": true
+ "strip_outer_array":"true",
+ "line_delimiter": "\\x02"
}
}
}
@@ -200,6 +300,96 @@ PROPERTIES (
}
}
```
-> 注意：```format``` 参数默认值：```csv```，如果想要使用```json```格式，需要在配置中指定```format```
为```json```.
+
+>备注：
+>
+>```json
+>"loadProps": {
+> "format": "json",
+> "strip_outer_array":"true",
+> "line_delimiter": "\\x02"
+>}
+>```
+>
+>1. 这里我们使用了 JSON 格式导入数据
+>2. `line_delimiter` 默认是换行符，可能会和数据中的值冲突，我们可以使用一些特殊字符或者不可见字符，避免导入错误
+>3. strip_outer_array ：在一批导入数据中表示多行数据，Doris 在解析时会将数组展开，然后依次解析其中的每一个 Object
作为一行数据
+>4. 更多 Stream load 参数请参照 [Stream load文档]([Stream load - Apache
Doris](https://doris.apache.org/zh-CN/docs/dev/data-operate/import/import-way/stream-load-manual))
+>5. 如果是 CSV 格式我们可以这样使用
+>
+>```json
+>"loadProps": {
+> "format": "csv",
+> "column_separator": "\\x01",
+> "line_delimiter": "\\x02"
+>}
+>```
+>
+>**CSV 格式要特别注意行列分隔符，避免和数据中的特殊字符冲突，这里建议使用隐藏字符，默认列分隔符是：\t，行分隔符：\n**
4.执行datax任务，具体参考
[datax官网](https://github.com/alibaba/DataX/blob/master/userGuid.md)
+
+```
+python bin/datax.py my_import.json
+```
+
+执行之后我们可以看到下面的信息
+
+```
+2022-11-16 14:28:54.012 [job-0] INFO JobContainer - jobContainer starts to do
prepare ...
+2022-11-16 14:28:54.012 [job-0] INFO JobContainer - DataX Reader.Job
[mysqlreader] do prepare work .
+2022-11-16 14:28:54.013 [job-0] INFO JobContainer - DataX Writer.Job
[doriswriter] do prepare work .
+2022-11-16 14:28:54.020 [job-0] INFO JobContainer - jobContainer starts to do
split ...
+2022-11-16 14:28:54.020 [job-0] INFO JobContainer - Job set Channel-Number to
1 channels.
+2022-11-16 14:28:54.023 [job-0] INFO JobContainer - DataX Reader.Job
[mysqlreader] splits to [1] tasks.
+2022-11-16 14:28:54.023 [job-0] INFO JobContainer - DataX Writer.Job
[doriswriter] splits to [1] tasks.
+2022-11-16 14:28:54.033 [job-0] INFO JobContainer - jobContainer starts to do
schedule ...
+2022-11-16 14:28:54.036 [job-0] INFO JobContainer - Scheduler starts [1]
taskGroups.
+2022-11-16 14:28:54.037 [job-0] INFO JobContainer - Running by standalone
Mode.
+2022-11-16 14:28:54.041 [taskGroup-0] INFO TaskGroupContainer -
taskGroupId=[0] start [1] channels for [1] tasks.
+2022-11-16 14:28:54.043 [taskGroup-0] INFO Channel - Channel set
byte_speed_limit to -1, No bps activated.
+2022-11-16 14:28:54.043 [taskGroup-0] INFO Channel - Channel set
record_speed_limit to -1, No tps activated.
+2022-11-16 14:28:54.049 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0]
taskId[0] attemptCount[1] is started
+2022-11-16 14:28:54.052 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to
read record by Sql: [select
taskid,projectid,taskflowid,templateid,template_name,status_task from
dwd_universal_tb_task
+]
jdbcUrl:[jdbc:mysql://localhost:3306/demo?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
+Wed Nov 16 14:28:54 GMT+08:00 2022 WARN: Establishing SSL connection without
server's identity verification is not recommended. According to MySQL 5.5.45+,
5.6.26+ and 5.7.6+ requirements SSL connection must be established by default
if explicit option isn't set. For compliance with existing applications not
using SSL the verifyServerCertificate property is set to 'false'. You need
either to explicitly disable SSL by setting useSSL=false, or set useSSL=true
and provide truststore for ser [...]
+2022-11-16 14:28:54.071 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished
read record by Sql: [select
taskid,projectid,taskflowid,templateid,template_name,status_task from
dwd_universal_tb_task
+]
jdbcUrl:[jdbc:mysql://localhost:3306/demo?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
+2022-11-16 14:28:54.104 [Thread-1] INFO DorisStreamLoadObserver - Start to
join batch data: rows[2] bytes[438]
label[datax_doris_writer_c4e08cb9-c157-4689-932f-db34acc45b6f].
+2022-11-16 14:28:54.104 [Thread-1] INFO DorisStreamLoadObserver - Executing
stream load to:
'http://127.0.0.1:8030/api/demo/dwd_universal_tb_task/_stream_load', size: '441'
+2022-11-16 14:28:54.224 [Thread-1] INFO DorisStreamLoadObserver - StreamLoad
response
:{"Status":"Success","BeginTxnTimeMs":0,"Message":"OK","NumberUnselectedRows":0,"CommitAndPublishTimeMs":17,"Label":"datax_doris_writer_c4e08cb9-c157-4689-932f-db34acc45b6f","LoadBytes":441,"StreamLoadPutTimeMs":1,"NumberTotalRows":2,"WriteDataTimeMs":11,"TxnId":217056,"LoadTimeMs":31,"TwoPhaseCommit":"false","ReadDataTimeMs":0,"NumberLoadedRows":2,"NumberFilteredRows":0}
+2022-11-16 14:28:54.225 [Thread-1] INFO DorisWriterManager - Async stream
load finished: label[datax_doris_writer_c4e08cb9-c157-4689-932f-db34acc45b6f].
+2022-11-16 14:28:54.249 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0]
taskId[0] is successed, used[201]ms
+2022-11-16 14:28:54.250 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0]
completed it's tasks.
+2022-11-16 14:29:04.048 [job-0] INFO StandAloneJobContainerCommunicator -
Total 2 records, 214 bytes | Speed 21B/s, 0 records/s | Error 0 records, 0
bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s |
Percentage 100.00%
+2022-11-16 14:29:04.049 [job-0] INFO AbstractScheduler - Scheduler
accomplished all tasks.
+2022-11-16 14:29:04.049 [job-0] INFO JobContainer - DataX Writer.Job
[doriswriter] do post work.
+Wed Nov 16 14:29:04 GMT+08:00 2022 WARN: Establishing SSL connection without
server's identity verification is not recommended. According to MySQL 5.5.45+,
5.6.26+ and 5.7.6+ requirements SSL connection must be established by default
if explicit option isn't set. For compliance with existing applications not
using SSL the verifyServerCertificate property is set to 'false'. You need
either to explicitly disable SSL by setting useSSL=false, or set useSSL=true
and provide truststore for ser [...]
+2022-11-16 14:29:04.187 [job-0] INFO DorisWriter$Job - Start to execute
preSqls:[select count(1) from dwd_universal_tb_task]. context
info:jdbc:mysql://172.16.0.13:9030/demo.
+2022-11-16 14:29:04.204 [job-0] INFO JobContainer - DataX Reader.Job
[mysqlreader] do post work.
+2022-11-16 14:29:04.204 [job-0] INFO JobContainer - DataX jobId [0] completed
successfully.
+2022-11-16 14:29:04.204 [job-0] INFO HookInvoker - No hook invoked, because
base dir not exists or is a file: /data/datax/hook
+2022-11-16 14:29:04.205 [job-0] INFO JobContainer -
+ [total cpu info] =>
+ averageCpu | maxDeltaCpu
| minDeltaCpu
+ -1.00% | -1.00%
| -1.00%
+
+
+ [total gc info] =>
+ NAME | totalGCCount | maxDeltaGCCount
| minDeltaGCCount | totalGCTime | maxDeltaGCTime |
minDeltaGCTime
+ PS MarkSweep | 1 | 1
| 1 | 0.017s | 0.017s | 0.017s
+ PS Scavenge | 1 | 1
| 1 | 0.007s | 0.007s | 0.007s
+
+2022-11-16 14:29:04.205 [job-0] INFO JobContainer - PerfTrace not enable!
+2022-11-16 14:29:04.206 [job-0] INFO StandAloneJobContainerCommunicator -
Total 2 records, 214 bytes | Speed 21B/s, 0 records/s | Error 0 records, 0
bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s |
Percentage 100.00%
+2022-11-16 14:29:04.206 [job-0] INFO JobContainer -
+任务启动时刻 : 2022-11-16 14:28:53
+任务结束时刻 : 2022-11-16 14:29:04
+任务总计耗时 : 10s
+任务平均流量 : 21B/s
+记录写入速度 : 0rec/s
+读出记录总数 : 2
+读写失败总数 : 0
+
+```
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
