This is an automated email from the ASF dual-hosted git repository. kassiez pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push: new 9e79896a08e fix load job (#2033) 9e79896a08e is described below commit 9e79896a08eddb1d2255b9381fdd841a5f345a1c Author: echo-dundun <50286010+echo-...@users.noreply.github.com> AuthorDate: Mon Feb 24 17:17:26 2025 +0800 fix load job (#2033) ## Versions - [x] dev - [x] 3.0 - [x] 2.1 - [x] 2.0 ## Languages - [x] Chinese - [x] English ## Docs Checklist - [ ] Checked by AI - [ ] Test Cases Built --- .../load-and-export/CREATE-SYNC-JOB.md | 167 +++++++++---------- .../load-and-export/PAUSE-SYNC-JOB.md | 31 ++-- .../load-and-export/RESUME-SYNC-JOB.md | 33 ++-- .../load-and-export/SHOW-SYNC-JOB.md | 34 ++-- .../load-and-export/STOP-SYNC-JOB.md | 32 ++-- .../data-modification/load-and-export/SYNC.md | 20 ++- .../load-and-export/CREATE-SYNC-JOB.md | 174 +++++++++----------- .../load-and-export/PAUSE-SYNC-JOB.md | 29 ++-- .../load-and-export/RESUME-SYNC-JOB.md | 30 ++-- .../load-and-export/SHOW-SYNC-JOB.md | 42 +++-- .../load-and-export/STOP-SYNC-JOB.md | 31 ++-- .../data-modification/load-and-export/SYNC.md | 22 +-- .../Load/PAUSE-SYNC-JOB.md | 36 ++--- .../sql-reference/Utility-Statements/SYNC.md | 27 ++-- .../load-and-export/CREATE-SYNC-JOB.md | 176 +++++++++------------ .../load-and-export/PAUSE-SYNC-JOB.md | 28 ++-- .../load-and-export/RESUME-SYNC-JOB.md | 30 ++-- .../load-and-export/SHOW-SYNC-JOB.md | 37 ++--- .../load-and-export/STOP-SYNC-JOB.md | 31 ++-- .../data-modification/load-and-export/SYNC.md | 23 +-- .../load-and-export/CREATE-SYNC-JOB.md | 174 +++++++++----------- .../load-and-export/PAUSE-SYNC-JOB.md | 29 ++-- .../load-and-export/RESUME-SYNC-JOB.md | 31 ++-- .../load-and-export/SHOW-SYNC-JOB.md | 39 +++-- .../load-and-export/STOP-SYNC-JOB.md | 31 ++-- .../data-modification/load-and-export/SYNC.md | 23 ++- .../Load/PAUSE-SYNC-JOB.md | 39 ++--- .../sql-reference/Utility-Statements/SYNC.md | 29 ++-- .../load-and-export/CREATE-SYNC-JOB.md | 165 +++++++++---------- .../load-and-export/PAUSE-SYNC-JOB.md | 31 ++-- .../load-and-export/RESUME-SYNC-JOB.md | 32 ++-- .../load-and-export/SHOW-SYNC-JOB.md | 32 ++-- .../load-and-export/STOP-SYNC-JOB.md | 32 ++-- .../data-modification/load-and-export/SYNC.md | 19 +-- .../load-and-export/CREATE-SYNC-JOB.md | 167 +++++++++---------- .../load-and-export/PAUSE-SYNC-JOB.md | 31 ++-- .../load-and-export/RESUME-SYNC-JOB.md | 33 ++-- .../load-and-export/SHOW-SYNC-JOB.md | 34 ++-- .../load-and-export/STOP-SYNC-JOB.md | 32 ++-- .../data-modification/load-and-export/SYNC.md | 20 ++- 40 files changed, 985 insertions(+), 1071 deletions(-) diff --git a/docs/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md b/docs/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md index 73c1a9426ad..c8c170f4bdc 100644 --- a/docs/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md +++ b/docs/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md @@ -25,133 +25,110 @@ under the License. --> - ## Description -The data synchronization (Sync Job) function supports users to submit a resident data synchronization job, and incrementally synchronizes the CDC (Change Data Capture) of the user's data update operation in the Mysql database by reading the Binlog log from the specified remote address. Features. - -Currently, the data synchronization job only supports connecting to Canal, obtaining the parsed Binlog data from the Canal Server and importing it into Doris. +The data synchronization (Sync Job) function allows users to submit a persistent data synchronization job. It incrementally synchronizes the CDC (Change Data Capture) of data update operations from a MySQL database by reading the Binlog from a specified remote source. Currently, the synchronization job supports connecting to Canal, obtaining parsed Binlog data from the Canal server, and importing it into Doris. -Users can view the data synchronization job status through [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB). +Users can view the status of synchronization jobs via [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB). -grammar: +## Syntax ```sql -CREATE SYNC [db.]job_name - ( - channel_desc, - channel_desc - ... - ) +CREATE SYNC [<db>.]<job_name> +(<channel_desc> [, ... ]) +<binlog_desc> +``` +where: +```sql +channel_desc + : FROM <mysql_db>.<src_tbl> INTO <des_tbl> [ <columns_mapping> ] +``` +```sql binlog_desc + : FROM BINLOG ("<key>" = "<value>" [, ... ]) ``` -1. `job_name` +## Required Parameters - The synchronization job name is the unique identifier of the job in the current database. Only one job with the same `job_name` can be running. +**1. `<job_name>`** -2. `channel_desc` +> Specifies the unique name of the synchronization job within the current database. Only one job with the same `<job_name>` can be running at a time. - The data channel under the job is used to describe the mapping relationship between the mysql source table and the doris target table. +**2. `<channel_desc>`** - grammar: +> Describes the mapping relationship between the MySQL source table and the Doris target table. +> +> +> - **`<mysql_db.src_tbl>`**: Specifies the source table in MySQL (including the database name). +> - **`<des_tbl>`**: Specifies the target table in Doris. The target table must be unique, and its batch delete function must be enabled. +> - **`<columns_mapping>`** (Optional): Defines the mapping between columns of the source and target tables. If omitted, columns are mapped one-to-one in order. Note that the form `col_name = expr` is not supported. - ```sql - FROM mysql_db.src_tbl INTO des_tbl - [columns_mapping] - ``` - - 1. `mysql_db.src_tbl` - - Specify the database and source table on the mysql side. - - 2. `des_tbl` - - Specify the target table on the doris side. Only unique tables are supported, and the batch delete function of the table needs to be enabled (see the 'batch delete function' of help alter table for how to enable it). - - 4. `column_mapping` - - Specifies the mapping relationship between the columns of the mysql source table and the doris target table. If not specified, FE will default the columns of the source table and the target table to one-to-one correspondence in order. - - The form col_name = expr is not supported for columns. - - Example: - - ``` - Suppose the target table column is (k1, k2, v1), - - Change the order of columns k1 and k2 - (k2, k1, v1) - - Ignore the fourth column of the source data - (k2, k1, v1, dummy_column) - ``` - -3. `binlog_desc` - - Used to describe the remote data source, currently only one canal is supported. - - grammar: +**3. `<binlog_desc>`** - ```sql - FROM BINLOG - ( - "key1" = "value1", - "key2" = "value2" - ) - ``` +> Describes the remote data source for the Binlog. +> +> The properties for the Canal data source (keys prefixed with `canal.`) include: +> +> - **`canal.server.ip`**: Address of the Canal server. +> - **`canal.server.port`**: Port of the Canal server. +> - **`canal.destination`**: Identifier of the Canal instance. +> - **`canal.batchSize`**: Maximum batch size to fetch (default is 8192). +> - **`canal.username`**: Username for the Canal instance. +> - **`canal.password`**: Password for the Canal instance. +> - **`canal.debug`** (Optional): If set to true, prints detailed batch and row information. + +## Usage Notes + +- Currently, the synchronization job only supports connecting to a Canal server. +- Only one synchronization job with the same `<job_name>` can run concurrently within a database. +- The target table specified in `<channel_desc>` must have its batch delete function enabled. - 1. The properties corresponding to the Canal data source, prefixed with `canal.` +## Access Control Requirements - 1. canal.server.ip: address of canal server - 2. canal.server.port: the port of the canal server - 3. canal.destination: the identity of the instance - 4. canal.batchSize: The maximum batch size obtained, the default is 8192 - 5. canal.username: username of instance - 6. canal.password: the password of the instance - 7. canal.debug: optional, when set to true, the batch and details of each row of data will be printed out +Users executing this SQL command must have at least the following privileges: +| Privilege | Object | Notes | +| :---------------- | :------------- | :---------------------------- | +| LOAD_PRIV | Table | This operation can only be performed by users or roles who have the LOAD_PRIV privilege for the imported table. | -## Example +## Examples -1. Simply create a data synchronization job named `job1` for `test_tbl` of `test_db`, connect to the local Canal server, corresponding to the Mysql source table `mysql_db1.tbl1`. +1. **Create a simple synchronization job** - ```SQL + Create a synchronization job named `job1` in the `test_db` database that maps the MySQL source table `mysql_db1.tbl1` to the Doris target table `test_tbl`, connecting to a local Canal server. + + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db1`.`tbl1` INTO `test_tbl` + FROM `mysql_db1`.`tbl1` INTO `test_tbl` ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "127.0.0.1", - "canal.server.port" = "11111", - "canal.destination" = "example", - "canal.username" = "", - "canal.password" = "" + "type" = "canal", + "canal.server.ip" = "127.0.0.1", + "canal.server.port" = "11111", + "canal.destination" = "example", + "canal.username" = "", + "canal.password" = "" ); ``` -2. Create a data synchronization job named `job1` for multiple tables of `test_db`, corresponding to multiple Mysql source tables one-to-one, and explicitly specify the column mapping. +2. **Create a synchronization job with multiple channels and explicit column mapping** + + Create a synchronization job named `job1` in the `test_db` database for multiple MySQL source tables with one-to-one mapping and explicitly specified column orders. - ```SQL + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1) , - FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) + FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), + FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "xx.xxx.xxx.xx", - "canal.server.port" = "12111", - "canal.destination" = "example", - "canal.username" = "username", - "canal.password" = "password" + "type" = "canal", + "canal.server.ip" = "xx.xxx.xxx.xx", + "canal.server.port" = "12111", + "canal.destination" = "example", + "canal.username" = "username", + "canal.password" = "password" ); - ``` - -## Keywords - - CREATE, SYNC, JOB - -## Best Practice + ``` \ No newline at end of file diff --git a/docs/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md b/docs/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md index e37c6300b80..56a51d682d7 100644 --- a/docs/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md +++ b/docs/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md @@ -24,29 +24,34 @@ specific language governing permissions and limitations under the License. --> - - ## Description -Pause a running resident data synchronization job in a database via `job_name`. The suspended job will stop synchronizing data and keep the latest position of consumption until it is resumed by the user. +Pause a running resident data synchronization job in a database identified by `job_name`. The suspended job will stop synchronizing data while retaining its latest consumption position until it is resumed by the user. -grammar: +## Syntax ```sql -PAUSE SYNC JOB [db.]job_name +PAUSE SYNC JOB [<db>.]<job_name> ``` -## Example +## Required Parameters -1. Pause the data sync job named `job_name`. +**1. `<job_name>`** - ```sql - PAUSE SYNC JOB `job_name`; - ``` +> Specifies the name of the synchronization job to be paused. -## Keywords +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. - PAUSE, SYNC, JOB +## Access Control Requirements + +Any user or role can perform this operation. + +## Example -## Best Practice +1. Pause the data synchronization job named `job_name`. + ```sql + PAUSE SYNC JOB `job_name`; + ``` diff --git a/docs/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md b/docs/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md index 5bac52a3254..ae48b30279f 100644 --- a/docs/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md +++ b/docs/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md @@ -24,29 +24,36 @@ specific language governing permissions and limitations under the License. --> - - ## Description -Resume a resident data synchronization job whose current database has been suspended by `job_name`, and the job will continue to synchronize data from the latest position before the last suspension. +Resume a resident data synchronization job that has been suspended in a database by its `job_name`. Once resumed, the job continues to synchronize data starting from the latest position before the suspension. -grammar: +## Syntax ```sql -RESUME SYNC JOB [db.]job_name +RESUME SYNC JOB [<db>.]<job_name> ``` -## Example +## Required Parameters + +**1. `<job_name>`** + +> Specifies the name of the data synchronization job to be resumed. + +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. + -1. Resume the data synchronization job named `job_name` +## Access Control Requirements - ```sql - RESUME SYNC JOB `job_name`; - ``` +Any user or role can perform this operation. -## Keywords - RESUME, SYNC, LOAD +## Examples -## Best Practice +1. Resume the data synchronization job named `job_name`. + ```sql + RESUME SYNC JOB `job_name`; + ``` diff --git a/docs/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md b/docs/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md index a575351668b..7cc403ab2d9 100644 --- a/docs/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md +++ b/docs/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md @@ -3,8 +3,6 @@ "title": "SHOW SYNC JOB", "language": "en" } - - --- <!-- @@ -26,18 +24,29 @@ specific language governing permissions and limitations under the License. --> - ## Description -This command is used to currently display the status of resident data synchronization jobs in all databases. +This statement displays the status of resident data synchronization jobs in all databases. -grammar: +## Syntax ```sql -SHOW SYNC JOB [FROM db_name] +SHOW SYNC JOB [FROM <db_name>] ``` -## Example +## Optional Parameters + +**1. `<db_name>`** +> `<db_name>`represents the database name, which is used to specify the database from which the information of the synchronization jobs is to be shown. + +## Access Control Requirements +Users executing this SQL command must have at least one of the following privileges: + +| Privilege | Object | Notes | +| :------------------------------------------------------------------------ | :------------- | :------------------------------------- | +| ADMIN_PRIV, SELECT_PRIV, LOAD_PRIV, ALTER_PRIV, CREATE_PRIV, DROP_PRIV, SHOW_VIEW_PRIV | Database `db_name` | This operation requires at least one of the listed privileges on the target database. | + +## Examples 1. Display the status of all data synchronization jobs in the current database. @@ -45,15 +54,8 @@ SHOW SYNC JOB [FROM db_name] SHOW SYNC JOB; ``` -2. Display the status of all data synchronization jobs under the database `test_db`. +2. Display the status of all data synchronization jobs in the `test_db` database. ```sql SHOW SYNC JOB FROM `test_db`; - ``` - -## Keywords - - SHOW, SYNC, JOB - -## Best Practice - + ``` \ No newline at end of file diff --git a/docs/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md b/docs/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md index 140058b43a2..3c82b662a28 100644 --- a/docs/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md +++ b/docs/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md @@ -24,28 +24,36 @@ specific language governing permissions and limitations under the License. --> - ## Description -Stop a non-stop resident data synchronization job in a database by `job_name`. +Stop a running resident data synchronization job in a database by specifying its `job_name`. Once stopped, the job will cease synchronizing data and release its occupied resources. -grammar: +## Syntax ```sql -STOP SYNC JOB [db.]job_name +STOP SYNC JOB [<db>.]<job_name> ``` -## Example +## Required Parameters + +**1. `<job_name>`** + +> Specifies the name of the data synchronization job to be stopped. -1. Stop the data sync job named `job_name` +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. - ```sql - STOP SYNC JOB `job_name`; - ``` -## Keywords +## Access Control Requirements - STOP, SYNC, JOB +Any user or role can perform this operation. + + +## Example -## Best Practice +1. Stop the data synchronization job named `job_name`. + ```sql + STOP SYNC JOB `job_name`; + ``` diff --git a/docs/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md b/docs/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md index 53307c0626d..1f6762a0c49 100644 --- a/docs/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md +++ b/docs/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md @@ -26,25 +26,23 @@ under the License. ## Description -Used to synchronize metadata for fe non-master nodes. doris only master node can write fe metadata, other fe nodes write metadata operations will be forwarded to master. After master finishes metadata writing operation, there will be a short delay for non-master nodes to replay metadata, you can use this statement to synchronize metadata. +This statement is used to synchronize metadata for non-master Frontend (FE) nodes. In Apache Doris, only the master FE node can write metadata, while other FE nodes forward metadata write operations to the master. After the master completes the metadata writing operation, non-master nodes may experience a short delay in replaying the metadata. You can use this statement to force synchronization of metadata. -grammar: +## Syntax ```sql SYNC; ``` -## Example +## Access Control Requirements -1. Synchronized metadata: +Any user or role can perform this operation. - ```sql - SYNC; - ``` - -## Keywords - SYNC +## Examples -## Best Practice +Synchronize metadata: + ```sql + SYNC; + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md index 40238104596..14d8b72171f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md @@ -24,135 +24,117 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -数据同步 (Sync Job) 功能,支持用户提交一个常驻的数据同步作业,通过从指定的远端地址读取 Binlog 日志,增量同步用户在 Mysql 数据库的对数据更新操作的 CDC(Change Data Capture) 功能。 +数据同步 (Sync Job) 功能支持用户提交一个常驻的数据同步作业,通过从指定的远端地址读取 Binlog 日志,增量同步用户在 MySQL 数据库中数据更新操作的 CDC (Change Data Capture) 信息。 -目前数据同步作业只支持对接 Canal,从 Canal Server 上获取解析好的 Binlog 数据,导入到 Doris 内。 +用户可通过 [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB) 查看数据同步作业的状态。 -用户可通过 [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB) 查看数据同步作业状态。 - -语法: +## 语法 ```sql -CREATE SYNC [db.]job_name - ( - channel_desc, - channel_desc - ... - ) +CREATE SYNC [<db>.]<job_name> +(<channel_desc> [, ... ]) +<binlog_desc> +``` +where: +```sql +channel_desc + : FROM <mysql_db>.<src_tbl> INTO <des_tbl> [ <columns_mapping> ] +``` +```sql binlog_desc + : FROM BINLOG ("<key>" = "<value>" [, ... ]) ``` -1. `job_name` +## 必选参数 - 同步作业名称,是作业在当前数据库内的唯一标识,相同`job_name`的作业只能有一个在运行。 +**1. `<job_name>`** -2. `channel_desc` +> 同步作业名称,是当前数据库中作业的唯一标识。相同 `<job_name>` 的作业在同一时刻只能有一个在运行。 - 作业下的数据通道,用来描述 mysql 源表到 doris 目标表的映射关系。 +**2. `<channel_desc>`** - 语法: +> 用于描述 MySQL 源表到 Doris 目标表之间的映射关系。 +> +> +> - **`<mysql_db.src_tbl>`** +> 指定 MySQL 端的数据库及源表。 +> +> - **`<des_tbl>`** +> 指定 Doris 端的目标表。目标表必须为 Unique 表,并且需开启表的 batch delete 功能(详见 `help alter table` 中的“批量删除功能”)。 +> +> - **`<columns_mapping>`** (可选) +> 指定 MySQL 源表和 Doris 目标表之间的列映射关系。如果不指定,FE 会默认按照列顺序一一对应。 +> > **注意:** 不支持使用 `col_name = expr` 的形式指定列映射。 +> > +> > **示例:** +> > - 假设目标表列为 `(k1, k2, v1)`,可通过调整顺序实现 `(k2, k1, v1)`; +> > - 或者通过映射忽略源数据中的多余列,例如 `(k2, k1, v1, dummy_column)`。 - ```sql - FROM mysql_db.src_tbl INTO des_tbl - [columns_mapping] - ``` - - 1. `mysql_db.src_tbl` - - 指定 mysql 端的数据库和源表。 - - 2. `des_tbl` - - 指定 doris 端的目标表,只支持 Unique 表,且需开启表的 batch delete 功能 (开启方法请看 help alter table 的'批量删除功能')。 - - 4. `column_mapping` - - 指定 mysql 源表和 doris 目标表的列之间的映射关系。如果不指定,FE 会默认源表和目标表的列按顺序一一对应。 - - 不支持 col_name = expr 的形式表示列。 - - 示例: - - ``` - 假设目标表列为 (k1, k2, v1), - - 改变列 k1 和 k2 的顺序 - (k2, k1, v1) - - 忽略源数据的第四列 - (k2, k1, v1, dummy_column) - ``` - -3. `binlog_desc` - - 用来描述远端数据源,目前仅支持 canal 一种。 - - 语法: +**3. `<binlog_desc>`** - ```sql - FROM BINLOG - ( - "key1" = "value1", - "key2" = "value2" - ) - ``` +> 用来描述远端数据源,目前仅支持 Canal 数据源。 +> +> 对于 Canal 数据源,相关属性均以 `canal.` 为前缀: +> +> - **`canal.server.ip`**:Canal 服务器的地址 +> - **`canal.server.port`**:Canal 服务器的端口 +> - **`canal.destination`**:实例的标识 +> - **`canal.batchSize`**:获取数据的最大 batch 大小(默认值为 8192) +> - **`canal.username`**:实例的用户名 +> - **`canal.password`**:实例的密码 +> - **`canal.debug`** (可选):设置为 true 时,会打印出每个 batch 及每行数据的详细信息 + +## 注意事项 + +- 当前数据同步作业仅支持连接 Canal 服务器。 +- 同一数据库中,相同 `<job_name>` 的作业在同一时刻只能有一个运行。 +- Doris 目标表必须为 Unique 表,且需启用 batch delete 功能,否则数据同步可能失败。 + +## 权限控制 - 1. Canal 数据源对应的属性,以`canal.`为前缀 +执行此 SQL 命令的用户必须至少具有以下权限: - 1. canal.server.ip: canal server 的地址 - 2. canal.server.port: canal server 的端口 - 3. canal.destination: instance 的标识 - 4. canal.batchSize: 获取的 batch 大小的最大值,默认 8192 - 5. canal.username: instance 的用户名 - 6. canal.password: instance 的密码 - 7. canal.debug: 可选,设置为 true 时,会将 batch 和每一行数据的详细信息都打印出来 +| 权限 | 对象 | 说明 | +|---------|------------|-----------------------------------------| +| LOAD_PRIV | 表 | 该操作只能由拥有导入表的 LOAD_PRIV 权限的用户或角色执行。 | ## 示例 -1. 简单为 `test_db` 的 `test_tbl` 创建一个名为 `job1` 的数据同步作业,连接本地的 Canal 服务器,对应 Mysql 源表 `mysql_db1.tbl1`。 +1. **简单示例:** 为 `test_db` 数据库的目标表 `test_tbl` 创建一个名为 `job1` 的数据同步作业,连接本地 Canal 服务器,对应 MySQL 源表 `mysql_db1.tbl1`。 - ```SQL + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db1`.`tbl1` INTO `test_tbl ` + FROM `mysql_db1`.`tbl1` INTO `test_tbl` ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "127.0.0.1", - "canal.server.port" = "11111", - "canal.destination" = "example", - "canal.username" = "", - "canal.password" = "" + "type" = "canal", + "canal.server.ip" = "127.0.0.1", + "canal.server.port" = "11111", + "canal.destination" = "example", + "canal.username" = "", + "canal.password" = "" ); ``` -2. 为 `test_db` 的多张表创建一个名为 `job1` 的数据同步作业,一一对应多张 Mysql 源表,并显式的指定列映射。 +2. **多表同步及列映射示例:** 为 `test_db` 数据库的多张表创建一个名为 `job1` 的数据同步作业,对应多个 MySQL 源表,并显式指定列映射。 - ```SQL + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), - FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) + FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), + FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "xx.xxx.xxx.xx", - "canal.server.port" = "12111", - "canal.destination" = "example", - "canal.username" = "username", - "canal.password" = "password" + "type" = "canal", + "canal.server.ip" = "xx.xxx.xxx.xx", + "canal.server.port" = "12111", + "canal.destination" = "example", + "canal.username" = "username", + "canal.password" = "password" ); - ``` - -## 关键词 - - CREATE, SYNC, JOB - -### 最佳实践 + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md index a5e3169808c..540d6529ebf 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md @@ -24,19 +24,29 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -通过 `job_name` 暂停一个数据库内正在运行的常驻数据同步作业,被暂停的作业将停止同步数据,保持消费的最新位置,直到被用户恢复。 +通过 `job_name` 暂停一个数据库内正在运行的常驻数据同步作业。被暂停的作业将停止同步数据,并保持消费的最新位置,直到用户恢复该作业。 -语法: +## 语法 ```sql -PAUSE SYNC JOB [db.]job_name +PAUSE SYNC JOB [<db>.]<job_name> ``` +## 必选参数 +**1. `<job_name>`** +> 要暂停的同步作业的名称。 + +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 + +## 权限控制 + +任意用户或角色都可以执行该操作 + + ## 示例 1. 暂停名称为 `job_name` 的数据同步作业。 @@ -44,10 +54,3 @@ PAUSE SYNC JOB [db.]job_name ```sql PAUSE SYNC JOB `job_name`; ``` - -## 关键词 - - PAUSE, SYNC, JOB - -### 最佳实践 - diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md index 9584b171364..0fe1735b2ea 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md @@ -24,30 +24,32 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -通过 `job_name`恢复一个当前数据库已被暂停的常驻数据同步作业,作业将从上一次被暂停前最新的位置继续同步数据。 +通过 `job_name` 恢复当前数据库中已暂停的常驻数据同步作业。恢复后,作业将从上一次暂停前保存的最新位置继续同步数据。 -语法: +## 语法 ```sql -RESUME SYNC JOB [db.]job_name +RESUME SYNC JOB [<db>.]<job_name> ``` -## 示例 +## 必选参数 +**1. `<job_name>`** +> 指定要恢复的数据同步作业的名称。 -1. 恢复名称为 `job_name` 的数据同步作业 +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 - ```sql - RESUME SYNC JOB `job_name`; - ``` +## 权限控制 -## 关键词 +任意用户或角色都可以执行该操作 - RESUME, SYNC, LOAD +## 示例 -### 最佳实践 +1. 恢复名称为 `job_name` 的数据同步作业。 + ```sql + RESUME SYNC JOB `job_name`; + ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md index 6ff20e07cbb..7889e771533 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md @@ -3,7 +3,6 @@ "title": "SHOW SYNC JOB", "language": "zh-CN" } - --- <!-- @@ -25,37 +24,36 @@ specific language governing permissions and limitations under the License. --> - - - - ## 描述 -此命令用于当前显示所有数据库内的常驻数据同步作业状态。 +此语句用于显示所有数据库中的常驻数据同步作业状态。 -语法: +## 语法 ```sql -SHOW SYNC JOB [FROM db_name] +SHOW SYNC JOB [FROM <db_name>] ``` +## 可选参数 +**1. `<db_name>`** +> 显示指定数据库下的所有数据同步作业状态。 -## 示例 - -1. 展示当前数据库的所有数据同步作业状态。 +## 权限控制 +执行此 SQL 命令的用户必须至少具有以下权限之一: - ```sql - SHOW SYNC JOB; - ``` +| 权限 | 对象 | 说明 | +|--------------------------------------------------------------------|------------|-----------------------------------------| +| ADMIN_PRIV, SELECT_PRIV, LOAD_PRIV, ALTER_PRIV, CREATE_PRIV, DROP_PRIV, SHOW_VIEW_PRIV | 数据库 `db_name` | 执行此操作需至少拥有上述权限中的一项。 | -2. 展示数据库 `test_db` 下的所有数据同步作业状态。 - - ```sql - SHOW SYNC JOB FROM `test_db`; - ``` +## 示例 -## 关键词 +1. 显示当前数据库的所有数据同步作业状态。 - SHOW, SYNC, JOB + ```sql + SHOW SYNC JOB; + ``` -### 最佳实践 +2. 显示 `test_db` 数据库下的所有数据同步作业状态。 + ```sql + SHOW SYNC JOB FROM `test_db`; + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md index 3edff223499..3803adbba42 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md @@ -24,30 +24,31 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -通过 `job_name` 停止一个数据库内非停止状态的常驻数据同步作业。 +此语句通过 `job_name` 停止一个数据库内非停止状态的常驻数据同步作业。 -语法: +## 语法 ```sql -STOP SYNC JOB [db.]job_name +STOP SYNC JOB [<db>.]<job_name> ``` -## 示例 - -1. 停止名称为 `job_name` 的数据同步作业 +## 必选参数 +**1. `<job_name>`** +> 要暂停的同步作业的名称。 - ```sql - STOP SYNC JOB `job_name`; - ``` +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 -## 关键词 +## 权限控制 +任意用户或角色都可以执行该操作 - STOP, SYNC, JOB +## 示例 -### 最佳实践 +1. 停止名称为 `job_name` 的数据同步作业 + ```sql + STOP SYNC JOB `job_name`; + ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md index 0c24f1d9081..937a6820a80 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md @@ -24,30 +24,24 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -用于 fe 非 master 节点同步元数据。doris 只有 master 节点才能写 fe 元数据,其他 fe 节点写元数据的操作都会转发到 master 节点。在 master 完成元数据写入操作后,非 master 节点 replay 元数据会有短暂的延迟,可以使用该语句同步元数据。 +该语句用于同步非 Master Frontend(FE)节点的元数据。在 Apache Doris 中,只有 Master FE 节点可以写入元数据,其他 FE 节点的元数据写入操作都会转发至 Master 节点。在 Master 节点完成元数据写入操作后,非 Master 节点会存在短暂的元数据同步延迟。可以使用该语句强制同步元数据。 -语法: +## 语法 ```sql SYNC; ``` +## 权限控制 + +任意用户或角色都可以执行该操作 + ## 示例 -1. 同步元数据 +同步元数据: ```sql SYNC; - ``` - -## 关键词 - - SYNC - -### 最佳实践 - + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Data-Manipulation-Statements/Load/PAUSE-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Data-Manipulation-Statements/Load/PAUSE-SYNC-JOB.md index 116c6ce163f..540d6529ebf 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Data-Manipulation-Statements/Load/PAUSE-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Data-Manipulation-Statements/Load/PAUSE-SYNC-JOB.md @@ -1,6 +1,6 @@ --- { - "title": "PAUSE-SYNC-JOB", + "title": "PAUSE SYNC JOB", "language": "zh-CN" } --- @@ -24,33 +24,33 @@ specific language governing permissions and limitations under the License. --> -## PAUSE-SYNC-JOB - -### Name - -PAUSE SYNC JOB - ## 描述 -通过 `job_name` 暂停一个数据库内正在运行的常驻数据同步作业,被暂停的作业将停止同步数据,保持消费的最新位置,直到被用户恢复。 +通过 `job_name` 暂停一个数据库内正在运行的常驻数据同步作业。被暂停的作业将停止同步数据,并保持消费的最新位置,直到用户恢复该作业。 -语法: +## 语法 ```sql -PAUSE SYNC JOB [db.]job_name +PAUSE SYNC JOB [<db>.]<job_name> ``` -## 举例 +## 必选参数 +**1. `<job_name>`** +> 要暂停的同步作业的名称。 -1. 暂停名称为 `job_name` 的数据同步作业。 +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 - ```sql - PAUSE SYNC JOB `job_name`; - ``` +## 权限控制 + +任意用户或角色都可以执行该操作 -### Keywords - PAUSE, SYNC, JOB +## 示例 -### Best Practice +1. 暂停名称为 `job_name` 的数据同步作业。 + ```sql + PAUSE SYNC JOB `job_name`; + ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Utility-Statements/SYNC.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Utility-Statements/SYNC.md index f630f76076d..937a6820a80 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Utility-Statements/SYNC.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/sql-manual/sql-reference/Utility-Statements/SYNC.md @@ -24,33 +24,24 @@ specific language governing permissions and limitations under the License. --> -## SYNC - -### Name - -SYNC - ## 描述 -用于fe非master节点同步元数据。doris只有master节点才能写fe元数据,其他fe节点写元数据的操作都会转发到master节点。在master完成元数据写入操作后,非master节点replay元数据会有短暂的延迟,可以使用该语句同步元数据。 +该语句用于同步非 Master Frontend(FE)节点的元数据。在 Apache Doris 中,只有 Master FE 节点可以写入元数据,其他 FE 节点的元数据写入操作都会转发至 Master 节点。在 Master 节点完成元数据写入操作后,非 Master 节点会存在短暂的元数据同步延迟。可以使用该语句强制同步元数据。 -语法: +## 语法 ```sql SYNC; ``` -## 举例 +## 权限控制 -1. 同步元数据 +任意用户或角色都可以执行该操作 - ```sql - SYNC; - ``` +## 示例 -### Keywords - - SYNC - -### Best Practice +同步元数据: + ```sql + SYNC; + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md index 5d0924186b5..14d8b72171f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md @@ -24,141 +24,117 @@ specific language governing permissions and limitations under the License. --> - - ## 描述 +数据同步 (Sync Job) 功能支持用户提交一个常驻的数据同步作业,通过从指定的远端地址读取 Binlog 日志,增量同步用户在 MySQL 数据库中数据更新操作的 CDC (Change Data Capture) 信息。 -数据同步 (Sync Job) 功能,支持用户提交一个常驻的数据同步作业,通过从指定的远端地址读取 Binlog 日志,增量同步用户在 Mysql 数据库的对数据更新操作的 CDC(Change Data Capture) 功能。 - +用户可通过 [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB) 查看数据同步作业的状态。 -用户可通过 [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB) 查看数据同步作业状态。 - -语法: +## 语法 ```sql -CREATE SYNC [db.]job_name - ( - channel_desc, - channel_desc - ... - ) +CREATE SYNC [<db>.]<job_name> +(<channel_desc> [, ... ]) +<binlog_desc> +``` +where: +```sql +channel_desc + : FROM <mysql_db>.<src_tbl> INTO <des_tbl> [ <columns_mapping> ] +``` +```sql binlog_desc + : FROM BINLOG ("<key>" = "<value>" [, ... ]) ``` -1. `job_name` +## 必选参数 - 同步作业名称,是作业在当前数据库内的唯一标识,相同`job_name`的作业只能有一个在运行。 +**1. `<job_name>`** -2. `channel_desc` +> 同步作业名称,是当前数据库中作业的唯一标识。相同 `<job_name>` 的作业在同一时刻只能有一个在运行。 - 作业下的数据通道,用来描述 mysql 源表到 Doris 目标表的映射关系。 +**2. `<channel_desc>`** +> 用于描述 MySQL 源表到 Doris 目标表之间的映射关系。 +> +> +> - **`<mysql_db.src_tbl>`** +> 指定 MySQL 端的数据库及源表。 +> +> - **`<des_tbl>`** +> 指定 Doris 端的目标表。目标表必须为 Unique 表,并且需开启表的 batch delete 功能(详见 `help alter table` 中的“批量删除功能”)。 +> +> - **`<columns_mapping>`** (可选) +> 指定 MySQL 源表和 Doris 目标表之间的列映射关系。如果不指定,FE 会默认按照列顺序一一对应。 +> > **注意:** 不支持使用 `col_name = expr` 的形式指定列映射。 +> > +> > **示例:** +> > - 假设目标表列为 `(k1, k2, v1)`,可通过调整顺序实现 `(k2, k1, v1)`; +> > - 或者通过映射忽略源数据中的多余列,例如 `(k2, k1, v1, dummy_column)`。 - 语法: +**3. `<binlog_desc>`** - ```sql - FROM mysql_db.src_tbl INTO des_tbl - [columns_mapping] - ``` - - 1. `mysql_db.src_tbl` - - 指定 mysql 端的数据库和源表。 - - 2. `des_tbl` - - 指定 doris 端的目标表,只支持 Unique 表,且需开启表的 batch delete 功能 (开启方法请看 help alter table 的'批量删除功能')。 - - 4. `column_mapping` - - 指定 mysql 源表和 doris 目标表的列之间的映射关系。如果不指定,FE 会默认源表和目标表的列按顺序一一对应。 - - 指定 mysql 端的数据库和源表。 - - 不支持 col_name = expr 的形式表示列。 - - 示例: - - ``` - - 假设目标表列为 (k1, k2, v1), - - 改变列 k1 和 k2 的顺序 - - (k2, k1, v1) - - 忽略源数据的第四列 - (k2, k1, v1, dummy_column) - ``` - -3. `binlog_desc` - - - 用来描述远端数据源,目前仅支持 canal 一种。 - - 语法: +> 用来描述远端数据源,目前仅支持 Canal 数据源。 +> +> 对于 Canal 数据源,相关属性均以 `canal.` 为前缀: +> +> - **`canal.server.ip`**:Canal 服务器的地址 +> - **`canal.server.port`**:Canal 服务器的端口 +> - **`canal.destination`**:实例的标识 +> - **`canal.batchSize`**:获取数据的最大 batch 大小(默认值为 8192) +> - **`canal.username`**:实例的用户名 +> - **`canal.password`**:实例的密码 +> - **`canal.debug`** (可选):设置为 true 时,会打印出每个 batch 及每行数据的详细信息 - ```sql - FROM BINLOG - ( - "key1" = "value1", - "key2" = "value2" - ) - ``` +## 注意事项 + +- 当前数据同步作业仅支持连接 Canal 服务器。 +- 同一数据库中,相同 `<job_name>` 的作业在同一时刻只能有一个运行。 +- Doris 目标表必须为 Unique 表,且需启用 batch delete 功能,否则数据同步可能失败。 + +## 权限控制 + +执行此 SQL 命令的用户必须至少具有以下权限: - Canal 数据源对应的属性,以`canal.`为前缀。 - - 1. canal.server.ip: canal server 的地址 - 2. canal.server.port: canal server 的端口 - 3. canal.destination: instance 的标识 - 4. canal.batchSize: 获取的 batch 大小的最大值,默认 8192 - 5. canal.username: instance 的用户名 - 6. canal.password: instance 的密码 - 7. canal.debug: 可选,设置为 true 时,会将 batch 和每一行数据的详细信息都打印出来 +| 权限 | 对象 | 说明 | +|---------|------------|-----------------------------------------| +| LOAD_PRIV | 表 | 该操作只能由拥有导入表的 LOAD_PRIV 权限的用户或角色执行。 | ## 示例 -1. 简单为 `test_db` 的 `test_tbl` 创建一个名为 `job1` 的数据同步作业,连接本地的 Canal 服务器,对应 Mysql 源表 `mysql_db1.tbl1`。 +1. **简单示例:** 为 `test_db` 数据库的目标表 `test_tbl` 创建一个名为 `job1` 的数据同步作业,连接本地 Canal 服务器,对应 MySQL 源表 `mysql_db1.tbl1`。 ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db1`.`tbl1` INTO `test_tbl ` + FROM `mysql_db1`.`tbl1` INTO `test_tbl` ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "127.0.0.1", - "canal.server.port" = "11111", - "canal.destination" = "example", - "canal.username" = "", - "canal.password" = "" + "type" = "canal", + "canal.server.ip" = "127.0.0.1", + "canal.server.port" = "11111", + "canal.destination" = "example", + "canal.username" = "", + "canal.password" = "" ); ``` - -2. 为 `test_db` 的多张表创建一个名为 `job1` 的数据同步作业,一一对应多张 Mysql 源表,并显式的指定列映射。 +2. **多表同步及列映射示例:** 为 `test_db` 数据库的多张表创建一个名为 `job1` 的数据同步作业,对应多个 MySQL 源表,并显式指定列映射。 ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), - FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) + FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), + FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "xx.xxx.xxx.xx", - "canal.server.port" = "12111", - "canal.destination" = "example", - "canal.username" = "username", - "canal.password" = "password" + "type" = "canal", + "canal.server.ip" = "xx.xxx.xxx.xx", + "canal.server.port" = "12111", + "canal.destination" = "example", + "canal.username" = "username", + "canal.password" = "password" ); - ``` - -## 关键词 - - CREATE, SYNC, JOB - -## 最佳实践 + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md index 07dc6600a09..540d6529ebf 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md @@ -24,18 +24,29 @@ specific language governing permissions and limitations under the License. --> - - ## 描述 -通过 `job_name` 暂停一个数据库内正在运行的常驻数据同步作业,被暂停的作业将停止同步数据,保持消费的最新位置,直到被用户恢复。 +通过 `job_name` 暂停一个数据库内正在运行的常驻数据同步作业。被暂停的作业将停止同步数据,并保持消费的最新位置,直到用户恢复该作业。 -语法: +## 语法 ```sql -PAUSE SYNC JOB [db.]job_name +PAUSE SYNC JOB [<db>.]<job_name> ``` +## 必选参数 +**1. `<job_name>`** +> 要暂停的同步作业的名称。 + +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 + +## 权限控制 + +任意用户或角色都可以执行该操作 + + ## 示例 1. 暂停名称为 `job_name` 的数据同步作业。 @@ -43,10 +54,3 @@ PAUSE SYNC JOB [db.]job_name ```sql PAUSE SYNC JOB `job_name`; ``` - -## 关键词 - - PAUSE, SYNC, JOB - -## 最佳实践 - diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md index 01bc01dc3d0..56156283e3a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md @@ -24,29 +24,33 @@ specific language governing permissions and limitations under the License. --> - - ## 描述 -通过 `job_name`恢复一个当前数据库已被暂停的常驻数据同步作业,作业将从上一次被暂停前最新的位置继续同步数据。 +通过 `job_name` 恢复当前数据库中已暂停的常驻数据同步作业。恢复后,作业将从上一次暂停前保存的最新位置继续同步数据。 -语法: +## 语法 ```sql -RESUME SYNC JOB [db.]job_name +RESUME SYNC JOB [<db>.]<job_name> ``` -## 示例 +## 必选参数 +**1. `<job_name>`** +> 指定要恢复的数据同步作业的名称。 -1. 恢复名称为 `job_name` 的数据同步作业 +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 - ```sql - RESUME SYNC JOB `job_name`; - ``` -## 关键词 +## 权限控制 - RESUME, SYNC, LOAD +任意用户或角色都可以执行该操作 -## 最佳实践 +## 示例 + +1. 恢复名称为 `job_name` 的数据同步作业。 + ```sql + RESUME SYNC JOB `job_name`; + ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md index cf20d4cece3..0c9f0e450a2 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md @@ -3,7 +3,6 @@ "title": "SHOW SYNC JOB", "language": "zh-CN" } - --- <!-- @@ -25,35 +24,37 @@ specific language governing permissions and limitations under the License. --> - - ## 描述 -此命令用于当前显示所有数据库内的常驻数据同步作业状态。 +此语句用于显示所有数据库中的常驻数据同步作业状态。 -语法: +## 语法 ```sql SHOW SYNC JOB [FROM db_name] ``` -## 示例 - -1. 展示当前数据库的所有数据同步作业状态。 +## 必选参数 +**1. `<db_name>`** +> 显示指定数据库下的所有数据同步作业状态。 - ```sql - SHOW SYNC JOB; - ``` +## 权限控制 +执行此 SQL 命令的用户必须至少具有以下权限之一: -2. 展示数据库 `test_db` 下的所有数据同步作业状态。 +| 权限 | 对象 | 说明 | +|--------------------------------------------------------------------|------------|-----------------------------------------| +| ADMIN_PRIV, SELECT_PRIV, LOAD_PRIV, ALTER_PRIV, CREATE_PRIV, DROP_PRIV, SHOW_VIEW_PRIV | 数据库 `db_name` | 执行此操作需至少拥有上述权限中的一项。 | - ```sql - SHOW SYNC JOB FROM `test_db`; - ``` +## 示例 -## 关键词 +1. 显示当前数据库的所有数据同步作业状态。 - SHOW, SYNC, JOB + ```sql + SHOW SYNC JOB; + ``` -## 最佳实践 +2. 显示 `test_db` 数据库下的所有数据同步作业状态。 + ```sql + SHOW SYNC JOB FROM `test_db`; + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md index 6425c12c30c..3803adbba42 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md @@ -24,30 +24,31 @@ specific language governing permissions and limitations under the License. --> - - ## 描述 -通过 `job_name` 停止一个数据库内非停止状态的常驻数据同步作业。 - -语法: +此语句通过 `job_name` 停止一个数据库内非停止状态的常驻数据同步作业。 +## 语法 ```sql -STOP SYNC JOB [db.]job_name +STOP SYNC JOB [<db>.]<job_name> ``` -## 示例 - -1. 停止名称为 `job_name` 的数据同步作业 +## 必选参数 +**1. `<job_name>`** +> 要暂停的同步作业的名称。 - ```sql - STOP SYNC JOB `job_name`; - ``` +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 -## 关键词 +## 权限控制 +任意用户或角色都可以执行该操作 - STOP, SYNC, JOB +## 示例 -## 最佳实践 +1. 停止名称为 `job_name` 的数据同步作业 + ```sql + STOP SYNC JOB `job_name`; + ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md index 1d97abbbb80..937a6820a80 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md @@ -24,31 +24,24 @@ specific language governing permissions and limitations under the License. --> - - ## 描述 +该语句用于同步非 Master Frontend(FE)节点的元数据。在 Apache Doris 中,只有 Master FE 节点可以写入元数据,其他 FE 节点的元数据写入操作都会转发至 Master 节点。在 Master 节点完成元数据写入操作后,非 Master 节点会存在短暂的元数据同步延迟。可以使用该语句强制同步元数据。 -用于 fe 非 master 节点同步元数据。Doris 只有 master 节点才能写 fe 元数据,其他 fe 节点写元数据的操作都会转发到 master 节点。在 master 完成元数据写入操作后,非 master 节点 replay 元数据会有短暂的延迟,可以使用该语句同步元数据。 - - -语法: +## 语法 ```sql SYNC; ``` +## 权限控制 + +任意用户或角色都可以执行该操作 + ## 示例 -1. 同步元数据 +同步元数据: ```sql SYNC; - ``` - -## 关键词 - - SYNC - -## 最佳实践 - + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md index 40238104596..14d8b72171f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md @@ -24,135 +24,117 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -数据同步 (Sync Job) 功能,支持用户提交一个常驻的数据同步作业,通过从指定的远端地址读取 Binlog 日志,增量同步用户在 Mysql 数据库的对数据更新操作的 CDC(Change Data Capture) 功能。 +数据同步 (Sync Job) 功能支持用户提交一个常驻的数据同步作业,通过从指定的远端地址读取 Binlog 日志,增量同步用户在 MySQL 数据库中数据更新操作的 CDC (Change Data Capture) 信息。 -目前数据同步作业只支持对接 Canal,从 Canal Server 上获取解析好的 Binlog 数据,导入到 Doris 内。 +用户可通过 [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB) 查看数据同步作业的状态。 -用户可通过 [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB) 查看数据同步作业状态。 - -语法: +## 语法 ```sql -CREATE SYNC [db.]job_name - ( - channel_desc, - channel_desc - ... - ) +CREATE SYNC [<db>.]<job_name> +(<channel_desc> [, ... ]) +<binlog_desc> +``` +where: +```sql +channel_desc + : FROM <mysql_db>.<src_tbl> INTO <des_tbl> [ <columns_mapping> ] +``` +```sql binlog_desc + : FROM BINLOG ("<key>" = "<value>" [, ... ]) ``` -1. `job_name` +## 必选参数 - 同步作业名称,是作业在当前数据库内的唯一标识,相同`job_name`的作业只能有一个在运行。 +**1. `<job_name>`** -2. `channel_desc` +> 同步作业名称,是当前数据库中作业的唯一标识。相同 `<job_name>` 的作业在同一时刻只能有一个在运行。 - 作业下的数据通道,用来描述 mysql 源表到 doris 目标表的映射关系。 +**2. `<channel_desc>`** - 语法: +> 用于描述 MySQL 源表到 Doris 目标表之间的映射关系。 +> +> +> - **`<mysql_db.src_tbl>`** +> 指定 MySQL 端的数据库及源表。 +> +> - **`<des_tbl>`** +> 指定 Doris 端的目标表。目标表必须为 Unique 表,并且需开启表的 batch delete 功能(详见 `help alter table` 中的“批量删除功能”)。 +> +> - **`<columns_mapping>`** (可选) +> 指定 MySQL 源表和 Doris 目标表之间的列映射关系。如果不指定,FE 会默认按照列顺序一一对应。 +> > **注意:** 不支持使用 `col_name = expr` 的形式指定列映射。 +> > +> > **示例:** +> > - 假设目标表列为 `(k1, k2, v1)`,可通过调整顺序实现 `(k2, k1, v1)`; +> > - 或者通过映射忽略源数据中的多余列,例如 `(k2, k1, v1, dummy_column)`。 - ```sql - FROM mysql_db.src_tbl INTO des_tbl - [columns_mapping] - ``` - - 1. `mysql_db.src_tbl` - - 指定 mysql 端的数据库和源表。 - - 2. `des_tbl` - - 指定 doris 端的目标表,只支持 Unique 表,且需开启表的 batch delete 功能 (开启方法请看 help alter table 的'批量删除功能')。 - - 4. `column_mapping` - - 指定 mysql 源表和 doris 目标表的列之间的映射关系。如果不指定,FE 会默认源表和目标表的列按顺序一一对应。 - - 不支持 col_name = expr 的形式表示列。 - - 示例: - - ``` - 假设目标表列为 (k1, k2, v1), - - 改变列 k1 和 k2 的顺序 - (k2, k1, v1) - - 忽略源数据的第四列 - (k2, k1, v1, dummy_column) - ``` - -3. `binlog_desc` - - 用来描述远端数据源,目前仅支持 canal 一种。 - - 语法: +**3. `<binlog_desc>`** - ```sql - FROM BINLOG - ( - "key1" = "value1", - "key2" = "value2" - ) - ``` +> 用来描述远端数据源,目前仅支持 Canal 数据源。 +> +> 对于 Canal 数据源,相关属性均以 `canal.` 为前缀: +> +> - **`canal.server.ip`**:Canal 服务器的地址 +> - **`canal.server.port`**:Canal 服务器的端口 +> - **`canal.destination`**:实例的标识 +> - **`canal.batchSize`**:获取数据的最大 batch 大小(默认值为 8192) +> - **`canal.username`**:实例的用户名 +> - **`canal.password`**:实例的密码 +> - **`canal.debug`** (可选):设置为 true 时,会打印出每个 batch 及每行数据的详细信息 + +## 注意事项 + +- 当前数据同步作业仅支持连接 Canal 服务器。 +- 同一数据库中,相同 `<job_name>` 的作业在同一时刻只能有一个运行。 +- Doris 目标表必须为 Unique 表,且需启用 batch delete 功能,否则数据同步可能失败。 + +## 权限控制 - 1. Canal 数据源对应的属性,以`canal.`为前缀 +执行此 SQL 命令的用户必须至少具有以下权限: - 1. canal.server.ip: canal server 的地址 - 2. canal.server.port: canal server 的端口 - 3. canal.destination: instance 的标识 - 4. canal.batchSize: 获取的 batch 大小的最大值,默认 8192 - 5. canal.username: instance 的用户名 - 6. canal.password: instance 的密码 - 7. canal.debug: 可选,设置为 true 时,会将 batch 和每一行数据的详细信息都打印出来 +| 权限 | 对象 | 说明 | +|---------|------------|-----------------------------------------| +| LOAD_PRIV | 表 | 该操作只能由拥有导入表的 LOAD_PRIV 权限的用户或角色执行。 | ## 示例 -1. 简单为 `test_db` 的 `test_tbl` 创建一个名为 `job1` 的数据同步作业,连接本地的 Canal 服务器,对应 Mysql 源表 `mysql_db1.tbl1`。 +1. **简单示例:** 为 `test_db` 数据库的目标表 `test_tbl` 创建一个名为 `job1` 的数据同步作业,连接本地 Canal 服务器,对应 MySQL 源表 `mysql_db1.tbl1`。 - ```SQL + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db1`.`tbl1` INTO `test_tbl ` + FROM `mysql_db1`.`tbl1` INTO `test_tbl` ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "127.0.0.1", - "canal.server.port" = "11111", - "canal.destination" = "example", - "canal.username" = "", - "canal.password" = "" + "type" = "canal", + "canal.server.ip" = "127.0.0.1", + "canal.server.port" = "11111", + "canal.destination" = "example", + "canal.username" = "", + "canal.password" = "" ); ``` -2. 为 `test_db` 的多张表创建一个名为 `job1` 的数据同步作业,一一对应多张 Mysql 源表,并显式的指定列映射。 +2. **多表同步及列映射示例:** 为 `test_db` 数据库的多张表创建一个名为 `job1` 的数据同步作业,对应多个 MySQL 源表,并显式指定列映射。 - ```SQL + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), - FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) + FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), + FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "xx.xxx.xxx.xx", - "canal.server.port" = "12111", - "canal.destination" = "example", - "canal.username" = "username", - "canal.password" = "password" + "type" = "canal", + "canal.server.ip" = "xx.xxx.xxx.xx", + "canal.server.port" = "12111", + "canal.destination" = "example", + "canal.username" = "username", + "canal.password" = "password" ); - ``` - -## 关键词 - - CREATE, SYNC, JOB - -### 最佳实践 + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md index a5e3169808c..540d6529ebf 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md @@ -24,19 +24,29 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -通过 `job_name` 暂停一个数据库内正在运行的常驻数据同步作业,被暂停的作业将停止同步数据,保持消费的最新位置,直到被用户恢复。 +通过 `job_name` 暂停一个数据库内正在运行的常驻数据同步作业。被暂停的作业将停止同步数据,并保持消费的最新位置,直到用户恢复该作业。 -语法: +## 语法 ```sql -PAUSE SYNC JOB [db.]job_name +PAUSE SYNC JOB [<db>.]<job_name> ``` +## 必选参数 +**1. `<job_name>`** +> 要暂停的同步作业的名称。 + +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 + +## 权限控制 + +任意用户或角色都可以执行该操作 + + ## 示例 1. 暂停名称为 `job_name` 的数据同步作业。 @@ -44,10 +54,3 @@ PAUSE SYNC JOB [db.]job_name ```sql PAUSE SYNC JOB `job_name`; ``` - -## 关键词 - - PAUSE, SYNC, JOB - -### 最佳实践 - diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md index 9584b171364..56156283e3a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md @@ -24,30 +24,33 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -通过 `job_name`恢复一个当前数据库已被暂停的常驻数据同步作业,作业将从上一次被暂停前最新的位置继续同步数据。 +通过 `job_name` 恢复当前数据库中已暂停的常驻数据同步作业。恢复后,作业将从上一次暂停前保存的最新位置继续同步数据。 -语法: +## 语法 ```sql -RESUME SYNC JOB [db.]job_name +RESUME SYNC JOB [<db>.]<job_name> ``` -## 示例 +## 必选参数 +**1. `<job_name>`** +> 指定要恢复的数据同步作业的名称。 -1. 恢复名称为 `job_name` 的数据同步作业 +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 - ```sql - RESUME SYNC JOB `job_name`; - ``` -## 关键词 +## 权限控制 + +任意用户或角色都可以执行该操作 - RESUME, SYNC, LOAD +## 示例 -### 最佳实践 +1. 恢复名称为 `job_name` 的数据同步作业。 + ```sql + RESUME SYNC JOB `job_name`; + ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md index 6ff20e07cbb..ee030f8aafa 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md @@ -3,7 +3,6 @@ "title": "SHOW SYNC JOB", "language": "zh-CN" } - --- <!-- @@ -25,37 +24,37 @@ specific language governing permissions and limitations under the License. --> - - - - ## 描述 -此命令用于当前显示所有数据库内的常驻数据同步作业状态。 +此语句用于显示所有数据库中的常驻数据同步作业状态。 -语法: +## 语法 ```sql SHOW SYNC JOB [FROM db_name] ``` -## 示例 - -1. 展示当前数据库的所有数据同步作业状态。 +## 可选参数 +**1. `<db_name>`** +> 显示指定数据库下的所有数据同步作业状态。 - ```sql - SHOW SYNC JOB; - ``` +## 权限控制 +执行此 SQL 命令的用户必须至少具有以下权限之一: -2. 展示数据库 `test_db` 下的所有数据同步作业状态。 +| 权限 | 对象 | 说明 | +|--------------------------------------------------------------------|------------|-----------------------------------------| +| ADMIN_PRIV, SELECT_PRIV, LOAD_PRIV, ALTER_PRIV, CREATE_PRIV, DROP_PRIV, SHOW_VIEW_PRIV | 数据库 `db_name` | 执行此操作需至少拥有上述权限中的一项。 | - ```sql - SHOW SYNC JOB FROM `test_db`; - ``` +## 示例 -## 关键词 +1. 显示当前数据库的所有数据同步作业状态。 - SHOW, SYNC, JOB + ```sql + SHOW SYNC JOB; + ``` -### 最佳实践 +2. 显示 `test_db` 数据库下的所有数据同步作业状态。 + ```sql + SHOW SYNC JOB FROM `test_db`; + ``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md index 3edff223499..3803adbba42 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md @@ -24,30 +24,31 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -通过 `job_name` 停止一个数据库内非停止状态的常驻数据同步作业。 +此语句通过 `job_name` 停止一个数据库内非停止状态的常驻数据同步作业。 -语法: +## 语法 ```sql -STOP SYNC JOB [db.]job_name +STOP SYNC JOB [<db>.]<job_name> ``` -## 示例 - -1. 停止名称为 `job_name` 的数据同步作业 +## 必选参数 +**1. `<job_name>`** +> 要暂停的同步作业的名称。 - ```sql - STOP SYNC JOB `job_name`; - ``` +## 可选参数 + **1. `<db>`** + > 如果使用[<db>.]前缀指定了一个数据库,那么该作业将处于指定的数据库中;否则,将使用当前数据库。 -## 关键词 +## 权限控制 +任意用户或角色都可以执行该操作 - STOP, SYNC, JOB +## 示例 -### 最佳实践 +1. 停止名称为 `job_name` 的数据同步作业 + ```sql + STOP SYNC JOB `job_name`; + ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md index 0c24f1d9081..76befca874f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md @@ -24,30 +24,25 @@ specific language governing permissions and limitations under the License. --> - - - ## 描述 -用于 fe 非 master 节点同步元数据。doris 只有 master 节点才能写 fe 元数据,其他 fe 节点写元数据的操作都会转发到 master 节点。在 master 完成元数据写入操作后,非 master 节点 replay 元数据会有短暂的延迟,可以使用该语句同步元数据。 +该语句用于同步非 Master Frontend(FE)节点的元数据。在 Apache Doris 中,只有 Master FE 节点可以写入元数据,其他 FE 节点的元数据写入操作都会转发至 Master 节点。在 Master 节点完成元数据写入操作后,非 Master 节点会存在短暂的元数据同步延迟。可以使用该语句强制同步元数据。 -语法: +## 语法 ```sql SYNC; ``` -## 示例 - -1. 同步元数据 - ```sql - SYNC; - ``` +## 权限控制 -## 关键词 +任意用户或角色都可以执行该操作 - SYNC +## 示例 -### 最佳实践 +同步元数据: + ```sql + SYNC; + ``` \ No newline at end of file diff --git a/versioned_docs/version-2.0/sql-manual/sql-reference/Data-Manipulation-Statements/Load/PAUSE-SYNC-JOB.md b/versioned_docs/version-2.0/sql-manual/sql-reference/Data-Manipulation-Statements/Load/PAUSE-SYNC-JOB.md index c2f6b125b93..56a51d682d7 100644 --- a/versioned_docs/version-2.0/sql-manual/sql-reference/Data-Manipulation-Statements/Load/PAUSE-SYNC-JOB.md +++ b/versioned_docs/version-2.0/sql-manual/sql-reference/Data-Manipulation-Statements/Load/PAUSE-SYNC-JOB.md @@ -1,6 +1,6 @@ --- { - "title": "PAUSE-SYNC-JOB", + "title": "PAUSE SYNC JOB", "language": "en" } --- @@ -24,33 +24,34 @@ specific language governing permissions and limitations under the License. --> -## PAUSE-SYNC-JOB +## Description -### Name +Pause a running resident data synchronization job in a database identified by `job_name`. The suspended job will stop synchronizing data while retaining its latest consumption position until it is resumed by the user. -PAUSE SYNC JOB - -### Description - -Pause a running resident data synchronization job in a database via `job_name`. The suspended job will stop synchronizing data and keep the latest position of consumption until it is resumed by the user. - -grammar: +## Syntax ```sql -PAUSE SYNC JOB [db.]job_name +PAUSE SYNC JOB [<db>.]<job_name> ``` -### Example +## Required Parameters + +**1. `<job_name>`** + +> Specifies the name of the synchronization job to be paused. -1. Pause the data sync job named `job_name`. +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. - ```sql - PAUSE SYNC JOB `job_name`; - ``` +## Access Control Requirements -### Keywords +Any user or role can perform this operation. - PAUSE, SYNC, JOB +## Example -### Best Practice +1. Pause the data synchronization job named `job_name`. + ```sql + PAUSE SYNC JOB `job_name`; + ``` diff --git a/versioned_docs/version-2.0/sql-manual/sql-reference/Utility-Statements/SYNC.md b/versioned_docs/version-2.0/sql-manual/sql-reference/Utility-Statements/SYNC.md index 276a7b49eb1..1d0cab91a2b 100644 --- a/versioned_docs/version-2.0/sql-manual/sql-reference/Utility-Statements/SYNC.md +++ b/versioned_docs/version-2.0/sql-manual/sql-reference/Utility-Statements/SYNC.md @@ -24,33 +24,24 @@ specific language governing permissions and limitations under the License. --> -## SYNC +## Description -### Name +This statement is used to synchronize metadata for non-master Frontend (FE) nodes. In Apache Doris, only the master FE node can write metadata, while other FE nodes forward metadata write operations to the master. After the master completes the metadata writing operation, non-master nodes may experience a short delay in replaying the metadata. You can use this statement to force synchronization of metadata. -SYNC - -### Description - -Used to synchronize metadata for fe non-master nodes. doris only master node can write fe metadata, other fe nodes write metadata operations will be forwarded to master. After master finishes metadata writing operation, there will be a short delay for non-master nodes to replay metadata, you can use this statement to synchronize metadata. - -grammar: +## Syntax ```sql SYNC; ``` -### Example +## Access Control Requirements -1. Synchronized metadata: +Any user or role can perform this operation. - ```sql - SYNC; - ``` +## Examples -### Keywords - - SYNC - -### Best Practice +Synchronize metadata: + ```sql + SYNC; + ``` \ No newline at end of file diff --git a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md index 906a6b9597f..c8c170f4bdc 100644 --- a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md +++ b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md @@ -25,133 +25,110 @@ under the License. --> - ## Description -The data synchronization (Sync Job) function supports users to submit a resident data synchronization job, and incrementally synchronizes the CDC (Change Data Capture) of the user's data update operation in the Mysql database by reading the Binlog log from the specified remote address. Features. - -Currently, the data synchronization job only supports connecting to Canal, obtaining the parsed Binlog data from the Canal Server and importing it into Doris. +The data synchronization (Sync Job) function allows users to submit a persistent data synchronization job. It incrementally synchronizes the CDC (Change Data Capture) of data update operations from a MySQL database by reading the Binlog from a specified remote source. Currently, the synchronization job supports connecting to Canal, obtaining parsed Binlog data from the Canal server, and importing it into Doris. -Users can view the data synchronization job status through [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB). +Users can view the status of synchronization jobs via [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB). -grammar: +## Syntax ```sql -CREATE SYNC [db.]job_name - ( - channel_desc, - channel_desc - ... - ) +CREATE SYNC [<db>.]<job_name> +(<channel_desc> [, ... ]) +<binlog_desc> +``` +where: +```sql +channel_desc + : FROM <mysql_db>.<src_tbl> INTO <des_tbl> [ <columns_mapping> ] +``` +```sql binlog_desc + : FROM BINLOG ("<key>" = "<value>" [, ... ]) ``` -1. `job_name` +## Required Parameters - The synchronization job name is the unique identifier of the job in the current database. Only one job with the same `job_name` can be running. +**1. `<job_name>`** -2. `channel_desc` +> Specifies the unique name of the synchronization job within the current database. Only one job with the same `<job_name>` can be running at a time. - The data channel under the job is used to describe the mapping relationship between the mysql source table and the doris target table. +**2. `<channel_desc>`** - grammar: +> Describes the mapping relationship between the MySQL source table and the Doris target table. +> +> +> - **`<mysql_db.src_tbl>`**: Specifies the source table in MySQL (including the database name). +> - **`<des_tbl>`**: Specifies the target table in Doris. The target table must be unique, and its batch delete function must be enabled. +> - **`<columns_mapping>`** (Optional): Defines the mapping between columns of the source and target tables. If omitted, columns are mapped one-to-one in order. Note that the form `col_name = expr` is not supported. - ```sql - FROM mysql_db.src_tbl INTO des_tbl - [columns_mapping] - ``` - - 1. `mysql_db.src_tbl` - - Specify the database and source table on the mysql side. - - 2. `des_tbl` - - Specify the target table on the doris side. Only unique tables are supported, and the batch delete function of the table needs to be enabled (see the 'batch delete function' of help alter table for how to enable it). - - 4. `column_mapping` - - Specifies the mapping relationship between the columns of the mysql source table and the doris target table. If not specified, FE will default the columns of the source table and the target table to one-to-one correspondence in order. - - The form col_name = expr is not supported for columns. - - Example: - - ``` - Suppose the target table column is (k1, k2, v1), - - Change the order of columns k1 and k2 - (k2, k1, v1) - - Ignore the fourth column of the source data - (k2, k1, v1, dummy_column) - ``` - -3. `binlog_desc` - - Used to describe the remote data source, currently only one canal is supported. - - grammar: +**3. `<binlog_desc>`** - ```sql - FROM BINLOG - ( - "key1" = "value1", - "key2" = "value2" - ) - ``` +> Describes the remote data source for the Binlog. +> +> The properties for the Canal data source (keys prefixed with `canal.`) include: +> +> - **`canal.server.ip`**: Address of the Canal server. +> - **`canal.server.port`**: Port of the Canal server. +> - **`canal.destination`**: Identifier of the Canal instance. +> - **`canal.batchSize`**: Maximum batch size to fetch (default is 8192). +> - **`canal.username`**: Username for the Canal instance. +> - **`canal.password`**: Password for the Canal instance. +> - **`canal.debug`** (Optional): If set to true, prints detailed batch and row information. + +## Usage Notes + +- Currently, the synchronization job only supports connecting to a Canal server. +- Only one synchronization job with the same `<job_name>` can run concurrently within a database. +- The target table specified in `<channel_desc>` must have its batch delete function enabled. - 1. The properties corresponding to the Canal data source, prefixed with `canal.` +## Access Control Requirements - 1. canal.server.ip: address of canal server - 2. canal.server.port: the port of the canal server - 3. canal.destination: the identity of the instance - 4. canal.batchSize: The maximum batch size obtained, the default is 8192 - 5. canal.username: username of instance - 6. canal.password: the password of the instance - 7. canal.debug: optional, when set to true, the batch and details of each row of data will be printed out +Users executing this SQL command must have at least the following privileges: +| Privilege | Object | Notes | +| :---------------- | :------------- | :---------------------------- | +| LOAD_PRIV | Table | This operation can only be performed by users or roles who have the LOAD_PRIV privilege for the imported table. | ## Examples -1. Simply create a data synchronization job named `job1` for `test_tbl` of `test_db`, connect to the local Canal server, corresponding to the Mysql source table `mysql_db1.tbl1`. +1. **Create a simple synchronization job** - ```SQL + Create a synchronization job named `job1` in the `test_db` database that maps the MySQL source table `mysql_db1.tbl1` to the Doris target table `test_tbl`, connecting to a local Canal server. + + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db1`.`tbl1` INTO `test_tbl` + FROM `mysql_db1`.`tbl1` INTO `test_tbl` ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "127.0.0.1", - "canal.server.port" = "11111", - "canal.destination" = "example", - "canal.username" = "", - "canal.password" = "" + "type" = "canal", + "canal.server.ip" = "127.0.0.1", + "canal.server.port" = "11111", + "canal.destination" = "example", + "canal.username" = "", + "canal.password" = "" ); ``` -2. Create a data synchronization job named `job1` for multiple tables of `test_db`, corresponding to multiple Mysql source tables one-to-one, and explicitly specify the column mapping. +2. **Create a synchronization job with multiple channels and explicit column mapping** + + Create a synchronization job named `job1` in the `test_db` database for multiple MySQL source tables with one-to-one mapping and explicitly specified column orders. - ```SQL + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1) , - FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) + FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), + FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "xx.xxx.xxx.xx", - "canal.server.port" = "12111", - "canal.destination" = "example", - "canal.username" = "username", - "canal.password" = "password" + "type" = "canal", + "canal.server.ip" = "xx.xxx.xxx.xx", + "canal.server.port" = "12111", + "canal.destination" = "example", + "canal.username" = "username", + "canal.password" = "password" ); - ``` - -## Keywords - - CREATE, SYNC, JOB - -## Best Practice + ``` \ No newline at end of file diff --git a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md index d094d52f3a6..56a51d682d7 100644 --- a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md +++ b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md @@ -24,29 +24,34 @@ specific language governing permissions and limitations under the License. --> - - ## Description -Pause a running resident data synchronization job in a database via `job_name`. The suspended job will stop synchronizing data and keep the latest position of consumption until it is resumed by the user. +Pause a running resident data synchronization job in a database identified by `job_name`. The suspended job will stop synchronizing data while retaining its latest consumption position until it is resumed by the user. -grammar: +## Syntax ```sql -PAUSE SYNC JOB [db.]job_name +PAUSE SYNC JOB [<db>.]<job_name> ``` -## Examples +## Required Parameters + +**1. `<job_name>`** + +> Specifies the name of the synchronization job to be paused. -1. Pause the data sync job named `job_name`. +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. - ```sql - PAUSE SYNC JOB `job_name`; - ``` +## Access Control Requirements -## Keywords +Any user or role can perform this operation. - PAUSE, SYNC, JOB +## Example -## Best Practice +1. Pause the data synchronization job named `job_name`. + ```sql + PAUSE SYNC JOB `job_name`; + ``` diff --git a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md index a8fce4a2fa0..32c86ab62e7 100644 --- a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md +++ b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md @@ -24,29 +24,35 @@ specific language governing permissions and limitations under the License. --> - - ## Description -Resume a resident data synchronization job whose current database has been suspended by `job_name`, and the job will continue to synchronize data from the latest position before the last suspension. +Resume a resident data synchronization job that has been suspended in a database by its `job_name`. Once resumed, the job continues to synchronize data starting from the latest position before the suspension. -grammar: +## Syntax ```sql -RESUME SYNC JOB [db.]job_name +RESUME SYNC JOB [<db>.]<job_name> ``` -## Examples +## Required Parameters + +**1. `<job_name>`** -1. Resume the data synchronization job named `job_name` +> Specifies the name of the data synchronization job to be resumed. - ```sql - RESUME SYNC JOB `job_name`; - ``` +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. -## Keywords +## Access Control Requirements - RESUME, SYNC, LOAD +Any user or role can perform this operation. + + +## Examples -## Best Practice +1. Resume the data synchronization job named `job_name`. + ```sql + RESUME SYNC JOB `job_name`; + ``` diff --git a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md index c1b8c5688a1..7cc403ab2d9 100644 --- a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md +++ b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md @@ -3,8 +3,6 @@ "title": "SHOW SYNC JOB", "language": "en" } - - --- <!-- @@ -26,17 +24,28 @@ specific language governing permissions and limitations under the License. --> - ## Description -This command is used to currently display the status of resident data synchronization jobs in all databases. +This statement displays the status of resident data synchronization jobs in all databases. -grammar: +## Syntax ```sql -SHOW SYNC JOB [FROM db_name] +SHOW SYNC JOB [FROM <db_name>] ``` +## Optional Parameters + +**1. `<db_name>`** +> `<db_name>`represents the database name, which is used to specify the database from which the information of the synchronization jobs is to be shown. + +## Access Control Requirements +Users executing this SQL command must have at least one of the following privileges: + +| Privilege | Object | Notes | +| :------------------------------------------------------------------------ | :------------- | :------------------------------------- | +| ADMIN_PRIV, SELECT_PRIV, LOAD_PRIV, ALTER_PRIV, CREATE_PRIV, DROP_PRIV, SHOW_VIEW_PRIV | Database `db_name` | This operation requires at least one of the listed privileges on the target database. | + ## Examples 1. Display the status of all data synchronization jobs in the current database. @@ -45,15 +54,8 @@ SHOW SYNC JOB [FROM db_name] SHOW SYNC JOB; ``` -2. Display the status of all data synchronization jobs under the database `test_db`. +2. Display the status of all data synchronization jobs in the `test_db` database. ```sql SHOW SYNC JOB FROM `test_db`; - ``` - -## Keywords - - SHOW, SYNC, JOB - -## Best Practice - + ``` \ No newline at end of file diff --git a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md index aff7d2d29f5..3c82b662a28 100644 --- a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md +++ b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md @@ -24,28 +24,36 @@ specific language governing permissions and limitations under the License. --> - ## Description -Stop a non-stop resident data synchronization job in a database by `job_name`. +Stop a running resident data synchronization job in a database by specifying its `job_name`. Once stopped, the job will cease synchronizing data and release its occupied resources. -grammar: +## Syntax ```sql -STOP SYNC JOB [db.]job_name +STOP SYNC JOB [<db>.]<job_name> ``` -## Examples +## Required Parameters + +**1. `<job_name>`** + +> Specifies the name of the data synchronization job to be stopped. + +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. + -1. Stop the data sync job named `job_name` +## Access Control Requirements - ```sql - STOP SYNC JOB `job_name`; - ``` +Any user or role can perform this operation. -## Keywords - STOP, SYNC, JOB +## Example -## Best Practice +1. Stop the data synchronization job named `job_name`. + ```sql + STOP SYNC JOB `job_name`; + ``` diff --git a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md index e3410df413c..1d0cab91a2b 100644 --- a/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md +++ b/versioned_docs/version-2.1/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md @@ -26,25 +26,22 @@ under the License. ## Description -Used to synchronize metadata for fe non-master nodes. doris only master node can write fe metadata, other fe nodes write metadata operations will be forwarded to master. After master finishes metadata writing operation, there will be a short delay for non-master nodes to replay metadata, you can use this statement to synchronize metadata. +This statement is used to synchronize metadata for non-master Frontend (FE) nodes. In Apache Doris, only the master FE node can write metadata, while other FE nodes forward metadata write operations to the master. After the master completes the metadata writing operation, non-master nodes may experience a short delay in replaying the metadata. You can use this statement to force synchronization of metadata. -grammar: +## Syntax ```sql SYNC; ``` +## Access Control Requirements + +Any user or role can perform this operation. + ## Examples -1. Synchronized metadata: +Synchronize metadata: ```sql SYNC; - ``` - -## Keywords - - SYNC - -## Best Practice - + ``` \ No newline at end of file diff --git a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md index 73c1a9426ad..c8c170f4bdc 100644 --- a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md +++ b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/CREATE-SYNC-JOB.md @@ -25,133 +25,110 @@ under the License. --> - ## Description -The data synchronization (Sync Job) function supports users to submit a resident data synchronization job, and incrementally synchronizes the CDC (Change Data Capture) of the user's data update operation in the Mysql database by reading the Binlog log from the specified remote address. Features. - -Currently, the data synchronization job only supports connecting to Canal, obtaining the parsed Binlog data from the Canal Server and importing it into Doris. +The data synchronization (Sync Job) function allows users to submit a persistent data synchronization job. It incrementally synchronizes the CDC (Change Data Capture) of data update operations from a MySQL database by reading the Binlog from a specified remote source. Currently, the synchronization job supports connecting to Canal, obtaining parsed Binlog data from the Canal server, and importing it into Doris. -Users can view the data synchronization job status through [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB). +Users can view the status of synchronization jobs via [SHOW SYNC JOB](../../../../sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB). -grammar: +## Syntax ```sql -CREATE SYNC [db.]job_name - ( - channel_desc, - channel_desc - ... - ) +CREATE SYNC [<db>.]<job_name> +(<channel_desc> [, ... ]) +<binlog_desc> +``` +where: +```sql +channel_desc + : FROM <mysql_db>.<src_tbl> INTO <des_tbl> [ <columns_mapping> ] +``` +```sql binlog_desc + : FROM BINLOG ("<key>" = "<value>" [, ... ]) ``` -1. `job_name` +## Required Parameters - The synchronization job name is the unique identifier of the job in the current database. Only one job with the same `job_name` can be running. +**1. `<job_name>`** -2. `channel_desc` +> Specifies the unique name of the synchronization job within the current database. Only one job with the same `<job_name>` can be running at a time. - The data channel under the job is used to describe the mapping relationship between the mysql source table and the doris target table. +**2. `<channel_desc>`** - grammar: +> Describes the mapping relationship between the MySQL source table and the Doris target table. +> +> +> - **`<mysql_db.src_tbl>`**: Specifies the source table in MySQL (including the database name). +> - **`<des_tbl>`**: Specifies the target table in Doris. The target table must be unique, and its batch delete function must be enabled. +> - **`<columns_mapping>`** (Optional): Defines the mapping between columns of the source and target tables. If omitted, columns are mapped one-to-one in order. Note that the form `col_name = expr` is not supported. - ```sql - FROM mysql_db.src_tbl INTO des_tbl - [columns_mapping] - ``` - - 1. `mysql_db.src_tbl` - - Specify the database and source table on the mysql side. - - 2. `des_tbl` - - Specify the target table on the doris side. Only unique tables are supported, and the batch delete function of the table needs to be enabled (see the 'batch delete function' of help alter table for how to enable it). - - 4. `column_mapping` - - Specifies the mapping relationship between the columns of the mysql source table and the doris target table. If not specified, FE will default the columns of the source table and the target table to one-to-one correspondence in order. - - The form col_name = expr is not supported for columns. - - Example: - - ``` - Suppose the target table column is (k1, k2, v1), - - Change the order of columns k1 and k2 - (k2, k1, v1) - - Ignore the fourth column of the source data - (k2, k1, v1, dummy_column) - ``` - -3. `binlog_desc` - - Used to describe the remote data source, currently only one canal is supported. - - grammar: +**3. `<binlog_desc>`** - ```sql - FROM BINLOG - ( - "key1" = "value1", - "key2" = "value2" - ) - ``` +> Describes the remote data source for the Binlog. +> +> The properties for the Canal data source (keys prefixed with `canal.`) include: +> +> - **`canal.server.ip`**: Address of the Canal server. +> - **`canal.server.port`**: Port of the Canal server. +> - **`canal.destination`**: Identifier of the Canal instance. +> - **`canal.batchSize`**: Maximum batch size to fetch (default is 8192). +> - **`canal.username`**: Username for the Canal instance. +> - **`canal.password`**: Password for the Canal instance. +> - **`canal.debug`** (Optional): If set to true, prints detailed batch and row information. + +## Usage Notes + +- Currently, the synchronization job only supports connecting to a Canal server. +- Only one synchronization job with the same `<job_name>` can run concurrently within a database. +- The target table specified in `<channel_desc>` must have its batch delete function enabled. - 1. The properties corresponding to the Canal data source, prefixed with `canal.` +## Access Control Requirements - 1. canal.server.ip: address of canal server - 2. canal.server.port: the port of the canal server - 3. canal.destination: the identity of the instance - 4. canal.batchSize: The maximum batch size obtained, the default is 8192 - 5. canal.username: username of instance - 6. canal.password: the password of the instance - 7. canal.debug: optional, when set to true, the batch and details of each row of data will be printed out +Users executing this SQL command must have at least the following privileges: +| Privilege | Object | Notes | +| :---------------- | :------------- | :---------------------------- | +| LOAD_PRIV | Table | This operation can only be performed by users or roles who have the LOAD_PRIV privilege for the imported table. | -## Example +## Examples -1. Simply create a data synchronization job named `job1` for `test_tbl` of `test_db`, connect to the local Canal server, corresponding to the Mysql source table `mysql_db1.tbl1`. +1. **Create a simple synchronization job** - ```SQL + Create a synchronization job named `job1` in the `test_db` database that maps the MySQL source table `mysql_db1.tbl1` to the Doris target table `test_tbl`, connecting to a local Canal server. + + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db1`.`tbl1` INTO `test_tbl` + FROM `mysql_db1`.`tbl1` INTO `test_tbl` ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "127.0.0.1", - "canal.server.port" = "11111", - "canal.destination" = "example", - "canal.username" = "", - "canal.password" = "" + "type" = "canal", + "canal.server.ip" = "127.0.0.1", + "canal.server.port" = "11111", + "canal.destination" = "example", + "canal.username" = "", + "canal.password" = "" ); ``` -2. Create a data synchronization job named `job1` for multiple tables of `test_db`, corresponding to multiple Mysql source tables one-to-one, and explicitly specify the column mapping. +2. **Create a synchronization job with multiple channels and explicit column mapping** + + Create a synchronization job named `job1` in the `test_db` database for multiple MySQL source tables with one-to-one mapping and explicitly specified column orders. - ```SQL + ```sql CREATE SYNC `test_db`.`job1` ( - FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1) , - FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) + FROM `mysql_db`.`t1` INTO `test1` (k1, k2, v1), + FROM `mysql_db`.`t2` INTO `test2` (k3, k4, v2) ) FROM BINLOG ( - "type" = "canal", - "canal.server.ip" = "xx.xxx.xxx.xx", - "canal.server.port" = "12111", - "canal.destination" = "example", - "canal.username" = "username", - "canal.password" = "password" + "type" = "canal", + "canal.server.ip" = "xx.xxx.xxx.xx", + "canal.server.port" = "12111", + "canal.destination" = "example", + "canal.username" = "username", + "canal.password" = "password" ); - ``` - -## Keywords - - CREATE, SYNC, JOB - -## Best Practice + ``` \ No newline at end of file diff --git a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md index e37c6300b80..56a51d682d7 100644 --- a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md +++ b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/PAUSE-SYNC-JOB.md @@ -24,29 +24,34 @@ specific language governing permissions and limitations under the License. --> - - ## Description -Pause a running resident data synchronization job in a database via `job_name`. The suspended job will stop synchronizing data and keep the latest position of consumption until it is resumed by the user. +Pause a running resident data synchronization job in a database identified by `job_name`. The suspended job will stop synchronizing data while retaining its latest consumption position until it is resumed by the user. -grammar: +## Syntax ```sql -PAUSE SYNC JOB [db.]job_name +PAUSE SYNC JOB [<db>.]<job_name> ``` -## Example +## Required Parameters -1. Pause the data sync job named `job_name`. +**1. `<job_name>`** - ```sql - PAUSE SYNC JOB `job_name`; - ``` +> Specifies the name of the synchronization job to be paused. -## Keywords +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. - PAUSE, SYNC, JOB +## Access Control Requirements + +Any user or role can perform this operation. + +## Example -## Best Practice +1. Pause the data synchronization job named `job_name`. + ```sql + PAUSE SYNC JOB `job_name`; + ``` diff --git a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md index 5bac52a3254..ae48b30279f 100644 --- a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md +++ b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/RESUME-SYNC-JOB.md @@ -24,29 +24,36 @@ specific language governing permissions and limitations under the License. --> - - ## Description -Resume a resident data synchronization job whose current database has been suspended by `job_name`, and the job will continue to synchronize data from the latest position before the last suspension. +Resume a resident data synchronization job that has been suspended in a database by its `job_name`. Once resumed, the job continues to synchronize data starting from the latest position before the suspension. -grammar: +## Syntax ```sql -RESUME SYNC JOB [db.]job_name +RESUME SYNC JOB [<db>.]<job_name> ``` -## Example +## Required Parameters + +**1. `<job_name>`** + +> Specifies the name of the data synchronization job to be resumed. + +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. + -1. Resume the data synchronization job named `job_name` +## Access Control Requirements - ```sql - RESUME SYNC JOB `job_name`; - ``` +Any user or role can perform this operation. -## Keywords - RESUME, SYNC, LOAD +## Examples -## Best Practice +1. Resume the data synchronization job named `job_name`. + ```sql + RESUME SYNC JOB `job_name`; + ``` diff --git a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md index a575351668b..7cc403ab2d9 100644 --- a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md +++ b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SHOW-SYNC-JOB.md @@ -3,8 +3,6 @@ "title": "SHOW SYNC JOB", "language": "en" } - - --- <!-- @@ -26,18 +24,29 @@ specific language governing permissions and limitations under the License. --> - ## Description -This command is used to currently display the status of resident data synchronization jobs in all databases. +This statement displays the status of resident data synchronization jobs in all databases. -grammar: +## Syntax ```sql -SHOW SYNC JOB [FROM db_name] +SHOW SYNC JOB [FROM <db_name>] ``` -## Example +## Optional Parameters + +**1. `<db_name>`** +> `<db_name>`represents the database name, which is used to specify the database from which the information of the synchronization jobs is to be shown. + +## Access Control Requirements +Users executing this SQL command must have at least one of the following privileges: + +| Privilege | Object | Notes | +| :------------------------------------------------------------------------ | :------------- | :------------------------------------- | +| ADMIN_PRIV, SELECT_PRIV, LOAD_PRIV, ALTER_PRIV, CREATE_PRIV, DROP_PRIV, SHOW_VIEW_PRIV | Database `db_name` | This operation requires at least one of the listed privileges on the target database. | + +## Examples 1. Display the status of all data synchronization jobs in the current database. @@ -45,15 +54,8 @@ SHOW SYNC JOB [FROM db_name] SHOW SYNC JOB; ``` -2. Display the status of all data synchronization jobs under the database `test_db`. +2. Display the status of all data synchronization jobs in the `test_db` database. ```sql SHOW SYNC JOB FROM `test_db`; - ``` - -## Keywords - - SHOW, SYNC, JOB - -## Best Practice - + ``` \ No newline at end of file diff --git a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md index 140058b43a2..3c82b662a28 100644 --- a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md +++ b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/STOP-SYNC-JOB.md @@ -24,28 +24,36 @@ specific language governing permissions and limitations under the License. --> - ## Description -Stop a non-stop resident data synchronization job in a database by `job_name`. +Stop a running resident data synchronization job in a database by specifying its `job_name`. Once stopped, the job will cease synchronizing data and release its occupied resources. -grammar: +## Syntax ```sql -STOP SYNC JOB [db.]job_name +STOP SYNC JOB [<db>.]<job_name> ``` -## Example +## Required Parameters + +**1. `<job_name>`** + +> Specifies the name of the data synchronization job to be stopped. -1. Stop the data sync job named `job_name` +## Optional Parameters +**1. `<db>`** +> If a database is specified using the `[<db>.]` prefix, the job is located in that database; otherwise, the current database is used. - ```sql - STOP SYNC JOB `job_name`; - ``` -## Keywords +## Access Control Requirements - STOP, SYNC, JOB +Any user or role can perform this operation. + + +## Example -## Best Practice +1. Stop the data synchronization job named `job_name`. + ```sql + STOP SYNC JOB `job_name`; + ``` diff --git a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md index 53307c0626d..1f6762a0c49 100644 --- a/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md +++ b/versioned_docs/version-3.0/sql-manual/sql-statements/data-modification/load-and-export/SYNC.md @@ -26,25 +26,23 @@ under the License. ## Description -Used to synchronize metadata for fe non-master nodes. doris only master node can write fe metadata, other fe nodes write metadata operations will be forwarded to master. After master finishes metadata writing operation, there will be a short delay for non-master nodes to replay metadata, you can use this statement to synchronize metadata. +This statement is used to synchronize metadata for non-master Frontend (FE) nodes. In Apache Doris, only the master FE node can write metadata, while other FE nodes forward metadata write operations to the master. After the master completes the metadata writing operation, non-master nodes may experience a short delay in replaying the metadata. You can use this statement to force synchronization of metadata. -grammar: +## Syntax ```sql SYNC; ``` -## Example +## Access Control Requirements -1. Synchronized metadata: +Any user or role can perform this operation. - ```sql - SYNC; - ``` - -## Keywords - SYNC +## Examples -## Best Practice +Synchronize metadata: + ```sql + SYNC; + ``` \ No newline at end of file --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org