This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push: new 464830437a [doc] add cloudcanal docs (#10813) 464830437a is described below commit 464830437acc861d1637b138e5f9b1a0a6a51fe8 Author: KamiWan <344277...@qq.com> AuthorDate: Sun Jul 17 11:50:40 2022 +0800 [doc] add cloudcanal docs (#10813) --- .../public/images/cloudcanal/cloudcanal-1.jpg | Bin 0 -> 169852 bytes .../public/images/cloudcanal/cloudcanal-2.jpg | Bin 0 -> 369963 bytes .../public/images/cloudcanal/cloudcanal-3.jpg | Bin 0 -> 420489 bytes .../public/images/cloudcanal/cloudcanal-4.png | Bin 0 -> 239668 bytes .../public/images/cloudcanal/cloudcanal-5.png | Bin 0 -> 250224 bytes .../public/images/cloudcanal/cloudcanal-6.png | Bin 0 -> 339062 bytes .../public/images/cloudcanal/cloudcanal-7.png | Bin 0 -> 357200 bytes .../public/images/cloudcanal/cloudcanal-8.jpg | Bin 0 -> 308720 bytes docs/zh-CN/docs/ecosystem/cloudcanal.md | 90 +++++++++++++++++++++ 9 files changed, 90 insertions(+) diff --git a/docs/.vuepress/public/images/cloudcanal/cloudcanal-1.jpg b/docs/.vuepress/public/images/cloudcanal/cloudcanal-1.jpg new file mode 100644 index 0000000000..978fdd95ad Binary files /dev/null and b/docs/.vuepress/public/images/cloudcanal/cloudcanal-1.jpg differ diff --git a/docs/.vuepress/public/images/cloudcanal/cloudcanal-2.jpg b/docs/.vuepress/public/images/cloudcanal/cloudcanal-2.jpg new file mode 100644 index 0000000000..2e58d25ca5 Binary files /dev/null and b/docs/.vuepress/public/images/cloudcanal/cloudcanal-2.jpg differ diff --git a/docs/.vuepress/public/images/cloudcanal/cloudcanal-3.jpg b/docs/.vuepress/public/images/cloudcanal/cloudcanal-3.jpg new file mode 100644 index 0000000000..e3050e59ed Binary files /dev/null and b/docs/.vuepress/public/images/cloudcanal/cloudcanal-3.jpg differ diff --git a/docs/.vuepress/public/images/cloudcanal/cloudcanal-4.png b/docs/.vuepress/public/images/cloudcanal/cloudcanal-4.png new file mode 100644 index 0000000000..379c95018e Binary files /dev/null and b/docs/.vuepress/public/images/cloudcanal/cloudcanal-4.png differ diff --git a/docs/.vuepress/public/images/cloudcanal/cloudcanal-5.png b/docs/.vuepress/public/images/cloudcanal/cloudcanal-5.png new file mode 100644 index 0000000000..ec9518bf95 Binary files /dev/null and b/docs/.vuepress/public/images/cloudcanal/cloudcanal-5.png differ diff --git a/docs/.vuepress/public/images/cloudcanal/cloudcanal-6.png b/docs/.vuepress/public/images/cloudcanal/cloudcanal-6.png new file mode 100644 index 0000000000..c1e7be356c Binary files /dev/null and b/docs/.vuepress/public/images/cloudcanal/cloudcanal-6.png differ diff --git a/docs/.vuepress/public/images/cloudcanal/cloudcanal-7.png b/docs/.vuepress/public/images/cloudcanal/cloudcanal-7.png new file mode 100644 index 0000000000..62ae33a555 Binary files /dev/null and b/docs/.vuepress/public/images/cloudcanal/cloudcanal-7.png differ diff --git a/docs/.vuepress/public/images/cloudcanal/cloudcanal-8.jpg b/docs/.vuepress/public/images/cloudcanal/cloudcanal-8.jpg new file mode 100644 index 0000000000..aa4cae7c5e Binary files /dev/null and b/docs/.vuepress/public/images/cloudcanal/cloudcanal-8.jpg differ diff --git a/docs/zh-CN/docs/ecosystem/cloudcanal.md b/docs/zh-CN/docs/ecosystem/cloudcanal.md new file mode 100644 index 0000000000..da6ecde790 --- /dev/null +++ b/docs/zh-CN/docs/ecosystem/cloudcanal.md @@ -0,0 +1,90 @@ +# CloudCanal 数据导入 + +## 介绍 + +CloudCanal 社区版是一款由 [ClouGence 公司](https://www.clougence.com) 发行的集结构迁移、数据全量迁移/校验/订正、增量实时同步为一体的免费数据迁移同步平台。产品包含完整的产品化能力,助力企业打破数据孤岛、完成数据互融互通,从而更好的使用数据。 + + +## 下载安装 + +[CloudCanal 最新版下载地址](https://clougence.com) + +[CloudCanal 社区版安装部署参考文档](https://doc-cloudcanal.clougence.com/operation/install_linux) + +## 功能说明 + +- 推荐使用2.2.5.0及以上的CloudCanal版本写入Doris +- 建议您在使用 CloudCanal 将 **增量数据** 导入至 Doris 时,控制导入的频率,CloudCanal写入Doris的默认导入频率可以通过参数`realFlushPauseSec`调整,默认为10秒。 +- 当前社区版本最大的内存配置为2g,如果同步任务运行产生OOM异常或者GC停顿严重可以调小以下参数减少批次大小从而减少内存占用。全量参数为`fullBatchSize`和`fullRingBufferSize`,增量参数为`increBatchSize`和`increRingBufferSize` +- 支持的源端以及功能项: + + | 数据源 \ 功能项 | 结构迁移 | 全量数据迁移 | 增量实时同步 | 数据校验 | + | --- | --- | --- | --- | --- | + | Oracle 源端 | 支持 | 支持 | 支持 | 支持 | + | PostgreSQL 源端 | 支持 | 支持 | 支持 | 支持 | + | Greenplum 源端 | 支持 | 支持 | 不支持 | 支持 | + | MySQL 源端 | 支持 | 支持 | 支持 | 支持 | + +## 使用方法 + +CloudCanal 提供了完整的产品化能力,用户在可视化界面完成数据源添加和任务创建即可自动完成结构迁移、全量迁移、增量实时同步。下文演示如何将 MySQL 数据库中的数据迁移同步到对端 Doris 中。其他源端同步到 Doris 也可以按照类似的方式进行。 + +### 前置条件 + +首先参考 [CloudCanal 安装部署参考文档](https://www.askcug.com/topic/75) 完成 CloudCanal 社区版的安装和部署。 + +### 添加数据源 + +- 登录 CloudCanal 平台 +- 数据源管理-> 新增数据源 +- 选择自建数据库中 Doris + + + +> Tips: +> +> - Client 地址: 为 Doris 提供给 MySQL Client 的服务端口,CloudCanal 主要用其查询库表的元数据信息 +> +> - Http 地址: Http 地址主要用于接收 CloudCanal 数据导入的请求 + +### 任务创建 + +添加好数据源之后可以按照如下步骤进行数据迁移、同步任务的创建。 + +- **任务管理**-> **任务创建** +- 选择 **源** 和 **目标** 数据库 +- 点击 下一步 + + + +- 选择 **增量同步**,并且启用 **全量数据初始化** +- 不勾选 DDL 同步(暂不支持) +- 点击下一步 + + + +- 选择订阅的表,**结构迁移自动创建的表为主键模型的表,因此暂不支持无主键表** +- 点击下一步 + + + +- 配置列映射 +- 点击下一步 + + + +- 创建任务 + + + +- 查看任务状态。任务创建后,会自动完成结构迁移、全量、增量阶段。 + + + +## 参考资料 + +更多关于 CloudCanal 同步 Doris 的资料,可以查看 + +- [5 分钟搞定 MySQL/PostgreSQL/Oracle 到 Doris 数据迁移同步-CloudCanal 实战](https://www.askcug.com/topic/262) + +- [CloudCanal 官方社区](https://www.askcug.com/) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org