This is an automated email from the ASF dual-hosted git repository. liaoxin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push: new c810a39cbae [doc](routine load) introduce routine load job auto resume (#2137) c810a39cbae is described below commit c810a39cbae9a8eebd6ed47fe525202c85370db2 Author: hui lai <1353307...@qq.com> AuthorDate: Wed Mar 5 12:33:15 2025 +0800 [doc](routine load) introduce routine load job auto resume (#2137) --- .../import/import-way/routine-load-manual.md | 14 ++++++++++++++ .../import/import-way/routine-load-manual.md | 14 ++++++++++++++ .../import/import-way/routine-load-manual.md | 18 ++++++++++++++++++ .../import/import-way/routine-load-manual.md | 18 ++++++++++++++++++ .../import/import-way/routine-load-manual.md | 18 ++++++++++++++++++ .../import/import-way/routine-load-manual.md | 18 ++++++++++++++++++ 6 files changed, 100 insertions(+) diff --git a/docs/data-operate/import/import-way/routine-load-manual.md b/docs/data-operate/import/import-way/routine-load-manual.md index 584601ec383..e4a2602e61e 100644 --- a/docs/data-operate/import/import-way/routine-load-manual.md +++ b/docs/data-operate/import/import-way/routine-load-manual.md @@ -76,6 +76,20 @@ The specific import process of Routine Load is shown in the following diagram: 5. The newly generated Routine Load Tasks continue to be scheduled by the Task Scheduler in a continuous cycle. +### Auto Resume + +To ensure high availability of jobs, an auto-resume mechanism has been introduced. In the event of an unexpected pause, the Routine Load Scheduler thread will attempt to auto-resume the job. For unexpected Kafka outages or other scenarios where the system is unable to function, the auto-resume mechanism ensures that once Kafka is restored, the routine load job can continue running normally without manual intervention. + +Situations where auto-resume will not occur: + +- The user manually executes the PAUSE ROUTINE LOAD command. + +- There are issues with data quality. + +- Situations where resuming is not possible, such as when a database table is deleted. + +Apart from these three situations, other paused jobs will attempt to resume automatically. + ## Quick Start ### Create Job diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/routine-load-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/routine-load-manual.md index b0e829fd147..6db61788841 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/routine-load-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/routine-load-manual.md @@ -76,6 +76,20 @@ Routine Load 的导入具体流程如下图所示: 5. 新生成的 Routine Load Task 由 Task Scheduler 继续调度,不断循环。 +### 自动恢复 + +为了确保作业的高可用性,引入了自动恢复机制。在非预期暂停的情况下,Routine Load Scheduler 调度线程会尝试自动恢复作业。对于 Kafka 侧的意外宕机或其他无法工作的情况,自动恢复机制可以确保在 Kafka 恢复后,无需人工干预,导入作业能够继续正常运行。 + +不会自动恢复的情况: + +- 用户手动执行 PAUSE ROUTINE LOAD 命令。 + +- 数据质量存在问题。 + +- 无法自动恢复的情况,例如库表被删除。 + +除了上述三种情况,其他暂停状态的作业都会尝试自动恢复。 + ## 快速上手 ### 创建导入作业 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/import/import-way/routine-load-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/import/import-way/routine-load-manual.md index b0e829fd147..156f05320e0 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/import/import-way/routine-load-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/import/import-way/routine-load-manual.md @@ -76,6 +76,24 @@ Routine Load 的导入具体流程如下图所示: 5. 新生成的 Routine Load Task 由 Task Scheduler 继续调度,不断循环。 +### 自动恢复 + +为了确保作业的高可用性,引入了自动恢复机制。在非预期暂停的情况下,Routine Load Scheduler 调度线程会尝试自动恢复作业。对于 Kafka 侧的意外宕机或其他无法工作的情况,自动恢复机制可以确保在 Kafka 恢复后,无需人工干预,导入作业能够继续正常运行。 + +不会自动恢复的情况: + +- 用户手动执行 PAUSE ROUTINE LOAD 命令。 + +- 数据质量存在问题。 + +- 无法自动恢复的情况,例如库表被删除。 + +除了上述三种情况,其他暂停状态的作业都会尝试自动恢复。 + +#### FAQ + +1. 对于集群重启或升级,自动恢复可能会遇到一些问题。在 2.1.7 版本之前,集群重启或升级有较高的概率导致任务暂停后无法自动恢复。自 2.1.7 版本起,集群重启或升级导致任务暂停后无法自动恢复的概率有所降低。 + ## 快速上手 ### 创建导入作业 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/data-operate/import/import-way/routine-load-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/data-operate/import/import-way/routine-load-manual.md index b0e829fd147..2535c5a5140 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/data-operate/import/import-way/routine-load-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/data-operate/import/import-way/routine-load-manual.md @@ -76,6 +76,24 @@ Routine Load 的导入具体流程如下图所示: 5. 新生成的 Routine Load Task 由 Task Scheduler 继续调度,不断循环。 +### 自动恢复 + +为了确保作业的高可用性,引入了自动恢复机制。在非预期暂停的情况下,Routine Load Scheduler 调度线程会尝试自动恢复作业。对于 Kafka 侧的意外宕机或其他无法工作的情况,自动恢复机制可以确保在 Kafka 恢复后,无需人工干预,导入作业能够继续正常运行。 + +不会自动恢复的情况: + +- 用户手动执行 PAUSE ROUTINE LOAD 命令。 + +- 数据质量存在问题。 + +- 无法自动恢复的情况,例如库表被删除。 + +除了上述三种情况,其他暂停状态的作业都会尝试自动恢复。 + +#### FAQ + +1. 对于集群重启或升级,自动恢复可能会遇到一些问题。在 3.0.2 版本之前,集群重启或升级同样有较高的概率导致任务暂停后无法自动恢复。自 3.0.2 版本起,即使集群重启或升级导致任务暂停,预期也能实现自动恢复。 + ## 快速上手 ### 创建导入作业 diff --git a/versioned_docs/version-2.1/data-operate/import/import-way/routine-load-manual.md b/versioned_docs/version-2.1/data-operate/import/import-way/routine-load-manual.md index 584601ec383..3e727ad9e9d 100644 --- a/versioned_docs/version-2.1/data-operate/import/import-way/routine-load-manual.md +++ b/versioned_docs/version-2.1/data-operate/import/import-way/routine-load-manual.md @@ -76,6 +76,24 @@ The specific import process of Routine Load is shown in the following diagram: 5. The newly generated Routine Load Tasks continue to be scheduled by the Task Scheduler in a continuous cycle. +### Auto Resume + +To ensure high availability of jobs, an auto-resume mechanism has been introduced. In the event of an unexpected pause, the Routine Load Scheduler thread will attempt to auto-resume the job. For unexpected Kafka outages or other scenarios where the system is unable to function, the auto-resume mechanism ensures that once Kafka is restored, the routine load job can continue running normally without manual intervention. + +Situations where auto-resume will not occur: + +- The user manually executes the PAUSE ROUTINE LOAD command. + +- There are issues with data quality. + +- Situations where resuming is not possible, such as when a database table is deleted. + +Apart from these three situations, other paused jobs will attempt to resume automatically. + +### FAQ + +Auto-resume may encounter some issues during cluster restarts or upgrades. Before version 2.1.7, there was a high probability that tasks would not automatically resume after being paused due to cluster restarts or upgrades. Since version 2.1.7, the likelihood of tasks not resuming automatically after such events has decreased. + ## Quick Start ### Create Job diff --git a/versioned_docs/version-3.0/data-operate/import/import-way/routine-load-manual.md b/versioned_docs/version-3.0/data-operate/import/import-way/routine-load-manual.md index 584601ec383..ef0dfe9d3b8 100644 --- a/versioned_docs/version-3.0/data-operate/import/import-way/routine-load-manual.md +++ b/versioned_docs/version-3.0/data-operate/import/import-way/routine-load-manual.md @@ -76,6 +76,24 @@ The specific import process of Routine Load is shown in the following diagram: 5. The newly generated Routine Load Tasks continue to be scheduled by the Task Scheduler in a continuous cycle. +### Auto Resume + +To ensure high availability of jobs, an auto-resume mechanism has been introduced. In the event of an unexpected pause, the Routine Load Scheduler thread will attempt to auto-resume the job. For unexpected Kafka outages or other scenarios where the system is unable to function, the auto-resume mechanism ensures that once Kafka is restored, the routine load job can continue running normally without manual intervention. + +Situations where auto-resume will not occur: + +- The user manually executes the PAUSE ROUTINE LOAD command. + +- There are issues with data quality. + +- Situations where resuming is not possible, such as when a database table is deleted. + +Apart from these three situations, other paused jobs will attempt to resume automatically. + +### FAQ + +Auto-resume may encounter some issues during cluster restarts or upgrades. Before version 3.0.2, there was similarly a high probability that tasks would not automatically resume after being paused due to cluster restarts or upgrades. Since version 3.0.2, even if tasks are paused due to cluster restarts or upgrades, auto-resume is expected to be achievable. + ## Quick Start ### Create Job --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org