This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new a1615d9654e Add AWS Kinesis Routine Load docs in EN/ZH/JA (#3521)
a1615d9654e is described below
commit a1615d9654e713929cfaacf634e9165e0e027b21
Author: Refrain <[email protected]>
AuthorDate: Sat Oct 17 05:40:26 2026 +0800
Add AWS Kinesis Routine Load docs in EN/ZH/JA (#3521)
## Summary
Document Kinesis Routine Load for the dev version in English and
Chinese.
- Provide a complete setup and JSON ingestion example, including
target-table DDL, credentials, IAM/Doris permissions, a producer
command, and expected query output.
- Use the actual `kinesis_stream` property. Document the supported
shard/sequence-number positions and their mutually exclusive
default-position setting; timestamp position strings are not supported.
- Explain task progress, lag, pause/resume/stop behavior, errors, and
throughput considerations.
- Restore the current sidebar structure and add the page under import
data sources. Do not resurrect the retired `ja-source` staging tree.
## Version scope
This is a **dev-only feature**. Current source branches `branch-4.0`,
`branch-4.1`, and `branch-4.2` do not contain Kinesis Routine Load, so a
4.x counterpart would describe unavailable functionality. The guide
states this limitation in both languages.
## Validation
- Checked FE `KinesisDataSourceProperties`, `KinesisRoutineLoadJob`, and
BE `data_consumer.cpp` / `kinesis_conf.cpp` at Doris master.
- Compared the existing Kinesis regression tests for creation, shard
positions, and SHOW fields.
- Rebased on current website master and resolved navigation/deleted-file
conflicts.
- `git diff --check` and Docusaurus Markdown/MDX compilation pass.
- Documentation governance checks introduce no errors. The missing-4.x
warning is intentional and explained above.
- No live AWS ingestion test was run during this documentation audit.
---------
Co-authored-by: morningman <[email protected]>
---
.../data-operate/import/data-source/aws-kinesis.md | 138 +++++++++++++++++++++
.../data-operate/import/data-source/aws-kinesis.md | 138 +++++++++++++++++++++
sidebars.ts | 1 +
3 files changed, 277 insertions(+)
diff --git a/docs/data-operate/import/data-source/aws-kinesis.md
b/docs/data-operate/import/data-source/aws-kinesis.md
new file mode 100644
index 00000000000..3f3d4aa0add
--- /dev/null
+++ b/docs/data-operate/import/data-source/aws-kinesis.md
@@ -0,0 +1,138 @@
+---
+{
+ "title": "AWS Kinesis",
+ "language": "en",
+ "description": "Continuously load AWS Kinesis Data Streams into Apache
Doris with Routine Load: setup, authentication, shard positions, and
troubleshooting."
+}
+---
+
+Routine Load can continuously read AWS Kinesis Data Streams and load records
into Doris tables. A stream contains shards; Doris tracks progress using a
sequence number for each shard.
+
+:::info Version
+Kinesis Routine Load is available in the dev version. The 4.x source branches
do not include this data source; this guide does not apply to those versions.
+:::
+
+## Prerequisites
+
+- Use a Doris build that includes Kinesis Routine Load. BEs must be able to
reach the Kinesis endpoint.
+- Prepare a Kinesis stream and credentials that allow `kinesis:ListShards`,
`kinesis:GetShardIterator`, and `kinesis:GetRecords` on it.
+- Use a Doris account with database/table creation permissions and `LOAD_PRIV`
on the target table. Administrative operations on an existing job also require
the appropriate load permission.
+- For the producer command below, configure the AWS CLI with credentials that
have `kinesis:PutRecord` permission. Use a dedicated test stream, or account
for existing records in the stream.
+
+Doris supports an access-key/secret-key pair, temporary credentials with a
session token, an assumed IAM role, or the AWS default credential chain
available to the BE process. For general AWS credential guidance, see [AWS
Authentication](../../../admin-manual/auth/integrations/aws-authentication-and-authorization.md).
+
+## Quick Start
+
+### 1. Create a database and target table
+
+Run the following SQL with a MySQL client. The example uses one replica for a
single-BE development cluster; choose an appropriate replica count for
production.
+
+```sql
+CREATE DATABASE IF NOT EXISTS kinesis_demo;
+USE kinesis_demo;
+
+CREATE TABLE kinesis_events (
+ id BIGINT NOT NULL,
+ name STRING
+)
+UNIQUE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+```
+
+### 2. Create a Routine Load job
+
+Replace the stream name, region, and credentials before running this
statement. The property is `kinesis_stream`, without an `aws.` prefix.
`TRIM_HORIZON` reads from the oldest records still retained by the stream.
+
+```sql
+CREATE ROUTINE LOAD kinesis_demo.kinesis_job ON kinesis_events
+COLUMNS(id, name)
+PROPERTIES (
+ "format" = "json",
+ "desired_concurrent_number" = "1"
+)
+FROM KINESIS (
+ "aws.region" = "us-east-1",
+ "kinesis_stream" = "my-test-stream",
+ "aws.access_key" = "<access_key>",
+ "aws.secret_key" = "<secret_key>",
+ "property.kinesis_default_pos" = "TRIM_HORIZON"
+);
+```
+
+For BE environments that already have an appropriate default AWS credential
chain, omit the access-key and secret-key properties. See the parameter
reference for role and temporary-credential settings.
+
+### 3. Write a test record and verify it
+
+From a shell with AWS CLI v2 configured for the test stream:
+
+```shell
+aws kinesis put-record \
+ --region us-east-1 \
+ --stream-name my-test-stream \
+ --partition-key 1 \
+ --cli-binary-format raw-in-base64-out \
+ --data '{"id":1,"name":"Alice"}'
+```
+
+The command returns a shard ID and sequence number. Routine Load runs
asynchronously. After the job consumes and commits the record, query Doris:
+
+```sql
+SELECT id, name FROM kinesis_demo.kinesis_events WHERE id = 1;
+```
+
+```text
++----+-------+
+| id | name |
++----+-------+
+| 1 | Alice |
++----+-------+
+```
+
+## Parameters
+
+All properties below belong to the `FROM KINESIS (...)` clause. Omitted
optional properties retain their defaults. General job properties such as
`format`, `jsonpaths`, and `desired_concurrent_number` belong to `PROPERTIES
(...)`; see [Routine Load](../import-way/routine-load-manual.md).
+
+| Parameter | Type / Required | Default | Description |
+| --- | --- | --- | --- |
+| `aws.region` | String; required | None | Region containing the stream, such
as `us-east-1`. |
+| `kinesis_stream` | String; required | None | Stream name. Do not use
`aws.kinesis_stream`. |
+| `aws.endpoint` | String; optional | AWS endpoint for the region | Custom
Kinesis endpoint, for example a LocalStack endpoint. The legacy alias
`kinesis_endpoint` is also accepted; prefer `aws.endpoint`. |
+| `aws.access_key` | String; optional | AWS default credential chain when no
explicit credentials are supplied | Access-key ID. Must be supplied together
with `aws.secret_key`. |
+| `aws.secret_key` | String; optional | None | Secret key paired with
`aws.access_key`. |
+| `aws.session_key` | String; optional | None | Session token for temporary
credentials. |
+| `aws.role_arn` | String; optional | None | IAM role to assume; the
credential identity must be allowed to assume it. |
+| `kinesis_shards` | Comma-separated string; optional | Automatically discover
shards | Shard IDs to consume. |
+| `kinesis_shards_pos` | Comma-separated string; optional | `LATEST` for each
explicitly listed shard | Positions corresponding one-to-one to
`kinesis_shards`. Each value is `TRIM_HORIZON`, `LATEST`, or a sequence number.
A sequence number starts consumption **after** that record. |
+| `property.kinesis_default_pos` | String; optional | `LATEST` | Default start
position: `TRIM_HORIZON`, `LATEST`, or a sequence number. Cannot be combined
with `kinesis_shards_pos`. Timestamp strings are not supported. |
+| Other `property.*` | String; optional | Depends on the BE option | Forwarded
to the BE after stripping `property.`. For example, `property.aws.external.id`
supplies an external ID for role assumption and requires a role ARN. Only use
options supported by the BE. |
+
+## Monitor and Manage the Job
+
+```sql
+SHOW ROUTINE LOAD FOR kinesis_demo.kinesis_job;
+```
+
+| Field | Expected meaning |
+| --- | --- |
+| `State` | `RUNNING` after the job is scheduled and can read the stream.
Inspect the pause reason if the job is `PAUSED`. |
+| `DataSourceType` | `KINESIS`. |
+| `DataSourceProperties` | Region, stream, endpoint, and configured shards. |
+| `Progress` | Per-shard sequence-number progress. Sequence numbers are not
Kafka offsets and cannot be added together to count records. |
+| `Lag` | Per-shard lag in milliseconds behind the latest record, when
available. |
+
+```sql
+PAUSE ROUTINE LOAD FOR kinesis_demo.kinesis_job;
+RESUME ROUTINE LOAD FOR kinesis_demo.kinesis_job;
+STOP ROUTINE LOAD FOR kinesis_demo.kinesis_job;
+```
+
+`PAUSE` stops scheduling new tasks and allows the job to be resumed. `STOP`
permanently ends the job; a stopped job cannot be resumed.
+
+## Caveats and Best Practices
+
+- The default `LATEST` does not replay historical records. Use `TRIM_HORIZON`
for the oldest retained data, or explicit shard positions for controlled
replay. Data already expired from Kinesis cannot be recovered by changing the
start position.
+- Explicit shard IDs and positions must have the same number of entries. Do
not combine `kinesis_shards_pos` and `property.kinesis_default_pos`.
+- If creation reports an invalid property, check the spelling and the
distinction between job properties and source properties. In particular,
`aws.kinesis_stream` and timestamp start-position strings are invalid.
+- If the job pauses, inspect `SHOW ROUTINE LOAD` and BE logs for
authentication, endpoint, shard, and data-format errors. JSON records must
match the target columns or the configured mappings.
+- Use representative stream throughput to size concurrency. Kinesis read
limits and shard distribution can limit throughput; adding tasks cannot bypass
AWS per-shard limits.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/data-source/aws-kinesis.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/data-source/aws-kinesis.md
new file mode 100644
index 00000000000..a289e35c553
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/data-source/aws-kinesis.md
@@ -0,0 +1,138 @@
+---
+{
+ "title": "AWS Kinesis",
+ "language": "zh-CN",
+ "description": "通过 Routine Load 将 AWS Kinesis Data Streams 持续导入 Apache
Doris，涵盖准备、认证、分片起始位置和故障处理。"
+}
+---
+
+Routine Load 可以持续读取 AWS Kinesis Data Streams，并将记录导入 Doris 表。一个 Stream 包含多个
Shard，Doris 使用每个 Shard 的 Sequence Number 跟踪消费进度。
+
+:::info 版本说明
+Kinesis Routine Load 在 dev 版本提供。4.x 源码分支尚未包含该数据源，本文不适用于这些版本。
+:::
+
+## 前置条件
+
+- 使用包含 Kinesis Routine Load 的 Doris 构建版本。BE 必须能够访问 Kinesis 服务端点。
+- 准备 Kinesis Stream 和凭据，授予目标 Stream 的
`kinesis:ListShards`、`kinesis:GetShardIterator`、`kinesis:GetRecords` 权限。
+- Doris 账户需要创建数据库和表的权限，以及目标表的 `LOAD_PRIV`。管理已有任务也需要相应的导入权限。
+- 以下生产消息的命令需要配置 AWS CLI，并授予 `kinesis:PutRecord` 权限。请使用专门的测试 Stream，或者考虑 Stream
中已有的记录。
+
+Doris 支持 Access Key / Secret Key、携带 Session Token 的临时凭据、承担 IAM Role，以及 BE
进程可以使用的 AWS 默认凭据链。通用 AWS 认证说明参见 [AWS
认证与鉴权](../../../admin-manual/auth/integrations/aws-authentication-and-authorization.md)。
+
+## 快速开始
+
+### 1. 创建数据库和目标表
+
+使用 MySQL 客户端执行以下 SQL。示例使用单副本，可在单 BE 开发集群运行；生产环境应选择适当的副本数。
+
+```sql
+CREATE DATABASE IF NOT EXISTS kinesis_demo;
+USE kinesis_demo;
+
+CREATE TABLE kinesis_events (
+ id BIGINT NOT NULL,
+ name STRING
+)
+UNIQUE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+```
+
+### 2. 创建 Routine Load 任务
+
+执行前替换 Stream 名、区域和凭据。属性名为 `kinesis_stream`，没有 `aws.` 前缀。`TRIM_HORIZON` 从
Stream 中仍被保留的最早记录开始读取。
+
+```sql
+CREATE ROUTINE LOAD kinesis_demo.kinesis_job ON kinesis_events
+COLUMNS(id, name)
+PROPERTIES (
+ "format" = "json",
+ "desired_concurrent_number" = "1"
+)
+FROM KINESIS (
+ "aws.region" = "us-east-1",
+ "kinesis_stream" = "my-test-stream",
+ "aws.access_key" = "<access_key>",
+ "aws.secret_key" = "<secret_key>",
+ "property.kinesis_default_pos" = "TRIM_HORIZON"
+);
+```
+
+如果 BE 已配置可用的 AWS 默认凭据链，可以省略 Access Key 和 Secret Key 属性。角色和临时凭据的配置参见参数表。
+
+### 3. 写入测试记录并验证
+
+在已为测试 Stream 配置 AWS CLI v2 的 Shell 中执行：
+
+```shell
+aws kinesis put-record \
+ --region us-east-1 \
+ --stream-name my-test-stream \
+ --partition-key 1 \
+ --cli-binary-format raw-in-base64-out \
+ --data '{"id":1,"name":"Alice"}'
+```
+
+命令返回 Shard ID 和 Sequence Number。Routine Load 异步运行；任务消费并提交该记录后，在 Doris 中查询：
+
+```sql
+SELECT id, name FROM kinesis_demo.kinesis_events WHERE id = 1;
+```
+
+```text
++----+-------+
+| id | name |
++----+-------+
+| 1 | Alice |
++----+-------+
+```
+
+## 参数
+
+下表属性均放在 `FROM KINESIS (...)`
中。可选属性省略时使用默认值。`format`、`jsonpaths`、`desired_concurrent_number` 等通用任务属性放在
`PROPERTIES (...)` 中，参见 [Routine Load](../import-way/routine-load-manual.md)。
+
+| 参数 | 类型 / 是否必填 | 默认值 | 说明 |
+| --- | --- | --- | --- |
+| `aws.region` | 字符串；必填 | 无 | Stream 所在区域，例如 `us-east-1`。 |
+| `kinesis_stream` | 字符串；必填 | 无 | Stream 名称。不要使用 `aws.kinesis_stream`。 |
+| `aws.endpoint` | 字符串；可选 | 对应区域的 AWS 服务端点 | 自定义 Kinesis 服务端点，例如 LocalStack
地址。兼容旧别名 `kinesis_endpoint`，建议使用 `aws.endpoint`。 |
+| `aws.access_key` | 字符串；可选 | 未显式提供凭据时使用 AWS 默认凭据链 | Access Key ID，必须与
`aws.secret_key` 一起设置。 |
+| `aws.secret_key` | 字符串；可选 | 无 | 与 `aws.access_key` 配对的 Secret Key。 |
+| `aws.session_key` | 字符串；可选 | 无 | 临时凭据的 Session Token。 |
+| `aws.role_arn` | 字符串；可选 | 无 | 要承担的 IAM Role；凭据身份需要有承担该角色的权限。 |
+| `kinesis_shards` | 逗号分隔字符串；可选 | 自动发现 Shard | 要消费的 Shard ID。 |
+| `kinesis_shards_pos` | 逗号分隔字符串；可选 | 显式列出 Shard 时，每个 Shard 使用 `LATEST` | 与
`kinesis_shards` 一一对应的位置；每项为 `TRIM_HORIZON`、`LATEST` 或 Sequence Number。指定
Sequence Number 时，从该记录**之后**开始消费。 |
+| `property.kinesis_default_pos` | 字符串；可选 | `LATEST` | 默认起始位置，可为
`TRIM_HORIZON`、`LATEST` 或 Sequence Number。不能与 `kinesis_shards_pos`
同时使用，不支持时间戳字符串。 |
+| 其他 `property.*` | 字符串；可选 | 取决于 BE 选项 | 去除 `property.` 前缀后透传给 BE。例如
`property.aws.external.id` 提供承担角色的 External ID，需要同时配置 Role ARN。仅使用 BE 支持的选项。 |
+
+## 查看和管理任务
+
+```sql
+SHOW ROUTINE LOAD FOR kinesis_demo.kinesis_job;
+```
+
+| 字段 | 预期含义 |
+| --- | --- |
+| `State` | 任务被调度且可读取 Stream 后为 `RUNNING`。若为 `PAUSED`，请检查暂停原因。 |
+| `DataSourceType` | `KINESIS`。 |
+| `DataSourceProperties` | 区域、Stream、服务端点和配置的 Shard。 |
+| `Progress` | 每个 Shard 的 Sequence Number 进度。Sequence Number 不是 Kafka
Offset，不能相加计算记录数。 |
+| `Lag` | 可获取时，返回各 Shard 落后于最新记录的毫秒数。 |
+
+```sql
+PAUSE ROUTINE LOAD FOR kinesis_demo.kinesis_job;
+RESUME ROUTINE LOAD FOR kinesis_demo.kinesis_job;
+STOP ROUTINE LOAD FOR kinesis_demo.kinesis_job;
+```
+
+`PAUSE` 停止调度新任务，之后可以恢复。`STOP` 永久结束任务，不能恢复已停止的任务。
+
+## 注意事项和最佳实践
+
+- 默认的 `LATEST` 不会重放历史记录。如需读取最早保留的数据，使用 `TRIM_HORIZON`；如需控制重放位置，指定各 Shard
的位置。Kinesis 已过期的数据无法通过修改起始位置恢复。
+- 显式 Shard ID 和位置的数量必须一致。不要同时设置 `kinesis_shards_pos` 和
`property.kinesis_default_pos`。
+- 创建时若提示属性无效，请检查拼写以及任务属性、数据源属性的区别。`aws.kinesis_stream` 和时间戳形式的起始位置均无效。
+- 任务暂停时，检查 `SHOW ROUTINE LOAD` 和 BE 日志中的认证、服务端点、Shard 和数据格式错误。JSON
记录需要与目标列或列映射相匹配。
+- 根据实际 Stream 吞吐量设置并发度。Kinesis 读取限制和 Shard 分布会限制吞吐，增加任务数无法绕过 AWS 的单 Shard 限制。
diff --git a/sidebars.ts b/sidebars.ts
index c04c166a0a9..79f280b247f 100644
--- a/sidebars.ts
+++ b/sidebars.ts
@@ -338,6 +338,7 @@ const sidebars: SidebarsConfig = {
'data-operate/import/data-source/local-file',
'data-operate/import/data-source/kafka',
'data-operate/import/data-source/aws-msk',
+ 'data-operate/import/data-source/aws-kinesis',
'data-operate/import/data-source/flink',
'data-operate/import/data-source/hdfs',
'data-operate/import/data-source/amazon-s3',
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
