This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ec8fbbd4c62 Add IAM Assumed Role example (#2898)
ec8fbbd4c62 is described below
commit ec8fbbd4c62ce53245766ffb722dc1c6213de9e2
Author: Calvin Kirs <[email protected]>
AuthorDate: Sat Sep 20 01:51:27 2025 +0800
Add IAM Assumed Role example (#2898)
## Versions
- [x] dev
- [x] 3.0
- [x] 2.1
- [ ] 2.0
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
Co-authored-by: Mingyu Chen (Rayner) <[email protected]>
---
docs/lakehouse/catalogs/hive-catalog.mdx | 9 +++
docs/lakehouse/catalogs/iceberg-catalog.mdx | 21 ++++++
docs/lakehouse/catalogs/paimon-catalog.mdx | 77 +++++++++++++++++++++-
.../current/lakehouse/catalogs/hive-catalog.mdx | 9 +++
.../current/lakehouse/catalogs/iceberg-catalog.mdx | 21 ++++++
.../current/lakehouse/catalogs/paimon-catalog.mdx | 76 ++++++++++++++++++++-
.../lakehouse/catalogs/hive-catalog.mdx | 9 +++
.../lakehouse/catalogs/iceberg-catalog.mdx | 21 ++++++
.../lakehouse/catalogs/paimon-catalog.mdx | 76 ++++++++++++++++++++-
.../lakehouse/catalogs/hive-catalog.mdx | 9 +++
.../lakehouse/catalogs/iceberg-catalog.mdx | 21 ++++++
.../lakehouse/catalogs/paimon-catalog.mdx | 76 ++++++++++++++++++++-
scripts/check_move.js | 3 +-
.../lakehouse/catalogs/hive-catalog.mdx | 9 +++
.../lakehouse/catalogs/iceberg-catalog.mdx | 21 ++++++
.../lakehouse/catalogs/paimon-catalog.mdx | 77 +++++++++++++++++++++-
.../lakehouse/catalogs/hive-catalog.mdx | 9 +++
.../lakehouse/catalogs/iceberg-catalog.mdx | 21 ++++++
.../lakehouse/catalogs/paimon-catalog.mdx | 77 +++++++++++++++++++++-
19 files changed, 629 insertions(+), 13 deletions(-)
diff --git a/docs/lakehouse/catalogs/hive-catalog.mdx
b/docs/lakehouse/catalogs/hive-catalog.mdx
index afccfe5f5cb..1eb12eb12e0 100644
--- a/docs/lakehouse/catalogs/hive-catalog.mdx
+++ b/docs/lakehouse/catalogs/hive-catalog.mdx
@@ -211,6 +211,15 @@ Hive transactional tables are supported from version 3.x
onwards. For details, r
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG hive_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
Access HMS without Kerberos authentication
diff --git a/docs/lakehouse/catalogs/iceberg-catalog.mdx
b/docs/lakehouse/catalogs/iceberg-catalog.mdx
index 8db8b28d5dc..366f0e425eb 100644
--- a/docs/lakehouse/catalogs/iceberg-catalog.mdx
+++ b/docs/lakehouse/catalogs/iceberg-catalog.mdx
@@ -191,6 +191,17 @@ The current Iceberg dependency is version 1.6.1, which is
compatible with higher
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
Access HMS without Kerberos authentication
@@ -871,6 +882,16 @@ The current Iceberg dependency is version 1.6.1, which is
compatible with higher
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hadoop',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='COS' label='COS'>
```sql
diff --git a/docs/lakehouse/catalogs/paimon-catalog.mdx
b/docs/lakehouse/catalogs/paimon-catalog.mdx
index 2ffe69f8fe5..43ee177d22a 100644
--- a/docs/lakehouse/catalogs/paimon-catalog.mdx
+++ b/docs/lakehouse/catalogs/paimon-catalog.mdx
@@ -167,6 +167,17 @@ The currently dependent Paimon version is 1.0.0.
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket/warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -429,7 +440,7 @@ The currently dependent Paimon version is 1.0.0.
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -445,6 +456,16 @@ The currently dependent Paimon version is 1.0.0.
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -523,7 +544,7 @@ The currently dependent Paimon version is 1.0.0.
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -1050,3 +1071,55 @@ Result:
## Appendix
+### FAQ
+
+1. `Could not find a file io implementation for scheme 's3a' in the classpath`
+
+ In versions prior to 3.1, when using HMS as metadata storage with 's3a' or
's3' protocol on the storage side, the following error occurs:
+
+ ```
+ java.io.UncheckedIOException:
org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io
implementation for scheme 's3a' in the classpath. Hadoop FileSystem also cannot
access this path 's3a://`.
+ ```
+
+ Versions prior to 3.1 do not support the `s3a` protocol. If you must use
it, you can temporarily add `s3a` protocol-related parameters as a workaround.
This operation is not needed for versions 3.1 and later.
+
+ For example:
+
+ ```sql
+ CREATE CATALOG `paimon_hms_on_s3a` PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://172.20.48.119:9383"
+ "warehouse" = "s3a://bucket/paimon_warehouse",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>",
+ "s3.region" = "ap-east-1",
+ "s3.endpoint" = "s3.ap-east-1.amazonaws.com",
+ -- explicitly add s3a parameters
+ "fs.s3a.access.key"="<ak>",
+ "fs.s3a.secret.key"="<sk>",
+ "fs.s3a.endpoint.region"="ap-east-1",
+ "fs.s3a.endpoint"="s3.ap-east-1.amazonaws.com",
+ "fs.s3a.impl"= "org.apache.hadoop.fs.s3a.S3AFileSystem",
+ );
+ ```
+
+ When using the `s3` protocol, you need to explicitly add Paimon S3-related
parameters, i.e., add the `paimon.` prefix. For example:
+
+ ```sql
+ CREATE CATALOG test_paimon_on_hms_aws_catalog PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket-hk/paimon_warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'ap-east-1',
+ 's3.endpoint' = 's3.ap-east-1.amazonaws.com',
+ 's3.access_key' = '<ak>',
+ 's3.secret_key' = '<sk>',
+ -- explicitly add s3a parameters
+ 'paimon.s3.region' = 'ap-east-1',
+ 'paimon.s3.endpoint' = 'ap-east-1',
+ 'paimon.s3.access-key' = '<ak>',
+ 'paimon.s3.secret-key' = '<sk>'
+ );
+ ```
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx
index d5153174e62..e85042b18d9 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx
@@ -220,6 +220,15 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG hive_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
访问未开启 Kerberos 认证的 HMS
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx
index d3283c7812f..b74a54933ed 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx
@@ -203,6 +203,17 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
访问未开启 Kerberos 认证的 HMS
@@ -883,6 +894,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hadoop',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='COS' label='COS'>
```sql
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx
index d6a2d494a40..69dc5ef0fc4 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx
@@ -166,6 +166,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.access_key' = '<ak>',
's3.secret_key' = '<sk>'
);
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket/warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
```
</TabItem>
<TabItem value='OSS' label='OSS'>
@@ -429,7 +439,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -445,6 +455,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -523,7 +543,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -1049,3 +1069,55 @@ SELECT `bucket` , COUNT(*) as file_count,
SUM(file_size_in_bytes)/1024/1024 as t
## 附录
+### FAQ
+
+1. `Could not find a file io implementation for scheme 's3a' in the classpath`
+
+ 3.1 之前的版本，使用 HMS 作为元数据存储，存储端 使用 's3a' 或 's3' 协议的时候，出现如下报错：
+
+ ```
+ java.io.UncheckedIOException:
org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io
implementation for scheme 's3a' in the classpath. Hadoop FileSystem also cannot
access this path 's3a://`.
+ ```
+
+ 3.1 之前的不支持 `s3a` 协议，如果必须使用，可以临时显示添加 `s3a` 协议相关参数临时解决。3.1 之后的版本无需此操作。
+
+ 如：
+
+ ```sql
+ CREATE CATALOG `paimon_hms_on_s3a` PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://172.20.48.119:9383"
+ "warehouse" = "s3a://bucket/paimon_warehouse",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>",
+ "s3.region" = "ap-east-1",
+ "s3.endpoint" = "s3.ap-east-1.amazonaws.com",
+ -- explicitly add s3a parameters
+ "fs.s3a.access.key"="<ak>",
+ "fs.s3a.secret.key"="<sk>",
+ "fs.s3a.endpoint.region"="ap-east-1",
+ "fs.s3a.endpoint"="s3.ap-east-1.amazonaws.com",
+ "fs.s3a.impl"= "org.apache.hadoop.fs.s3a.S3AFileSystem",
+ );
+ ```
+
+ 使用 `s3` 协议的时候，需要显示添加 Paimon S3 相关参数，即添加 `paimon.` 前缀。如
+
+ ```sql
+ CREATE CATALOG test_paimon_on_hms_aws_catalog PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket-hk/paimon_warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'ap-east-1',
+ 's3.endpoint' = 's3.ap-east-1.amazonaws.com',
+ 's3.access_key' = '<ak>',
+ 's3.secret_key' = '<sk>',
+ -- explicitly add s3 parameters
+ 'paimon.s3.region' = 'ap-east-1',
+ 'paimon.s3.endpoint' = 'ap-east-1',
+ 'paimon.s3.access-key' = '<ak>',
+ 'paimon.s3.secret-key' = '<sk>'
+ );
+ ```
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/hive-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/hive-catalog.mdx
index d5153174e62..e85042b18d9 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/hive-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/hive-catalog.mdx
@@ -220,6 +220,15 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG hive_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
访问未开启 Kerberos 认证的 HMS
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx
index d3283c7812f..b74a54933ed 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx
@@ -203,6 +203,17 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
访问未开启 Kerberos 认证的 HMS
@@ -883,6 +894,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hadoop',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='COS' label='COS'>
```sql
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx
index d6a2d494a40..69dc5ef0fc4 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx
@@ -166,6 +166,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.access_key' = '<ak>',
's3.secret_key' = '<sk>'
);
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket/warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
```
</TabItem>
<TabItem value='OSS' label='OSS'>
@@ -429,7 +439,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -445,6 +455,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -523,7 +543,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -1049,3 +1069,55 @@ SELECT `bucket` , COUNT(*) as file_count,
SUM(file_size_in_bytes)/1024/1024 as t
## 附录
+### FAQ
+
+1. `Could not find a file io implementation for scheme 's3a' in the classpath`
+
+ 3.1 之前的版本，使用 HMS 作为元数据存储，存储端 使用 's3a' 或 's3' 协议的时候，出现如下报错：
+
+ ```
+ java.io.UncheckedIOException:
org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io
implementation for scheme 's3a' in the classpath. Hadoop FileSystem also cannot
access this path 's3a://`.
+ ```
+
+ 3.1 之前的不支持 `s3a` 协议，如果必须使用，可以临时显示添加 `s3a` 协议相关参数临时解决。3.1 之后的版本无需此操作。
+
+ 如：
+
+ ```sql
+ CREATE CATALOG `paimon_hms_on_s3a` PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://172.20.48.119:9383"
+ "warehouse" = "s3a://bucket/paimon_warehouse",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>",
+ "s3.region" = "ap-east-1",
+ "s3.endpoint" = "s3.ap-east-1.amazonaws.com",
+ -- explicitly add s3a parameters
+ "fs.s3a.access.key"="<ak>",
+ "fs.s3a.secret.key"="<sk>",
+ "fs.s3a.endpoint.region"="ap-east-1",
+ "fs.s3a.endpoint"="s3.ap-east-1.amazonaws.com",
+ "fs.s3a.impl"= "org.apache.hadoop.fs.s3a.S3AFileSystem",
+ );
+ ```
+
+ 使用 `s3` 协议的时候，需要显示添加 Paimon S3 相关参数，即添加 `paimon.` 前缀。如
+
+ ```sql
+ CREATE CATALOG test_paimon_on_hms_aws_catalog PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket-hk/paimon_warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'ap-east-1',
+ 's3.endpoint' = 's3.ap-east-1.amazonaws.com',
+ 's3.access_key' = '<ak>',
+ 's3.secret_key' = '<sk>',
+ -- explicitly add s3 parameters
+ 'paimon.s3.region' = 'ap-east-1',
+ 'paimon.s3.endpoint' = 'ap-east-1',
+ 'paimon.s3.access-key' = '<ak>',
+ 'paimon.s3.secret-key' = '<sk>'
+ );
+ ```
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/hive-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/hive-catalog.mdx
index d5153174e62..e85042b18d9 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/hive-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/hive-catalog.mdx
@@ -220,6 +220,15 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG hive_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
访问未开启 Kerberos 认证的 HMS
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/iceberg-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/iceberg-catalog.mdx
index d3283c7812f..b74a54933ed 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/iceberg-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/iceberg-catalog.mdx
@@ -203,6 +203,17 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
访问未开启 Kerberos 认证的 HMS
@@ -883,6 +894,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hadoop',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='COS' label='COS'>
```sql
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/paimon-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/paimon-catalog.mdx
index d6a2d494a40..69dc5ef0fc4 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/paimon-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/lakehouse/catalogs/paimon-catalog.mdx
@@ -166,6 +166,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.access_key' = '<ak>',
's3.secret_key' = '<sk>'
);
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket/warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
```
</TabItem>
<TabItem value='OSS' label='OSS'>
@@ -429,7 +439,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -445,6 +455,16 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
's3.secret_key' = '<sk>'
);
```
+ 使用 IAM Assumed Role 的方式获取 S3 访问凭证 (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -523,7 +543,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -1049,3 +1069,55 @@ SELECT `bucket` , COUNT(*) as file_count,
SUM(file_size_in_bytes)/1024/1024 as t
## 附录
+### FAQ
+
+1. `Could not find a file io implementation for scheme 's3a' in the classpath`
+
+ 3.1 之前的版本，使用 HMS 作为元数据存储，存储端 使用 's3a' 或 's3' 协议的时候，出现如下报错：
+
+ ```
+ java.io.UncheckedIOException:
org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io
implementation for scheme 's3a' in the classpath. Hadoop FileSystem also cannot
access this path 's3a://`.
+ ```
+
+ 3.1 之前的不支持 `s3a` 协议，如果必须使用，可以临时显示添加 `s3a` 协议相关参数临时解决。3.1 之后的版本无需此操作。
+
+ 如：
+
+ ```sql
+ CREATE CATALOG `paimon_hms_on_s3a` PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://172.20.48.119:9383"
+ "warehouse" = "s3a://bucket/paimon_warehouse",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>",
+ "s3.region" = "ap-east-1",
+ "s3.endpoint" = "s3.ap-east-1.amazonaws.com",
+ -- explicitly add s3a parameters
+ "fs.s3a.access.key"="<ak>",
+ "fs.s3a.secret.key"="<sk>",
+ "fs.s3a.endpoint.region"="ap-east-1",
+ "fs.s3a.endpoint"="s3.ap-east-1.amazonaws.com",
+ "fs.s3a.impl"= "org.apache.hadoop.fs.s3a.S3AFileSystem",
+ );
+ ```
+
+ 使用 `s3` 协议的时候，需要显示添加 Paimon S3 相关参数，即添加 `paimon.` 前缀。如
+
+ ```sql
+ CREATE CATALOG test_paimon_on_hms_aws_catalog PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket-hk/paimon_warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'ap-east-1',
+ 's3.endpoint' = 's3.ap-east-1.amazonaws.com',
+ 's3.access_key' = '<ak>',
+ 's3.secret_key' = '<sk>',
+ -- explicitly add s3 parameters
+ 'paimon.s3.region' = 'ap-east-1',
+ 'paimon.s3.endpoint' = 'ap-east-1',
+ 'paimon.s3.access-key' = '<ak>',
+ 'paimon.s3.secret-key' = '<sk>'
+ );
+ ```
diff --git a/scripts/check_move.js b/scripts/check_move.js
index 27f6727e1ee..2ec9b85b1af 100644
--- a/scripts/check_move.js
+++ b/scripts/check_move.js
@@ -45,7 +45,8 @@ function isLocalLink(link) {
!link.startsWith("https://";) &&
!link.startsWith("mailto:";) &&
!link.startsWith("#") &&
- !path.isAbsolute(link);
+ !path.isAbsolute(link)&&
+ !/.*@.*\..*/.test(link);
}
function removeCodeBlocks(content) {
diff --git a/versioned_docs/version-2.1/lakehouse/catalogs/hive-catalog.mdx
b/versioned_docs/version-2.1/lakehouse/catalogs/hive-catalog.mdx
index afccfe5f5cb..1eb12eb12e0 100644
--- a/versioned_docs/version-2.1/lakehouse/catalogs/hive-catalog.mdx
+++ b/versioned_docs/version-2.1/lakehouse/catalogs/hive-catalog.mdx
@@ -211,6 +211,15 @@ Hive transactional tables are supported from version 3.x
onwards. For details, r
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG hive_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
Access HMS without Kerberos authentication
diff --git a/versioned_docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx
b/versioned_docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx
index 8db8b28d5dc..366f0e425eb 100644
--- a/versioned_docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx
+++ b/versioned_docs/version-2.1/lakehouse/catalogs/iceberg-catalog.mdx
@@ -191,6 +191,17 @@ The current Iceberg dependency is version 1.6.1, which is
compatible with higher
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
Access HMS without Kerberos authentication
@@ -871,6 +882,16 @@ The current Iceberg dependency is version 1.6.1, which is
compatible with higher
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hadoop',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='COS' label='COS'>
```sql
diff --git a/versioned_docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx
b/versioned_docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx
index 2ffe69f8fe5..43ee177d22a 100644
--- a/versioned_docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx
+++ b/versioned_docs/version-2.1/lakehouse/catalogs/paimon-catalog.mdx
@@ -167,6 +167,17 @@ The currently dependent Paimon version is 1.0.0.
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket/warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -429,7 +440,7 @@ The currently dependent Paimon version is 1.0.0.
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -445,6 +456,16 @@ The currently dependent Paimon version is 1.0.0.
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -523,7 +544,7 @@ The currently dependent Paimon version is 1.0.0.
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -1050,3 +1071,55 @@ Result:
## Appendix
+### FAQ
+
+1. `Could not find a file io implementation for scheme 's3a' in the classpath`
+
+ In versions prior to 3.1, when using HMS as metadata storage with 's3a' or
's3' protocol on the storage side, the following error occurs:
+
+ ```
+ java.io.UncheckedIOException:
org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io
implementation for scheme 's3a' in the classpath. Hadoop FileSystem also cannot
access this path 's3a://`.
+ ```
+
+ Versions prior to 3.1 do not support the `s3a` protocol. If you must use
it, you can temporarily add `s3a` protocol-related parameters as a workaround.
This operation is not needed for versions 3.1 and later.
+
+ For example:
+
+ ```sql
+ CREATE CATALOG `paimon_hms_on_s3a` PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://172.20.48.119:9383"
+ "warehouse" = "s3a://bucket/paimon_warehouse",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>",
+ "s3.region" = "ap-east-1",
+ "s3.endpoint" = "s3.ap-east-1.amazonaws.com",
+ -- explicitly add s3a parameters
+ "fs.s3a.access.key"="<ak>",
+ "fs.s3a.secret.key"="<sk>",
+ "fs.s3a.endpoint.region"="ap-east-1",
+ "fs.s3a.endpoint"="s3.ap-east-1.amazonaws.com",
+ "fs.s3a.impl"= "org.apache.hadoop.fs.s3a.S3AFileSystem",
+ );
+ ```
+
+ When using the `s3` protocol, you need to explicitly add Paimon S3-related
parameters, i.e., add the `paimon.` prefix. For example:
+
+ ```sql
+ CREATE CATALOG test_paimon_on_hms_aws_catalog PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket-hk/paimon_warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'ap-east-1',
+ 's3.endpoint' = 's3.ap-east-1.amazonaws.com',
+ 's3.access_key' = '<ak>',
+ 's3.secret_key' = '<sk>',
+ -- explicitly add s3a parameters
+ 'paimon.s3.region' = 'ap-east-1',
+ 'paimon.s3.endpoint' = 'ap-east-1',
+ 'paimon.s3.access-key' = '<ak>',
+ 'paimon.s3.secret-key' = '<sk>'
+ );
+ ```
diff --git a/versioned_docs/version-3.0/lakehouse/catalogs/hive-catalog.mdx
b/versioned_docs/version-3.0/lakehouse/catalogs/hive-catalog.mdx
index afccfe5f5cb..1eb12eb12e0 100644
--- a/versioned_docs/version-3.0/lakehouse/catalogs/hive-catalog.mdx
+++ b/versioned_docs/version-3.0/lakehouse/catalogs/hive-catalog.mdx
@@ -211,6 +211,15 @@ Hive transactional tables are supported from version 3.x
onwards. For details, r
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG hive_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
Access HMS without Kerberos authentication
diff --git a/versioned_docs/version-3.0/lakehouse/catalogs/iceberg-catalog.mdx
b/versioned_docs/version-3.0/lakehouse/catalogs/iceberg-catalog.mdx
index 8db8b28d5dc..366f0e425eb 100644
--- a/versioned_docs/version-3.0/lakehouse/catalogs/iceberg-catalog.mdx
+++ b/versioned_docs/version-3.0/lakehouse/catalogs/iceberg-catalog.mdx
@@ -191,6 +191,17 @@ The current Iceberg dependency is version 1.6.1, which is
compatible with higher
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
Access HMS without Kerberos authentication
@@ -871,6 +882,16 @@ The current Iceberg dependency is version 1.6.1, which is
compatible with higher
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG iceberg_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'hadoop',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='COS' label='COS'>
```sql
diff --git a/versioned_docs/version-3.0/lakehouse/catalogs/paimon-catalog.mdx
b/versioned_docs/version-3.0/lakehouse/catalogs/paimon-catalog.mdx
index 2ffe69f8fe5..43ee177d22a 100644
--- a/versioned_docs/version-3.0/lakehouse/catalogs/paimon-catalog.mdx
+++ b/versioned_docs/version-3.0/lakehouse/catalogs/paimon-catalog.mdx
@@ -167,6 +167,17 @@ The currently dependent Paimon version is 1.0.0.
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_hms_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket/warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -429,7 +440,7 @@ The currently dependent Paimon version is 1.0.0.
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -445,6 +456,16 @@ The currently dependent Paimon version is 1.0.0.
's3.secret_key' = '<sk>'
);
```
+ Obtaining S3 access credentials using an IAM Assumed Role (3.1.1+)
+ ```sql
+ CREATE CATALOG paimon_fs_on_s3_iamrole PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://bucket/warehouse',
+ 's3.region' = 'us-east-1',
+ 's3.role_arn' = 'arn:aws:iam::543815668950:role/role'
+ );
+ ```
</TabItem>
<TabItem value='OSS' label='OSS'>
```sql
@@ -523,7 +544,7 @@ The currently dependent Paimon version is 1.0.0.
'type' = 'paimon',
'paimon.catalog.type' = 'filesystem',
'warehouse' = 'hdfs://127.0.0.1:8020/user/doris/paimon1',
- 'fs.default.fs' = 'hdfs://127.0.0.1:8020'
+ 'fs.defaultFS' = 'hdfs://127.0.0.1:8020'
);
```
</TabItem>
@@ -1050,3 +1071,55 @@ Result:
## Appendix
+### FAQ
+
+1. `Could not find a file io implementation for scheme 's3a' in the classpath`
+
+ In versions prior to 3.1, when using HMS as metadata storage with 's3a' or
's3' protocol on the storage side, the following error occurs:
+
+ ```
+ java.io.UncheckedIOException:
org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io
implementation for scheme 's3a' in the classpath. Hadoop FileSystem also cannot
access this path 's3a://`.
+ ```
+
+ Versions prior to 3.1 do not support the `s3a` protocol. If you must use
it, you can temporarily add `s3a` protocol-related parameters as a workaround.
This operation is not needed for versions 3.1 and later.
+
+ For example:
+
+ ```sql
+ CREATE CATALOG `paimon_hms_on_s3a` PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://172.20.48.119:9383"
+ "warehouse" = "s3a://bucket/paimon_warehouse",
+ "s3.access_key" = "<ak>",
+ "s3.secret_key" = "<sk>",
+ "s3.region" = "ap-east-1",
+ "s3.endpoint" = "s3.ap-east-1.amazonaws.com",
+ -- explicitly add s3a parameters
+ "fs.s3a.access.key"="<ak>",
+ "fs.s3a.secret.key"="<sk>",
+ "fs.s3a.endpoint.region"="ap-east-1",
+ "fs.s3a.endpoint"="s3.ap-east-1.amazonaws.com",
+ "fs.s3a.impl"= "org.apache.hadoop.fs.s3a.S3AFileSystem",
+ );
+ ```
+
+ When using the `s3` protocol, you need to explicitly add Paimon S3-related
parameters, i.e., add the `paimon.` prefix. For example:
+
+ ```sql
+ CREATE CATALOG test_paimon_on_hms_aws_catalog PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 's3://bucket-hk/paimon_warehouse',
+ 'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
+ 's3.region' = 'ap-east-1',
+ 's3.endpoint' = 's3.ap-east-1.amazonaws.com',
+ 's3.access_key' = '<ak>',
+ 's3.secret_key' = '<sk>',
+ -- explicitly add s3a parameters
+ 'paimon.s3.region' = 'ap-east-1',
+ 'paimon.s3.endpoint' = 'ap-east-1',
+ 'paimon.s3.access-key' = '<ak>',
+ 'paimon.s3.secret-key' = '<sk>'
+ );
+ ```
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
