This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new bf940c148e5 [doc](iceberg) add Iceberg system table (#2514)
bf940c148e5 is described below
commit bf940c148e5d766659375e498edf2d2d24f42297
Author: Socrates <[email protected]>
AuthorDate: Wed Jun 25 18:52:28 2025 +0800
[doc](iceberg) add Iceberg system table (#2514)
relate pr: https://github.com/apache/doris/pull/51190
## Versions
- [x] dev
- [ ] 3.0
- [ ] 2.1
- [ ] 2.0
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
---------
Co-authored-by: Mingyu Chen (Rayner) <[email protected]>
---
docs/lakehouse/catalogs/iceberg-catalog.md | 247 +++++++++++++++++----
docs/lakehouse/catalogs/paimon-catalog.md | 43 ++++
.../current/lakehouse/catalogs/iceberg-catalog.md | 244 ++++++++++++++++----
.../current/lakehouse/catalogs/paimon-catalog.md | 43 ++++
.../version-2.1/releasenotes/v3.0/release-3.0.6.md | 4 +-
.../version-3.0/releasenotes/v3.0/release-3.0.6.md | 4 +-
6 files changed, 504 insertions(+), 81 deletions(-)
diff --git a/docs/lakehouse/catalogs/iceberg-catalog.md
b/docs/lakehouse/catalogs/iceberg-catalog.md
index cc156d0d819..94d04a8ebb3 100644
--- a/docs/lakehouse/catalogs/iceberg-catalog.md
+++ b/docs/lakehouse/catalogs/iceberg-catalog.md
@@ -248,7 +248,7 @@ CREATE CATALOG iceberg_s3 PROPERTIES (
);
```
-Please refer to [Integration with AWS S3
Tables](../best-practices/doris-aws-s3tables.md)。
+Please refer to [Integration with AWS S3
Tables](../best-practices/doris-aws-s3tables.md).
## Query Operations
@@ -330,6 +330,213 @@ SELECT * FROM iceberg_tbl FOR VERSION AS OF 'tag1';
For the `FOR VERSION AS OF` syntax, Doris will automatically determine whether
the parameter is a timestamp or a Branch/Tag name.
+## System Tables
+
+> This feature is supported since version 3.1.0
+
+Doris supports querying Iceberg system tables to retrieve metadata information
about tables. You can use system tables to view snapshot history, manifest
files, data files, partitions, and other metadata.
+
+To access metadata of an Iceberg table, append a `$` symbol followed by the
system table name to the table name:
+
+```sql
+SELECT * FROM iceberg_table$system_table_name;
+```
+
+For example, to view the table's history, you can execute:
+
+```sql
+SELECT * FROM iceberg_table$history;
+```
+
+> Currently, the `all_manifests` and `position_deletes` system tables are not
yet supported and are planned to be supported in future versions.
+
+### entries
+
+Shows all manifest entries for the current snapshot of the table:
+
+`all_entries` and `entries` are similar, with the difference that
`all_entries` contains entries from all snapshots, while `entries` only
contains entries from the current snapshot.
+
+```sql
+SELECT * FROM iceberg_table$entries;
+```
+
+Result:
+
+```text
++--------+---------------------+-----------------+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| status | snapshot_id | sequence_number | file_sequence_number |
data_file
[...]
++--------+---------------------+-----------------+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| 2 | 4890031351138056789 | 1 | 1 |
{"content":0,
"file_path":"s3://.../iceberg_table/data/id=1/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00001.parquet",
"file_format":"PARQUET", "spec_id":0, "partition":{"id":1}, "record_count":1,
"file_size_in_bytes":625, "column_sizes":{1:36, 2:41}, "value_counts":{1:1,
2:1}, "null_value_counts":{1:0, 2:0}, "nan_value_counts":{},
"lower_bounds":{1:" ", 2:"Alice"}, "upper_bounds":{1:" ", 2:"Alice"}, "key_
[...]
+| 0 | 1851184769713369003 | 1 | 1 |
{"content":0,
"file_path":"s3://.../iceberg_table/data/id=2/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00002.parquet",
"file_format":"PARQUET", "spec_id":0, "partition":{"id":2}, "record_count":1,
"file_size_in_bytes":611, "column_sizes":{1:36, 2:39}, "value_counts":{1:1,
2:1}, "null_value_counts":{1:0, 2:0}, "nan_value_counts":{},
"lower_bounds":{1:" ", 2:"Bob"}, "upper_bounds":{1:" ", 2:"Bob"}, "key_meta
[...]
++--------+---------------------+-----------------+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+```
+
+### files
+
+Shows file list for the current snapshot of the table:
+
+```sql
+SELECT * FROM iceberg_table$files;
+```
+
+Result:
+
+```text
++---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+---------+-----------+--------------+--------------------+--------------+--------------+-------------------+------------------+-----------------------+-----------------------+--------------+---------------+--------------+---------------+--------------+----------------------+----------------+-----------
[...]
+| content | file_path
| file_format | spec_id | partition | record_count |
file_size_in_bytes | column_sizes | value_counts | null_value_counts |
nan_value_counts | lower_bounds | upper_bounds | key_metadata
| split_offsets | equality_ids | sort_order_id | first_row_id |
referenced_data_file | content_offset | content_si [...]
++---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+---------+-----------+--------------+--------------------+--------------+--------------+-------------------+------------------+-----------------------+-----------------------+--------------+---------------+--------------+---------------+--------------+----------------------+----------------+-----------
[...]
+| 0 |
s3://.../iceberg_table/data/id=2/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00002.parquet
| PARQUET | 0 | {"id":2} | 1 | 611 |
{1:36, 2:39} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Bob"} | {1:" ", 2:"Bob"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
+| 0 |
s3://.../iceberg_table/data/id=4/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00004.parquet
| PARQUET | 0 | {"id":4} | 1 | 618 |
{1:36, 2:40} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Dave"} | {1:" ", 2:"Dave"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
+| 0 |
s3://.../iceberg_table/data/id=6/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00006.parquet
| PARQUET | 0 | {"id":6} | 1 | 625 |
{1:36, 2:41} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Frank"} | {1:" ", 2:"Frank"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
+| 0 |
s3://.../iceberg_table/data/id=8/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00008.parquet
| PARQUET | 0 | {"id":8} | 1 | 625 |
{1:36, 2:41} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Heidi"} | {1:" ", 2:"Heidi"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
+| 0 |
s3://.../iceberg_table/data/id=10/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00010.parquet
| PARQUET | 0 | {"id":10} | 1 | 618 |
{1:36, 2:40} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Judy"} | {1:" ", 2:"Judy"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
++---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+---------+-----------+--------------+--------------------+--------------+--------------+-------------------+------------------+-----------------------+-----------------------+--------------+---------------+--------------+---------------+--------------+----------------------+----------------+-----------
[...]
+```
+
+Similar to the `files` system table, you can also query specific file type
information through the following system tables:
+
+```sql
+-- Query data files for the current snapshot
+SELECT * FROM iceberg_table$data_files;
+
+-- Query delete files for the current snapshot
+SELECT * FROM iceberg_table$delete_files;
+
+-- Query all files (including data and delete files) from all snapshots
+SELECT * FROM iceberg_table$all_files;
+
+-- Query data files from all snapshots
+SELECT * FROM iceberg_table$all_data_files;
+
+-- Query delete files from all snapshots
+SELECT * FROM iceberg_table$all_delete_files;
+```
+
+The result format of these tables is similar to the `files` system table, but
each focuses specifically on data files or delete files. System tables with the
`all_` prefix contain files from all snapshots, not just files from the current
snapshot.
+
+Note: When specific types of files do not exist in the table (for example,
querying `delete_files` when there are no delete files in the table), the query
result may be empty.
+
+### history
+
+Shows all history of the table:
+
+```sql
+SELECT * FROM iceberg_table$history;
+```
+
+Result:
+
+```text
++----------------------------+---------------------+---------------------+---------------------+
+| made_current_at | snapshot_id | parent_id |
is_current_ancestor |
++----------------------------+---------------------+---------------------+---------------------+
+| 2025-06-12 22:29:16.357000 | 1851184769713369003 | NULL |
1 |
+| 2025-06-12 22:29:39.922000 | 4890031351138056789 | 1851184769713369003 |
1 |
++----------------------------+---------------------+---------------------+---------------------+
+```
+
+### manifests
+
+Shows manifest file info of the table:
+
+```sql
+SELECT * FROM iceberg_table$manifests;
+```
+
+Result:
+
+```text
++---------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------+-----------------------------+----------------------------+--------------------------------------------------------------------------------+
+| content | path
|
length | partition_spec_id | added_snapshot_id | added_data_files_count |
existing_data_files_count | deleted_data_files_count | added_delete_files_count
| existing_delete_files_count | deleted_delete_files_count |
partition_summaries |
++---------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------+-----------------------------+----------------------------+--------------------------------------------------------------------------------+
+| 0 |
s3://.../iceberg_table/metadata/3194eb8b-5ea4-4cbe-95ba-073229458e7b-m0.avro |
7138 | 0 | 4890031351138056789 | 0 |
5 | 5 | 0 |
0 | 0 | [{"contains_null":0,
"contains_nan":0, "lower_bound":"1", "upper_bound":"10"}] |
++---------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------+-----------------------------+----------------------------+--------------------------------------------------------------------------------+
+```
+
+### metadata_log_entries
+
+Shows meta logs of the table:
+
+```sql
+SELECT * FROM iceberg_table$metadata_log_entries;
+```
+
+Result:
+
+```text
++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+------------------+------------------------+
+| timestamp | file
| latest_snapshot_id | latest_schema_id |
latest_sequence_number |
++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+------------------+------------------------+
+| 2025-06-12 22:29:06.948000 |
s3://.../iceberg_table/metadata/00000-e373aa16-15f1-4e69-ae7d-5ed64199cf9a.metadata.json
| NULL | NULL | NULL |
+| 2025-06-12 22:29:16.357000 |
s3://.../iceberg_table/metadata/00001-bbc8e244-e41c-4958-92f4-63b8c3ee1196.metadata.json
| 1851184769713369003 | 0 | 1 |
+| 2025-06-12 22:29:39.922000 |
s3://.../iceberg_table/metadata/00002-7dc00d6a-6269-4200-9d28-5f8c1c6b9f99.metadata.json
| 4890031351138056789 | 0 | 2 |
++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+------------------+------------------------+
+```
+
+### partitions
+
+Shows partitions of the table:
+
+```sql
+SELECT * FROM iceberg_table$partitions;
+```
+
+Result:
+
+```text
++-----------+---------+--------------+------------+-------------------------------+------------------------------+----------------------------+------------------------------+----------------------------+----------------------------+--------------------------+
+| partition | spec_id | record_count | file_count |
total_data_file_size_in_bytes | position_delete_record_count |
position_delete_file_count | equality_delete_record_count |
equality_delete_file_count | last_updated_at |
last_updated_snapshot_id |
++-----------+---------+--------------+------------+-------------------------------+------------------------------+----------------------------+------------------------------+----------------------------+----------------------------+--------------------------+
+| {"id":8} | 0 | 1 | 1 |
625 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
+| {"id":6} | 0 | 1 | 1 |
625 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
+| {"id":10} | 0 | 1 | 1 |
618 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
+| {"id":4} | 0 | 1 | 1 |
618 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
+| {"id":2} | 0 | 1 | 1 |
611 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
++-----------+---------+--------------+------------+-------------------------------+------------------------------+----------------------------+------------------------------+----------------------------+----------------------------+--------------------------+
+```
+
+Note:
+
+1. For non-partitioned tables, the `partitions` table will not contain the
`partition` and `spec_id` fields.
+2. The `partitions` table shows partitions that contain data files or delete
files in the current snapshot. However, delete files are not applied, so in
some cases, a partition may still be displayed even if all data rows in the
partition have been marked as deleted by delete files.
+
+### refs
+
+Shows all known snapshot references (branches and tags) for the table:
+
+```sql
+SELECT * FROM iceberg_table$refs;
+```
+
+Result:
+
+```text
++------+--------+---------------------+-------------------------+-----------------------+------------------------+
+| name | type | snapshot_id | max_reference_age_in_ms |
min_snapshots_to_keep | max_snapshot_age_in_ms |
++------+--------+---------------------+-------------------------+-----------------------+------------------------+
+| main | BRANCH | 4890031351138056789 | NULL |
NULL | NULL |
++------+--------+---------------------+-------------------------+-----------------------+------------------------+
+```
+
+### snapshots
+
+Shows all snapshots of the table:
+
+```sql
+SELECT * FROM iceberg_table$snapshots;
+```
+
+Result:
+
+```text
++----------------------------+---------------------+---------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| committed_at | snapshot_id | parent_id |
operation | manifest_list
| summary
[...]
++----------------------------+---------------------+---------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| 2025-06-12 22:29:16.357000 | 1851184769713369003 | NULL |
append |
s3://.../iceberg_table/metadata/snap-1851184769713369003-1-82059f57-821a-4983-b083-002cc2cde313.avro
| {"spark.app.id":"application_1738810850199_0472", "added-data-files":"10",
"added-records":"10", "added-files-size":"6200",
"changed-partition-count":"10", "total-records":"10",
"total-files-size":"6200", "total-data-files":"10", "total-delete-files":"0",
"total-position-deletes":"0", "total-equality- [...]
+| 2025-06-12 22:29:39.922000 | 4890031351138056789 | 1851184769713369003 |
overwrite |
s3://.../iceberg_table/metadata/snap-4890031351138056789-1-3194eb8b-5ea4-4cbe-95ba-073229458e7b.avro
| {"spark.app.id":"application_1738810850199_0472", "deleted-data-files":"5",
"deleted-records":"5", "removed-files-size":"3103",
"changed-partition-count":"5", "total-records":"5", "total-files-size":"3097",
"total-data-files":"5", "total-delete-files":"0", "total-position-deletes":"0",
"total-equality [...]
++----------------------------+---------------------+---------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+```
+
## Write Operations
### INSERT INTO
@@ -540,44 +747,6 @@ For an Iceberg Database, you must first drop all tables
under the database befor
* Object storage
-## System Tables
-
-Doris supports following system tables for iceberg table. These system tables
contains various metadata of Iceberg table.
-
-### snapshots
-
-Provides a detailed view of snapshots of the Iceberg table.
-
-```sql
-SELECT * FROM iceberg_meta(
- 'table' = 'iceberg_ctl.iceberg_db.iceberg_tbl',
- 'query_type' = 'snapshots'
-)
-```
-
-Or(Since 3.1.x):
-
-```
-SELECT * FROM iceberg_ctl.iceberg_db.iceberg_tbl$snapshost;
-```
-
-```
-*************************** 1. row ***************************
- committed_at: 2024-11-28 11:07:29
- snapshot_id: 8903826400153112036
- parent_id: -1
- operation: append
-manifest_list:
oss://path/to/metadata/snap-8903826400153112036-1-3835e66d-9a18-4cb0-b9b0-9ec80527ad8d.avro
- summary:
{"added-data-files":"2","added-records":"3","added-files-size":"2742","changed-partition-count":"2","total-records":"3","total-files-size":"2742","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}
-*************************** 2. row ***************************
- committed_at: 2024-11-28 11:10:11
- snapshot_id: 6099853805930794326
- parent_id: 8903826400153112036
- operation: append
-manifest_list:
oss://path/to/metadata/snap-6099853805930794326-1-dd46a1bd-219b-4fb0-bb46-ac441d8b3105.avro
- summary:
{"added-data-files":"1","added-records":"1","added-files-size":"1367","changed-partition-count":"1","total-records":"4","total-files-size":"4109","total-data-files":"3","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}
-```
-
## Appendix
### Change Log
diff --git a/docs/lakehouse/catalogs/paimon-catalog.md
b/docs/lakehouse/catalogs/paimon-catalog.md
index 66b847d2f6d..d23135cb8b7 100644
--- a/docs/lakehouse/catalogs/paimon-catalog.md
+++ b/docs/lakehouse/catalogs/paimon-catalog.md
@@ -221,5 +221,48 @@ SELECT * FROM paimon_tbl LIMIT 10;
SELECT * FROM paimon_ctl.paimon_db.paimon_tbl LIMIT 10;
```
+### Batch Incremental Query
+
+> This feature is supported since version 3.1.0
+
+Supports [Batch
Incremental](https://paimon.apache.org/docs/master/flink/sql-query/#batch-incremental)
queries for Paimon, similar to Flink.
+
+Supports querying incremental data within specified snapshot or timestamp
intervals. The interval is left-closed and right-open.
+
+```sql
+-- read from snapshot 2
+SELECT * FROM paimon_table@incr('startSnapshotId'='2');
+
+-- between snapshots [0, 5)
+SELECT * FROM paimon_table@incr('startSnapshotId'='0', 'endSnapshotId'='5');
+
+-- between snapshots [0, 5) with specified scan mode
+SELECT * FROM paimon_table@incr('startSnapshotId'='0', 'endSnapshotId'='5',
'incrementalBetweenScanMode'='diff');
+
+-- read from start timestamp
+SELECT * FROM paimon_table@incr('startTimestamp'='1750844949');
+
+-- read between timestamp
+SELECT * FROM paimon_table@incr('startTimestamp'='1750844949',
'endTimestamp'='1750944949');
+```
+
+Parameter:
+
+| Parameter | Description | Example |
+| --- | --- | -- |
+| `startSnapshotId` | Starting snapshot ID, must be greater than 0 |
`'startSnapshotId'='3'` |
+| `endSnapshotId` | Ending snapshot ID, must be greater than
`startSnapshotId`. Optional, if not specified, reads from `startSnapshotId` to
the latest snapshot | `'endSnapshotId'='10'` |
+| `incrementalBetweenScanMode` | Specifies the incremental read mode, default
is `auto`, supports `delta`, `changelog` and `diff` |
`'incrementalBetweenScanMode'='delta'` |
+| `startTimestamp` | Starting snapshot timestamp, must be greater than or
equal to 0 | `'startTimestamp'='1750844949'` |
+| `endTimestamp` | Ending snapshot timestamp, must be greater than
`startTimestamp`. Optional, if not specified, reads from `startTimestamp` to
the latest snapshot | `'endTimestamp'='1750944949'` |
+
+> `startSnapshotId` and `endSnapshotId` will compose the Paimon parameter
`'incremental-between'='3,10'`
+
+> `startTimestamp` and `endTimestamp` will compose the Paimon parameter
`'incremental-between-timestamp'='1750844949,1750944949'`
+
+> `incrementalBetweenScanMode` corresponds to the Paimon parameter
`incremental-between-scan-mode`.
+
+Refer to the [Paimon
documentation](https://paimon.apache.org/docs/master/maintenance/configurations/)
for further details about these parameters.
+
## Appendix
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.md
index 3b035735d5b..7e2138128fd 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.md
@@ -340,6 +340,212 @@ SELECT * FROM iceberg_tbl FOR VERSION AS OF 'tag1';
对于 `FOR VERSION AS OF` 语法，Doris 会根据后面的参数，自动判断是时间戳还是 Branch/Tag 名称。
+## 系统表
+
+> 该功能自 3.1.0 版本支持
+
+Doris 支持查询 Iceberg 系统表，用来查询表的相关元信息。支持使用系统表查看快照历史、清单文件、数据文件、分区等信息。
+
+要访问 Iceberg 表的元数据，可以在表名后添加 `$` 符号，后跟系统表名称：
+
+```sql
+SELECT * FROM iceberg_table$system_table_name;
+```
+
+例如，要查看表的历史记录，可以执行：
+
+```sql
+SELECT * FROM iceberg_table$history;
+```
+
+> 目前 `all_manifests` 和 `position_deletes` 系统表尚未支持，计划在以后版本中支持。
+
+### entries
+
+显示表当前快照的所有清单条目：
+
+`all_entries` 和 `entries` 类似，区别在于 `all_entries` 包含了所有快照的条目，而 `entries`
只包含当前快照的条目。
+
+```sql
+SELECT * FROM iceberg_table$entries;
+```
+
+结果：
+
+```text
++--------+---------------------+-----------------+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| status | snapshot_id | sequence_number | file_sequence_number |
data_file
[...]
++--------+---------------------+-----------------+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| 2 | 4890031351138056789 | 1 | 1 |
{"content":0,
"file_path":"s3://.../iceberg_table/data/id=1/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00001.parquet",
"file_format":"PARQUET", "spec_id":0, "partition":{"id":1}, "record_count":1,
"file_size_in_bytes":625, "column_sizes":{1:36, 2:41}, "value_counts":{1:1,
2:1}, "null_value_counts":{1:0, 2:0}, "nan_value_counts":{},
"lower_bounds":{1:" ", 2:"Alice"}, "upper_bounds":{1:" ", 2:"Alice"}, "key_
[...]
+| 0 | 1851184769713369003 | 1 | 1 |
{"content":0,
"file_path":"s3://.../iceberg_table/data/id=2/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00002.parquet",
"file_format":"PARQUET", "spec_id":0, "partition":{"id":2}, "record_count":1,
"file_size_in_bytes":611, "column_sizes":{1:36, 2:39}, "value_counts":{1:1,
2:1}, "null_value_counts":{1:0, 2:0}, "nan_value_counts":{},
"lower_bounds":{1:" ", 2:"Bob"}, "upper_bounds":{1:" ", 2:"Bob"}, "key_meta
[...]
++--------+---------------------+-----------------+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+```
+
+### files
+
+显示当前快照的文件信息：
+
+```sql
+SELECT * FROM iceberg_table$files;
+```
+
+结果：
+```text
++---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+---------+-----------+--------------+--------------------+--------------+--------------+-------------------+------------------+-----------------------+-----------------------+--------------+---------------+--------------+---------------+--------------+----------------------+----------------+-----------
[...]
+| content | file_path
| file_format | spec_id | partition | record_count |
file_size_in_bytes | column_sizes | value_counts | null_value_counts |
nan_value_counts | lower_bounds | upper_bounds | key_metadata
| split_offsets | equality_ids | sort_order_id | first_row_id |
referenced_data_file | content_offset | content_si [...]
++---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+---------+-----------+--------------+--------------------+--------------+--------------+-------------------+------------------+-----------------------+-----------------------+--------------+---------------+--------------+---------------+--------------+----------------------+----------------+-----------
[...]
+| 0 |
s3://.../iceberg_table/data/id=2/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00002.parquet
| PARQUET | 0 | {"id":2} | 1 | 611 |
{1:36, 2:39} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Bob"} | {1:" ", 2:"Bob"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
+| 0 |
s3://.../iceberg_table/data/id=4/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00004.parquet
| PARQUET | 0 | {"id":4} | 1 | 618 |
{1:36, 2:40} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Dave"} | {1:" ", 2:"Dave"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
+| 0 |
s3://.../iceberg_table/data/id=6/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00006.parquet
| PARQUET | 0 | {"id":6} | 1 | 625 |
{1:36, 2:41} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Frank"} | {1:" ", 2:"Frank"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
+| 0 |
s3://.../iceberg_table/data/id=8/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00008.parquet
| PARQUET | 0 | {"id":8} | 1 | 625 |
{1:36, 2:41} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Heidi"} | {1:" ", 2:"Heidi"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
+| 0 |
s3://.../iceberg_table/data/id=10/00000-16-79ef2fd7-9997-47eb-a91a-9f7af8201315-0-00010.parquet
| PARQUET | 0 | {"id":10} | 1 | 618 |
{1:36, 2:40} | {1:1, 2:1} | {1:0, 2:0} | {} | {1:" ",
2:"Judy"} | {1:" ", 2:"Judy"} | NULL | [4] | NULL
| 0 | NULL | NULL | NULL |
NULL | {"id":{"column_size":36, "value_count":1, "null_value_ [...]
++---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+---------+-----------+--------------+--------------------+--------------+--------------+-------------------+------------------+-----------------------+-----------------------+--------------+---------------+--------------+---------------+--------------+----------------------+----------------+-----------
[...]
+```
+
+类似于 `files` 系统表，还可以通过以下系统表查询特定类型的文件信息：
+
+```sql
+-- 查询当前快照的数据文件
+SELECT * FROM iceberg_table$data_files;
+
+-- 查询当前快照的删除文件
+SELECT * FROM iceberg_table$delete_files;
+
+-- 查询所有快照的所有文件（包括数据和删除文件）
+SELECT * FROM iceberg_table$all_files;
+
+-- 查询所有快照的数据文件
+SELECT * FROM iceberg_table$all_data_files;
+
+-- 查询所有快照的删除文件
+SELECT * FROM iceberg_table$all_delete_files;
+```
+
+这些表的结果格式与 `files` 系统表相似，但分别专注于数据文件或删除文件。`all_` 前缀的系统表包含了所有快照中的文件，而不仅仅是当前快照的文件。
+
+注意：当表中不存在特定类型的文件时（例如，表中没有删除文件时查询 `delete_files`），查询结果可能为空。
+
+### history
+
+显示表的所有历史记录：
+
+```sql
+SELECT * FROM iceberg_table$history;
+```
+
+结果：
+
+```text
++----------------------------+---------------------+---------------------+---------------------+
+| made_current_at | snapshot_id | parent_id |
is_current_ancestor |
++----------------------------+---------------------+---------------------+---------------------+
+| 2025-06-12 22:29:16.357000 | 1851184769713369003 | NULL |
1 |
+| 2025-06-12 22:29:39.922000 | 4890031351138056789 | 1851184769713369003 |
1 |
++----------------------------+---------------------+---------------------+---------------------+
+```
+
+### manifests
+
+显示表的当前 manifest 文件信息：
+
+```sql
+SELECT * FROM iceberg_table$manifests;
+```
+
+结果：
+
+```text
++---------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------+-----------------------------+----------------------------+--------------------------------------------------------------------------------+
+| content | path
|
length | partition_spec_id | added_snapshot_id | added_data_files_count |
existing_data_files_count | deleted_data_files_count | added_delete_files_count
| existing_delete_files_count | deleted_delete_files_count |
partition_summaries |
++---------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------+-----------------------------+----------------------------+--------------------------------------------------------------------------------+
+| 0 |
s3://.../iceberg_table/metadata/3194eb8b-5ea4-4cbe-95ba-073229458e7b-m0.avro |
7138 | 0 | 4890031351138056789 | 0 |
5 | 5 | 0 |
0 | 0 | [{"contains_null":0,
"contains_nan":0, "lower_bound":"1", "upper_bound":"10"}] |
++---------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------+-----------------------------+----------------------------+--------------------------------------------------------------------------------+
+```
+
+### metadata_log_entries
+
+显示表的元数据日志条目：
+
+```sql
+SELECT * FROM iceberg_table$metadata_log_entries;
+```
+
+结果：
+
+```text
++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+------------------+------------------------+
+| timestamp | file
| latest_snapshot_id | latest_schema_id |
latest_sequence_number |
++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+------------------+------------------------+
+| 2025-06-12 22:29:06.948000 |
s3://.../iceberg_table/metadata/00000-e373aa16-15f1-4e69-ae7d-5ed64199cf9a.metadata.json
| NULL | NULL | NULL |
+| 2025-06-12 22:29:16.357000 |
s3://.../iceberg_table/metadata/00001-bbc8e244-e41c-4958-92f4-63b8c3ee1196.metadata.json
| 1851184769713369003 | 0 | 1 |
+| 2025-06-12 22:29:39.922000 |
s3://.../iceberg_table/metadata/00002-7dc00d6a-6269-4200-9d28-5f8c1c6b9f99.metadata.json
| 4890031351138056789 | 0 | 2 |
++----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+------------------+------------------------+
+```
+
+### partitions
+
+显示表的当前分区信息：
+
+```sql
+SELECT * FROM iceberg_table$partitions;
+```
+
+结果：
+
+```text
++-----------+---------+--------------+------------+-------------------------------+------------------------------+----------------------------+------------------------------+----------------------------+----------------------------+--------------------------+
+| partition | spec_id | record_count | file_count |
total_data_file_size_in_bytes | position_delete_record_count |
position_delete_file_count | equality_delete_record_count |
equality_delete_file_count | last_updated_at |
last_updated_snapshot_id |
++-----------+---------+--------------+------------+-------------------------------+------------------------------+----------------------------+------------------------------+----------------------------+----------------------------+--------------------------+
+| {"id":8} | 0 | 1 | 1 |
625 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
+| {"id":6} | 0 | 1 | 1 |
625 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
+| {"id":10} | 0 | 1 | 1 |
618 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
+| {"id":4} | 0 | 1 | 1 |
618 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
+| {"id":2} | 0 | 1 | 1 |
611 | 0 | 0 |
0 | 0 | 2025-06-12 22:29:16.357000 |
1851184769713369003 |
++-----------+---------+--------------+------------+-------------------------------+------------------------------+----------------------------+------------------------------+----------------------------+----------------------------+--------------------------+
+```
+
+注意：
+
+1. 对于非分区表，`partitions` 表将不包含 `partition` 和 `spec_id` 字段。
+2. `partitions`
表显示当前快照中包含数据文件或删除文件的分区。但是，删除文件未应用，因此在某些情况下，即使分区的所有数据行都已被删除文件标记为已删除，分区仍可能显示。
+
+### refs
+
+显示表的所有已知快照引用（分支和标签）：
+
+```sql
+SELECT * FROM iceberg_table$refs;
+```
+
+结果：
+
+```text
++------+--------+---------------------+-------------------------+-----------------------+------------------------+
+| name | type | snapshot_id | max_reference_age_in_ms |
min_snapshots_to_keep | max_snapshot_age_in_ms |
++------+--------+---------------------+-------------------------+-----------------------+------------------------+
+| main | BRANCH | 4890031351138056789 | NULL |
NULL | NULL |
++------+--------+---------------------+-------------------------+-----------------------+------------------------+
+```
+
+### snapshots
+
+显示表的所有有效快照：
+
+```sql
+SELECT * FROM iceberg_table$snapshots;
+```
+
+结果：
+
+```text
++----------------------------+---------------------+---------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| committed_at | snapshot_id | parent_id |
operation | manifest_list
| summary
[...]
++----------------------------+---------------------+---------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| 2025-06-12 22:29:16.357000 | 1851184769713369003 | NULL |
append |
s3://.../iceberg_table/metadata/snap-1851184769713369003-1-82059f57-821a-4983-b083-002cc2cde313.avro
| {"spark.app.id":"application_1738810850199_0472", "added-data-files":"10",
"added-records":"10", "added-files-size":"6200",
"changed-partition-count":"10", "total-records":"10",
"total-files-size":"6200", "total-data-files":"10", "total-delete-files":"0",
"total-position-deletes":"0", "total-equality- [...]
+| 2025-06-12 22:29:39.922000 | 4890031351138056789 | 1851184769713369003 |
overwrite |
s3://.../iceberg_table/metadata/snap-4890031351138056789-1-3194eb8b-5ea4-4cbe-95ba-073229458e7b.avro
| {"spark.app.id":"application_1738810850199_0472", "deleted-data-files":"5",
"deleted-records":"5", "removed-files-size":"3103",
"changed-partition-count":"5", "total-records":"5", "total-files-size":"3097",
"total-data-files":"5", "total-delete-files":"0", "total-position-deletes":"0",
"total-equality [...]
++----------------------------+---------------------+---------------------+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+```
+
## 写入操作
### INSERT INTO
@@ -550,44 +756,6 @@ DROP DATABASE [IF EXISTS] iceberg.iceberg_db;
* 对象存储
-## 系统表
-
-Doris 支持以下 Iceberg 系统表，用于查看 Iceberg 表的各类元数据信息。
-
-### snapshots
-
-查询指定 Iceberg 表的 Snapshot 信息。
-
-```sql
-SELECT * FROM iceberg_meta(
- 'table' = 'iceberg_ctl.iceberg_db.iceberg_tbl',
- 'query_type' = 'snapshots'
-)
-```
-
-或（3.1.x 版本支持）：
-
-```
-SELECT * FROM iceberg_ctl.iceberg_db.iceberg_tbl$snapshost;
-```
-
-```
-*************************** 1. row ***************************
- committed_at: 2024-11-28 11:07:29
- snapshot_id: 8903826400153112036
- parent_id: -1
- operation: append
-manifest_list:
oss://path/to/metadata/snap-8903826400153112036-1-3835e66d-9a18-4cb0-b9b0-9ec80527ad8d.avro
- summary:
{"added-data-files":"2","added-records":"3","added-files-size":"2742","changed-partition-count":"2","total-records":"3","total-files-size":"2742","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}
-*************************** 2. row ***************************
- committed_at: 2024-11-28 11:10:11
- snapshot_id: 6099853805930794326
- parent_id: 8903826400153112036
- operation: append
-manifest_list:
oss://path/to/metadata/snap-6099853805930794326-1-dd46a1bd-219b-4fb0-bb46-ac441d8b3105.avro
- summary:
{"added-data-files":"1","added-records":"1","added-files-size":"1367","changed-partition-count":"1","total-records":"4","total-files-size":"4109","total-data-files":"3","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}
-```
-
## 附录
### 版本更新记录
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.md
index 1fac333c52f..af7070883f6 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.md
@@ -221,5 +221,48 @@ SELECT * FROM paimon_tbl LIMIT 10;
SELECT * FROM paimon_ctl.paimon_db.paimon_tbl LIMIT 10;
```
+### 增量查询
+
+> 该功能自 3.1.0 版本支持
+
+支持类似 Flink 针对 Paimon 的 [Batch
Incremental](https://paimon.apache.org/docs/master/flink/sql-query/#batch-incremental)
查询。
+
+支持查询指定的快照或时间戳区间内的增量数据。区间为左闭右开区间。
+
+```sql
+-- read from snapshot 2
+SELECT * FROM paimon_table@incr('startSnapshotId'='2');
+
+-- between snapshots [0, 5)
+SELECT * FROM paimon_table@incr('startSnapshotId'='0', 'endSnapshotId'='5');
+
+-- between snapshots [0, 5) with specified scan mode
+SELECT * FROM paimon_table@incr('startSnapshotId'='0', 'endSnapshotId'='5',
'incrementalBetweenScanMode'='diff');
+
+-- read from start timestamp
+SELECT * FROM paimon_table@incr('startTimestamp'='1750844949');
+
+-- read between timestamp
+SELECT * FROM paimon_table@incr('startTimestamp'='1750844949',
'endTimestamp'='1750944949');
+```
+
+参数说明：
+
+| 参数 | 说明 | 示例 |
+| --- | --- | -- |
+| `startSnapshotId` | 起始快照 ID，必须大于 0 | `'startSnapshotId'='3'` |
+| `endSnapshotId` | 结束快照 ID，必须大于 `startSnapshotId`。可选，如不指定，则表示从
`startSnapshotId` 开始读取到最新的快照 | `'endSnapshotId'='10'` |
+| `incrementalBetweenScanMode` | 指定增量读取的模式，默认 `auto`，支持 `delta`， `changelog` 和
`diff` | `'incrementalBetweenScanMode'='delta'` |
+| `startTimestamp` | 起始快照时间，必须大于等于 0 | `'startTimestamp'='1750844949'` |
+| `endTimestamp` | 结束快照时间，必须大于 `startTimestamp`。可选，如不指定，则表示从 `startTimestamp`
开始读取到最新的快照 | `'endTimestamp'='1750944949'` |
+
+> `startSnapshotId` 和 `endSnapshotId` 会组成 Paimon 参数
`'incremental-between'='3,10'`
+
+> `startTimestamp` 和 `endTimestamp` 会组成 Paimon 参数
`'incremental-between-timestamp'='1750844949,1750944949'`
+
+> `incrementalBetweenScanMode` 对应 Paimon 参数 `incremental-between-scan-mode`。
+
+可参阅 [Paimon
文档](https://paimon.apache.org/docs/master/maintenance/configurations/)
进一步了解这些参数。
+
## 附录
diff --git a/versioned_docs/version-2.1/releasenotes/v3.0/release-3.0.6.md
b/versioned_docs/version-2.1/releasenotes/v3.0/release-3.0.6.md
index 89c57af3e9e..940a4215ed4 100644
--- a/versioned_docs/version-2.1/releasenotes/v3.0/release-3.0.6.md
+++ b/versioned_docs/version-2.1/releasenotes/v3.0/release-3.0.6.md
@@ -37,7 +37,7 @@ Dear community members, the Apache Doris 3.0.6 version was
officially released o
- `xpath_string`
- For detailed information, please refer to
[documentation](https://doris.apache.org/docs/3.0/sql-manual/sql-functions/scalar-functions/string-functions/xpath-string)
- `str_to_map`
-- For detailed information, please refer to
[documentation](https://doris.apache.org/docs/3.0/sql-manual/sql-functions/scalar-functions/map-functions/str-to-map)
+ - For detailed information, please refer to
[documentation](https://doris.apache.org/docs/3.0/sql-manual/sql-functions/scalar-functions/map-functions/str-to-map)
- `months_between`
- For detailed information, please refer to
[documentation](https://doris.apache.org/docs/3.0/sql-manual/sql-functions/scalar-functions/date-time-functions/months-between)
- `next_day`
@@ -182,4 +182,4 @@ Dear community members, the Apache Doris 3.0.6 version was
officially released o
- Fixed audit log loss in high-concurrency scenarios
[#50357](https://github.com/apache/doris/pull/50357)
- Prevented metadata replay failures during dynamic partition creation
[#49569](https://github.com/apache/doris/pull/49569)
- Solved Global UDF loss after restart
[#50279](https://github.com/apache/doris/pull/50279)
-- Aligned View metadata return format with MySQL
[#51058](https://github.com/apache/doris/pull/51058)
\ No newline at end of file
+- Aligned View metadata return format with MySQL
[#51058](https://github.com/apache/doris/pull/51058)
diff --git a/versioned_docs/version-3.0/releasenotes/v3.0/release-3.0.6.md
b/versioned_docs/version-3.0/releasenotes/v3.0/release-3.0.6.md
index 89c57af3e9e..940a4215ed4 100644
--- a/versioned_docs/version-3.0/releasenotes/v3.0/release-3.0.6.md
+++ b/versioned_docs/version-3.0/releasenotes/v3.0/release-3.0.6.md
@@ -37,7 +37,7 @@ Dear community members, the Apache Doris 3.0.6 version was
officially released o
- `xpath_string`
- For detailed information, please refer to
[documentation](https://doris.apache.org/docs/3.0/sql-manual/sql-functions/scalar-functions/string-functions/xpath-string)
- `str_to_map`
-- For detailed information, please refer to
[documentation](https://doris.apache.org/docs/3.0/sql-manual/sql-functions/scalar-functions/map-functions/str-to-map)
+ - For detailed information, please refer to
[documentation](https://doris.apache.org/docs/3.0/sql-manual/sql-functions/scalar-functions/map-functions/str-to-map)
- `months_between`
- For detailed information, please refer to
[documentation](https://doris.apache.org/docs/3.0/sql-manual/sql-functions/scalar-functions/date-time-functions/months-between)
- `next_day`
@@ -182,4 +182,4 @@ Dear community members, the Apache Doris 3.0.6 version was
officially released o
- Fixed audit log loss in high-concurrency scenarios
[#50357](https://github.com/apache/doris/pull/50357)
- Prevented metadata replay failures during dynamic partition creation
[#49569](https://github.com/apache/doris/pull/49569)
- Solved Global UDF loss after restart
[#50279](https://github.com/apache/doris/pull/50279)
-- Aligned View metadata return format with MySQL
[#51058](https://github.com/apache/doris/pull/51058)
\ No newline at end of file
+- Aligned View metadata return format with MySQL
[#51058](https://github.com/apache/doris/pull/51058)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
