This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 24e25d3d38 [docs] Document bucketed append incremental clustering
(#8437)
24e25d3d38 is described below
commit 24e25d3d380721d0ea213b0ee87fe65cba402da1
Author: LsomeYeah <[email protected]>
AuthorDate: Thu Jul 2 20:00:21 2026 +0800
[docs] Document bucketed append incremental clustering (#8437)
---
docs/docs/append-table/bucketed.mdx | 9 ++--
docs/docs/append-table/incremental-clustering.mdx | 50 ++++++++++++++++++++++-
docs/docs/learn-paimon/scenario-guide.mdx | 6 ++-
3 files changed, 59 insertions(+), 6 deletions(-)
diff --git a/docs/docs/append-table/bucketed.mdx
b/docs/docs/append-table/bucketed.mdx
index eb77931e37..59493ea5ae 100644
--- a/docs/docs/append-table/bucketed.mdx
+++ b/docs/docs/append-table/bucketed.mdx
@@ -53,9 +53,12 @@ CREATE TABLE my_table (
## Data Skipping
The primary and most significant advantage of a bucketed append table is
**data skipping**. When queries contain
-equality (`=`) or `IN` filter conditions on the `bucket-key`, Paimon can
efficiently push these predicates down to
-skip irrelevant bucket files entirely. This means a large number of files that
do not match the filter are pruned
-before reading, drastically reducing I/O and accelerating queries.
+equality (`=`) or `IN` filter conditions on the complete `bucket-key`, Paimon
can efficiently push these predicates
+down to skip irrelevant bucket files entirely. This means a large number of
files that do not match the filter are
+pruned before reading, drastically reducing I/O and accelerating queries.
+
+For a composite `bucket-key`, the query predicate must cover all bucket-key
columns with finite equality or `IN`
+values to determine the target buckets.
For example, if `bucket-key` is `product_id` and you query:
diff --git a/docs/docs/append-table/incremental-clustering.mdx
b/docs/docs/append-table/incremental-clustering.mdx
index 54c146c4ba..688ee58290 100644
--- a/docs/docs/append-table/incremental-clustering.mdx
+++ b/docs/docs/append-table/incremental-clustering.mdx
@@ -45,7 +45,9 @@ Incremental Clustering supports:
- Support changing clustering keys; newly ingested data is clustered according
to the latest clustering keys.
- Provide a full mode; when selected, the entire dataset will be reclustered.
-**Only append unaware-bucket table supports Incremental Clustering.**
+Incremental Clustering is supported for append tables in both unaware-bucket
mode (`bucket = -1`) and
+bucketed mode (`bucket > 0`). For bucketed append tables, additional
requirements apply because
+clustering gives up the ordered append guarantee within buckets.
## Enable Incremental Clustering
@@ -93,6 +95,43 @@ To enable Incremental Clustering, the following
configuration needs to be set fo
</table>
+For bucketed append tables (`bucket > 0`), you must also set the following
option:
+
+<table className="table table-bordered">
+ <thead>
+ <tr>
+ <th className="text-left" style={{width: "20%"}}>Option</th>
+ <th className="text-left" style={{width: "10%"}}>Value</th>
+ <th className="text-left" style={{width: "5%"}}>Required</th>
+ <th className="text-left" style={{width: "10%"}}>Type</th>
+ <th className="text-left" style={{width: "55%"}}>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>bucket-append-ordered</h5></td>
+ <td>false</td>
+ <td style={{wordWrap: "break-word"}}>Yes</td>
+ <td>Boolean</td>
+ <td>Must be set to false for bucketed append tables with incremental
clustering.</td>
+ </tr>
+ </tbody>
+
+</table>
+
+Bucketed append tables with Incremental Clustering do not support
`deletion-vectors.enabled = true`.
+
+Example:
+
+```sql
+ALTER TABLE T SET (
+ 'bucket-append-ordered' = 'false',
+ 'clustering.incremental' = 'true',
+ 'clustering.columns' = 'event_time,user_id',
+ 'clustering.strategy' = 'zorder'
+);
+```
+
Once Incremental Clustering for a table is enabled, you can run Incremental
Clustering in batch mode periodically
to continuously optimizes data layout of the table and deliver better query
performance.
@@ -103,7 +142,7 @@ clustering and small-file merging must be performed
exclusively via Incremental
## Run Incremental Clustering
:::info
-only support running Incremental Clustering in batch mode.
+The following examples submit batch compact jobs. They are the recommended way
to run Incremental Clustering explicitly.
:::
@@ -201,6 +240,13 @@ You can use `-D execution.runtime-mode=batch` or `-yD
execution.runtime-mode=bat
</Tabs>
## Auto-Clustering For Historical Partition
+:::info
+
+Auto-clustering for historical partitions currently applies only to
unaware-bucket append tables (`bucket = -1`).
+Bucketed append clustering does not use the `clustering.history-partition.*`
table options.
+
+:::
+
While performing incremental clustering on recently active partitions, Paimon
can automatically detect historical and
inactive partitions and evaluate whether their data layout has reached an
optimal state.
For those historical partitions that have not yet achieved optimal layout,
Paimon will also perform full clustering on them
diff --git a/docs/docs/learn-paimon/scenario-guide.mdx
b/docs/docs/learn-paimon/scenario-guide.mdx
index a60dc51583..412cec1650 100644
--- a/docs/docs/learn-paimon/scenario-guide.mdx
+++ b/docs/docs/learn-paimon/scenario-guide.mdx
@@ -287,6 +287,10 @@ SELECT * FROM product_logs WHERE product_id IN (1, 2, 3);
See [Bucketed Append — Data Skipping](../append-table/bucketed#data-skipping).
+If queries also frequently filter on columns other than the `bucket-key` and
ordered streaming is not required, you can
+enable [Incremental Clustering](../append-table/incremental-clustering) on the
bucketed append table by setting
+`bucket-append-ordered = false`.
+
**Bucketed Join Bonus:** If two bucketed tables share the same `bucket-key`
and bucket count, Spark can join them
**without shuffle**, significantly accelerating batch join queries:
@@ -330,7 +334,7 @@ See [Bucketed
Streaming](../append-table/bucketed#bucketed-streaming).
| Mode | Config | Data Skipping | Bucketed Join | Ordered Streaming |
Incremental Clustering |
|---|---|---|---|---|---|
| Unaware-Bucket (default) | No bucket config | Via min-max / file index | No
| No | Yes |
-| Bucketed | `bucket = N, bucket-key = col` | **Bucket-key filter pushdown** |
Yes | Yes | No |
+| Bucketed | `bucket = N, bucket-key = col` | **Bucket-key filter pushdown** |
Yes | Yes, unless `bucket-append-ordered = false` | Yes, requires
`bucket-append-ordered = false` |
---
