This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 8f5ce6b844 [docs] Organize maintenance guides by task (#9713)
8f5ce6b844 is described below
commit 8f5ce6b84496492bacbc3fc44fe8270a2107af2d
Author: Jingsong Lee <[email protected]>
AuthorDate: Thu Sep 10 13:32:49 2026 +0800
[docs] Organize maintenance guides by task (#9713)
---
docs/docs/maintenance/dedicated-compaction.mdx | 18 ++-
docs/docs/maintenance/index.md | 70 ++++++++++
docs/docs/maintenance/manage-branches.mdx | 2 +-
docs/docs/maintenance/manage-partitions.md | 3 +
docs/docs/maintenance/manage-snapshots.mdx | 9 +-
docs/docs/maintenance/manage-tags.mdx | 6 +
docs/docs/maintenance/metrics.md | 173 +++++++++++++------------
docs/docs/maintenance/write-performance.md | 8 +-
docs/sidebars.js | 48 +++++--
9 files changed, 241 insertions(+), 96 deletions(-)
diff --git a/docs/docs/maintenance/dedicated-compaction.mdx
b/docs/docs/maintenance/dedicated-compaction.mdx
index 153118053f..d1fee22e9a 100644
--- a/docs/docs/maintenance/dedicated-compaction.mdx
+++ b/docs/docs/maintenance/dedicated-compaction.mdx
@@ -29,6 +29,22 @@ under the License.
# Dedicated Compaction
+Use this guide to run compaction separately from writers. For compaction
behavior and tuning, see
+[Compaction](../primary-key-table/compaction). To inspect compaction activity,
see
+[Compaction Metrics](./metrics#compaction-metrics).
+
+Start with the job scope, then choose any target filters or special compaction
strategy. Each section describes
+its supported table types, execution modes, and engine commands.
+
+| Task | Guide |
+| --- | --- |
+| Compact one table | [Dedicated Compaction Job](#dedicated-compaction-job) |
+| Compact tables across databases | [Database Compaction
Job](#database-compaction-job) |
+| Select buckets in a fixed-bucket table | [Compact Specified
Buckets](#compact-specified-buckets) |
+| Compact partitions that have been idle | [Historical Partition
Compact](#historical-partition-compact) |
+| Avoid compacting expired partitions | [Skip Expired
Partitions](#skip-expired-partitions-in-compaction-job) |
+| Sort data during compaction to improve queries | [Sort
Compact](#sort-compact) |
+
Paimon's snapshot management supports writing with multiple writers.
:::info
@@ -585,4 +601,4 @@ CALL sys.compact_database(
</TabItem>
-</Tabs>
\ No newline at end of file
+</Tabs>
diff --git a/docs/docs/maintenance/index.md b/docs/docs/maintenance/index.md
index 21f5d313d4..7c776a2bc6 100644
--- a/docs/docs/maintenance/index.md
+++ b/docs/docs/maintenance/index.md
@@ -21,3 +21,73 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
+
+# Maintenance
+
+Use these guides to manage table history, clean up stored data, run compaction
jobs, and tune and monitor workloads.
+Storage setup and configuration references are also collected here.
+
+## Choose a Task
+
+| Task | Start here |
+| --- | --- |
+| Reduce storage usage | [Choose a cleanup operation](#clean-up-stored-data) |
+| Keep daily versions available for queries | [Automatic tag creation and
retention](./manage-tags#automatic-creation) |
+| Recover from an incorrect write | [Choose a recovery
operation](#recover-table-data) |
+| Validate changes on a separate branch | [Manage Branches](./manage-branches)
|
+| Run compaction separately from writers | [Dedicated
Compaction](./dedicated-compaction) |
+| Investigate slow writes or memory pressure | [Choose
metrics](./metrics#choose-metrics-for-a-task), then see [Write
Performance](./write-performance) |
+| Connect to HDFS or an object store | [Filesystems](./filesystems) |
+| Look up a table, catalog, or connector option |
[Configurations](./configurations) |
+
+## Clean Up Stored Data
+
+Choose the operation based on what you want to remove:
+
+- [Expire partitions](./manage-partitions#expiring-partitions) to remove old
partitions from the latest table state.
+ Physical file deletion depends on snapshot expiration.
+- [Expire snapshots](./manage-snapshots#expire-snapshots) to limit retained
history and remove files that are no longer
+ needed. Review the retention requirements of batch queries and streaming
readers before changing the policy.
+- [Remove orphan files](./manage-snapshots#remove-orphan-files) to clean up
files that are no longer referenced.
+ Follow the cleanup guide's age cutoff to account for files being added by
active writers.
+
+[Tags](./manage-tags) preserve historical data independently of snapshot
expiration. Review tag retention as part of
+your storage policy. To remove empty directories left after file deletion, see
the
+`snapshot.clean-empty-directories` option in [Expire
Snapshots](./manage-snapshots#expire-snapshots).
+
+## Recover Table Data
+
+Choose a recovery mode, then follow the instructions for a snapshot ID or a
tag.
+The linked guides include the supported engine commands and operation-specific
limitations.
+
+| Recovery mode | Effect on table history | Instructions |
+| --- | --- | --- |
+| Rollback | Restore the target state and remove snapshots and tags after the
target. | [Snapshot](./manage-snapshots#rollback-to-snapshot) or
[tag](./manage-tags#rollback-to-tag) |
+| Rollback as latest | Restore the target state as a new latest snapshot,
preserving later snapshots and tags. |
[Snapshot](./manage-snapshots#rollback-to-snapshot-as-latest) or
[tag](./manage-tags#rollback-to-tag-as-latest) |
+
+For a workflow that validates corrected data on a separate branch before
updating the main branch, see
+[Manage Branches](./manage-branches) and [Fast
Forward](./manage-branches#fast-forward).
+
+## Browse by Topic
+
+### Data Lifecycle & Versioning
+
+- [Manage Snapshots](./manage-snapshots): retain or expire table history, roll
back, and remove orphan files.
+- [Manage Tags](./manage-tags): preserve named versions, automate tag
retention, and restore tagged data.
+- [Manage Branches](./manage-branches): create isolated branches, validate
changes, and fast-forward the main branch.
+- [Manage Partitions](./manage-partitions): expire partitions and mark
partitions ready for downstream consumers.
+
+### Compaction & Data Layout
+
+- [Dedicated Compaction](./dedicated-compaction): run table or database
compaction jobs and select compaction targets.
+- [Rescale Bucket](./rescale-bucket): change bucket counts and reorganize
existing data.
+
+### Performance & Monitoring
+
+- [Write Performance](./write-performance): tune parallelism, buffering, file
formats, and memory usage.
+- [Metrics](./metrics): access metrics through Flink and choose metrics for
investigation.
+
+### Storage & Configuration
+
+- [Filesystems](./filesystems): install filesystem dependencies and configure
storage access.
+- [Configurations](./configurations): look up table, catalog, connector, and
file-format options.
diff --git a/docs/docs/maintenance/manage-branches.mdx
b/docs/docs/maintenance/manage-branches.mdx
index 3fb79d950b..8b96d10ad5 100644
--- a/docs/docs/maintenance/manage-branches.mdx
+++ b/docs/docs/maintenance/manage-branches.mdx
@@ -93,7 +93,7 @@ You can delete branch by its name.
:::warning
-__Note:__ The `Delete Branches` operation only deletes the metadata file. If
you want to clear the data written during the branch, use
[remove_orphan_files](../flink/procedures)
+__Note:__ The `Delete Branches` operation only deletes the metadata file. If
you want to clear the data written during the branch, see [Remove Orphan
Files](./manage-snapshots#remove-orphan-files).
:::
diff --git a/docs/docs/maintenance/manage-partitions.md
b/docs/docs/maintenance/manage-partitions.md
index b5a471a6b9..50870c1e9a 100644
--- a/docs/docs/maintenance/manage-partitions.md
+++ b/docs/docs/maintenance/manage-partitions.md
@@ -26,6 +26,9 @@ under the License.
Paimon provides multiple ways to manage partitions, including expire
historical partitions by different strategies or
mark a partition done to notify the downstream application that the partition
has finished writing.
+For how partition expiration relates to snapshot retention and file cleanup,
see
+[Clean Up Stored Data](./#clean-up-stored-data).
+
## Expiring Partitions
You can set `partition.expiration-time` when creating a partitioned table.
Paimon streaming sink will periodically check
diff --git a/docs/docs/maintenance/manage-snapshots.mdx
b/docs/docs/maintenance/manage-snapshots.mdx
index 69b972cb26..205894d7b7 100644
--- a/docs/docs/maintenance/manage-snapshots.mdx
+++ b/docs/docs/maintenance/manage-snapshots.mdx
@@ -27,7 +27,10 @@ under the License.
# Manage Snapshots
-This section will describe the management and behavior related to snapshots.
+Use this guide to [expire snapshots](#expire-snapshots), [roll back to a
snapshot](#rollback-to-snapshot),
+or [remove orphan files](#remove-orphan-files). For cleanup across partitions,
snapshots, and tags, see
+[Clean Up Stored Data](./#clean-up-stored-data). To compare snapshot and tag
recovery modes, see
+[Recover Table Data](./#recover-table-data).
## Expire Snapshots
@@ -294,6 +297,10 @@ CALL sys.expire_snapshots(table =>
'database_name.table_name', retain_max => 10,
Rollback a table to a specific snapshot ID.
+To restore the target state as a new latest snapshot while preserving later
snapshots and tags, see
+[Rollback to Snapshot as Latest](#rollback-to-snapshot-as-latest). To restore
a named tag, see
+[Rollback to Tag](./manage-tags#rollback-to-tag).
+
<Tabs groupId="rollback-to">
<TabItem value="flink-sql" label="Flink SQL">
diff --git a/docs/docs/maintenance/manage-tags.mdx
b/docs/docs/maintenance/manage-tags.mdx
index f53538c3a8..490f264a41 100644
--- a/docs/docs/maintenance/manage-tags.mdx
+++ b/docs/docs/maintenance/manage-tags.mdx
@@ -34,6 +34,9 @@ data files, and the historical data of expired snapshots
cannot be queried anymo
To solve this problem, you can create a tag based on a snapshot. The tag will
maintain the manifests and data files of the
snapshot. A typical usage is creating tags daily, then you can maintain the
historical data of each day for batch reading.
+For the relationship between tag retention and other cleanup operations, see
[Clean Up Stored Data](./#clean-up-stored-data).
+To compare snapshot and tag recovery modes, see [Recover Table
Data](./#recover-table-data).
+
## Automatic Creation
Paimon supports automatic creation of tags in writing job.
@@ -235,6 +238,9 @@ CALL sys.delete_tag(table => 'test.t', tag => 'test_tag');
Rollback table to a specific tag. All snapshots and tags whose snapshot id is
larger than the tag will be deleted (and
the data will be deleted too).
+To preserve later snapshots and tags, see [Rollback to Tag as
Latest](#rollback-to-tag-as-latest).
+To restore a snapshot by ID, see [Rollback to
Snapshot](./manage-snapshots#rollback-to-snapshot).
+
<Tabs groupId="rollback-to">
<TabItem value="flink-sql" label="Flink SQL">
diff --git a/docs/docs/maintenance/metrics.md b/docs/docs/maintenance/metrics.md
index e484257e70..efd78e520d 100644
--- a/docs/docs/maintenance/metrics.md
+++ b/docs/docs/maintenance/metrics.md
@@ -35,6 +35,95 @@ There are three types of metrics provided in the Paimon
metric system, `Gauge`,
Paimon has supported built-in metrics to measure operations of **commits**,
**scans**, **writes**, **compactions** and **lookups**, which can be bridged to
any computing engine that supports, like Flink, Spark etc.
+## Bridging To Flink
+
+Paimon has implemented bridging metrics to Flink's metrics system, which can
be reported by Flink, and the lifecycle of metric groups are managed by Flink.
+
+Please join the `<scope>.<infix>.<metric_name>` to get the complete metric
identifier when using Flink to access Paimon, `metric_name` can be got from
[Metric List](./metrics#metrics-list).
+
+For example, the identifier of metric `lastPartitionsWritten` for table
`word_count` in Flink job named `insert_word_count` is:
+
+`localhost.taskmanager.localhost:60340-775a20.insert_word_count.Global
Committer : word_count.0.paimon.table.word_count.commit.lastPartitionsWritten`.
+
+From Flink Web-UI, go to the committer operator's metrics, it's shown as:
+
+`0.Global_Committer___word_count.paimon.table.word_count.commit.lastPartitionsWritten`.
+
+:::info
+
+1. Please refer to [System
Scope](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#system-scope)
to understand Flink `scope`
+2. Scan metrics are only supported by Flink versions >= 1.18
+
+:::
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 130pt"></th>
+ <th class="text-left" style="width: 280pt">Scope</th>
+ <th class="text-left" style="width: 250pt">Infix</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Scan Metrics</td>
+ <td><host>.jobmanager.<job_name></td>
+ <td><source_operator_name>.coordinator.
enumerator.paimon.table.<table_name>.scan</td>
+ </tr>
+ <tr>
+ <td>Lookup Metrics</td>
+
<td><host>.taskmanager.<tm_id>.<job_name>.<lookup_operator_name>.<subtask_index></td>
+ <td>paimon.table.<table_name>.lookup</td>
+ </tr>
+ <tr>
+ <td>Commit Metrics</td>
+
<td><host>.taskmanager.<tm_id>.<job_name>.<committer_operator_name>.<subtask_index></td>
+ <td>paimon.table.<table_name>.commit</td>
+ </tr>
+ <tr>
+ <td>Write Metrics</td>
+
<td><host>.taskmanager.<tm_id>.<job_name>.<writer_operator_name>.<subtask_index></td>
+
<td>paimon.table.<table_name>.partition.<partition_string>.bucket.<bucket_index>.writer</td>
+ </tr>
+ <tr>
+ <td>Write Buffer Metrics</td>
+
<td><host>.taskmanager.<tm_id>.<job_name>.<writer_operator_name>.<subtask_index></td>
+ <td>paimon.table.<table_name>.writeBuffer</td>
+ </tr>
+ <tr>
+ <td>Blob Fetch Metrics</td>
+
<td><host>.taskmanager.<tm_id>.<job_name>.<writer_operator_name>.<subtask_index></td>
+ <td>paimon.table.<table_name>.blobFetch</td>
+ </tr>
+ <tr>
+ <td>Compaction Metrics</td>
+
<td><host>.taskmanager.<tm_id>.<job_name>.<writer_operator_name>.<subtask_index></td>
+
<td>paimon.table.<table_name>.partition.<partition_string>.bucket.<bucket_index>.compaction</td>
+ </tr>
+ <tr>
+ <td>Flink Source Metrics</td>
+
<td><host>.taskmanager.<tm_id>.<job_name>.<source_operator_name>.<subtask_index></td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>Flink Sink Metrics</td>
+
<td><host>.taskmanager.<tm_id>.<job_name>.<committer_operator_name>.<subtask_index></td>
+ <td>-</td>
+ </tr>
+ </tbody>
+</table>
+
+## Choose Metrics for a Task
+
+Use these groups to identify the operation to investigate. For tuning options,
follow the related guide.
+
+| Task | Metrics to inspect | Related guide |
+| --- | --- | --- |
+| Investigate slow commits | [Commit duration and attempts](#commit-metrics) |
[Commit Memory](./write-performance#commit-memory) |
+| Investigate writer memory pressure | [Write buffer
usage](#write-buffer-metrics) | [Write
Memory](./write-performance#write-memory) |
+| Check whether compaction is keeping up | [Level 0 file counts and compaction
activity](#compaction-metrics) | [Compaction](../primary-key-table/compaction)
and [Dedicated Compaction](./dedicated-compaction) |
+| Track Flink source latency or sink throughput | [Flink Connector Standard
Metrics](#flink-connector-standard-metrics) | [Write
Performance](./write-performance) |
+
## Metrics List
Below is lists of Paimon built-in metrics. They are summarized into types of
scan metrics, commit metrics, write metrics, write buffer metrics, blob fetch
metrics, compaction metrics and lookup metrics.
@@ -445,89 +534,11 @@ Lookup metrics are available for local partial lookup.
They are reported at look
</tbody>
</table>
-## Bridging To Flink
-
-Paimon has implemented bridging metrics to Flink's metrics system, which can
be reported by Flink, and the lifecycle of metric groups are managed by Flink.
-
-Please join the `<scope>.<infix>.<metric_name>` to get the complete metric
identifier when using Flink to access Paimon, `metric_name` can be got from
[Metric List](./metrics#metrics-list).
-
-For example, the identifier of metric `lastPartitionsWritten` for table
`word_count` in Flink job named `insert_word_count` is:
-
-`localhost.taskmanager.localhost:60340-775a20.insert_word_count.Global
Committer : word_count.0.paimon.table.word_count.commit.lastPartitionsWritten`.
-
-From Flink Web-UI, go to the committer operator's metrics, it's shown as:
-
-`0.Global_Committer___word_count.paimon.table.word_count.commit.lastPartitionsWritten`.
-
-:::info
-
-1. Please refer to [System
Scope](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#system-scope)
to understand Flink `scope`
-2. Scan metrics are only supported by Flink versions >= 1.18
-
-:::
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 130pt"></th>
- <th class="text-left" style="width: 280pt">Scope</th>
- <th class="text-left" style="width: 250pt">Infix</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>Scan Metrics</td>
- <td><host>.jobmanager.<job_name></td>
- <td><source_operator_name>.coordinator.
enumerator.paimon.table.<table_name>.scan</td>
- </tr>
- <tr>
- <td>Lookup Metrics</td>
-
<td><host>.taskmanager.<tm_id>.<job_name>.<lookup_operator_name>.<subtask_index></td>
- <td>paimon.table.<table_name>.lookup</td>
- </tr>
- <tr>
- <td>Commit Metrics</td>
-
<td><host>.taskmanager.<tm_id>.<job_name>.<committer_operator_name>.<subtask_index></td>
- <td>paimon.table.<table_name>.commit</td>
- </tr>
- <tr>
- <td>Write Metrics</td>
-
<td><host>.taskmanager.<tm_id>.<job_name>.<writer_operator_name>.<subtask_index></td>
-
<td>paimon.table.<table_name>.partition.<partition_string>.bucket.<bucket_index>.writer</td>
- </tr>
- <tr>
- <td>Write Buffer Metrics</td>
-
<td><host>.taskmanager.<tm_id>.<job_name>.<writer_operator_name>.<subtask_index></td>
- <td>paimon.table.<table_name>.writeBuffer</td>
- </tr>
- <tr>
- <td>Blob Fetch Metrics</td>
-
<td><host>.taskmanager.<tm_id>.<job_name>.<writer_operator_name>.<subtask_index></td>
- <td>paimon.table.<table_name>.blobFetch</td>
- </tr>
- <tr>
- <td>Compaction Metrics</td>
-
<td><host>.taskmanager.<tm_id>.<job_name>.<writer_operator_name>.<subtask_index></td>
-
<td>paimon.table.<table_name>.partition.<partition_string>.bucket.<bucket_index>.compaction</td>
- </tr>
- <tr>
- <td>Flink Source Metrics</td>
-
<td><host>.taskmanager.<tm_id>.<job_name>.<source_operator_name>.<subtask_index></td>
- <td>-</td>
- </tr>
- <tr>
- <td>Flink Sink Metrics</td>
-
<td><host>.taskmanager.<tm_id>.<job_name>.<committer_operator_name>.<subtask_index></td>
- <td>-</td>
- </tr>
- </tbody>
-</table>
-
-### Flink Connector Standard Metrics
+## Flink Connector Standard Metrics
When using Flink to read and write, Paimon has implemented some key standard
Flink connector metrics to measure the source latency and output of sink, see
[FLIP-33: Standardize Connector
Metrics](https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics).
Flink source / sink metrics implemented are listed here.
-#### Source Metrics (Flink)
+### Source Metrics (Flink)
<table class="table table-bordered">
<thead>
@@ -566,7 +577,7 @@ Please note that if you specified `consumer-id` in your
streaming query, the lev
:::
-#### Sink Metrics (Flink)
+### Sink Metrics (Flink)
<table class="table table-bordered">
<thead>
diff --git a/docs/docs/maintenance/write-performance.md
b/docs/docs/maintenance/write-performance.md
index fb8bfa4738..44410446dc 100644
--- a/docs/docs/maintenance/write-performance.md
+++ b/docs/docs/maintenance/write-performance.md
@@ -24,6 +24,10 @@ under the License.
# Write Performance
+Start with [Metrics](./metrics#choose-metrics-for-a-task) to identify the
operation to investigate, then use the
+relevant tuning guidance below. For running compaction separately from
writers, see
+[Dedicated Compaction](./dedicated-compaction).
+
Paimon's write performance is closely related to checkpoint, so if you need
greater write throughput:
1. Flink Configuration (`'flink-conf.yaml'/'config.yaml'` or `SET` in SQL):
Increase the checkpoint interval
@@ -31,7 +35,7 @@ Paimon's write performance is closely related to checkpoint,
so if you need grea
(`'execution.checkpointing.max-concurrent-checkpoints'`), or just use batch
mode.
2. Increase `write-buffer-size`.
3. Enable `write-buffer-spillable`.
-4. Rescale bucket number if you are using Fixed-Bucket mode.
+4. [Rescale bucket number](./rescale-bucket) if you are using Fixed-Bucket
mode.
Option `'changelog-producer' = 'lookup' or 'full-compaction'`, and option
`'full-compaction.delta-commits'` have a
large impact on write performance, if it is a snapshot / full synchronization
phase you can unset these options and
@@ -175,4 +179,4 @@ You can use fine-grained-resource-management of Flink to
increase committer heap
1. Configure Flink Configuration
`cluster.fine-grained-resource-management.enabled: true`. (This is default
after Flink 1.18)
2. Configure Paimon Table Options: `sink.committer-memory`, for example 300
MB, depends on your `TaskManager`.
(`sink.committer-cpu` is also supported)
-3. If you use Flink batch job write data into Paimon or run dedicated
compaction, Configure Flink Configuration
`fine-grained.shuffle-mode.all-blocking: true`.
\ No newline at end of file
+3. If you use Flink batch job write data into Paimon or run dedicated
compaction, Configure Flink Configuration
`fine-grained.shuffle-mode.all-blocking: true`.
diff --git a/docs/sidebars.js b/docs/sidebars.js
index d6f18ae5a8..a4aa47d801 100644
--- a/docs/sidebars.js
+++ b/docs/sidebars.js
@@ -248,16 +248,44 @@ const sidebars = {
"id": "maintenance/index"
},
"items": [
- "maintenance/filesystems",
- "maintenance/write-performance",
- "maintenance/dedicated-compaction",
- "maintenance/manage-snapshots",
- "maintenance/rescale-bucket",
- "maintenance/manage-tags",
- "maintenance/metrics",
- "maintenance/manage-branches",
- "maintenance/manage-partitions",
- "maintenance/configurations"
+ {
+ type: "category",
+ "label": "Data Lifecycle & Versioning",
+ "collapsed": true,
+ "items": [
+ "maintenance/manage-snapshots",
+ "maintenance/manage-tags",
+ "maintenance/manage-branches",
+ "maintenance/manage-partitions"
+ ]
+ },
+ {
+ type: "category",
+ "label": "Compaction & Data Layout",
+ "collapsed": true,
+ "items": [
+ "maintenance/dedicated-compaction",
+ "maintenance/rescale-bucket"
+ ]
+ },
+ {
+ type: "category",
+ "label": "Performance & Monitoring",
+ "collapsed": true,
+ "items": [
+ "maintenance/write-performance",
+ "maintenance/metrics"
+ ]
+ },
+ {
+ type: "category",
+ "label": "Storage & Configuration",
+ "collapsed": true,
+ "items": [
+ "maintenance/filesystems",
+ "maintenance/configurations"
+ ]
+ }
]
},
{
