This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 90a43c01f5 [docs] Reorganize Iceberg compatibility guides and add SVG
diagrams (#9732)
90a43c01f5 is described below
commit 90a43c01f5cf1dde72bc97c90fff91f70590b971
Author: Jingsong Lee <[email protected]>
AuthorDate: Fri Sep 11 13:39:36 2026 +0800
[docs] Reorganize Iceberg compatibility guides and add SVG diagrams (#9732)
---
docs/docs/iceberg/append-table.mdx | 179 +++++-----
docs/docs/iceberg/catalogs.md | 91 ++++++
docs/docs/iceberg/configurations.mdx | 35 +-
docs/docs/iceberg/{index.md => data-types.md} | 76 +----
docs/docs/iceberg/ecosystem.mdx | 141 ++++----
docs/docs/iceberg/hive-catalog.md | 183 +++++++----
docs/docs/iceberg/iceberg-tags.md | 84 +++--
docs/docs/iceberg/index.md | 166 ++++------
docs/docs/iceberg/primary-key-table.mdx | 364 +++++++++------------
docs/docs/iceberg/rest-catalog.mdx | 172 +++++-----
docs/sidebars.js | 24 +-
docs/static/img/iceberg-metadata-layout.svg | 52 +++
docs/static/img/iceberg-primary-key-visibility.svg | 53 +++
docs/static/img/iceberg-publication.svg | 56 ++++
14 files changed, 971 insertions(+), 705 deletions(-)
diff --git a/docs/docs/iceberg/append-table.mdx
b/docs/docs/iceberg/append-table.mdx
index 8e5b180138..e2b474e50e 100644
--- a/docs/docs/iceberg/append-table.mdx
+++ b/docs/docs/iceberg/append-table.mdx
@@ -1,5 +1,5 @@
---
-title: "Append Table"
+title: "Append Tables"
sidebar_position: 2
---
@@ -27,147 +27,150 @@ under the License.
# Append Tables
-Let's walk through a simple example, where we query Paimon tables with Iceberg
connectors in Flink and Spark.
-Before trying out this example, make sure that your compute engine already
supports Iceberg.
-Please refer to Iceberg's document if you haven't set up Iceberg.
-* Flink: [Preparation when using Flink SQL
Client](https://iceberg.apache.org/docs/latest/flink/#preparation-when-using-flink-sql-client)
-* Spark: [Using Iceberg in Spark
3](https://iceberg.apache.org/docs/latest/spark-getting-started/#using-iceberg-in-spark-3)
+This walkthrough creates an append table through Paimon and reads the same
data through an Iceberg
+Hadoop catalog. It also provides the catalog setup used by the [primary key
examples](./primary-key-table.mdx).
-Let's now create a Paimon append only table with Iceberg compatibility enabled
and insert some data.
+## Before You Begin
-<Tabs groupId="create-paimon-append-only-table">
+Install the Paimon connector for your engine and an Iceberg runtime compatible
with that engine.
+See the [Paimon Flink quick start](../flink/quick-start.mdx),
+[Paimon Spark quick start](../spark/quick-start.mdx), and Iceberg's
+[Flink setup](https://iceberg.apache.org/docs/latest/flink/) or
+[Spark setup](https://iceberg.apache.org/docs/latest/spark-getting-started/).
-<TabItem value="flink-sql" label="Flink SQL">
+Replace the path and version placeholders below. Both connectors must be able
to access the same
+warehouse. In a cluster, use shared storage accessible to every worker.
+
+## Prepare Catalogs
+
+<Tabs groupId="iceberg-engine">
+<TabItem value="flink" label="Flink SQL">
+
+Install the Paimon and Iceberg Flink runtime JARs before starting the cluster
and SQL client.
+Use batch mode for the bounded examples and synchronous DML so each insert
finishes before its query.
```sql
+SET 'execution.runtime-mode' = 'batch';
+SET 'table.dml-sync' = 'true';
+
CREATE CATALOG paimon_catalog WITH (
'type' = 'paimon',
'warehouse' = '<path-to-warehouse>'
);
-CREATE TABLE paimon_catalog.`default`.cities (
- country STRING,
- name STRING
-) WITH (
- 'metadata.iceberg.storage' = 'hadoop-catalog'
-);
+CREATE DATABASE IF NOT EXISTS paimon_catalog.`default`;
-INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'new york'),
('germany', 'berlin'), ('usa', 'chicago'), ('germany', 'hamburg');
+CREATE CATALOG iceberg_catalog WITH (
+ 'type' = 'iceberg',
+ 'catalog-type' = 'hadoop',
+ 'warehouse' = '<path-to-warehouse>/iceberg',
+ 'cache-enabled' = 'false'
+);
```
</TabItem>
+<TabItem value="spark" label="Spark SQL">
-<TabItem value="spark-sql" label="Spark SQL">
-
-Start `spark-sql` with the following command line.
+Start `spark-sql` with both catalogs. The Iceberg artifact name contains the
Spark major/minor and
+Scala binary versions; its Maven version follows the colon. Select a
combination published by
+Iceberg and supported by your Paimon Spark connector.
```bash
-spark-sql --jars <path-to-paimon-jar> \
+spark-sql \
+ --jars <path-to-paimon-spark-jar> \
+ --packages
org.apache.iceberg:iceberg-spark-runtime-<spark-major.minor>_<scala-binary-version>:<iceberg-version>
\
--conf
spark.sql.catalog.paimon_catalog=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.paimon_catalog.warehouse=<path-to-warehouse> \
- --packages org.apache.iceberg:iceberg-spark-runtime-<iceberg-version> \
--conf
spark.sql.catalog.iceberg_catalog=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.iceberg_catalog.type=hadoop \
--conf
spark.sql.catalog.iceberg_catalog.warehouse=<path-to-warehouse>/iceberg \
- --conf spark.sql.catalog.iceberg_catalog.cache-enabled=false \ # disable
iceberg catalog caching to quickly see the result
+ --conf spark.sql.catalog.iceberg_catalog.cache-enabled=false \
--conf
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
```
-Run the following Spark SQL to create Paimon table and insert data.
-
```sql
-CREATE TABLE paimon_catalog.`default`.cities (
- country STRING,
- name STRING
-) TBLPROPERTIES (
- 'metadata.iceberg.storage' = 'hadoop-catalog'
-);
-
-INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'new york'),
('germany', 'berlin'), ('usa', 'chicago'), ('germany', 'hamburg');
+CREATE DATABASE IF NOT EXISTS paimon_catalog.`default`;
```
</TabItem>
-
</Tabs>
-Now let's query this Paimon table with Iceberg connector.
+The examples disable Iceberg catalog caching to make newly published metadata
easier to observe.
+For Hive or REST access, use the corresponding [catalog guide](./catalogs.md).
-<Tabs groupId="query-paimon-append-only-table">
+## Create the Paimon Table
-<TabItem value="flink-sql" label="Flink SQL">
+<Tabs groupId="iceberg-engine">
+<TabItem value="flink" label="Flink SQL">
```sql
-CREATE CATALOG iceberg_catalog WITH (
- 'type' = 'iceberg',
- 'catalog-type' = 'hadoop',
- 'warehouse' = '<path-to-warehouse>/iceberg',
- 'cache-enabled' = 'false' -- disable iceberg catalog caching to quickly
see the result
+CREATE TABLE paimon_catalog.`default`.cities (
+ country STRING,
+ name STRING
+) WITH (
+ 'metadata.iceberg.storage' = 'hadoop-catalog'
);
-
-SELECT * FROM iceberg_catalog.`default`.cities WHERE country = 'germany';
-/*
-+----+--------------------------------+--------------------------------+
-| op | country | name |
-+----+--------------------------------+--------------------------------+
-| +I | germany | berlin |
-| +I | germany | hamburg |
-+----+--------------------------------+--------------------------------+
-*/
```
</TabItem>
-
-<TabItem value="spark-sql" label="Spark SQL">
+<TabItem value="spark" label="Spark SQL">
```sql
-SELECT * FROM iceberg_catalog.`default`.cities WHERE country = 'germany';
-/*
-germany berlin
-germany hamburg
-*/
+CREATE TABLE paimon_catalog.`default`.cities (
+ country STRING,
+ name STRING
+) TBLPROPERTIES (
+ 'metadata.iceberg.storage' = 'hadoop-catalog'
+);
```
</TabItem>
-
</Tabs>
-Let's insert more data and query again.
+## Write with Paimon, Read with Iceberg
-<Tabs groupId="query-paimon-append-only-table-again">
-
-<TabItem value="flink-sql" label="Flink SQL">
+Run the following SQL in either engine. The catalog name determines which
connector handles each
+statement.
```sql
-INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'houston'),
('germany', 'munich');
-
-SELECT * FROM iceberg_catalog.`default`.cities WHERE country = 'germany';
-/*
-+----+--------------------------------+--------------------------------+
-| op | country | name |
-+----+--------------------------------+--------------------------------+
-| +I | germany | munich |
-| +I | germany | berlin |
-| +I | germany | hamburg |
-+----+--------------------------------+--------------------------------+
-*/
+INSERT INTO paimon_catalog.`default`.cities VALUES
+ ('usa', 'new york'),
+ ('germany', 'berlin'),
+ ('usa', 'chicago'),
+ ('germany', 'hamburg');
+
+SELECT country, name
+FROM iceberg_catalog.`default`.cities
+WHERE country = 'germany'
+ORDER BY name;
```
-</TabItem>
+Expected rows:
-<TabItem value="spark-sql" label="Spark SQL">
+```text
+country name
+germany berlin
+germany hamburg
+```
+
+Insert another row through Paimon, then read the refreshed Iceberg
representation:
```sql
-INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'houston'),
('germany', 'munich');
-
-SELECT * FROM iceberg_catalog.`default`.cities WHERE country = 'germany';
-/*
-germany munich
-germany berlin
-germany hamburg
-*/
-```
+INSERT INTO paimon_catalog.`default`.cities VALUES ('germany', 'munich');
-</TabItem>
+SELECT country, name
+FROM iceberg_catalog.`default`.cities
+WHERE country = 'germany'
+ORDER BY name;
+```
-</Tabs>
+```text
+country name
+germany berlin
+germany hamburg
+germany munich
+```
+If the table or new rows are missing, first check that the insert completed,
the Iceberg warehouse
+ends in `/iceberg`, and the reader can access the published metadata and
original data files.
+For updates and deletes, continue with [primary key
tables](./primary-key-table.mdx).
diff --git a/docs/docs/iceberg/catalogs.md b/docs/docs/iceberg/catalogs.md
new file mode 100644
index 0000000000..42d11c3eaa
--- /dev/null
+++ b/docs/docs/iceberg/catalogs.md
@@ -0,0 +1,91 @@
+---
+title: "Catalogs and Metadata Layout"
+sidebar_position: 4
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Catalogs and Metadata Layout
+
+Choose how readers discover the Iceberg table, then choose where Paimon writes
its Iceberg metadata.
+These are separate decisions: `metadata.iceberg.storage` selects the
publication mode, while
+`metadata.iceberg.storage-location` controls the filesystem layout.
+
+## Choose a Publication Mode
+
+| `metadata.iceberg.storage` | Reader access | Default metadata location |
Additional setup |
+| --- | --- | --- | --- |
+| `disabled` | No Iceberg representation | None | Default setting |
+| `table-location` | Load an individual table by path | Under the Paimon table
| Iceberg reader with path-based access |
+| `hadoop-catalog` | Iceberg Hadoop catalog | Separate Iceberg warehouse |
[Flink or Spark catalog](./append-table.mdx#prepare-catalogs) |
+| `hive-catalog` | Iceberg Hive catalog | Separate Iceberg warehouse | [Hive
metastore and client](./hive-catalog.md) |
+| `rest-catalog` | Iceberg REST catalog | Separate Iceberg warehouse, plus
REST publication | [REST endpoint and Paimon Iceberg JAR](./rest-catalog.mdx) |
+
+Choose `hadoop-catalog` for the walkthroughs. Use Hive or REST when your
readers discover tables
+through those services. Use `table-location` for an individual table that
readers load by path.
+
+## Metadata Layout
+
+For a Paimon table at `<warehouse>/default.db/cities`, the two layouts are:
+
+| `metadata.iceberg.storage-location` | Iceberg metadata directory |
+| --- | --- |
+| `catalog-location` | `<warehouse>/iceberg/default/cities/metadata` |
+| `table-location` | `<warehouse>/default.db/cities/metadata` |
+
+
+
+The override is optional. By default, `table-location` storage uses the table
layout; Hadoop, Hive,
+and REST storage use the separate catalog layout. Changing the layout does not
move Paimon's data
+files.
+
+The default Hadoop catalog walkthrough points the Iceberg reader at
`<warehouse>/iceberg`.
+A reader that loads a table directly uses the directory containing `metadata`,
for example
+`<warehouse>/default.db/cities` with the table layout.
+
+### Databases with Custom Locations
+
+The separate catalog layout derives its path from a Paimon database directory
ending in `.db`.
+For a database with a nonstandard location, store metadata alongside the table
instead. For example,
+combine Hive registration with the table layout:
+
+```sql
+'metadata.iceberg.storage' = 'hive-catalog',
+'metadata.iceberg.storage-location' = 'table-location',
+'metadata.iceberg.uri' = 'thrift://<metastore-host>:9083'
+```
+
+Use the registered Hive table to discover this location; the default
+`<warehouse>/iceberg` Hadoop catalog path no longer describes this table's
metadata.
+
+## Keep Writer and Reader Settings Aligned
+
+Paimon options configure publication. Iceberg connector options configure
reading; they use
+different names even when they describe the same service.
+
+| Setting | Paimon table option | Iceberg reader setting |
+| --- | --- | --- |
+| Hive metastore | `metadata.iceberg.uri` | Hive catalog `uri` (Flink/Spark)
or `hive.metastore.uri` (Trino) |
+| REST endpoint | `metadata.iceberg.rest.uri` | REST catalog `uri` |
+| REST warehouse | `metadata.iceberg.rest.warehouse` | REST catalog
`warehouse` |
+| Hadoop warehouse, with default layout | Derived from the Paimon table
location | `<paimon-warehouse>/iceberg` |
+
+See [Hive Catalog](./hive-catalog.md) and [REST Catalog](./rest-catalog.mdx)
for complete examples,
+and [configuration reference](./configurations.mdx) for the available Paimon
options.
diff --git a/docs/docs/iceberg/configurations.mdx
b/docs/docs/iceberg/configurations.mdx
index cd20534b62..8381bb6c18 100644
--- a/docs/docs/iceberg/configurations.mdx
+++ b/docs/docs/iceberg/configurations.mdx
@@ -1,6 +1,6 @@
---
-title: "Configurations"
-sidebar_position: 7
+title: "Configuration Reference"
+sidebar_position: 10
---
import ConfigTable from '@site/src/components/ConfigTable';
@@ -25,8 +25,35 @@ specific language governing permissions and limitations
under the License.
-->
-# Configurations
+# Configuration Reference
-Options for Iceberg Compatibility.
+Set these options on the **Paimon table** to control Iceberg metadata
publication. Reader-side
+Iceberg catalog properties are configured separately.
+
+## Find the Right Options
+
+| Task | Options and guide |
+| --- | --- |
+| Enable publication and choose its location | `metadata.iceberg.storage`,
`metadata.iceberg.storage-location`; [catalogs and metadata
layout](./catalogs.md) |
+| Register in Hive | `metadata.iceberg.uri`, `metadata.iceberg.hive-*`,
`metadata.iceberg.database`, `metadata.iceberg.table`; [Hive
Catalog](./hive-catalog.md) |
+| Register in REST | `metadata.iceberg.rest.*`; [REST
Catalog](./rest-catalog.mdx) |
+| Publish deletion vectors | `metadata.iceberg.format-version` plus
`deletion-vectors.*`; [primary key
tables](./primary-key-table.mdx#deletion-vector-support) |
+| Adjust manifest compatibility | `metadata.iceberg.manifest-compression`,
`metadata.iceberg.manifest-legacy-version`; [query engines](./ecosystem.mdx) |
+| Control metadata cleanup | `metadata.iceberg.delete-after-commit.enabled`,
`metadata.iceberg.previous-versions-max` |
+
+Metadata retention options control generated metadata files, not Paimon
data-file retention.
+Use Paimon's [snapshot management](../maintenance/manage-snapshots.mdx) and
+[tag management](../maintenance/manage-tags.mdx) for table history.
+
+## Iceberg Table Options
<ConfigTable html={icebergConfigurationHtml} />
+
+## Prefixed Properties
+
+`metadata.iceberg.rest.<key>` passes `<key>` to the REST catalog client. It is
a property prefix,
+so the individual service-specific keys are not listed in the generated table
above.
+
+`metadata.iceberg.table-properties.<key>` supplies custom table properties
during REST publication.
+Iceberg-reserved properties are ignored. Use the dedicated Paimon options for
format version
+and publication behavior.
diff --git a/docs/docs/iceberg/index.md b/docs/docs/iceberg/data-types.md
similarity index 54%
copy from docs/docs/iceberg/index.md
copy to docs/docs/iceberg/data-types.md
index 1f94e90ef8..44d8b8c0bc 100644
--- a/docs/docs/iceberg/index.md
+++ b/docs/docs/iceberg/data-types.md
@@ -1,6 +1,6 @@
---
-title: "Iceberg Metadata"
-sidebar_position: 98
+title: "Data Types"
+sidebar_position: 9
---
<!--
@@ -22,64 +22,11 @@ specific language governing permissions and limitations
under the License.
-->
-# Overview
-
-Paimon supports generating Iceberg compatible metadata,
-so that Paimon tables can be consumed directly by Iceberg readers.
-
-Set the following table options, so that Paimon tables can generate Iceberg
compatible metadata.
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style={{width: "20%"}}>Option</th>
- <th class="text-left" style={{width: "5%"}}>Default</th>
- <th class="text-left" style={{width: "10%"}}>Type</th>
- <th class="text-left" style={{width: "60%"}}>Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>metadata.iceberg.storage</h5></td>
- <td style={{wordWrap: "break-word"}}>disabled</td>
- <td>Enum</td>
- <td>
- When set, produce Iceberg metadata after a snapshot is committed, so
that Iceberg readers can read Paimon's raw data files.
- <ul>
- <li><code>disabled</code>: Disable Iceberg compatibility
support.</li>
- <li><code>table-location</code>: Store Iceberg metadata in each
table's directory.</li>
- <li><code>hadoop-catalog</code>: Store Iceberg metadata in a
separate directory. This directory can be specified as the warehouse directory
of an Iceberg Hadoop catalog.</li>
- <li><code>hive-catalog</code>: Not only store Iceberg metadata like
hadoop-catalog, but also create Iceberg external table in Hive.</li>
- </ul>
- </td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.storage-location</h5></td>
- <td style={{wordWrap: "break-word"}}>(none)</td>
- <td>Enum</td>
- <td>
- Specifies where to store Iceberg metadata files. If not set, the
storage location will default based on the selected metadata.iceberg.storage
type.
- <ul>
- <li><code>table-location</code>: Store Iceberg metadata in each
table's directory. Useful for standalone Iceberg tables or Iceberg Java API
access. Can also be used with Hive Catalog.</li>
- <li><code>catalog-location</code>: Store Iceberg metadata in a
separate directory. This is the default behavior when using Hive Catalog or
Hadoop Catalog.</li>
- </ul>
- </td>
- </tr>
- </tbody>
-</table>
-
-For most SQL users, we recommend setting `'metadata.iceberg.storage' =
'hadoop-catalog'`
-or `'metadata.iceberg.storage' = 'hive-catalog'`,
-so that all tables can be visited as an Iceberg warehouse.
-For Iceberg Java API users, you might consider setting
`'metadata.iceberg.storage' = 'table-location'`,
-so you can visit each table with its table path.
-When using `metadata.iceberg.storage = hadoop-catalog` or `hive-catalog`,
-you can optionally configure `metadata.iceberg.storage-location` to control
where the metadata is stored.
-If not set, the default behavior depends on the storage type.
-
-## Supported Types
-
-Paimon Iceberg compatibility currently supports the following data types.
+# Data Types
+
+
+The following mappings describe the metadata Paimon publishes. The Iceberg
reader must also
+support the table's file format, format version, and column types.
| Paimon Data Type | Iceberg Data Type |
|----------------|-------------------|
@@ -126,3 +73,12 @@ compatibility publishes only millisecond time values.
- Iceberg REST catalog publication does not yet support geospatial columns.
Use `table-location`, `hadoop-catalog`, or `hive-catalog` metadata storage
instead.
:::
+
+
+## Existing Tables
+
+Enabling Iceberg metadata validates historical schemas as well as the current
schema. Changing or
+dropping an incompatible column in the latest schema alone may therefore be
insufficient. Check
+schema history when enabling publication fails with a timestamp, time, or
geospatial type error.
+
+For connector-specific restrictions, see [query engines](./ecosystem.mdx).
diff --git a/docs/docs/iceberg/ecosystem.mdx b/docs/docs/iceberg/ecosystem.mdx
index a111ae8f4f..dd99bc16f3 100644
--- a/docs/docs/iceberg/ecosystem.mdx
+++ b/docs/docs/iceberg/ecosystem.mdx
@@ -1,11 +1,8 @@
---
-title: "Ecosystem"
-sidebar_position: 6
+title: "Query Engines"
+sidebar_position: 8
---
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -25,92 +22,98 @@ specific language governing permissions and limitations
under the License.
-->
-# Iceberg Ecosystems
-
-## AWS Athena
+# Query Engines
-AWS Athena may use old manifest reader to read Iceberg manifest by names, we
should let Paimon producing legacy Iceberg
-manifest list file, you can enable:
`'metadata.iceberg.manifest-legacy-version'`.
-
-## DuckDB
+An engine's Iceberg connector can read Paimon's published Iceberg
representation if it supports
+the catalog, data file format, column types, and Iceberg features used by the
table. Configure the
+reader's access to both metadata and the original Paimon data files.
-Duckdb may rely on files placed in the `root/data` directory, while Paimon is
usually placed directly in the `root`
-directory, so you can configure this parameter for the table to achieve
compatibility:
-`'data-file.path-directory' = 'data'`.
+For Flink and Spark, start with the [append-table
walkthrough](./append-table.mdx). The examples
+below use Iceberg connectors; all writes and maintenance remain in Paimon.
## Trino Iceberg
-In this example, we use Trino Iceberg connector to access Paimon table through
Iceberg Hive catalog.
-Before trying out this example, make sure that you have configured Trino
Iceberg connector.
-See [Trino's
document](https://trino.io/docs/current/connector/iceberg.html#general-configuration)
for more information.
+First publish the `animals` table using the [Hive catalog
example](./hive-catalog.md#publish-a-table).
+Configure Trino to connect to the same Hive metastore.
-Let's first create a Paimon table with Iceberg compatibility enabled.
+Create `etc/catalog/iceberg.properties` on the Trino server:
-<Tabs groupId="paimon-append-only-table-trino-1">
+```properties
+connector.name=iceberg
+iceberg.catalog.type=hive_metastore
+hive.metastore.uri=thrift://<metastore-host>:9083
+```
+
+Add the filesystem configuration and credentials required by your storage.
Trino requires a
+configured filesystem in addition to the metastore connection; see the
+[Iceberg connector
configuration](https://trino.io/docs/current/connector/iceberg.html#general-configuration).
-<TabItem value="flink-sql" label="Flink SQL">
+Query the registered table with the fully qualified catalog, schema, and table
name:
```sql
-CREATE CATALOG paimon_catalog WITH (
- 'type' = 'paimon',
- 'warehouse' = '<path-to-warehouse>'
-);
-
-CREATE TABLE paimon_catalog.`default`.animals (
- kind STRING,
- name STRING
-) WITH (
- 'metadata.iceberg.storage' = 'hive-catalog',
- 'metadata.iceberg.uri' = 'thrift://<host>:<port>'
-);
-
-INSERT INTO paimon_catalog.`default`.animals VALUES ('mammal', 'cat'),
('mammal', 'dog'), ('reptile', 'snake'), ('reptile', 'lizard');
+SELECT kind, name
+FROM iceberg.default.animals
+WHERE kind = 'mammal'
+ORDER BY name;
```
-</TabItem>
+```text
+kind name
+mammal cat
+mammal dog
+```
-<TabItem value="spark-sql" label="Spark SQL">
+If you configured `metadata.iceberg.database` or `metadata.iceberg.table`, use
those registration
+names in the Trino query.
-Start `spark-sql` with the following command line.
+## AWS Athena
-```bash
-spark-sql --jars <path-to-paimon-jar> \
- --conf
spark.sql.catalog.paimon_catalog=org.apache.paimon.spark.SparkCatalog \
- --conf spark.sql.catalog.paimon_catalog.warehouse=<path-to-warehouse> \
- --packages org.apache.iceberg:iceberg-spark-runtime-<iceberg-version> \
- --conf
spark.sql.catalog.iceberg_catalog=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.iceberg_catalog.type=hadoop \
- --conf
spark.sql.catalog.iceberg_catalog.warehouse=<path-to-warehouse>/iceberg \
- --conf spark.sql.catalog.iceberg_catalog.cache-enabled=false \ # disable
iceberg catalog caching to quickly see the result
- --conf
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
-```
+Athena discovers Iceberg tables through AWS Glue. Publish the table using a
properly configured
+[Glue Hive client](./hive-catalog.md#aws-glue-catalog), and check Athena's
+[Iceberg
limitations](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html).
+Athena documents support for Iceberg v2 tables. Use the full-compaction mode
for primary key tables
+when querying with a reader that cannot consume v3 deletion vectors.
-Run the following Spark SQL to create Paimon table, insert/update data, and
query with Iceberg catalog.
+For readers that require the legacy manifest field naming, enable this Paimon
table option:
```sql
-CREATE TABLE paimon_catalog.`default`.animals (
- kind STRING,
- name STRING
-) TBLPROPERTIES (
- 'metadata.iceberg.storage' = 'hive-catalog',
- 'metadata.iceberg.uri' = 'thrift://<host>:<port>'
-);
-
-INSERT INTO paimon_catalog.`default`.animals VALUES ('mammal', 'cat'),
('mammal', 'dog'), ('reptile', 'snake'), ('reptile', 'lizard');
+'metadata.iceberg.manifest-legacy-version' = 'true'
```
-</TabItem>
+This controls the manifest representation; it does not add support for
otherwise unsupported
+Iceberg versions or column types.
+
+## DuckDB
+
+Install and load DuckDB's [Iceberg
extension](https://duckdb.org/docs/current/core_extensions/iceberg/overview)
+and choose the catalog or metadata-file access method supported by your
extension version.
+Check its support for the format version and deletion information used by the
table.
-</Tabs>
+When troubleshooting a read, verify that the data paths referenced by the
manifests are reachable.
+Paimon's data files remain in the Paimon table directory even when the Iceberg
metadata is in a
+separate warehouse. Do not treat `<warehouse>/iceberg` as a copy of the data.
-Start Trino using Iceberg catalog and query from Paimon table.
+For integrations that require data files under a `data` subdirectory, Paimon
supports the following
+option for new writes:
```sql
-SELECT * FROM animals WHERE class = 'mammal';
-/*
- kind | name
---------+------
- mammal | cat
- mammal | dog
-*/
+'data-file.path-directory' = 'data'
```
+
+Changing this setting does not relocate existing data files. Use it only when
required by the
+reader or integration you deploy. Keep `metadata.iceberg.manifest-compression`
at its default
+`snappy` unless the reader supports the alternative codec.
+
+## Troubleshooting
+
+| Symptom | What to check |
+| --- | --- |
+| Table not found | Publication mode, Hadoop warehouse path, Hive/REST
registration name, and writer-side catalog dependencies |
+| New rows are missing | Completed Paimon commit, primary key compaction mode,
metadata publication, and reader cache |
+| Data files cannot be opened | Reader filesystem support, credentials, and
access to the original Paimon data paths |
+| Schema or manifest parsing fails | [Type restrictions](./data-types.md),
format version, manifest codec, and connector version |
+| Tag not found | Whether its snapshot exists in Iceberg metadata, reader
refresh, and [REST tag limitations](./iceberg-tags.md) |
+
+For Hive or REST tables missing from the service, inspect the writer logs as
well as the local
+metadata directory. A missing metadata committer dependency can leave local
Iceberg metadata files
+present without registering the table in the external catalog.
diff --git a/docs/docs/iceberg/hive-catalog.md
b/docs/docs/iceberg/hive-catalog.md
index 52bd5d6f2a..6420ed03be 100644
--- a/docs/docs/iceberg/hive-catalog.md
+++ b/docs/docs/iceberg/hive-catalog.md
@@ -1,5 +1,5 @@
---
-title: "Hive Catalogs"
+title: "Hive Catalog"
sidebar_position: 5
---
@@ -24,77 +24,118 @@ under the License.
# Hive Catalog
-When creating Paimon table, set `'metadata.iceberg.storage' = 'hive-catalog'`.
-This option value not only store Iceberg metadata like hadoop-catalog, but
also create Iceberg external table in Hive.
-This Paimon table can be accessed from Iceberg Hive catalog later.
-
-To provide information about Hive metastore,
-you also need to set some (or all) of the following table options when
creating Paimon table.
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Option</th>
- <th class="text-left" style="width: 5%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 60%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>metadata.iceberg.uri</h5></td>
- <td style="word-wrap: break-word;"></td>
- <td>String</td>
- <td>Hive metastore uri for Iceberg Hive catalog.</td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.hive-conf-dir</h5></td>
- <td style="word-wrap: break-word;"></td>
- <td>String</td>
- <td>hive-conf-dir for Iceberg Hive catalog.</td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.hadoop-conf-dir</h5></td>
- <td style="word-wrap: break-word;"></td>
- <td>String</td>
- <td>hadoop-conf-dir for Iceberg Hive catalog.</td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.manifest-compression</h5></td>
- <td style="word-wrap: break-word;">snappy</td>
- <td>String</td>
- <td>Compression for Iceberg manifest files.</td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.manifest-legacy-version</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td>Should use the legacy manifest version to generate Iceberg's 1.4
manifest files.</td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.hive-client-class</h5></td>
- <td style="word-wrap:
break-word;">org.apache.hadoop.hive.metastore.HiveMetaStoreClient</td>
- <td>String</td>
- <td>Hive client class name for Iceberg Hive Catalog.</td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.glue.skip-archive</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td>Skip archive for AWS Glue catalog.</td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.hive-skip-update-stats</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td>Skip updating Hive stats.</td>
- </tr>
- </tbody>
-</table>
+Use `hive-catalog` to publish an Iceberg table in a Hive metastore. Paimon
writes Iceberg metadata
+and updates the metastore entry to point to it. Iceberg readers then discover
the table through
+their own Hive catalog connector.
+
+## Before You Begin
+
+Prepare the Paimon and Iceberg engine connectors as described in the
+[append-table walkthrough](./append-table.mdx#before-you-begin). The Paimon
writer also needs the
+Paimon Hive catalog module and its Hive dependencies; see [Hive catalog
setup](../concepts/catalog.md#hive-catalog).
+Both the writer and reader need access to the metastore and the files
referenced by the table.
+
+## Publish a Table
+
+The following Flink SQL uses a filesystem Paimon catalog and registers the
Iceberg representation
+in Hive. Replace the warehouse and metastore placeholders.
+
+```sql
+SET 'execution.runtime-mode' = 'batch';
+SET 'table.dml-sync' = 'true';
+
+CREATE CATALOG paimon_catalog WITH (
+ 'type' = 'paimon',
+ 'warehouse' = '<path-to-warehouse>'
+);
+
+CREATE DATABASE IF NOT EXISTS paimon_catalog.`default`;
+
+CREATE TABLE paimon_catalog.`default`.animals (
+ kind STRING,
+ name STRING
+) WITH (
+ 'metadata.iceberg.storage' = 'hive-catalog',
+ 'metadata.iceberg.uri' = 'thrift://<metastore-host>:9083'
+);
+
+INSERT INTO paimon_catalog.`default`.animals VALUES
+ ('mammal', 'cat'), ('mammal', 'dog'),
+ ('reptile', 'snake'), ('reptile', 'lizard');
+```
+
+For Spark, use the same table options in `TBLPROPERTIES`; the
+[append-table walkthrough](./append-table.mdx) shows the corresponding syntax.
+
+## Read through Iceberg
+
+In Flink, connect an Iceberg Hive catalog to the same metastore:
+
+```sql
+CREATE CATALOG iceberg_hive WITH (
+ 'type' = 'iceberg',
+ 'catalog-type' = 'hive',
+ 'uri' = 'thrift://<metastore-host>:9083',
+ 'cache-enabled' = 'false'
+);
+
+SELECT kind, name FROM iceberg_hive.`default`.animals
+WHERE kind = 'mammal' ORDER BY name;
+```
+
+```text
+kind name
+mammal cat
+mammal dog
+```
+
+See [Trino](./ecosystem.mdx#trino-iceberg) for a reader that uses the same
Hive registration.
+For primary key tables, the [compaction or deletion-vector
requirements](./primary-key-table.mdx)
+also apply.
+
+## Table Names and Metadata Location
+
+By default, the Iceberg database and table names match the Paimon names. When
the Paimon catalog
+also uses the same Hive metastore, give the Iceberg representation a distinct
name to avoid
+reusing the Paimon table's metastore entry:
+
+```sql
+'metadata.iceberg.database' = 'iceberg_analytics',
+'metadata.iceberg.table' = 'animals'
+```
+
+Readers would then query `iceberg_hive.iceberg_analytics.animals`. For Hive
publication,
+`metadata.iceberg.database` also accepts multiple databases separated by
semicolons.
+Aliases change catalog registration names, not the physical metadata path.
+
+Metadata is stored under `<warehouse>/iceberg/<database>/<table>/metadata` by
default.
+Use `metadata.iceberg.storage-location = table-location` for metadata
alongside the Paimon table,
+including databases with custom locations. See [metadata
layout](./catalogs.md#metadata-layout).
+
+## Connection Options
+
+| Paimon table option | When to use it |
+| --- | --- |
+| `metadata.iceberg.uri` | Set the Hive metastore Thrift URI explicitly |
+| `metadata.iceberg.hive-conf-dir` | Load Hive configuration such as
`hive-site.xml` |
+| `metadata.iceberg.hadoop-conf-dir` | Load Hadoop configuration needed by the
Hive client |
+| `metadata.iceberg.hive-client-class` | Use a custom Hive metastore client |
+| `metadata.iceberg.hive-skip-update-stats` | Skip updating Hive statistics |
+
+If the URI is not set as a table option, provide `hive.metastore.uris` in the
loaded configuration.
+See [configuration reference](./configurations.mdx) for defaults and other
publication options.
## AWS Glue Catalog
-You can use Hive Catalog to connect AWS Glue metastore, you can use set
`'metadata.iceberg.hive-client-class'` to
-`'com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient'`.
+To publish through a Hive-compatible AWS Glue client, set:
+
+```sql
+'metadata.iceberg.hive-client-class' =
'com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient'
+```
+
+Install a Glue Hive client compatible with your Hive dependencies on the
writer classpath, and
+configure its AWS region and credentials. The [AWS Glue Data Catalog
client](https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore)
+provides build and configuration instructions.
`metadata.iceberg.glue.skip-archive` controls
+whether publication skips archiving Glue table versions.
-> **Note:** You can use this
[repo](https://github.com/promotedai/aws-glue-data-catalog-client-for-apache-hive-metastore)
to build the required jar, include it in your path and configure the
AWSCatalogMetastoreClient.
+Reader support has separate constraints; see
[Athena](./ecosystem.mdx#aws-athena).
diff --git a/docs/docs/iceberg/iceberg-tags.md
b/docs/docs/iceberg/iceberg-tags.md
index fd8159d65f..3cb106444a 100644
--- a/docs/docs/iceberg/iceberg-tags.md
+++ b/docs/docs/iceberg/iceberg-tags.md
@@ -1,6 +1,6 @@
---
-title: "Iceberg Tags"
-sidebar_position: 4
+title: "Tags"
+sidebar_position: 7
---
<!--
@@ -24,25 +24,67 @@ under the License.
# Iceberg Tags
-When enable iceberg compatibility, Paimon Tags will also be synced to [Iceberg
Tags](https://iceberg.apache.org/docs/nightly/branching/#historical-tags).
-Tags are only synced to Iceberg if the referenced snapshot exists in the
Iceberg table.
+A Paimon tag can be recorded as a named snapshot reference in the generated
Iceberg metadata.
+This allows Iceberg readers to query that historical state by name.
+
+## Publication Requirements
+
+The tagged snapshot must already exist in the current Iceberg metadata. If it
is absent, Paimon
+skips adding the Iceberg reference. For a primary key table, tagging a Paimon
snapshot does not
+make unpublished changes visible; the [read-mode
requirements](./primary-key-table.mdx) still apply.
+
+Tag creation and deletion update the generated metadata file. This is the file
read by
+`table-location`, Hadoop, and Hive access. Refresh the reader or disable its
catalog cache to
+observe a changed tag.
+
+:::info REST catalogs
+
+Do not rely on Paimon tag creation or deletion being synchronized to a REST
catalog. The tag
+callback updates the local metadata file, while REST publication maintains
separate catalog
+metadata. The example below uses Hadoop catalog access.
+
+:::
+
+## Create and Query a Tag in Flink
+
+Complete the [append-table walkthrough](./append-table.mdx) first, using the
Flink tab. It creates
+`paimon_catalog.default.cities` and an Iceberg Hadoop catalog with caching
disabled.
+
+Choose an existing snapshot ID from the Paimon snapshots table:
```sql
-CREATE CATALOG paimon WITH (
- 'type' = 'paimon',
- 'warehouse' = '<path-to-warehouse>'
-);
-
-CREATE CATALOG iceberg WITH (
- 'type' = 'iceberg',
- 'catalog-type' = 'hadoop',
- 'warehouse' = '<path-to-warehouse>/iceberg',
- 'cache-enabled' = 'false' -- disable iceberg catalog caching to quickly
see the result
-);
-
--- create tag for paimon table
-CALL paimon.sys.create_tag('default.T', 'tag1', 1);
-
--- query tag in iceberg table
-SELECT * FROM iceberg.`default`.T /*+ OPTIONS('tag'='tag1') */;
+SELECT snapshot_id FROM paimon_catalog.`default`.`cities$snapshots`
+ORDER BY snapshot_id;
```
+
+After verifying that snapshot 1 is present in the published Iceberg history,
create a tag and query it:
+
+```sql
+CALL paimon_catalog.sys.create_tag('default.cities', 'first_batch', 1);
+
+SELECT country, name
+FROM iceberg_catalog.`default`.cities /*+ OPTIONS('tag'='first_batch') */
+WHERE country = 'germany'
+ORDER BY name;
+```
+
+For the first insert in the walkthrough, the tagged rows are:
+
+```text
+country name
+germany berlin
+germany hamburg
+```
+
+If you reused an existing table, replace `1` with the snapshot you intend to
tag. Verify that snapshot
+in the Iceberg reader's snapshot history as well; a Paimon snapshot's
existence alone is insufficient.
+
+## Retention and Format Changes
+
+Manage tag lifecycle through [Paimon tags](../maintenance/manage-tags.mdx). A
tag selects a snapshot;
+it does not bypass the publication rules or convert the table's file format.
+
+Tags do not require format v3 by themselves. Upgrade
`metadata.iceberg.format-version` only when a
+feature such as deletion vectors requires it. A format change rebuilds Iceberg
metadata and can
+remove previously published history and tag references. Verify the snapshots
and tags available
+through Iceberg after rebuilding.
diff --git a/docs/docs/iceberg/index.md b/docs/docs/iceberg/index.md
index 1f94e90ef8..453f41410b 100644
--- a/docs/docs/iceberg/index.md
+++ b/docs/docs/iceberg/index.md
@@ -1,5 +1,5 @@
---
-title: "Iceberg Metadata"
+title: "Iceberg Compatibility"
sidebar_position: 98
---
@@ -22,107 +22,71 @@ specific language governing permissions and limitations
under the License.
-->
-# Overview
-
-Paimon supports generating Iceberg compatible metadata,
-so that Paimon tables can be consumed directly by Iceberg readers.
-
-Set the following table options, so that Paimon tables can generate Iceberg
compatible metadata.
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style={{width: "20%"}}>Option</th>
- <th class="text-left" style={{width: "5%"}}>Default</th>
- <th class="text-left" style={{width: "10%"}}>Type</th>
- <th class="text-left" style={{width: "60%"}}>Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>metadata.iceberg.storage</h5></td>
- <td style={{wordWrap: "break-word"}}>disabled</td>
- <td>Enum</td>
- <td>
- When set, produce Iceberg metadata after a snapshot is committed, so
that Iceberg readers can read Paimon's raw data files.
- <ul>
- <li><code>disabled</code>: Disable Iceberg compatibility
support.</li>
- <li><code>table-location</code>: Store Iceberg metadata in each
table's directory.</li>
- <li><code>hadoop-catalog</code>: Store Iceberg metadata in a
separate directory. This directory can be specified as the warehouse directory
of an Iceberg Hadoop catalog.</li>
- <li><code>hive-catalog</code>: Not only store Iceberg metadata like
hadoop-catalog, but also create Iceberg external table in Hive.</li>
- </ul>
- </td>
- </tr>
- <tr>
- <td><h5>metadata.iceberg.storage-location</h5></td>
- <td style={{wordWrap: "break-word"}}>(none)</td>
- <td>Enum</td>
- <td>
- Specifies where to store Iceberg metadata files. If not set, the
storage location will default based on the selected metadata.iceberg.storage
type.
- <ul>
- <li><code>table-location</code>: Store Iceberg metadata in each
table's directory. Useful for standalone Iceberg tables or Iceberg Java API
access. Can also be used with Hive Catalog.</li>
- <li><code>catalog-location</code>: Store Iceberg metadata in a
separate directory. This is the default behavior when using Hive Catalog or
Hadoop Catalog.</li>
- </ul>
- </td>
- </tr>
- </tbody>
-</table>
-
-For most SQL users, we recommend setting `'metadata.iceberg.storage' =
'hadoop-catalog'`
-or `'metadata.iceberg.storage' = 'hive-catalog'`,
-so that all tables can be visited as an Iceberg warehouse.
-For Iceberg Java API users, you might consider setting
`'metadata.iceberg.storage' = 'table-location'`,
-so you can visit each table with its table path.
-When using `metadata.iceberg.storage = hadoop-catalog` or `hive-catalog`,
-you can optionally configure `metadata.iceberg.storage-location` to control
where the metadata is stored.
-If not set, the default behavior depends on the storage type.
+# Iceberg Compatibility
-## Supported Types
+Paimon can publish Iceberg metadata that points to its existing data files.
This lets applications
+query a Paimon table through an Iceberg connector while Paimon continues to
manage writes,
+compaction, and data retention.
+
+
+
+## Start Here
+
+| Task | Guide |
+| --- | --- |
+| Read your first Paimon table through Flink or Spark's Iceberg connector |
[Append tables](./append-table.mdx) |
+| Read updates and deletes from a primary key table | [Primary key
tables](./primary-key-table.mdx) |
+| Choose Hadoop, Hive, REST, or path-based access | [Catalogs and metadata
layout](./catalogs.md) |
+| Query a named historical snapshot | [Tags](./iceberg-tags.md) |
+| Connect Trino, Athena, or DuckDB | [Query engines](./ecosystem.mdx) |
+| Check column types and format requirements | [Data types](./data-types.md) |
+| Look up table options | [Configuration reference](./configurations.mdx) |
+
+## How Publication Works
+
+1. A writer commits a Paimon snapshot.
+2. Paimon generates Iceberg manifests and snapshot metadata for the files
eligible for Iceberg reads.
+3. With Hive or REST storage, Paimon also publishes the metadata to the
external catalog.
+4. An Iceberg reader loads the published metadata and reads the referenced
data files directly.
+
+Enable publication with the Paimon table option `metadata.iceberg.storage`;
its default is `disabled`.
+For a first example, use `hadoop-catalog`. The Iceberg warehouse is then
+`<paimon-warehouse>/iceberg`, using the default metadata layout.
+
+```sql
+'metadata.iceberg.storage' = 'hadoop-catalog'
+```
+
+Metadata publication does not copy the table's data. Iceberg readers therefore
need access to both
+the metadata location and the original Paimon data files, including the
required filesystem
+configuration and credentials.
-Paimon Iceberg compatibility currently supports the following data types.
-
-| Paimon Data Type | Iceberg Data Type |
-|----------------|-------------------|
-| `BOOLEAN` | `boolean` |
-| `INT` | `int` |
-| `BIGINT` | `long` |
-| `FLOAT` | `float` |
-| `DOUBLE` | `double` |
-| `DECIMAL` | `decimal` |
-| `CHAR` | `string` |
-| `VARCHAR` | `string` |
-| `BINARY` | `binary` |
-| `VARBINARY` | `binary` |
-| `DATE` | `date` |
-| `TIME` (precision 0-3) | `time` |
-| `TIME` (other precisions) | not supported |
-| `TIMESTAMP` (precision 3-6) | `timestamp` |
-| `TIMESTAMP_LTZ` (precision 3-6) | `timestamptz` |
-| `TIMESTAMP` (other precisions) | not supported |
-| `TIMESTAMP_LTZ` (other precisions) | not supported |
-| `GEOMETRY(crs)` | `geometry(crs)` |
-| `GEOGRAPHY(crs, algorithm)` | `geography(crs, algorithm)` |
-| `ARRAY` | `list` |
-| `MAP` | `map` |
-| `ROW` | `struct` |
-
-:::info
-
-**Note on Timestamp Types:**
-- `TIMESTAMP` and `TIMESTAMP_LTZ` types with precision from 3 to 6 are mapped
to standard Iceberg timestamp types
-- Any other precision is rejected while Iceberg metadata is enabled. A
precision above 6 is written
- as Parquet INT96, which Iceberg reads as a microsecond zoned timestamp
rather than the
- nanoseconds the column declares. Use a precision from 3 to 6.
-
-**Note on Time Types:**
-`TIME` types with a precision above 3 are rejected while Iceberg metadata is
enabled: Iceberg
-compatibility publishes only millisecond time values.
-
-**Note on Geospatial Types:**
-- `GEOMETRY` and `GEOGRAPHY` values use OGC Well-Known Binary (WKB). The
default CRS is `OGC:CRS84`, and the default geography edge algorithm is
`spherical`.
-- Geospatial columns require Parquet for data, per-level, and changelog files.
When Iceberg metadata is enabled, set `metadata.iceberg.format-version` to `3`.
-- Spark SQL supports geospatial columns in Spark 4.1 when
`spark.sql.geospatial.enabled=true`, for CRSs recognized by Spark, with the
`spherical` geography edge algorithm. Spark 3.x, Spark 4.0, and Flink SQL
reject these columns instead of exposing them as binary and losing the CRS or
edge algorithm.
-- When Iceberg metadata is enabled, a `GEOGRAPHY` CRS cannot contain a comma,
including in nested columns, because Iceberg's geospatial type grammar uses
commas to separate parameters.
-- Iceberg REST catalog publication does not yet support geospatial columns.
Use `table-location`, `hadoop-catalog`, or `hive-catalog` metadata storage
instead.
+## What Iceberg Readers See
+
+| Paimon table | Files eligible for incremental publication | When changes
become visible |
+| --- | --- | --- |
+| Append table | Data files in the committed snapshot | After metadata
publication and reader refresh |
+| Primary key table without Iceberg deletion vectors | Files at the highest
LSM level | After full compaction, metadata publication, and reader refresh |
+| Primary key table with Iceberg v3 deletion vectors | Files above L0,
together with deletion vectors | After changes reach those files and metadata
is published and refreshed |
+
+Initial publication and metadata rebuilds use snapshot splits that can be read
directly without
+Paimon's merge logic. The table above describes subsequent incremental
publication. Use compaction
+to establish a predictable visibility boundary for primary key tables.
+
+The [primary key guide](./primary-key-table.mdx) explains both modes and their
configuration.
+Disabling an Iceberg catalog's cache can help with interactive verification,
but cannot make
+uncompacted or unpublished changes visible.
+
+:::caution Manage the table through Paimon
+
+Use the Iceberg representation for reads. Perform writes, schema changes,
compaction, snapshot
+expiration, and file cleanup through Paimon. Both representations refer to
shared data files;
+independent Iceberg mutations or cleanup can invalidate Paimon's view of the
table.
:::
+
+## Supported Types
+
+Compatibility depends on the column types, data file format, Iceberg format
version, and reader.
+See [supported data types and precision limits](./data-types.md) before
enabling publication on an
+existing table. Primary key deletion vectors and geospatial columns require
Iceberg format v3.
diff --git a/docs/docs/iceberg/primary-key-table.mdx
b/docs/docs/iceberg/primary-key-table.mdx
index fb9a23223b..2e20762241 100644
--- a/docs/docs/iceberg/primary-key-table.mdx
+++ b/docs/docs/iceberg/primary-key-table.mdx
@@ -1,5 +1,5 @@
---
-title: "Primary Key Table"
+title: "Primary Key Tables"
sidebar_position: 3
---
@@ -27,82 +27,48 @@ under the License.
# Primary Key Tables
-Let's walk through a simple example, where we query Paimon tables with Iceberg
connectors in Flink and Spark.
-Before trying out this example, make sure that your compute engine already
supports Iceberg.
-Please refer to Iceberg's document if you haven't set up Iceberg.
-* Flink: [Preparation when using Flink SQL
Client](https://iceberg.apache.org/docs/latest/flink/#preparation-when-using-flink-sql-client)
-* Spark: [Using Iceberg in Spark
3](https://iceberg.apache.org/docs/latest/spark-getting-started/#using-iceberg-in-spark-3)
+Paimon merges changes to primary key tables using its LSM storage. Iceberg
readers use the files
+and deletion information published in Iceberg metadata, so the publication
mode determines how
+soon they see an update.
-<Tabs groupId="paimon-primary-key-table">
+## Choose a Read Mode
-<TabItem value="flink-sql" label="Flink SQL">
+| Mode | Required options | Files eligible for incremental publication |
+| --- | --- | --- |
+| Full compaction | Enable `metadata.iceberg.storage` | Highest-level LSM
files |
+| Deletion vectors | Enable storage, format v3, deletion vectors, and 64-bit
bitmaps | Files above L0 plus deletion vectors |
-```sql
-CREATE CATALOG paimon_catalog WITH (
- 'type' = 'paimon',
- 'warehouse' = '<path-to-warehouse>'
-);
+
+
+Initial publication and metadata rebuilds use snapshot splits that can be read
directly without
+Paimon's merge logic. These may include files outside the incremental rules
above. Use compaction
+to establish a predictable visibility boundary.
+
+Use the full-compaction mode when the reader does not support Iceberg v3
deletion vectors. Use the
+[deletion-vector mode](#deletion-vector-support) when the reader supports it
and you need to expose
+changes without waiting for full compaction to the highest level.
+
+## Full-Compaction Example
+First [prepare the Paimon and Iceberg
catalogs](./append-table.mdx#prepare-catalogs) for Flink or Spark.
+Create the following table through the Paimon catalog.
+
+<Tabs groupId="iceberg-engine">
+<TabItem value="flink" label="Flink SQL">
+
+```sql
CREATE TABLE paimon_catalog.`default`.orders (
order_id BIGINT,
status STRING,
payment DOUBLE,
PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
- 'metadata.iceberg.storage' = 'hadoop-catalog',
- 'compaction.optimization-interval' = '1ms' -- ATTENTION: this option is
only for testing, see "timeliness" section below for more information
-);
-
-INSERT INTO paimon_catalog.`default`.orders VALUES (1, 'SUBMITTED', CAST(NULL
AS DOUBLE)), (2, 'COMPLETED', 200.0), (3, 'SUBMITTED', CAST(NULL AS DOUBLE));
-
-CREATE CATALOG iceberg_catalog WITH (
- 'type' = 'iceberg',
- 'catalog-type' = 'hadoop',
- 'warehouse' = '<path-to-warehouse>/iceberg',
- 'cache-enabled' = 'false' -- disable iceberg catalog caching to quickly
see the result
+ 'metadata.iceberg.storage' = 'hadoop-catalog'
);
-
-SELECT * FROM iceberg_catalog.`default`.orders WHERE status = 'COMPLETED';
-/*
-+----+----------------------+--------------------------------+--------------------------------+
-| op | order_id | status |
payment |
-+----+----------------------+--------------------------------+--------------------------------+
-| +I | 2 | COMPLETED |
200.0 |
-+----+----------------------+--------------------------------+--------------------------------+
-*/
-
-INSERT INTO paimon_catalog.`default`.orders VALUES (1, 'COMPLETED', 100.0);
-
-SELECT * FROM iceberg_catalog.`default`.orders WHERE status = 'COMPLETED';
-/*
-+----+----------------------+--------------------------------+--------------------------------+
-| op | order_id | status |
payment |
-+----+----------------------+--------------------------------+--------------------------------+
-| +I | 1 | COMPLETED |
100.0 |
-| +I | 2 | COMPLETED |
200.0 |
-+----+----------------------+--------------------------------+--------------------------------+
-*/
```
</TabItem>
-
-<TabItem value="spark-sql" label="Spark SQL">
-
-Start `spark-sql` with the following command line.
-
-```bash
-spark-sql --jars <path-to-paimon-jar> \
- --conf
spark.sql.catalog.paimon_catalog=org.apache.paimon.spark.SparkCatalog \
- --conf spark.sql.catalog.paimon_catalog.warehouse=<path-to-warehouse> \
- --packages org.apache.iceberg:iceberg-spark-runtime-<iceberg-version> \
- --conf
spark.sql.catalog.iceberg_catalog=org.apache.iceberg.spark.SparkCatalog \
- --conf spark.sql.catalog.iceberg_catalog.type=hadoop \
- --conf
spark.sql.catalog.iceberg_catalog.warehouse=<path-to-warehouse>/iceberg \
- --conf spark.sql.catalog.iceberg_catalog.cache-enabled=false \ # disable
iceberg catalog caching to quickly see the result
- --conf
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
-```
-
-Run the following Spark SQL to create Paimon table, insert/update data, and
query with Iceberg catalog.
+<TabItem value="spark" label="Spark SQL">
```sql
CREATE TABLE paimon_catalog.`default`.orders (
@@ -111,186 +77,162 @@ CREATE TABLE paimon_catalog.`default`.orders (
payment DOUBLE
) TBLPROPERTIES (
'primary-key' = 'order_id',
- 'metadata.iceberg.storage' = 'hadoop-catalog',
- 'compaction.optimization-interval' = '1ms' -- ATTENTION: this option is
only for testing, see "timeliness" section below for more information
+ 'metadata.iceberg.storage' = 'hadoop-catalog'
);
+```
+
+</TabItem>
+</Tabs>
+
+Run these statements in either engine. Explicit full compaction makes the
visibility boundary
+clear without configuring a very short production compaction interval. Wait
for the compaction
+job to finish before querying.
+
+```sql
+INSERT INTO paimon_catalog.`default`.orders VALUES
+ (1, 'SUBMITTED', CAST(NULL AS DOUBLE)),
+ (2, 'COMPLETED', 200.0),
+ (3, 'SUBMITTED', CAST(NULL AS DOUBLE));
+
+CALL paimon_catalog.sys.compact(`table` => 'default.orders', compact_strategy
=> 'full');
+
+SELECT order_id, status, payment
+FROM iceberg_catalog.`default`.orders
+WHERE status = 'COMPLETED'
+ORDER BY order_id;
+```
-INSERT INTO paimon_catalog.`default`.orders VALUES (1, 'SUBMITTED', CAST(NULL
AS DOUBLE)), (2, 'COMPLETED', 200.0), (3, 'SUBMITTED', CAST(NULL AS DOUBLE));
+```text
+order_id status payment
+2 COMPLETED 200.0
+```
-SELECT * FROM iceberg_catalog.`default`.orders WHERE status = 'COMPLETED';
-/*
-2 COMPLETED 200.0
-*/
+Update order 1 through Paimon, compact again, and read the result:
+```sql
INSERT INTO paimon_catalog.`default`.orders VALUES (1, 'COMPLETED', 100.0);
-SELECT * FROM iceberg_catalog.`default`.orders WHERE status = 'COMPLETED';
-/*
-2 COMPLETED 200.0
-1 COMPLETED 100.0
-*/
+CALL paimon_catalog.sys.compact(`table` => 'default.orders', compact_strategy
=> 'full');
+
+SELECT order_id, status, payment
+FROM iceberg_catalog.`default`.orders
+WHERE status = 'COMPLETED'
+ORDER BY order_id;
```
-</TabItem>
+```text
+order_id status payment
+1 COMPLETED 100.0
+2 COMPLETED 200.0
+```
-</Tabs>
+### Timeliness
-Paimon primary key tables organize data files as LSM trees, so data files must
be merged in memory before querying.
-However, Iceberg readers are not able to merge data files, so they can only
query data files on the highest level of LSM trees.
-Data files on the highest level are produced by the full compaction process.
-So **to conclude, for primary key tables, Iceberg readers can only query data
after full compaction**.
-
-By default, there is no guarantee on how frequently Paimon will perform full
compaction.
-You can configure the following table option, so that Paimon is forced to
perform full compaction after several commits.
-
-<table className="table table-bordered">
- <thead>
- <tr>
- <th className="text-left" style={{width: "20%"}}>Option</th>
- <th className="text-left" style={{width: "5%"}}>Default</th>
- <th className="text-left" style={{width: "10%"}}>Type</th>
- <th className="text-left" style={{width: "60%"}}>Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>compaction.optimization-interval</h5></td>
- <td style={{wordWrap: "break-word"}}>(none)</td>
- <td>Duration</td>
- <td>Full compaction will be constantly triggered per time interval.
First compaction after the job starts will always be full compaction.</td>
- </tr>
- <tr>
- <td><h5>full-compaction.delta-commits</h5></td>
- <td style={{wordWrap: "break-word"}}>(none)</td>
- <td>Integer</td>
- <td>Full compaction will be constantly triggered after delta commits.
Only implemented in Flink.</td>
- </tr>
- </tbody>
-</table>
-
-Note that full compaction is a resource-consuming process, so the value of
this table option should not be too small.
-We recommend full compaction to be performed once or twice per hour.
+Without Iceberg deletion vectors, **incremental publication selects only the
highest LSM level**. A completed Paimon
+write alone does not guarantee that Iceberg can see the change. Full
compaction moves the merged
+result to that level.
-## Deletion Vector Support
+For continuous workloads, choose a compaction schedule based on the required
freshness and the
+cost of rewriting data:
-[Deletion vectors](../concepts/spec/tableindex#deletion-vectors) in Paimon are
used to store deleted records for each file.
-Under deletion-vector mode, paimon readers can directly filter out unnecessary
records during reading phase without merging data.
-Fortunately, Iceberg has supported [deletion
vectors](https://iceberg.apache.org/spec/?h=deletion#deletion-vectors) in
[Version 3](https://iceberg.apache.org/spec/?h=deletion#version-3).
-This means that if the Iceberg reader can recognize Paimon's deletion vectors,
it will be able to read all of Paimon's data, even without the ability to merge
data files.
-With Paimon's deletion vectors synchronized to Iceberg, Iceberg reader and
Paimon reader can achieve true real-time synchronization.
+| Option | Purpose |
+| --- | --- |
+| `compaction.optimization-interval` | Trigger optimization compaction based
on an elapsed interval |
+| `full-compaction.delta-commits` | Trigger full compaction after a number of
delta commits in Flink streaming writes |
+Neither option has a default interval/count. See [dedicated
compaction](../maintenance/dedicated-compaction.mdx)
+for scheduling and resource configuration. Reader caching and catalog
publication add to the
+end-to-end delay.
-If the following conditions are met, it will construct metadata about Paimon's
deletion vectors for Iceberg.
-* '`deletion-vectors.enabled`' and '`deletion-vectors.bitmap64`' should be set
to true. Because only 64-bit bitmap implementation of deletion vector in Paimon
is compatible with Iceberg.
-* '`metadata.iceberg.format-version`'(default value is 2) should be set to 3.
Because Iceberg only supports deletion vector in V3.
-* Version of Iceberg should be 1.8.0+.
-* JDK version should be 11+. Iceberg has stopped supporting JDK 8 since
version 1.7.0.
+## Deletion Vector Support
-Here is an example:
-<Tabs groupId="deletion-vector-table">
+Paimon can publish [deletion
vectors](../concepts/spec/tableindex.md#deletion-vectors) in the
+[Iceberg v3 format](https://iceberg.apache.org/spec/#deletion-vectors). A
deletion vector marks
+obsolete row positions so a compatible reader can filter them without merging
all versions of a key.
-<TabItem value="flink-sql" label="Flink SQL">
+Set all three options when creating the table:
```sql
--- flink version: 1.20.1
+'metadata.iceberg.format-version' = '3',
+'deletion-vectors.enabled' = 'true',
+'deletion-vectors.bitmap64' = 'true'
+```
-CREATE CATALOG paimon_catalog WITH (
- 'type' = 'paimon',
- 'warehouse' = '<path-to-warehouse>'
-);
+Use an Iceberg reader with v3 deletion-vector support. Iceberg introduced this
support in
+[1.8.0](https://iceberg.apache.org/releases/#180-release);
+verify support in the actual engine connector as well. Iceberg 1.8.x requires
JDK 11 or later.
--- Create a paimon table with primary key and enable deletion vector
-CREATE TABLE paimon_catalog.`default`.T
-(
- pt INT
- ,k INT
- ,v INT
- ,PRIMARY KEY (pt, k) NOT ENFORCED
-)PARTITIONED BY (pt)
-WITH (
- 'metadata.iceberg.storage' = 'hadoop-catalog'
- ,'metadata.iceberg.format-version' = '3'
- ,'deletion-vectors.enabled' = 'true'
- ,'deletion-vectors.bitmap64' = 'true'
-);
+:::info Visibility still depends on publication
+
+During incremental publication in this mode, Paimon selects primary key data
files with a level
+greater than zero. L0 files are not included in those incremental updates.
Changes must reach
+eligible files and their deletion vectors must be committed and
+published before a refreshed Iceberg reader sees them. This mode removes the
requirement to compact
+all the way to the highest level; it does not guarantee immediate visibility
of every write.
-INSERT INTO paimon_catalog.`default`.T
-VALUES (1, 9, 90), (1, 10, 100), (1, 11, 110), (2, 20, 200)
-;
+:::
+
+### Example: Read an Updated Key
--- iceberg version: 1.8.1
-CREATE CATALOG iceberg_catalog WITH (
- 'type' = 'iceberg',
- 'catalog-type' = 'hadoop',
- 'warehouse' = '<path-to-warehouse>/iceberg',
- 'cache-enabled' = 'false' -- disable iceberg catalog caching to quickly
see the result
+Use the Flink catalogs and batch settings from [the catalog
setup](./append-table.mdx#prepare-catalogs),
+with an Iceberg runtime that supports deletion vectors.
+
+```sql
+CREATE TABLE paimon_catalog.`default`.orders_dv (
+ order_id BIGINT,
+ status STRING,
+ PRIMARY KEY (order_id) NOT ENFORCED
+) WITH (
+ 'metadata.iceberg.storage' = 'hadoop-catalog',
+ 'metadata.iceberg.format-version' = '3',
+ 'deletion-vectors.enabled' = 'true',
+ 'deletion-vectors.bitmap64' = 'true'
);
-SELECT * FROM iceberg_catalog.`default`.T;
-/*
-+------------+------------+------------+
-| pt | k | v |
-+------------+------------+------------+
-| 2 | 20 | 200 |
-| 1 | 9 | 90 |
-| 1 | 10 | 100 |
-| 1 | 11 | 110 |
-+------------+------------+------------+
-*/
-
--- insert some data again, this will generate deletion vectors
-INSERT INTO paimon_catalog.`default`.T
-VALUES (1, 10, 101), (2, 20, 201), (1, 12, 121)
-;
-
--- select deletion-vector index in paimon
-SELECT * FROM paimon_catalog.`default`.`T$table_indexes` WHERE
index_type='DELETION_VECTORS';
-/*
-+------------+-----------+-------------------+------------------------
-----+------------+------------+--------------------------------+
-| partition | bucket | index_type | file_name
| file_size | row_count | dv_ranges |
-+------------+-----------+-------------------+------------------------
-----+------------+------------+--------------------------------+
-| {1} | 0 | DELETION_VECTORS | index-4ae44c5d-2fc6-40b0-9ff0~
| 43 | 1 | [(data-968fdf3a-2f44-41df-89b~ |
-+------------+-----------+-------------------+------------------------
-----+------------+------------+--------------------------------+
-*/
-
--- select in iceberg, the updates was successfully read by iceberg
-SELECT * FROM iceberg_catalog.`default`.T;
-/*
-+------------+------------+------------+
-| pt | k | v |
-+------------+------------+------------+
-| 1 | 9 | 90 |
-| 1 | 11 | 110 |
-| 2 | 20 | 201 |
-| 1 | 10 | 101 |
-| 1 | 12 | 121 |
-+------------+------------+------------+
-*/
+INSERT INTO paimon_catalog.`default`.orders_dv VALUES
+ (1, 'SUBMITTED'), (2, 'COMPLETED');
+INSERT INTO paimon_catalog.`default`.orders_dv VALUES (1, 'COMPLETED');
+
+SELECT order_id, status
+FROM iceberg_catalog.`default`.orders_dv
+ORDER BY order_id;
```
-</TabItem>
+Once the writes and metadata publication complete, the expected rows are:
-</Tabs>
+```text
+order_id status
+1 COMPLETED
+2 COMPLETED
+```
-:::info
+To inspect the deletion-vector indexes on the Paimon side:
-note1: Upgrade the implementation of deletion vector to 64-bit bitmap if
necessary.
+```sql
+SELECT * FROM paimon_catalog.`default`.`orders_dv$table_indexes`
+WHERE index_type = 'DELETION_VECTORS';
+```
-:::
+### Existing Tables with 32-Bit Deletion Vectors
-If your paimon table has already been in deletion-vector mode, but 32-bit
bitmap was used for deletion vector.
-You need to upgrade the implementation of deletion vector to 64-bit bitmap if
you want to synchronize deletion-vector metadata to iceberg.
-You can follow the following steps to upgrade to 64-bit deletion-vector:
-1. stop all the writing jobs of your paimon table.
-2. perform a [full
compaction](../maintenance/dedicated-compaction#dedicated-compaction-job) to
your paimon table.
-3. run `ALTER TABLE tableName SET ('deletion-vectors.bitmap64' = 'true')` to
upgrade to 64-bit deletion vector.
-4. restart your writing job. If meeting the all the conditions mentioned
above, deletion vector metadata will be synchronized to iceberg.
+Changing `deletion-vectors.bitmap64` does not convert existing 32-bit index
files. Before enabling
+Iceberg deletion-vector publication:
-:::info
+1. Stop all writers to the table.
+2. Run a [full
compaction](../maintenance/dedicated-compaction.mdx#dedicated-compaction-job)
and wait for it to finish.
+3. Set `deletion-vectors.bitmap64` to `true` and
`metadata.iceberg.format-version` to `3`.
+4. Restart the writers with the updated options and verify reads through the
Iceberg connector.
-note2: Upgrade the format version of iceberg to 3 if necessary.
+For Flink SQL, step 3 is:
-:::
-You can upgrade the format version of iceberg from 2 to 3 by setting
`'metadata.iceberg.format-version' = '3'`.
-This will recreate the iceberg metadata without using the base metadata.
+```sql
+ALTER TABLE paimon_catalog.`default`.orders_dv SET (
+ 'deletion-vectors.bitmap64' = 'true',
+ 'metadata.iceberg.format-version' = '3'
+);
+```
+
+Switching from format v2 to v3 rebuilds the Iceberg metadata on the next
publication instead of
+reusing the v2 base. Verify the resulting snapshots and
[tags](./iceberg-tags.md), and check every
+reader's v3 support before upgrading.
diff --git a/docs/docs/iceberg/rest-catalog.mdx
b/docs/docs/iceberg/rest-catalog.mdx
index 21d6fbc8fc..859663accd 100644
--- a/docs/docs/iceberg/rest-catalog.mdx
+++ b/docs/docs/iceberg/rest-catalog.mdx
@@ -1,6 +1,6 @@
---
-title: "Rest Catalog"
-sidebar_position: 5
+title: "REST Catalog"
+sidebar_position: 6
---
import Stable from '@site/src/components/Stable';
@@ -25,112 +25,134 @@ specific language governing permissions and limitations
under the License.
-->
-# Rest Catalog
+# REST Catalog
-When creating Paimon table, set `'metadata.iceberg.storage' = 'rest-catalog'`.
-This option value will not only store Iceberg metadata like hadoop-catalog,
but also create table in [iceberg rest
catalog](https://iceberg.apache.org/terms/#decoupling-using-the-rest-catalog).
-This Paimon table can be accessed from Iceberg Rest catalog later.
+Use `rest-catalog` to publish the Iceberg representation to an Iceberg REST
catalog service.
+Paimon first writes Iceberg metadata to its configured filesystem location,
then commits the
+corresponding metadata to the REST catalog.
-You need to provide information about Rest Catalog by setting options prefixed
with `'metadata.iceberg.rest.'`, such as
-`'metadata.iceberg.rest.uri' = 'https://localhost/'`. Paimon will try to use
these options to initialize an iceberg rest catalog,
-and use this rest catalog to commit metadata.
+## Dependencies
-:::warning
+The Paimon writer needs the `paimon-iceberg` JAR in addition to its engine
connector. The REST
+module requires JDK 11 or later.
-Tables containing `GEOMETRY` or `GEOGRAPHY` columns cannot use
`'metadata.iceberg.storage' = 'rest-catalog'` because the bundled Iceberg REST
client cannot parse Iceberg v3 geospatial types. Use `table-location`,
`hadoop-catalog`, or `hive-catalog` metadata storage instead.
+<Stable>
-:::
+Download
[paimon-iceberg-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-iceberg/@@VERSION@@/paimon-iceberg-@@VERSION@@.jar).
-**Dependency:**
+</Stable>
+<Unstable>
-This feature needs dependency:
+Download the matching [paimon-iceberg snapshot
JAR](https://repository.apache.org/snapshots/org/apache/paimon/paimon-iceberg/@@VERSION@@/).
-<Stable>
+</Unstable>
+For Flink, install the JAR on the cluster and SQL client classpaths before
starting them.
+For Spark, include it in `--jars` alongside the Paimon Spark JAR. Iceberg
readers still need
+their own engine-compatible Iceberg runtime.
+To build this module from the repository root with JDK 11 or later, activate
its Maven profile:
-[paimon-iceberg-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-iceberg/@@VERSION@@/paimon-iceberg-@@VERSION@@.jar),
+```bash
+mvn -Ppaimon-iceberg -pl paimon-iceberg -am -DskipTests package
+```
+The bundled JAR is written to
`paimon-iceberg/target/paimon-iceberg-@@VERSION@@.jar`.
+## Configure the Service
-</Stable>
+Paimon passes table options prefixed with `metadata.iceberg.rest.` to the
Iceberg REST client after
+removing the prefix. For example:
-<Unstable>
+| Paimon table option | REST client property |
+| --- | --- |
+| `metadata.iceberg.rest.uri` | `uri` |
+| `metadata.iceberg.rest.warehouse` | `warehouse` |
+| `metadata.iceberg.rest.clients` | `clients` |
+Set the endpoint and any warehouse or authentication properties required by
your service. Configure
+the reader independently with the corresponding Iceberg catalog properties.
Both sides must select
+the same REST warehouse and namespace.
+## Publish and Read an Append Table
-[paimon-iceberg-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-iceberg/@@VERSION@@/),
+This Flink example uses an append table so you can verify REST publication
without first configuring
+primary key compaction. Replace the placeholders and add the authentication
settings your service
+requires to both catalogs.
+```sql
+SET 'execution.runtime-mode' = 'batch';
+SET 'table.dml-sync' = 'true';
+CREATE CATALOG paimon_catalog WITH (
+ 'type' = 'paimon',
+ 'warehouse' = '<path-to-paimon-warehouse>'
+);
-</Unstable>
+CREATE DATABASE IF NOT EXISTS paimon_catalog.`default`;
-and JDK version should be 11+.
+CREATE TABLE paimon_catalog.`default`.cities_rest (
+ country STRING,
+ name STRING
+) WITH (
+ 'metadata.iceberg.storage' = 'rest-catalog',
+ 'metadata.iceberg.rest.uri' = 'https://<rest-catalog-host>',
+ 'metadata.iceberg.rest.warehouse' = '<rest-warehouse>'
+);
-You can also manually build the jar from the source code.(need JDK 11+)
+INSERT INTO paimon_catalog.`default`.cities_rest VALUES
+ ('germany', 'berlin'), ('germany', 'hamburg');
-To build from source code, [clone the git repository](@@GITHUB_REPO@@).
+CREATE CATALOG iceberg_rest WITH (
+ 'type' = 'iceberg',
+ 'catalog-type' = 'rest',
+ 'uri' = 'https://<rest-catalog-host>',
+ 'warehouse' = '<rest-warehouse>',
+ 'cache-enabled' = 'false'
+);
-Build bundled jar with the following command.
-- `mvn clean install -DskipTests`
+SELECT country, name FROM iceberg_rest.`default`.cities_rest ORDER BY name;
+```
-You can find the jar in
`./paimon-iceberg/target/paimon-iceberg-@@VERSION@@.jar`.
+```text
+country name
+germany berlin
+germany hamburg
+```
-**Example:**
+For primary key tables, also select a [compaction or deletion-vector
mode](./primary-key-table.mdx).
+The REST service does not remove those visibility requirements.
-Here is an example using flink sql:
-```sql
--- create a paimon table
-CREATE TABLE `paimon`.`default`.`T` (
- pt INT,
- k INT,
- v INT,
- PRIMARY KEY (pt, k) NOT ENFORCED
-) PARTITIONED BY (pt) WITH (
- 'metadata.iceberg.storage' = 'rest-catalog',
- 'metadata.iceberg.rest.uri' = 'http://localhost:55807/',
- 'metadata.iceberg.rest.warehouse' = 'rck_warehouse',
- 'metadata.iceberg.rest.clients' = '1'
-);
+## Publication and Recovery
--- insert some data
-INSERT INTO `paimon`.`default`.`T` VALUES(1, 9, 90),(1, 10, 100),(1, 11,
110),(2, 20, 200);
-
--- create an iceberg rest catalog
-CREATE CATALOG `iceberg` WITH (
- 'type' = 'iceberg',
- 'catalog-type' = 'rest',
- 'uri' = 'http://localhost:55807/',
- 'clients' = '1',
- 'cache-enabled' = 'false'
-)
-
--- verify the data in iceberg rest-catalog
-SELECT v, k, pt FROM `iceberg`.`default`.`T` ORDER BY pt, k;
-/*
-the query results:
- 90, 9, 1
-100, 10, 1
-110, 11, 1
-200, 20, 2
-*/
-```
+The local metadata uses the [separate catalog
layout](./catalogs.md#metadata-layout) by default.
+REST publication uses Paimon's generated metadata as its source of truth:
-**Schema compatibility and Partition evolution:**
+| REST table state | Publication behavior |
+| --- | --- |
+| Table is absent | Create or register the table, then publish its state |
+| The same snapshot and commit identity are already published | Treat the
retry as complete |
+| Table exists without a snapshot after an incomplete creation | Complete
publication without an unnecessary drop/create cycle |
+| Current state matches the expected base | Commit the metadata update |
+| Current state conflicts with the expected base | Rebuild or re-register the
Iceberg representation from Paimon's metadata |
-There is a fundamental difference between Paimon and Iceberg regarding the
starting fieldId. Paimon uses fieldId 0, while Iceberg uses fieldId 1. If we
create an Iceberg table using a Paimon schema directly, it will shift all
fieldIds by +1, causing field disorder. However, it is possible to update the
schema after table creation and start the schema from fieldId 0.
+Recovery can remove and recreate the REST catalog entry. Removal uses
non-purging semantics: it
+does not delete the shared data files. Some v3 recovery paths must register a
metadata file directly
+to preserve row IDs; Paimon checks the service's registration support before
removing the existing entry.
-Table creation attempts to minimize issues with fieldId disorder and partition
evolution by following a 2 option logic:
+Keep writes and table maintenance in Paimon. Independent changes through the
Iceberg catalog can
+conflict with later publication and be replaced during recovery.
-- Partition fieldId = 0: Paimon creates an empty schema first and then updates
the schema to the actual one. Partition evolution is unavoidable.
-- Partition fieldId > 0: Paimon creates an initial dummy schema first,
offsetting partition fields correctly, and then updates the schema to the
actual one, avoiding partition evolution.
+## Schema and Feature Compatibility
-**Note:**
+Paimon field IDs start at zero, while Iceberg table creation assigns IDs
starting at one. Paimon
+handles this difference during REST table creation and schema publication. If
the first field is
+also a partition field, creation may require an intermediate unpartitioned
schema and partition
+spec evolution. Do not manually renumber the fields in the Iceberg
representation.
-Paimon will firstly write iceberg metadata in a separate directory like
hadoop-catalog, and then commit metadata to iceberg rest catalog.
-If the two are incompatible, we take the metadata stored in the separate
directory as the reference.
+Tables containing `GEOMETRY` or `GEOGRAPHY` columns cannot use REST
publication because the bundled
+REST client cannot parse those Iceberg v3 types. Use Hadoop, Hive, or
table-location storage;
+see [data types](./data-types.md).
-There are some cases when committing to iceberg rest catalog:
-1. table not exists in iceberg rest-catalog. It'll create the table in rest
catalog first, and commit metadata.
-2. table exists in iceberg rest-catalog and is compatible with the base
metadata stored in the separate directory. It'll directly get the table and
commit metadata.
-3. table exists, and isn't compatible with the base metadata stored in the
separate directory. It'll **drop the table and recreate the table**, then
commit metadata.
+Paimon tag creation and deletion are not guaranteed to propagate to the REST
catalog. See
+[tags and publication limits](./iceberg-tags.md).
diff --git a/docs/sidebars.js b/docs/sidebars.js
index 099fb7f0b7..89b2926a90 100644
--- a/docs/sidebars.js
+++ b/docs/sidebars.js
@@ -631,19 +631,33 @@ const sidebars = {
},
{
type: "category",
- "label": "Iceberg Metadata",
+ "label": "Iceberg Compatibility",
"collapsed": true,
"link": {
type: "doc",
"id": "iceberg/index"
},
"items": [
- "iceberg/append-table",
- "iceberg/primary-key-table",
+ {
+ type: "category",
+ "label": "Read Tables",
+ "items": [
+ "iceberg/append-table",
+ "iceberg/primary-key-table"
+ ]
+ },
+ {
+ type: "category",
+ "label": "Catalogs",
+ "link": { type: "doc", "id": "iceberg/catalogs" },
+ "items": [
+ "iceberg/hive-catalog",
+ "iceberg/rest-catalog"
+ ]
+ },
"iceberg/iceberg-tags",
- "iceberg/hive-catalog",
- "iceberg/rest-catalog",
"iceberg/ecosystem",
+ "iceberg/data-types",
"iceberg/configurations"
]
},
diff --git a/docs/static/img/iceberg-metadata-layout.svg
b/docs/static/img/iceberg-metadata-layout.svg
new file mode 100644
index 0000000000..98bb9f7e11
--- /dev/null
+++ b/docs/static/img/iceberg-metadata-layout.svg
@@ -0,0 +1,52 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="960" height="534" viewBox="0 0
960 534" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">Choose where metadata lives</title>
+<desc id="desc">With catalog-location, Iceberg metadata is under
warehouse/iceberg/default/cities/metadata. With table-location, metadata is
under warehouse/default.db/cities/metadata. Both layouts reference Paimon data
files in the original table.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif">
+<rect x="1" y="1" width="958" height="532" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">Choose where metadata lives</text>
+<text x="28" y="75" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Example table: <warehouse>/default.db/cities</text>
+<rect x="28" y="103" width="440" height="360" rx="10" fill="#f8fafc"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="48" y="139" font-size="23" font-weight="700" text-anchor="start"
fill="#2463b4">catalog-location</text>
+<text x="48" y="167" font-size="17" font-weight="400" text-anchor="start"
fill="#526277">Default for Hadoop, Hive, and REST</text>
+<text x="48" y="210" font-size="18" font-weight="700" text-anchor="start"
fill="#172b4d" font-family="monospace"><warehouse>/</text>
+<rect x="492" y="103" width="440" height="360" rx="10" fill="#f8fafc"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="512" y="139" font-size="23" font-weight="700" text-anchor="start"
fill="#2463b4">table-location</text>
+<text x="512" y="167" font-size="17" font-weight="400" text-anchor="start"
fill="#526277">Store metadata alongside the table</text>
+<text x="512" y="210" font-size="18" font-weight="700" text-anchor="start"
fill="#172b4d" font-family="monospace"><warehouse>/</text>
+<text x="66" y="248" font-size="18" font-weight="400" text-anchor="start"
fill="#172b4d" font-family="monospace">default.db/cities/</text>
+<rect x="82" y="264" width="340" height="49" rx="7" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="100" y="295" font-size="18" font-weight="700" text-anchor="start"
fill="#10705d">Paimon data files</text>
+<text x="66" y="355" font-size="18" font-weight="400" text-anchor="start"
fill="#172b4d" font-family="monospace">iceberg/default/cities/</text>
+<rect x="82" y="371" width="340" height="49" rx="7" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="100" y="402" font-size="18" font-weight="700" text-anchor="start"
fill="#2463b4" font-family="monospace">metadata/</text>
+<text x="530" y="248" font-size="18" font-weight="400" text-anchor="start"
fill="#172b4d" font-family="monospace">default.db/cities/</text>
+<rect x="546" y="264" width="340" height="49" rx="7" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="564" y="295" font-size="18" font-weight="700" text-anchor="start"
fill="#10705d">Paimon data files</text>
+<rect x="546" y="371" width="340" height="49" rx="7" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="564" y="402" font-size="18" font-weight="700" text-anchor="start"
fill="#2463b4" font-family="monospace">metadata/</text>
+<path d="M 422 395 H 448 V 289 H 428" fill="none" stroke="#526277"
stroke-width="2" stroke-dasharray="6 5" marker-end="url(#arrow)"/>
+<text x="248" y="446" font-size="16" font-weight="400" text-anchor="middle"
fill="#526277">Metadata references the original data files</text>
+<path d="M 886 395 H 912 V 289 H 892" fill="none" stroke="#526277"
stroke-width="2" stroke-dasharray="6 5" marker-end="url(#arrow)"/>
+<text x="712" y="446" font-size="16" font-weight="400" text-anchor="middle"
fill="#526277">Metadata references the original data files</text>
+<text x="480" y="503" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">Catalog selection controls discovery. The layout option controls
metadata paths.</text>
+</g>
+</svg>
diff --git a/docs/static/img/iceberg-primary-key-visibility.svg
b/docs/static/img/iceberg-primary-key-visibility.svg
new file mode 100644
index 0000000000..fbbd09e45b
--- /dev/null
+++ b/docs/static/img/iceberg-primary-key-visibility.svg
@@ -0,0 +1,53 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="960" height="558" viewBox="0 0
960 558" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">When primary key changes reach Iceberg</title>
+<desc id="desc">For incremental publication without Iceberg deletion vectors,
full compaction makes highest-level files eligible. With v3 deletion vectors,
files above L0 and deletion vectors are eligible. Initial publication and
metadata rebuilds use raw-readable snapshot splits.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif">
+<rect x="1" y="1" width="958" height="556" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">When primary key changes reach Iceberg</text>
+<text x="28" y="75" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">File eligibility during incremental metadata publication</text>
+<rect x="28" y="102" width="440" height="362" rx="10" fill="#f8fafc"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="48" y="140" font-size="23" font-weight="700" text-anchor="start"
fill="#172b4d">Full-compaction mode</text>
+<rect x="48" y="162" width="400" height="71" rx="10" fill="#fff2d7"
stroke="#9a5200" stroke-width="1.5"/>
+<text x="248.0" y="194" font-size="21" font-weight="700" text-anchor="middle"
fill="#9a5200">L0: pending changes</text>
+<rect x="492" y="102" width="440" height="362" rx="10" fill="#f8fafc"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="512" y="140" font-size="23" font-weight="700" text-anchor="start"
fill="#172b4d">Iceberg v3 deletion vectors</text>
+<rect x="512" y="162" width="400" height="71" rx="10" fill="#fff2d7"
stroke="#9a5200" stroke-width="1.5"/>
+<text x="712.0" y="194" font-size="21" font-weight="700" text-anchor="middle"
fill="#9a5200">L0: pending changes</text>
+<path d="M 248 233 V 291" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="266" y="270" font-size="17" font-weight="400" text-anchor="start"
fill="#526277">full compaction</text>
+<path d="M 712 233 V 291" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="730" y="259" font-size="17" font-weight="400" text-anchor="start"
fill="#526277">lookup /</text>
+<text x="730" y="281" font-size="17" font-weight="400" text-anchor="start"
fill="#526277">compaction</text>
+<rect x="48" y="297" width="400" height="83" rx="10" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="248.0" y="329" font-size="21" font-weight="700" text-anchor="middle"
fill="#10705d">Highest LSM level</text>
+<text x="248.0" y="357" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Merged rows</text>
+<rect x="512" y="297" width="400" height="83" rx="10" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="712.0" y="329" font-size="21" font-weight="700" text-anchor="middle"
fill="#10705d">Files above L0 + deletion vectors</text>
+<text x="712.0" y="357" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Obsolete row positions are filtered</text>
+<path d="M 248 380 V 409" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<path d="M 712 380 V 409" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="248" y="443" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">Publish metadata → refresh reader</text>
+<text x="712" y="443" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">Publish metadata → refresh reader</text>
+<text x="28" y="504" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Initial publication and rebuilds use raw-readable snapshot
splits.</text>
+<text x="28" y="532" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Use compaction to establish a predictable visibility
boundary.</text>
+</g>
+</svg>
diff --git a/docs/static/img/iceberg-publication.svg
b/docs/static/img/iceberg-publication.svg
new file mode 100644
index 0000000000..46dc123dc4
--- /dev/null
+++ b/docs/static/img/iceberg-publication.svg
@@ -0,0 +1,56 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="960" height="500" viewBox="0 0
960 500" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">One data set, two read paths</title>
+<desc id="desc">A Paimon writer commits a snapshot and publishes Iceberg
metadata. Paimon and Iceberg readers both read the original data files. Hive or
REST publication also updates an external catalog.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif">
+<rect x="1" y="1" width="958" height="498" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">One data set, two read paths</text>
+<text x="28" y="75" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Paimon owns writes and maintenance. Iceberg metadata exposes a
readable view.</text>
+<rect x="28" y="108" width="220" height="90" rx="10" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="138.0" y="140" font-size="21" font-weight="700" text-anchor="middle"
fill="#2463b4">Paimon writer</text>
+<text x="138.0" y="168" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Flink / Spark</text>
+<path d="M 248 153 H 304" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<rect x="310" y="108" width="270" height="90" rx="10" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="445.0" y="140" font-size="21" font-weight="700" text-anchor="middle"
fill="#2463b4">1. Commit snapshot</text>
+<text x="445.0" y="168" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Paimon metadata</text>
+<path d="M 580 153 H 636" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<rect x="642" y="108" width="290" height="114" rx="10" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="787.0" y="140" font-size="21" font-weight="700" text-anchor="middle"
fill="#10705d">2. Publish metadata</text>
+<text x="787.0" y="168" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Iceberg manifests + snapshots</text>
+<text x="787.0" y="192" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Hive / REST registration</text>
+<path d="M 445 198 V 326" fill="none" stroke="#526277" stroke-width="2"
stroke-dasharray="6 5" marker-end="url(#arrow)"/>
+<text x="458" y="267" font-size="16" font-weight="400" text-anchor="start"
fill="#526277">references</text>
+<rect x="28" y="332" width="220" height="90" rx="10" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="138.0" y="364" font-size="21" font-weight="700" text-anchor="middle"
fill="#2463b4">Paimon reader</text>
+<text x="138.0" y="392" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Load Paimon metadata</text>
+<rect x="310" y="332" width="340" height="90" rx="10" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="480.0" y="364" font-size="21" font-weight="700" text-anchor="middle"
fill="#10705d">Shared data files</text>
+<text x="480.0" y="392" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Remain in the Paimon table</text>
+<rect x="712" y="332" width="220" height="90" rx="10" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="822.0" y="364" font-size="21" font-weight="700" text-anchor="middle"
fill="#2463b4">Iceberg reader</text>
+<text x="822.0" y="392" font-size="17" font-weight="400" text-anchor="middle"
fill="#526277">Load published metadata</text>
+<path d="M 248 377 H 304" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<path d="M 712 377 H 656" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<path d="M 822 222 V 326" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="837" y="274" font-size="16" font-weight="400" text-anchor="start"
fill="#526277">refresh</text>
+<text x="480" y="465" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">Publication creates metadata; it does not duplicate the table’s
data.</text>
+</g>
+</svg>
