This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/amoro.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f17795257 Regenerated Amoro latest docs page based on
6af75bbe23b79974900997b85f9629d7c1623311 commit
f17795257 is described below
commit f177952576a3749fb4f1d99c25f4bef7a8d22625
Author: GitHub Actions <[email protected]>
AuthorDate: Wed Oct 21 07:15:35 2026 +0000
Regenerated Amoro latest docs page based on
6af75bbe23b79974900997b85f9629d7c1623311 commit
---
.../docs/latest/flink-getting-started/index.html | 11 +-
output/docs/latest/iceberg-format/index.html | 28 +-
output/docs/latest/index.html | 351 ++++++++++++++++++++-
output/docs/latest/index.xml | 235 +++++++++++++-
output/docs/latest/managing-catalogs/index.html | 40 +++
output/docs/latest/managing-optimizers/index.html | 27 +-
output/docs/latest/search.json | 2 +-
output/docs/latest/sitemap.xml | 80 ++++-
8 files changed, 749 insertions(+), 25 deletions(-)
diff --git a/output/docs/latest/flink-getting-started/index.html
b/output/docs/latest/flink-getting-started/index.html
index 3fa591fe9..97c4470ab 100644
--- a/output/docs/latest/flink-getting-started/index.html
+++ b/output/docs/latest/flink-getting-started/index.html
@@ -703,15 +703,16 @@ for more information.</p>
<p><code>./mvnw clean package -pl ':amoro-mixed-flink-runtime-1.18' -am
-DskipTests</code></p>
<p>The Flink Runtime Jar is located in the
<code>amoro-format-mixed/amoro-mixed-flink/v1.18/amoro-mixed-flink-runtime-1.18/target</code>
directory.</p>
<h2 id="environment-preparation">Environment preparation</h2>
+<p>Use Hadoop 3 client libraries; Hadoop 2 is no longer supported. Remove any
existing Hadoop 2 uber JAR from Flink’s <code>lib</code> directory before
adding the Hadoop 3 JARs.</p>
<p>Download Flink and related dependencies, and download Flink 1.18/1.19/1.20
as needed. Taking Flink 1.18 as an example:</p>
<div class="highlight"><pre tabindex="0"
style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code
class="language-shell" data-lang="shell"><span
style="display:flex;"><span><span style="color:#75715e"># Replace version value
with the latest Amoro version if needed</span>
</span></span><span style="display:flex;"><span>AMORO_VERSION<span
style="color:#f92672">=</span>0.9.0-incubating
</span></span><span style="display:flex;"><span>FLINK_VERSION<span
style="color:#f92672">=</span>1.18.1
</span></span><span style="display:flex;"><span>FLINK_MAJOR_VERSION<span
style="color:#f92672">=</span>1.18
-</span></span><span
style="display:flex;"><span>FLINK_HADOOP_SHADE_VERSION<span
style="color:#f92672">=</span>2.7.5
+</span></span><span style="display:flex;"><span>HADOOP_VERSION<span
style="color:#f92672">=</span>3.4.3
</span></span><span style="display:flex;"><span>APACHE_FLINK_URL<span
style="color:#f92672">=</span>archive.apache.org/dist/flink
</span></span><span style="display:flex;"><span>MAVEN_URL<span
style="color:#f92672">=</span>https://repo1.maven.org/maven2
-</span></span><span style="display:flex;"><span>FLINK_CONNECTOR_URL<span
style="color:#f92672">=</span><span
style="color:#e6db74">${</span>MAVEN_URL<span
style="color:#e6db74">}</span>/org/apache/flink
+</span></span><span style="display:flex;"><span>HADOOP_URL<span
style="color:#f92672">=</span><span
style="color:#e6db74">${</span>MAVEN_URL<span
style="color:#e6db74">}</span>/org/apache/hadoop
</span></span><span style="display:flex;"><span>AMORO_CONNECTOR_URL<span
style="color:#f92672">=</span><span
style="color:#e6db74">${</span>MAVEN_URL<span
style="color:#e6db74">}</span>/org/apache/amoro
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#
Download FLink binary package</span>
@@ -721,12 +722,13 @@ for more information.</p>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>cd flink-<span
style="color:#e6db74">${</span>FLINK_VERSION<span style="color:#e6db74">}</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#
Download Flink Hadoop dependency</span>
-</span></span><span style="display:flex;"><span>wget <span
style="color:#e6db74">${</span>FLINK_CONNECTOR_URL<span
style="color:#e6db74">}</span>/flink-shaded-hadoop-2-uber/<span
style="color:#e6db74">${</span>FLINK_HADOOP_SHADE_VERSION<span
style="color:#e6db74">}</span>-10.0/flink-shaded-hadoop-2-uber-<span
style="color:#e6db74">${</span>FLINK_HADOOP_SHADE_VERSION<span
style="color:#e6db74">}</span>-10.0.jar
+</span></span><span style="display:flex;"><span>wget <span
style="color:#e6db74">${</span>HADOOP_URL<span
style="color:#e6db74">}</span>/hadoop-client-api/<span
style="color:#e6db74">${</span>HADOOP_VERSION<span
style="color:#e6db74">}</span>/hadoop-client-api-<span
style="color:#e6db74">${</span>HADOOP_VERSION<span
style="color:#e6db74">}</span>.jar
+</span></span><span style="display:flex;"><span>wget <span
style="color:#e6db74">${</span>HADOOP_URL<span
style="color:#e6db74">}</span>/hadoop-client-runtime/<span
style="color:#e6db74">${</span>HADOOP_VERSION<span
style="color:#e6db74">}</span>/hadoop-client-runtime-<span
style="color:#e6db74">${</span>HADOOP_VERSION<span
style="color:#e6db74">}</span>.jar
</span></span><span style="display:flex;"><span><span style="color:#75715e">#
Download Flink Amoro Connector</span>
</span></span><span style="display:flex;"><span>wget <span
style="color:#e6db74">${</span>AMORO_CONNECTOR_URL<span
style="color:#e6db74">}</span>/amoro-mixed-format-flink-runtime-<span
style="color:#e6db74">${</span>FLINK_MAJOR_VERSION<span
style="color:#e6db74">}</span>/<span
style="color:#e6db74">${</span>AMORO_VERSION<span
style="color:#e6db74">}</span>/amoro-mixed-format-flink-runtime-<span
style="color:#e6db74">${</span>FLINK_MAJOR_VERSION<span
style="color:#e6db74">}</span>-<span s [...]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#
Copy the necessary JAR files to the lib directory</span>
-</span></span><span style="display:flex;"><span>mv
flink-shaded-hadoop-2-uber-<span
style="color:#e6db74">${</span>FLINK_HADOOP_SHADE_VERSION<span
style="color:#e6db74">}</span>-10.0.jar lib
+</span></span><span style="display:flex;"><span>mv hadoop-client-api-<span
style="color:#e6db74">${</span>HADOOP_VERSION<span
style="color:#e6db74">}</span>.jar hadoop-client-runtime-<span
style="color:#e6db74">${</span>HADOOP_VERSION<span
style="color:#e6db74">}</span>.jar lib
</span></span><span style="display:flex;"><span>mv
amoro-mixed-format-flink-runtime-<span
style="color:#e6db74">${</span>FLINK_MAJOR_VERSION<span
style="color:#e6db74">}</span>-<span
style="color:#e6db74">${</span>AMORO_VERSION<span
style="color:#e6db74">}</span>.jar lib
</span></span></code></pre></div><p>Modify Flink related configuration
files:</p>
<div class="highlight"><pre tabindex="0"
style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code
class="language-shell" data-lang="shell"><span style="display:flex;"><span>cd
flink-1.18.1
@@ -741,7 +743,6 @@ for more information.</p>
</span></span><span style="display:flex;"><span>cp
examples/table/ChangelogSocketExample.jar lib
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>cp
../amoro-mixed-flink-runtime-<span
style="color:#e6db74">${</span>FLINK_MAJOR_VERSION<span
style="color:#e6db74">}</span>-<span
style="color:#e6db74">${</span>AMORO_VERSION<span
style="color:#e6db74">}</span>.jar lib
-</span></span><span style="display:flex;"><span>cp
../flink-shaded-hadoop-2-uber-<span
style="color:#e6db74">${</span>FLINK_HADOOP_SHADE_VERSION<span
style="color:#e6db74">}</span>-10.0.jar lib
</span></span></code></pre></div><h3 id="mixed-hive-format">Mixed-Hive
format</h3>
<p>Starting from Amoro version 0.3.1, Mixed-Hive format is supported, and data
in Amoro Mixed-Hive format tables can be read/written through Flink. When
operating on Mixed-Hive format tables through Flink, the following points
should be noted:</p>
<ol>
diff --git a/output/docs/latest/iceberg-format/index.html
b/output/docs/latest/iceberg-format/index.html
index 74649bb00..41af4556f 100644
--- a/output/docs/latest/iceberg-format/index.html
+++ b/output/docs/latest/iceberg-format/index.html
@@ -686,7 +686,23 @@ However, as data and delete files are written, the read
performance and availabi
<img src="../images/formats/iceberg_format.png" alt="Iceberg Format"
width="70%" height="70%">
<p>Starting from Amoro v0.4, Iceberg format including v1 and v2 is supported.
Users only need to register Iceberg’s catalog in Amoro to host the table
for Amoro maintenance. For detailed operation steps, please refer to <a
href="../managing-catalogs/">Managing Catalogs</a>.
Amoro maintains the performance and economic availability of Iceberg tables
with minimal read/write costs through means such as small file merging,
eq-delete file conversion to pos-delete files, duplicate data elimination, and
file cleaning, and Amoro has no intrusive impact on the functionality of
Iceberg.</p>
-<p>Iceberg format has full upward and downward compatibility features, and in
general, users do not have to worry about the compatibility of the Iceberg
version used by the engine client with the Iceberg version on which Amoro
depends.</p>
+<h3 id="format-version-compatibility">Format version compatibility</h3>
+<p>Amoro automatic maintenance supports Iceberg format versions 1 and 2.
Format version 3
+is not supported by the bundled Iceberg writer: a metadata commit can omit
fields required
+by newer V3 readers and make the table inaccessible to those readers.</p>
+<p>Amoro skips optimizing evaluation and planning for V3 tables and rejects
their maintenance
+and optimizing commits. This also applies to a table upgraded while a
transaction is committing.
+Recovery checks the table version before resuming optimizing jobs. If the
catalog lookup fails,
+recovery continues with a warning; the writer still checks compatibility
before committing.
+Snapshot expiration, dangling delete cleanup, data expiration, automatic tags,
and orphan file
+cleanup are all blocked for unsupported tables.</p>
+<p>The compatibility guard does not block catalog discovery or metadata reads
and does not
+change the table’s maintenance settings. Read-only table summary
collection remains available
+when optimizing is disabled and table summary collection is enabled. Loading
and reading still
+depend on the features supported by the bundled Iceberg reader. The table
name, unsupported
+version, and supported versions are reported in AMS logs; rejected maintenance
and optimizing
+processes closed by the guard also use the existing failure or closure
reporting. This protection
+does not repair metadata already written without required V3 fields.</p>
<p>Amoro supports all catalog types supported by Iceberg, including but not
limited to: Hadoop, Hive, Glue, JDBC, Nessie, Snowflake, and so on.</p>
<p>Amoro supports all storage types supported by Iceberg, including but not
limited to: Hadoop, S3, AliyunOSS, GCS, ECS, and so on.</p>
</div>
@@ -708,7 +724,15 @@ Amoro maintains the performance and economic availability
of Iceberg tables with
<div id="toc" class="markdown-body">
<div id="full">
- <nav id="TableOfContents"></nav>
+ <nav id="TableOfContents">
+ <ul>
+ <li>
+ <ul>
+ <li><a href="#format-version-compatibility">Format version
compatibility</a></li>
+ </ul>
+ </li>
+ </ul>
+</nav>
</div>
</div>
diff --git a/output/docs/latest/index.html b/output/docs/latest/index.html
index 21f4561a9..2bbc9581f 100644
--- a/output/docs/latest/index.html
+++ b/output/docs/latest/index.html
@@ -32,14 +32,13 @@
<!DOCTYPE html>
<html>
<head>
- <meta name="generator" content="Hugo 0.166.0">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
- <title>Amoro</title>
+ <title>Introduction</title>
@@ -84,6 +83,7 @@
+ <meta name="generator" content="Hugo 0.168.0">
</head>
<body>
@@ -389,7 +389,7 @@
<li>
- <a class="top-menu" id="" href="./">
+ <a class="top-menu" id="active" href="./">
<span>Introduction</span>
</a>
</li>
@@ -399,70 +399,253 @@
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed"
data-toggle="collapse" href="#Concepts">
<span>Concepts</span>
</a>
</li>
+ <div id="Concepts" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./catalogs/">Catalogs</a>
+ </li>
+
+ <li>
+ <a href="./self-optimizing/">Self-Optimizing</a>
+ </li>
+
+ <li>
+ <a href="./table-watermark/">Table Watermark</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed"
data-toggle="collapse" href="#Configuration">
<span>Configuration</span>
</a>
</li>
+ <div id="Configuration" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./ams-config/">AMS Configuration</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed"
data-toggle="collapse" href="#Admin_Guides">
<span>Admin Guides</span>
</a>
</li>
+ <div id="Admin_Guides" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./deployment/">Deployment</a>
+ </li>
+
+ <li>
+ <a href="./deployment-on-kubernetes/">Deployment
On Kubernetes</a>
+ </li>
+
+ <li>
+ <a href="./managing-catalogs/">Managing
Catalogs</a>
+ </li>
+
+ <li>
+ <a href="./managing-optimizers/">Managing
Optimizers</a>
+ </li>
+
+ <li>
+ <a
href="./using-customized-encryption-method/">Using Customized Encryption Method
for Configurations</a>
+ </li>
+
+ <li>
+ <a href="./using-kyuubi/">Using Kyuubi By
Terminal</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed"
data-toggle="collapse" href="#User_Guides">
<span>User Guides</span>
</a>
</li>
+ <div id="User_Guides" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./using-tables/">Using Tables</a>
+ </li>
+
+ <li>
+ <a href="./configurations/">Configurations</a>
+ </li>
+
+ <li>
+ <a href="./cdc-ingestion/">CDC Ingestion</a>
+ </li>
+
+ <li>
+ <a href="./metrics/">Metrics</a>
+ </li>
+
+ <li>
+ <a href="./rest-api/">REST API</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed"
data-toggle="collapse" href="#Formats">
<span>Formats</span>
</a>
</li>
+ <div id="Formats" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./formats-overview/">Overview</a>
+ </li>
+
+ <li>
+ <a href="./iceberg-format/">Iceberg</a>
+ </li>
+
+ <li>
+ <a href="./paimon-format/">Paimon</a>
+ </li>
+
+ <li>
+ <a href="./mixed-iceberg-format/">Mixed-Iceberg</a>
+ </li>
+
+ <li>
+ <a href="./mixed-hive-format/">Mixed-Hive</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed"
data-toggle="collapse" href="#Flink">
<span>Flink</span>
</a>
</li>
+ <div id="Flink" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./flink-getting-started/">Flink Getting
Started</a>
+ </li>
+
+ <li>
+ <a href="./flink-ddl/">Flink DDL</a>
+ </li>
+
+ <li>
+ <a href="./flink-dml/">Flink DML</a>
+ </li>
+
+ <li>
+ <a href="./flink-cdc-ingestion/">Flink CDC
Ingestion</a>
+ </li>
+
+ <li>
+ <a href="./flink-datastream/">Flink DataStream</a>
+ </li>
+
+ <li>
+ <a href="./flink-using-logstore/">Using
Logstore</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed"
data-toggle="collapse" href="#Spark">
<span>Spark</span>
</a>
</li>
+ <div id="Spark" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./spark-getting-started/">Spark Getting
Started</a>
+ </li>
+
+ <li>
+ <a href="./spark-configuration/">Spark
Configuration</a>
+ </li>
+
+ <li>
+ <a href="./spark-ddl/">Spark DDL</a>
+ </li>
+
+ <li>
+ <a href="./spark-queries/">Spark Queries</a>
+ </li>
+
+ <li>
+ <a href="./spark-writes/">Spark Writes</a>
+ </li>
+
+ </ul>
+ </div>
+
+
+
+
+
+ <li>
+ <a class="top-menu" id="" href="./trino/">
+ <span>Trino</span>
+ </a>
+ </li>
</ul>
@@ -482,7 +665,132 @@
<a
href="https://amoro.apache.org/docs/latest/../../docs/0.8.1/";><strong>0.8.1-incubating</strong></a>.
</p>
</div>
- </div>
+ <!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+<h1 id="introduction">Introduction</h1>
+<p>Amoro is a Lakehouse management system built on open data lake formats.
Working with compute engines including Flink, Spark, and Trino, Amoro brings
pluggable and self-managed features for Lakehouse to provide out-of-the-box
data warehouse experience, and helps data platforms or products easily build
infra-decoupled, stream-and-batch-fused and lake-native architecture.</p>
+<h2 id="architecture">Architecture</h2>
+<p>The architecture of Amoro is as follows:</p>
+<p align="center">
+<img src="images/introduce_amoro.png" alt="Architecture" width="90%"
height="90%">
+</p>
+<p>The core components of Amoro include:</p>
+<ul>
+<li>AMS: Amoro Management Service provides Lakehouse management features, like
self-optimizing, data expiration, etc.
+It also provides a unified catalog service for all compute engines, which can
also be combined with existing metadata services.</li>
+<li>Plugins: Amoro provides a wide selection of external plugins to meet
different scenarios.
+<ul>
+<li>Optimizers: The self-optimizing execution engine plugin asynchronously
performs merging, sorting, deduplication,
+layout optimization, and other operations on all type table format tables.</li>
+<li>Terminal: SQL command-line tools, provide various implementations like
local Spark and Kyuubi.</li>
+<li>LogStore: Provide millisecond to second level SLAs for real-time data
processing based on message queues like Kafka and Pulsar.</li>
+</ul>
+</li>
+</ul>
+<h2 id="supported-table-formats">Supported table formats</h2>
+<p>Amoro can manage tables of different table formats, similar to how
MySQL/ClickHouse can choose different storage engines.
+Amoro meets diverse user needs by using different table formats. Currently,
Amoro supports three table formats:</p>
+<ul>
+<li>Iceberg format: means using the native table format of the Apache Iceberg,
which has all the features and characteristics of Iceberg.</li>
+<li>Mixed-Iceberg format: built on top of Iceberg format, which can accelerate
data processing using LogStore
+and provides more efficient query performance and streaming read capability in
CDC scenarios.</li>
+<li>Mixed-Hive format: has the same features as the Mixed-Iceberg tables but
is compatible with a Hive table.
+Support upgrading Hive tables to Mixed-Hive tables, and allow Hive’s
native read and write methods after upgrading.</li>
+</ul>
+<h2 id="supported-engines">Supported engines</h2>
+<h3 id="iceberg-format">Iceberg format</h3>
+<p>Iceberg format tables use the engine integration method provided by the
Iceberg community.
+For details, please refer to: <a
href="https://iceberg.apache.org/docs/latest/";>Iceberg Docs</a>.</p>
+<h3 id="paimon-format">Paimon format</h3>
+<p>Paimon format tables use the engine integration method provided by the
Paimon community.
+For details, please refer to: <a
href="https://paimon.apache.org/docs/master/";>Paimon Docs</a>.</p>
+<h3 id="mixed-format">Mixed format</h3>
+<p>Amoro support multiple processing engines for Mixed format as below:</p>
+<table>
+ <thead>
+ <tr>
+ <th>Processing Engine</th>
+ <th>Version</th>
+ <th>Batch Read</th>
+ <th>Batch Write</th>
+ <th>Batch Overwrite</th>
+ <th>Streaming Read</th>
+ <th>Streaming Write</th>
+ <th>Create Table</th>
+ <th>Alter Table</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Flink</td>
+ <td>1.18.x, 1.19.x, 1.20.x</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ </tr>
+ <tr>
+ <td>Spark</td>
+ <td>3.3, 3.4, 3.5</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✔</td>
+ </tr>
+ <tr>
+ <td>Hive</td>
+ <td>2.x, 3.x</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ </tr>
+ <tr>
+ <td>Trino</td>
+ <td>406</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ </tr>
+ </tbody>
+</table>
+<h2 id="user-cases">User cases</h2>
+<h3 id="self-managed-streaming-lakehouse">Self-managed streaming Lakehouse</h3>
+<p>Amoro makes it easier for users to handle the challenges of writing to a
real-time data lake, such as ingesting append-only event logs or CDC data from
databases.
+In these scenarios, the rapid increase of fragment and redundant files cannot
be ignored.
+To address this issue, Amoro provides a pluggable streaming data
self-optimizing mechanism that automatically compacts fragment files and
removes expired data, ensuring high-quality table queries while reducing system
costs.</p>
+<h3 id="stream-and-batch-fused-data-pipeline">Stream-and-batch-fused data
pipeline</h3>
+<p>Whether in the AI or BI business field , the requirement for real-time
analysis is becoming increasingly high. The traditional approach of using one
streaming job to complete all data processing from the source to the end is no
longer applicable. There is an increasing demand for layered construction of
streaming data pipeline, and the traditional layered construction approach
based on message queues can cause a inconsistency problem between the streaming
and batch data processing. Bu [...]
+<h3 id="cloud-native-lakehouse">Cloud-native Lakehouse</h3>
+<p>Currently, most data platforms and products are tightly coupled with their
underlying infrastructure(such as the storage layer). The migration of
infrastructure, such as switching to cloud-native OSS, may require extensive
adaptation efforts or even be impossible. However, Amoro provides an
infra-decoupled, lake-native architecture built on top of the infrastructure.
This allows products based on Amoro to interact with the underlying
infrastructure through a unified interface (Amoro C [...]
+</div>
@@ -501,7 +809,26 @@
<div id="toc" class="markdown-body">
<div id="full">
- <nav id="TableOfContents"></nav>
+ <nav id="TableOfContents">
+ <ul>
+ <li><a href="#architecture">Architecture</a></li>
+ <li><a href="#supported-table-formats">Supported table formats</a></li>
+ <li><a href="#supported-engines">Supported engines</a>
+ <ul>
+ <li><a href="#iceberg-format">Iceberg format</a></li>
+ <li><a href="#paimon-format">Paimon format</a></li>
+ <li><a href="#mixed-format">Mixed format</a></li>
+ </ul>
+ </li>
+ <li><a href="#user-cases">User cases</a>
+ <ul>
+ <li><a href="#self-managed-streaming-lakehouse">Self-managed streaming
Lakehouse</a></li>
+ <li><a
href="#stream-and-batch-fused-data-pipeline">Stream-and-batch-fused data
pipeline</a></li>
+ <li><a href="#cloud-native-lakehouse">Cloud-native Lakehouse</a></li>
+ </ul>
+ </li>
+ </ul>
+</nav>
</div>
</div>
diff --git a/output/docs/latest/index.xml b/output/docs/latest/index.xml
index 5e773c706..9574a8e31 100644
--- a/output/docs/latest/index.xml
+++ b/output/docs/latest/index.xml
@@ -1,9 +1,9 @@
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom";>
<channel>
- <title>Amoro</title>
+ <title>Introduction on Amoro</title>
<link>https://amoro.apache.org/docs/latest/</link>
- <description>Recent content on Amoro</description>
+ <description>Recent content in Introduction on
Amoro</description>
<generator>Hugo</generator>
<language>en-us</language>
@@ -11,5 +11,236 @@
<atom:link
href="https://amoro.apache.org/docs/latest/index.xml"; rel="self"
type="application/rss+xml" />
+ <item>
+ <title></title>
+
<link>https://amoro.apache.org/docs/latest/readme/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/readme/</guid>
+ <description><h2
id="amoro-docs">Amoro Docs</h2>
<p>This directory
contains the documentation content of
Amoro.</p>
<ul>
<li>The documentation is written in
Markdown format.</li>
<li>The images referenced in the
documentation are saved in the <code>images</code>
folder.</li>
</ul>
<h3
id="style">Style</h3>
<ul>
<li>Proper
nouns should start w [...]
+ </item>
+ <item>
+ <title>AMS Configuration</title>
+
<link>https://amoro.apache.org/docs/latest/ams-config/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/ams-config/</guid>
+ <description><!--
 ~ Licensed to the
Apache Software Foundation (ASF) under one
 ~ or more contributor license
agreements. See the NOTICE file
 ~ distributed with this work for
additional information
 ~ regarding copyright ownership. The ASF licenses
this file
 ~ to you under the Apache License, Version 2.0 (the
 ~
"License"); you may not use this file except in compliance
 ~ with
the License. You may obtain a copy of the License at&#x [...]
+ </item>
+ <item>
+ <title>Catalogs</title>
+
<link>https://amoro.apache.org/docs/latest/catalogs/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/catalogs/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>CDC Ingestion</title>
+
<link>https://amoro.apache.org/docs/latest/cdc-ingestion/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/cdc-ingestion/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Configurations</title>
+
<link>https://amoro.apache.org/docs/latest/configurations/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/configurations/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Deployment</title>
+
<link>https://amoro.apache.org/docs/latest/deployment/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/deployment/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Deployment On Kubernetes</title>
+
<link>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink CDC Ingestion</title>
+
<link>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink DataStream</title>
+
<link>https://amoro.apache.org/docs/latest/flink-datastream/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/flink-datastream/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink DDL</title>
+
<link>https://amoro.apache.org/docs/latest/flink-ddl/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/flink-ddl/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink DML</title>
+
<link>https://amoro.apache.org/docs/latest/flink-dml/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/flink-dml/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink Getting Started</title>
+
<link>https://amoro.apache.org/docs/latest/flink-getting-started/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/flink-getting-started/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Iceberg</title>
+
<link>https://amoro.apache.org/docs/latest/iceberg-format/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/iceberg-format/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Managing Catalogs</title>
+
<link>https://amoro.apache.org/docs/latest/managing-catalogs/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/managing-catalogs/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Managing Optimizers</title>
+
<link>https://amoro.apache.org/docs/latest/managing-optimizers/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/managing-optimizers/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Metrics</title>
+
<link>https://amoro.apache.org/docs/latest/metrics/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/metrics/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Mixed-Hive</title>
+
<link>https://amoro.apache.org/docs/latest/mixed-hive-format/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/mixed-hive-format/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Mixed-Iceberg</title>
+
<link>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Overview</title>
+
<link>https://amoro.apache.org/docs/latest/formats-overview/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/formats-overview/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Paimon</title>
+
<link>https://amoro.apache.org/docs/latest/paimon-format/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/paimon-format/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>REST API</title>
+
<link>https://amoro.apache.org/docs/latest/rest-api/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/rest-api/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Self-Optimizing</title>
+
<link>https://amoro.apache.org/docs/latest/self-optimizing/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/self-optimizing/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark Configuration</title>
+
<link>https://amoro.apache.org/docs/latest/spark-configuration/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/spark-configuration/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark DDL</title>
+
<link>https://amoro.apache.org/docs/latest/spark-ddl/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/spark-ddl/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark Getting Started</title>
+
<link>https://amoro.apache.org/docs/latest/spark-getting-started/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/spark-getting-started/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark Queries</title>
+
<link>https://amoro.apache.org/docs/latest/spark-queries/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/spark-queries/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark Writes</title>
+
<link>https://amoro.apache.org/docs/latest/spark-writes/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/spark-writes/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Table Watermark</title>
+
<link>https://amoro.apache.org/docs/latest/table-watermark/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/table-watermark/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Trino</title>
+
<link>https://amoro.apache.org/docs/latest/trino/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/trino/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Using Customized Encryption Method for
Configurations</title>
+
<link>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Using Kyuubi By Terminal</title>
+
<link>https://amoro.apache.org/docs/latest/using-kyuubi/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/using-kyuubi/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Using Logstore</title>
+
<link>https://amoro.apache.org/docs/latest/flink-using-logstore/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/flink-using-logstore/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Using Tables</title>
+
<link>https://amoro.apache.org/docs/latest/using-tables/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00
+0000</pubDate>
+
<guid>https://amoro.apache.org/docs/latest/using-tables/</guid>
+ <description><!--
 - Licensed to the
Apache Software Foundation (ASF) under one or more
 - contributor license
agreements. See the NOTICE file distributed with
 - this work for
additional information regarding copyright ownership.
 - The ASF licenses
this file to You under the Apache License, Version 2.0
 - (the
"License"); you may not use this file except in compliance with
 -
the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
</channel>
</rss>
diff --git a/output/docs/latest/managing-catalogs/index.html
b/output/docs/latest/managing-catalogs/index.html
index 65f551ab8..e5790fbe0 100644
--- a/output/docs/latest/managing-catalogs/index.html
+++ b/output/docs/latest/managing-catalogs/index.html
@@ -726,6 +726,46 @@ please read the following guidelines and select the
appropriate creation accordi
</ul>
<h3 id="configure-table-properties">Configure table properties</h3>
<p>If you want to add the same table properties to all tables under a catalog,
you can add these table properties here on the catalog level. If you also
configure this property on the table level, the property on the table will take
effect.</p>
+<div class="info">
+ When configuring table-level properties on a catalog, you <strong>must
prefix the key with <code>table.</code></strong>. The prefix will be
automatically stripped when the property is applied to tables. For example, to
set the default <code>self-optimizing.group</code> for all tables under the
catalog, you need to configure <code>table.self-optimizing.group</code> instead
of <code>self-optimizing.group</code>. Properties without the
<code>table.</code> prefix will not be recognized a [...]
+</div>
+<p>The following table lists some examples. For the full list of available
table properties, see <a href="../user-guides/configurations.md">Table
configurations</a>.</p>
+<table>
+ <thead>
+ <tr>
+ <th>Catalog property key</th>
+ <th>Effective table property key</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+
<td><code>table.self-optimizing.enabled</code></td>
+
<td><code>self-optimizing.enabled</code></td>
+ <td>Enable/disable self-optimizing for
all tables</td>
+ </tr>
+ <tr>
+
<td><code>table.self-optimizing.group</code></td>
+
<td><code>self-optimizing.group</code></td>
+ <td>Set the default optimizer group for
all tables</td>
+ </tr>
+ <tr>
+
<td><code>table.table-expire.enabled</code></td>
+
<td><code>table-expire.enabled</code></td>
+ <td>Enable/disable table expiration for
all tables</td>
+ </tr>
+ <tr>
+
<td><code>table.clean-orphan-file.enabled</code></td>
+
<td><code>clean-orphan-file.enabled</code></td>
+ <td>Enable/disable orphan file cleaning
for all tables</td>
+ </tr>
+ <tr>
+
<td><code>table.log-store.enabled</code></td>
+ <td><code>log-store.enabled</code></td>
+ <td>Enable/disable LogStore for all
tables</td>
+ </tr>
+ </tbody>
+</table>
<h2 id="rest-catalog">REST Catalog</h2>
<p>When a user needs to create a Iceberg REST Catalog, they can choose
<strong>External Catalog Type</strong>、<strong>Custom Metastore
Type</strong>、<strong>Iceberg Table Format</strong>, configure properties
include:
<strong>catalog-impl=org.apache.iceberg.rest.RESTCatalog</strong>,
<strong>uri=$restCatalog_uri</strong>.</p>
diff --git a/output/docs/latest/managing-optimizers/index.html
b/output/docs/latest/managing-optimizers/index.html
index d02c34f00..4e9f57af7 100644
--- a/output/docs/latest/managing-optimizers/index.html
+++ b/output/docs/latest/managing-optimizers/index.html
@@ -1206,7 +1206,7 @@ contain several optimizers with the same container
implementation to facilitate
<td>All</td>
<td>No</td>
<td>false</td>
- <td>Whether to enable dynamic resource
allocation (AIP-5) for this group: the group then grows its optimizer count
automatically in response to optimizing demand, bounded by
<code>dynamic-allocation.max-parallelism</code>. Not supported on
externally-registered optimizers. Manual optimizer operations
(scale-out/release via the dashboard) are not recommended on groups with
dynamic allocation enabled.</td>
+ <td>Whether to enable dynamic resource
allocation for this group: the group then grows its optimizer count
automatically in response to optimizing demand, bounded by
<code>dynamic-allocation.max-parallelism</code>. Not supported on
externally-registered optimizers. Manual optimizer operations
(scale-out/release via the dashboard) are not recommended on groups with
dynamic allocation enabled.</td>
</tr>
<tr>
<td>dynamic-allocation.min-parallelism</td>
@@ -1296,7 +1296,29 @@ contain several optimizers with the same container
implementation to facilitate
<li>When using <code>shutdown-timeout-ms</code> with a Flink Optimizer, also
raise <code>flink-conf.task.cancellation.timeout</code> (default 180000)
accordingly — the graceful drain on job cancellation is capped below
Flink’s cancellation watchdog, which otherwise fails the whole
TaskManager.</li>
</ul>
</div>
-<h3 id="edit-optimizer-group">Edit optimizer group</h3>
+<h3 id="dynamic-resource-allocation">Dynamic resource allocation</h3>
+<p>Dynamic resource allocation (DRA) lets a group grow and shrink its
optimizer count automatically based on optimizing demand, instead of running a
fixed number of optimizers. Enable it by setting
<code>dynamic-allocation.enabled</code> to <code>true</code> on the group; once
enabled, <code>dynamic-allocation.min-parallelism</code> and
<code>dynamic-allocation.max-parallelism</code> take over from the deprecated
flat <code>min-parallelism</code>.</p>
+<p><strong>Scale-up</strong>: when optimizing demand (queued tasks) persists
for <code>dynamic-allocation.scheduler-backlog-timeout</code>, the group scales
out for the first time. While demand keeps persisting, it scales out again
every <code>dynamic-allocation.sustained-backlog-timeout</code>, up to
<code>dynamic-allocation.max-parallelism</code>.</p>
+<p><strong>Scale-down</strong>: an optimizer that has been idle for
<code>dynamic-allocation.executor-idle-timeout</code> becomes a scale-down
candidate. At most one optimizer is removed per evaluation round, and only
while the group has no scale-up demand, with at least
<code>dynamic-allocation.scale-down-cooldown</code> between removals. A
scaled-down optimizer is drained gracefully — it stops receiving new tasks and
is removed once its in-flight tasks finish; if draining exceeds <code [...]
+<div class="info">
+ <ul>
+<li>DRA is not supported for externally-registered optimizers.</li>
+<li>Avoid manual scale-out/release from the dashboard on a group with DRA
enabled — it fights with the auto scaler.</li>
+<li><code>dynamic-allocation.min-parallelism</code> and
<code>dynamic-allocation.max-parallelism</code> must both be reachable as
multiples of <code>dynamic-allocation.executor-parallelism</code>, the number
of threads added or removed per scaling step.</li>
+</ul>
+
+</div>
+<p>An example enabling DRA on a Kubernetes optimizer group, entered in the
group’s <code>properties</code>:</p>
+<div class="highlight"><pre tabindex="0"
style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code
class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.enabled</span>: <span
style="color:#66d9ef">true</span> <span
style="color:#75715e"># Turn on dynamic resource allocation</span>
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.min-parallelism</span>: <span
style="color:#ae81ff">8</span> <span
style="color:#75715e"># Keep at least 1 optimizer (executor-parallelism=8)
running</span>
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.max-parallelism</span>: <span
style="color:#ae81ff">64</span> <span
style="color:#75715e"># Scale out to at most 8 optimizers</span>
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.executor-parallelism</span>: <span
style="color:#ae81ff">8</span> <span style="color:#75715e">#
Threads per optimizer pod; 4-8 is recommended on Kubernetes</span>
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.scheduler-backlog-timeout</span>:
<span style="color:#ae81ff">1min </span> <span style="color:#75715e">#
Wait 1 min of sustained backlog before the first scale-out</span>
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.sustained-backlog-timeout</span>:
<span style="color:#ae81ff">30s </span> <span style="color:#75715e">#
Re-evaluate and scale out every 30s while backlog persists</span>
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.executor-idle-timeout</span>: <span
style="color:#ae81ff">10min </span> <span style="color:#75715e">#
Consider an optimizer idle after 10 min without tasks</span>
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.scale-down-cooldown</span>: <span
style="color:#ae81ff">1min </span> <span style="color:#75715e">#
Wait at least 1 min between scale-down removals</span>
+</span></span><span style="display:flex;"><span><span
style="color:#f92672">dynamic-allocation.drain-timeout</span>: <span
style="color:#ae81ff">15min </span> <span
style="color:#75715e"># Force-remove a draining optimizer after 15 min</span>
+</span></span></code></pre></div><h3 id="edit-optimizer-group">Edit optimizer
group</h3>
<p>You can click the <code>edit</code> button on the <code>Optimizer
Groups</code> page to modify the configuration of the Optimizer group.</p>
<h3 id="remove-optimizer-group">Remove optimizer group</h3>
<p>You can click the <code>remove</code> button on the <code>Optimizer
Groups</code> page to delete the optimizer group, but only if the group is
@@ -1514,6 +1536,7 @@ If the optimizer runs normally, you will see an optimizer
with the status <code>
<li><a href="#optimizer-group">Optimizer group</a>
<ul>
<li><a href="#add-optimizer-group">Add optimizer group</a></li>
+ <li><a href="#dynamic-resource-allocation">Dynamic resource
allocation</a></li>
<li><a href="#edit-optimizer-group">Edit optimizer group</a></li>
<li><a href="#remove-optimizer-group">Remove optimizer group</a></li>
</ul>
diff --git a/output/docs/latest/search.json b/output/docs/latest/search.json
index 0637a088a..fa5aaa7c6 100644
--- a/output/docs/latest/search.json
+++ b/output/docs/latest/search.json
@@ -1 +1 @@
-[]
\ No newline at end of file
+[{"categories":null,"content":"Amoro Docs This directory contains the
documentation content of Amoro.\nThe documentation is written in Markdown
format. The images referenced in the documentation are saved in the images
folder. Style Proper nouns should start with a capital letter, like
Hadoop、Hive、Iceberg、Amoro
","description":"","title":"","uri":"https://amoro.apache.org/docs/latest/readme/"},{"categories":null,"content":";
AMS Configuration Amoro Management Service Configuration The con [...]
\ No newline at end of file
diff --git a/output/docs/latest/sitemap.xml b/output/docs/latest/sitemap.xml
index 3ecb0371f..54c7a1740 100644
--- a/output/docs/latest/sitemap.xml
+++ b/output/docs/latest/sitemap.xml
@@ -2,10 +2,88 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";>
<url>
- <loc>https://amoro.apache.org/docs/latest/</loc>
+ <loc>https://amoro.apache.org/docs/latest/readme/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/admin-guides/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/ams-config/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/catalogs/</loc>
</url><url>
<loc>https://amoro.apache.org/docs/latest/categories/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/cdc-ingestion/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/concepts/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/configuration/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/configurations/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/deployment/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/engines/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-datastream/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-ddl/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-dml/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-getting-started/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/formats/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/iceberg-format/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/managing-catalogs/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/managing-optimizers/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/metrics/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/mixed-hive-format/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/formats-overview/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/paimon-format/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/rest-api/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/self-optimizing/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-configuration/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-ddl/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-getting-started/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-queries/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-writes/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/table-watermark/</loc>
</url><url>
<loc>https://amoro.apache.org/docs/latest/tags/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/trino/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/user-guides/</loc>
+ </url><url>
+
<loc>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/using-kyuubi/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-using-logstore/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/using-tables/</loc>
</url>
</urlset>
