Hi all,
I'd like to propose adding native Apache Iceberg support to Cloudberry
as part of the datalake_fdw extension, and to collect community feedback
before we open-source the implementation.
== Motivation ==
Cloudberry today lacks a transactional read/write entry point for lake
formats. The existing PXF-based FDW is read-mostly, has no Catalog
concept, and cannot share snapshots with the wider Iceberg ecosystem
(Spark / Trino / Flink). This proposal introduces Iceberg tables as
first-class "lake tables" in CB with full SELECT / INSERT / UPDATE /
DELETE / VACUUM, Schema Evolution, and Read Committed isolation, while
keeping metadata format fully compatible with the community.
== Key design points ==
* Iceberg Table AM — Iceberg tables register as a dedicated Table AM
(not a pure FDW), so UPDATE/DELETE/ctid and transactional semantics
come for free; data I/O is delegated to a Volume FDW.
* Catalog × Volume × Table abstraction — Catalog (Polaris / Hive /
Builtin) and Volume (S3 / HDFS) are two separate FDWs, freely
composable. A Builtin Catalog is provided for zero-dependency
deployments.
* Metadata delegated to an out-of-process Java agent — datalake_agent
wraps iceberg-java and runs as a jar launched and supervised by a PG
bgworker (datalake_proxy). The PG↔agent RPC channel is gRPC. This
keeps wire-format compatibility with the community and makes Iceberg
version upgrades a jar swap.
* Transactional semantics — a per-transaction Metadata Tracker maps
Iceberg's optimistic CAS onto PG's MVCC, supporting
Read-Your-Own-Writes, Read Committed across concurrent writers,
SAVEPOINT, and asynchronous cleanup of orphaned files.
* MPP execution — only the QD talks to the agent; data read/write is
parallel on QEs directly against object storage.
== Read more / discuss ==
The full design document (architecture, end-to-end flows, MPP model,
limitations and open questions) and the discussion thread are here:
https://github.com/apache/cloudberry/discussions/1683
The design doc ends with a list of review topics we're
most interested in hearing opinions on. Looking forward to your
feedback on the thread.
Thanks,
Xiaoyu Liu
