Xuanwo commented on code in PR #1854: URL: https://github.com/apache/iceberg-rust/pull/1854#discussion_r2545291104
########## docs/rfcs/0001_modularize_iceberg_implementations.md: ########## @@ -0,0 +1,196 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Modularize `iceberg` Implementations + +## Background + +Issue #1819 highlighted that the current `iceberg` crate mixes the Iceberg protocol abstractions (catalog/table/plan/transaction) with concrete runtime, storage, and execution implementations (Tokio runtime wrappers, opendal-based `FileIO`, Arrow readers, DataFusion helpers, etc.). This makes the crate heavy, couples unrelated dependencies, and prevents users from bringing their own engines or storage stacks. + +After recent maintainer discussions we agreed on two principles: +1. The `iceberg` crate itself remains the single source of truth for all protocol traits and data structures. +2. All concrete integrations (Tokio runtime, opendal `FileIO`, Arrow/DataFusion executors, catalog adapters, etc.) move out of `iceberg` into dedicated companion crates. Users who need a ready-made execution path can depend on those crates (for example `iceberg-datafusion`) while users building custom stacks can depend solely on `iceberg`. + +This RFC describes the plan to slim down `iceberg` into a pure protocol crate and to reorganize the workspace around pluggable companion crates. + +## Goals and Scope + +- **Keep `iceberg` as the protocol crate**: it exposes all traits (`Catalog`, `Table`, `Transaction`, `FileIO`, `Runtime`, `ScanPlan`, etc.) plus metadata/plan logic, but no longer ships concrete runtimes or storage adapters. +- **Detach embedded implementations**: move opendal-based IO, Tokio runtime helpers, Arrow converters, and similar code into separate crates under `crates/fileio/*`, `crates/runtime/*`, `crates/engine/*`, or existing integration crates. +- **Enable composable combinations**: users assemble the stack they need by combining `iceberg` with specific implementation crates (e.g., `iceberg-fileio-opendal`, `iceberg-runtime-tokio`, `iceberg-engine-arrow`, `iceberg-datafusion`). +- **Minimize breaking surfaces**: trait APIs stay in `iceberg`; downstream crates only adjust their dependency graph. + +Out of scope: changing the Iceberg table specification or rewriting catalog adapters’ external behavior. + +## Architecture Overview + +### Workspace Layout + +``` +crates/ + iceberg/ # core traits, metadata, planning, transactions + fileio/ + opendal/ # e.g. `iceberg-fileio-opendal` + fs/ # other FileIO implementations + runtime/ + tokio/ # e.g. `iceberg-runtime-tokio` + smol/ + engine/ + arrow/ # Arrow executor & schema helpers Review Comment: Makes sense to me, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
