liurenjie1024 commented on code in PR #1854: URL: https://github.com/apache/iceberg-rust/pull/1854#discussion_r2537005375
########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,192 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs). Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. +- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates. +- **Stable facade for existing users**: the top-level `iceberg` crate continues to expose the familiar API by re-exporting the kernel plus a default engine feature. + +Out of scope: changes to the Iceberg table specification or rewriting catalog adapters. + +## Architecture Overview + +### Workspace Layout + +``` +crates/ + kernel/ # new: pure protocols & planning logic + spec/ expr/ catalog/ table/ transaction/ scan/ runtime_api + io/traits.rs # FileIO traits (no opendal) Review Comment: Move this part to storage crate? ########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,192 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs). Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. +- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates. +- **Stable facade for existing users**: the top-level `iceberg` crate continues to expose the familiar API by re-exporting the kernel plus a default engine feature. Review Comment: I agree that we should keep public api as stable as possible, but it maybe not practical before we reach 1.0. Currently we reply on review to ensure it, but after reaching 1.0 we should use some autonomous checks to ensure that. ########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,192 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs). Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. +- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates. +- **Stable facade for existing users**: the top-level `iceberg` crate continues to expose the familiar API by re-exporting the kernel plus a default engine feature. + +Out of scope: changes to the Iceberg table specification or rewriting catalog adapters. + +## Architecture Overview + +### Workspace Layout + +``` +crates/ + kernel/ # new: pure protocols & planning logic + spec/ expr/ catalog/ table/ transaction/ scan/ runtime_api + io/traits.rs # FileIO traits (no opendal) + fileio/ + opendal/ # e.g. `iceberg-fileio-opendal` + fs/ # other FileIO implementations + runtime/ + tokio/ # e.g. `iceberg-runtime-tokio` + smol/ + iceberg/ # facade re-exporting kernel + default engine + catalog/* # depend on kernel (+ chosen FileIO/Runtime crates) + integrations/* # e.g. datafusion using facade or composing crates +``` + +### Trait Surfaces + +#### FileIO Review Comment: I don't think we should discuss details of `FileIO` crate here. @CTTY already have a draft pr and design doc, we should discuss it there. ########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,192 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs). Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. +- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates. +- **Stable facade for existing users**: the top-level `iceberg` crate continues to expose the familiar API by re-exporting the kernel plus a default engine feature. + +Out of scope: changes to the Iceberg table specification or rewriting catalog adapters. + +## Architecture Overview + +### Workspace Layout + +``` +crates/ + kernel/ # new: pure protocols & planning logic + spec/ expr/ catalog/ table/ transaction/ scan/ runtime_api + io/traits.rs # FileIO traits (no opendal) + fileio/ + opendal/ # e.g. `iceberg-fileio-opendal` + fs/ # other FileIO implementations + runtime/ + tokio/ # e.g. `iceberg-runtime-tokio` + smol/ + iceberg/ # facade re-exporting kernel + default engine + catalog/* # depend on kernel (+ chosen FileIO/Runtime crates) + integrations/* # e.g. datafusion using facade or composing crates +``` + +### Trait Surfaces + +#### FileIO + +```rust +pub struct FileMetadata { + pub size: u64, + ... +} + +pub type FileReader = Box<dyn FileRead>; + +#[async_trait::async_trait] +pub trait FileRead: Send + Sync + 'static { + async fn read(&self, range: Range<u64>) -> Result<Bytes>; +} + +pub type FileWriter = Box<dyn FileWrite>; + +#[async_trait::async_trait] +pub trait FileWrite: Send + Unpin + 'static { + async fn write(&mut self, bs: Bytes) -> Result<()>; + async fn close(&mut self) -> Result<FileMetadata>; +} + +pub type StorageFactory = fn(attrs: HashMap<String, String> -> Result<Arc<dyn Storage>>); + +#[async_trait::async_trait] +pub trait Storage: Clone + Send + Sync { + async fn reader(&self, path: &str) -> Result<FileReader>; + async fn writer(&self, path: &str) -> Result<FileWriter>; + async fn delete(&self, path: &str) -> Result<()>; + async fn exists(&self, path: &str) -> Result<bool>; + + ... +} + +pub struct FileIO { + registry: DashMap<String, StorageFactory>, +} + +impl FileIO { + fn register(scheme: &str, factory: StorageFactory); + + async fn read(path: &str) -> Result<Bytes>; + async fn reader(path: &str) -> Result<FileReader>; + async fn write(path: &str, bs: Bytes) -> Result<FileMetadata>; + async fn writer(path: &str) -> Result<FileWriter>; + + async fn delete(&self, path: &str) -> Result<()>; + ... +} +``` + +- The kernel only defines the trait and error types. +- `iceberg-fileio-opendal` (new crate) ships an opendal-based implementation; other backends can publish their own crates. + +#### Runtime + +```rust +pub trait Runtime: Send + Sync + 'static { + type JoinHandle<T>: Future<Output = T> + Send + 'static; + + fn spawn<F, T>(&self, fut: F) -> Self::JoinHandle<T> + where + F: Future<Output = T> + Send + 'static, + T: Send + 'static; + + fn sleep(&self, dur: Duration) -> Pin<Box<dyn Future<Output = ()> + Send>>; +} +``` + +- `TableScan` planning, metadata refresh, and `Transaction::commit` depend only on this trait. +- Crates such as `iceberg-runtime-tokio` provide concrete schedulers; consumers pick whichever runtime crate fits their stack. + +#### Catalog / Table / Transaction + +- The `Catalog` trait moves into the kernel and returns lightweight `TableHandle` objects (metadata + FileIO + Runtime). +- `TableHandle` no longer embeds Arrow helpers; Arrow-specific logic lives in engine crates. +- Transactions and their actions remain in the kernel, but rely on injected `Runtime` for retries/backoff. + +#### Scan / Planner + +- The kernel produces pure `TableScanPlan` descriptions (manifests, data-files, predicates, task graph). +- Engines provide executors (e.g., `ArrowExecutor`) that transform plans into record batches or other runtime-specific artifacts. + +### Facade Behavior + +- The top-level `iceberg` crate becomes a facade (`pub use iceberg_kernel::*`) that enables a *composition* of default crates (e.g. `iceberg-runtime-tokio`, `iceberg-fileio-opendal`, and a reference executor) behind feature flags. +- Existing convenience APIs (`Table::scan().to_arrow()`, `MemoryCatalog`, etc.) stay available but internally assemble the kernel with those default building blocks. + +## Migration Plan + +1. **Phase 1 – Create the kernel crate** + - Add `crates/kernel` and move `spec`, `expr`, `catalog`, `table`, `transaction`, `scan`, and supporting modules. + - Introduce temporary shim modules in the facade so existing imports keep working (mark them deprecated). + +2. **Phase 2 – Abstract runtime & IO** + - Define the `Runtime` and `FileIO` traits inside the kernel. + - Remove direct `tokio`/`opendal` dependencies from kernel modules. + - Introduce standalone crates (`iceberg-runtime-tokio`, `iceberg-fileio-opendal`, etc.) that implement the new traits. + +3. **Phase 3 – Detach Arrow/execution** + - Move `arrow` helpers and `ArrowReaderBuilder` into a reference executor crate (e.g. `iceberg-engine-arrow`). Review Comment: In fact, there are several parts in the `to_arrow` method: 1. `ArrowReaderBuilder` which is used to read one split of planned task. This is in fact compute engine independent, and we should leave it in `iceberg` crate. 2. An executor to use runtime to scan table in parallel. I think this part should be move to the new default engine crate. ########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,192 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs). Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. +- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates. +- **Stable facade for existing users**: the top-level `iceberg` crate continues to expose the familiar API by re-exporting the kernel plus a default engine feature. + +Out of scope: changes to the Iceberg table specification or rewriting catalog adapters. + +## Architecture Overview + +### Workspace Layout + +``` +crates/ + kernel/ # new: pure protocols & planning logic + spec/ expr/ catalog/ table/ transaction/ scan/ runtime_api Review Comment: Move `runtime_api` to `runtime` crate? ########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,192 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs). Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. +- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates. +- **Stable facade for existing users**: the top-level `iceberg` crate continues to expose the familiar API by re-exporting the kernel plus a default engine feature. + +Out of scope: changes to the Iceberg table specification or rewriting catalog adapters. + +## Architecture Overview + +### Workspace Layout + +``` +crates/ + kernel/ # new: pure protocols & planning logic + spec/ expr/ catalog/ table/ transaction/ scan/ runtime_api + io/traits.rs # FileIO traits (no opendal) + fileio/ + opendal/ # e.g. `iceberg-fileio-opendal` + fs/ # other FileIO implementations + runtime/ + tokio/ # e.g. `iceberg-runtime-tokio` + smol/ + iceberg/ # facade re-exporting kernel + default engine Review Comment: How about another approach: we keep iceberg part as now, and move the default `Executor` which produces an arrow record batch stream to another crate in `integrations`? This aligns with what we have in datafusion integration. ########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,192 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs). Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. +- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates. Review Comment: I'm thinking that we have a `iceberg-storage` crate, which contains all fileio related things, including `FileIO/InputFile/OutputFile` struct, `Storage` trait, some builtin storage implemention(enabled independently by feature). ########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,192 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs). Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. Review Comment: We already have a runtime module, which serves similar purpose. I'm thinking maybe we should split out a `iceberg-runtime` crate, and have the `iceberg` crate relying on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
