Xuanwo commented on code in PR #1854: URL: https://github.com/apache/iceberg-rust/pull/1854#discussion_r2533364420
########## docs/rfcs/0001_kernel.md: ########## @@ -0,0 +1,197 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# RFC: Extract `iceberg-kernel` for Pluggable Execution Layers + +## Background + +Issue #1819 proposes decoupling the protocol/metadata/plan logic that currently lives inside the `iceberg` crate so that it can serve as a reusable “kernel,” similar to the approach taken by delta-kernel-rs. Today the `iceberg` crate simultaneously exposes the public trait surface and the default engine (Tokio runtime, opendal-backed FileIO, Arrow readers, etc.). This tight coupling makes it difficult for downstream projects to embed Iceberg metadata while providing their own storage, runtime, or execution stack. + +## Goals and Scope + +- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.). +- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol. +- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates. +- **Stable facade for existing users**: the top-level `iceberg` crate continues to expose the familiar API by re-exporting the kernel plus a default engine feature. + +Out of scope: changes to the Iceberg table specification or rewriting catalog adapters. + +## Architecture Overview + +### Workspace Layout + +``` +crates/ + kernel/ # new: pure protocols & planning logic + spec/ expr/ catalog/ table/ transaction/ scan/ runtime_api + io/traits.rs # FileIO traits (no opendal) + fileio/ + opendal/ # e.g. `iceberg-fileio-opendal` + fs/ # other FileIO implementations + runtime/ + tokio/ # e.g. `iceberg-runtime-tokio` + smol/ + iceberg/ # facade re-exporting kernel + default engine + catalog/* # depend on kernel (+ chosen FileIO/Runtime crates) + integrations/* # e.g. datafusion using facade or composing crates +``` + +### Trait Surfaces + +#### FileIO + +```rust +pub struct FileMetadata { + pub size: u64, + ... +} + +pub type FileReader = Box<dyn FileRead>; + +#[async_trait::async_trait] +pub trait FileRead: Send + Sync + 'static { + async fn read(&self, range: Range<u64>) -> Result<Bytes>; +} + +pub type FileWriter = Box<dyn FileWrite>; + +#[async_trait::async_trait] +pub trait FileWrite: Send + Unpin + 'static { + async fn write(&mut self, bs: Bytes) -> Result<()>; + async fn close(&mut self) -> Result<FileMetadata>; +} + +pub type StorageFactory = fn(attrs: HashMap<String, String> -> Result<Arc<dyn Storage>>); + +#[async_trait::async_trait] +pub trait Storage: Clone + Send + Sync { + async fn reader(&self, path: &str) -> Result<FileReader>; + async fn writer(&self, path: &str) -> Result<FileWriter>; + async fn delete(&self, path: &str) -> Result<()>; + async fn exists(&self, path: &str) -> Result<bool>; + + ... +} + +pub struct FileIO { + registry: DashMap<String, StorageFactory>, +} + +impl FileIO { + fn register(scheme: &str, factory: StorageFactory); + + async fn read(path: &str) -> Result<Bytes>; + async fn reader(path: &str) -> Result<FileReader>; + async fn write(path: &str, bs: Bytes) -> Result<FileMetadata>; + async fn writer(path: &str) -> Result<FileWriter>; + + async fn delete(&self, path: &str) -> Result<()>; + ... +} +``` + +- The kernel only defines the trait and error types. +- `iceberg-fileio-opendal` (new crate) ships an opendal-based implementation; other backends can publish their own crates. + +#### Runtime + +```rust +pub trait Runtime: Send + Sync + 'static { + type JoinHandle<T>: Future<Output = T> + Send + 'static; + + fn spawn<F, T>(&self, fut: F) -> Self::JoinHandle<T> + where + F: Future<Output = T> + Send + 'static, + T: Send + 'static; + + fn spawn_blocking<F, T>(&self, f: F) -> Self::JoinHandle<T> Review Comment: Just want to align the current implementation, I'm fine removing them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
