NoahKusaba commented on code in PR #2638: URL: https://github.com/apache/datafusion-ballista/pull/2638#discussion_r4438568411
########## docs/source/user-guide/iceberg.md: ########## @@ -0,0 +1,224 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Apache Iceberg + +The `ballista-iceberg` crate (`integrations/iceberg`) lets Ballista scan +[Apache Iceberg](https://iceberg.apache.org/) tables. On top of +[iceberg-datafusion](https://github.com/apache/datafusion-iceberg), it adds the +codecs that send Iceberg tables and scans between the client, the scheduler and +the executors. + +The crate is not published yet. It depends on unreleased revisions of +iceberg-datafusion and iceberg-rust, and needs Rust 1.95. + +## Usage + +Ready the session with `register_iceberg`, then mount an Iceberg catalog: + +```rust +use std::sync::Arc; + +use ballista::prelude::{SessionConfigExt, SessionContextExt}; +use datafusion::execution::SessionStateBuilder; +use datafusion::prelude::{SessionConfig, SessionContext}; +use ballista_iceberg::register_iceberg; +use iceberg_datafusion::IcebergCatalogProvider; + +let state = SessionStateBuilder::new() + .with_config(SessionConfig::new_with_ballista()) + .with_default_features() + .build(); +let ctx = SessionContext::standalone_with_state(register_iceberg(state)).await?; + +ctx.register_catalog("rest", Arc::new(IcebergCatalogProvider::try_new(catalog).await?)); +ctx.sql("SELECT * FROM rest.ns.t").await?.show().await?; +``` + +A single table registered as an `IcebergCatalogTableProvider` works the same +way. Either kind reads the table's latest version in each query. + +For time travel, register an `IcebergStaticTableProvider`, which keeps one +version: `IcebergStaticTableProvider::try_new_from_table_snapshot(table, snapshot_id)`. + +The example registers both on a table in a temporary local warehouse, and shows +a commit reaching the live table but not the static one: + +```shell +cargo run -p ballista-iceberg --example standalone-iceberg-scan +``` + +## SQL clients + +Flight SQL clients (JDBC, ADBC, BI tools) send SQL to the scheduler, which plans +it, so they hold no catalog. To let them query Iceberg tables, mount catalogs +on the scheduler with `iceberg_session_builder`, which wraps the scheduler's +session builder (`default_session_builder` here, or your own), and give the +scheduler and executors the Iceberg codecs: + +```rust +use ballista_core::utils::default_session_builder; +use ballista_iceberg::{IcebergLogicalCodec, IcebergPhysicalCodec, iceberg_session_builder}; + +let rest = IcebergCatalogProvider::try_new(catalog).await?; +let mut config = SchedulerConfig::default() Review Comment: Added in 1105c595. `SchedulerConfigExt::with_iceberg` sets the codecs and mounts the catalogs, and `ExecutorProcessConfigExt::with_iceberg` sets the codecs on executors. Both wrap whatever session builder or codecs the config already has, and sit behind `scheduler` and `executor` features so clients don't pull in those crates. ```rust let config = SchedulerConfig::default() .with_flight_sql(true) .with_iceberg([("rest", Arc::new(rest) as _)]); let config = ExecutorProcessConfig::default().with_iceberg(); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
