Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2089469920 Thanks @marvinlanhenke for this pr, and @Fokko @Xuanwo @viirya @simonvandel @tshauck for review! -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 merged PR #324: URL: https://github.com/apache/iceberg-rust/pull/324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2089449018 > I haven't checked now, but from memory I believe py-iceberg uses pyspark to setup tables with actual data for proper integration testing? I think so. I think datafusion

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

viirya commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2085481329 Thanks for working on this @marvinlanhenke . Looks good to me. There are some more integrations with DataFusion I'm interested to work with like improving ExecutionPlan for IcebergTable

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584912205 ## crates/integrations/datafusion/README.md: ## @@ -0,0 +1,22 @@ + + +# Apache Iceberg Datafusion Integration + +This crate contains the integration of Apache Datafus

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2085319907 I've addressed those issues raised by @Xuanwo - thanks for the review. I think we can merge this PR for now - and continue working on #357; especially the caching issue

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584803007 ## crates/integrations/datafusion/src/schema.rs: ## @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584791992 ## crates/integrations/datafusion/src/schema.rs: ## @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584789483 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,136 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more c

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584781574 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584513718 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,97 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584505866 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584472188 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,136 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more co

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584368969 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,136 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more c

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

Fokko commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584261856 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,97 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

Xuanwo commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584230121 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,136 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584229759 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

Xuanwo commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1584213613 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1583019973 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more co

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2081488204 > Let's wait a moment to see what others think about the catalog snapshot problem. cc @Xuanwo @Fokko @viirya @sdd PTAL If it turns out, that the PR is okay (for now) - let

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582159023 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more c

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582158406 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2081482903 Let's wait a moment to see what others think about the catalog snapshot problem. cc @Xuanwo @Fokko @viirya @sdd PTAL -- This is an automated message from the Apache Git Service

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582152848 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more co

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582152303 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more co

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2081477394 > @liurenjie1024 @ZENOTME @viirya @simonvandel ...I just went ahead and pushed the recent updates; PTAL > > ## unresolved / todo: > * [ ] proper integration tests for t

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582151715 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582080221 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582083814 ## crates/integrations/datafusion/src/physical_plan/scan.rs: ## @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more c

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2081431552 @liurenjie1024 @ZENOTME @viirya @simonvandel ...I just went ahead and pushed the recent updates; PTAL ## unresolved / todo: - [ ] proper integration tests for table

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582080221 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582080221 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582055651 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582042879 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582042661 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582031879 ## crates/integrations/datafusion/src/schema.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1581362714 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1581362714 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1581219550 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor licens

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1581219550 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor licens

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1581067924 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580967711 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580510837 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580506460 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,73 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580505826 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1580024955 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579945951 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579940862 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579930294 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579919751 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

simonvandel commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579897701 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579883806 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579827548 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

tshauck commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579818649 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#issuecomment-2077663647 > I've left some comment to improve, but it looks great! I'll invite datafusion community to help review. @liurenjie1024 Thanks for the review. I fixed most of the

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579755826 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,73 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor l

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

Xuanwo commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579739285 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579728551 ## crates/integrations/datafusion/README.md: ## @@ -0,0 +1,22 @@ + + +# Apache Iceberg Integrations + +This crate contains the official Native Rust implementation of

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579728551 ## crates/integrations/datafusion/README.md: ## @@ -0,0 +1,22 @@ + + +# Apache Iceberg Integrations + +This crate contains the official Native Rust implementation of

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579713512 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579715245 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579715245 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579713512 ## crates/integrations/datafusion/Cargo.toml: ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579654225 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1579601294 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice