[PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

ludlows opened a new pull request, #10188: URL: https://github.com/apache/iceberg/pull/10188 Currently, the property needs to be set in `tez-site.xml`. However, according to [HIVE-25581](https://issues.apache.org/jira/browse/HIVE-25581), all Iceberg queries require this property to

Re: [I] Support virtual addressing style in PyArrowFileIO [iceberg-python]

github-actions[bot] commented on issue #21: URL: https://github.com/apache/iceberg-python/issues/21#issuecomment-2067414936 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] BoundType and BoundPredicate should match type [iceberg-python]

github-actions[bot] commented on issue #18: URL: https://github.com/apache/iceberg-python/issues/18#issuecomment-2067414948 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] BoundType and BoundPredicate should match type [iceberg-python]

github-actions[bot] closed issue #18: BoundType and BoundPredicate should match type URL: https://github.com/apache/iceberg-python/issues/18 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Support virtual addressing style in PyArrowFileIO [iceberg-python]

github-actions[bot] closed issue #21: Support virtual addressing style in PyArrowFileIO URL: https://github.com/apache/iceberg-python/issues/21 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Flink CDC iceberg table have duplicate rows [iceberg]

github-actions[bot] commented on issue #2610: URL: https://github.com/apache/iceberg/issues/2610#issuecomment-2067413975 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Feature Request: To query iceberg tables from BI tools(like tableau,DBVisualizer) using jdbc/odbc connectors [iceberg]

github-actions[bot] commented on issue #2605: URL: https://github.com/apache/iceberg/issues/2605#issuecomment-2067413958 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Hive aggregate query iceberg tables is failing with ArrayIndexOutOfBound exception using Hive Catalog [iceberg]

github-actions[bot] commented on issue #2601: URL: https://github.com/apache/iceberg/issues/2601#issuecomment-2067413943 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Allow Type Promotion to String [iceberg]

github-actions[bot] commented on issue #2594: URL: https://github.com/apache/iceberg/issues/2594#issuecomment-2067413933 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Build fails due to inaccessible palantir dependencies [iceberg]

github-actions[bot] commented on issue #2462: URL: https://github.com/apache/iceberg/issues/2462#issuecomment-2067413828 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Class not found error when use sink [iceberg]

github-actions[bot] commented on issue #2455: URL: https://github.com/apache/iceberg/issues/2455#issuecomment-2067413807 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Build fails due to inaccessible palantir dependencies [iceberg]

github-actions[bot] closed issue #2462: Build fails due to inaccessible palantir dependencies URL: https://github.com/apache/iceberg/issues/2462 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] Class not found error when use sink [iceberg]

github-actions[bot] closed issue #2455: Class not found error when use sink URL: https://github.com/apache/iceberg/issues/2455 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Move site-related things to a separate repo [iceberg]

github-actions[bot] commented on issue #2446: URL: https://github.com/apache/iceberg/issues/2446#issuecomment-2067413794 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] cannot insert value in hive command shell [iceberg]

github-actions[bot] commented on issue #2442: URL: https://github.com/apache/iceberg/issues/2442#issuecomment-2067413785 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Move site-related things to a separate repo [iceberg]

github-actions[bot] closed issue #2446: Move site-related things to a separate repo URL: https://github.com/apache/iceberg/issues/2446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] cannot insert value in hive command shell [iceberg]

github-actions[bot] closed issue #2442: cannot insert value in hive command shell URL: https://github.com/apache/iceberg/issues/2442 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[PR] Build: Bump requests-mock from 1.11.0 to 1.12.1 [iceberg-python]

dependabot[bot] opened a new pull request, #642: URL: https://github.com/apache/iceberg-python/pull/642 Bumps [requests-mock](https://github.com/jamielennox/requests-mock) from 1.11.0 to 1.12.1. Release notes Sourced from https://github.com/jamielennox/requests-mock/releases";>requ

[PR] Build: Bump boto3 from 1.34.34 to 1.34.69 [iceberg-python]

dependabot[bot] opened a new pull request, #641: URL: https://github.com/apache/iceberg-python/pull/641 Bumps [boto3](https://github.com/boto/boto3) from 1.34.34 to 1.34.69. Changelog Sourced from https://github.com/boto/boto3/blob/develop/CHANGELOG.rst";>boto3's changelog.

[PR] Build: Bump coverage from 7.4.2 to 7.4.4 [iceberg-python]

dependabot[bot] opened a new pull request, #640: URL: https://github.com/apache/iceberg-python/pull/640 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.4.2 to 7.4.4. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's chang

[PR] Build: Bump pytest from 7.4.4 to 8.1.1 [iceberg-python]

dependabot[bot] opened a new pull request, #639: URL: https://github.com/apache/iceberg-python/pull/639 Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.4 to 8.1.1. Release notes Sourced from https://github.com/pytest-dev/pytest/releases";>pytest's releases. 8.1

[PR] Build: Bump pydantic from 2.6.1 to 2.7.0 [iceberg-python]

dependabot[bot] opened a new pull request, #638: URL: https://github.com/apache/iceberg-python/pull/638 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.6.1 to 2.7.0. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

Re: [PR] Verify release quality of life improvements [iceberg-python]

kevinjqliu commented on code in PR #626: URL: https://github.com/apache/iceberg-python/pull/626#discussion_r1572919996 ## Makefile: ## @@ -63,6 +63,7 @@ test-coverage: docker compose -f dev/docker-compose-integration.yml up -d sh ./dev/run-azurite.sh sh .

Re: [PR] Verify release quality of life improvements [iceberg-python]

kevinjqliu commented on code in PR #626: URL: https://github.com/apache/iceberg-python/pull/626#discussion_r1572917220 ## Makefile: ## @@ -63,6 +63,7 @@ test-coverage: docker compose -f dev/docker-compose-integration.yml up -d sh ./dev/run-azurite.sh sh .

Re: [PR] Verify release quality of life improvements [iceberg-python]

kevinjqliu commented on code in PR #626: URL: https://github.com/apache/iceberg-python/pull/626#discussion_r1572915541 ## mkdocs/docs/verify-release.md: ## @@ -105,15 +105,17 @@ make test To run the full integration tests: ```sh -make test-s3 +make test-integration Review C

Re: [I] [feature request] Allow engines to time travel [iceberg-python]

gupteaj commented on issue #600: URL: https://github.com/apache/iceberg-python/issues/600#issuecomment-2067221356 Presto time travel reference - https://prestodb.io/docs/0.286/connector/iceberg.html#time-travel-using-version-system-version-and-timestamp-system-time Time travel for snapsho

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

sdd commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1572787996 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +239,27 @@ impl TableScan { .boxed()) } +fn create_manifest_eval_factory( +//&self, +id:

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

marvinlanhenke commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1572785208 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +239,27 @@ impl TableScan { .boxed()) } +fn create_manifest_eval_factory( +//&self, +

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#issuecomment-2067105023 Thanks for the reviews, @marvinlanhenke and @liurenjie1024! All comments addressed and ready for re-review 😄 -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

sdd commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1572771120 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +239,27 @@ impl TableScan { .boxed()) } +fn create_manifest_eval_factory( +//&self, +id:

Re: [PR] Simplify expression when doing `{and,or}` operations [iceberg-rust]

sdd commented on PR #339: URL: https://github.com/apache/iceberg-rust/pull/339#issuecomment-2067098098 Nice! 😎 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1572759976 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,393 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1572759092 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,393 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1572759566 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,393 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1572758803 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,393 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Core: Use 'delete' if OverwriteFiles only deletes data files [iceberg]

amogh-jahagirdar commented on code in PR #10150: URL: https://github.com/apache/iceberg/pull/10150#discussion_r1572688551 ## core/src/main/java/org/apache/iceberg/BaseOverwriteFiles.java: ## @@ -48,6 +48,10 @@ protected OverwriteFiles self() { @Override protected String

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

pvary commented on code in PR #10148: URL: https://github.com/apache/iceberg/pull/10148#discussion_r1572688579 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergWithHiveAutogatherEnable.java: ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

pvary commented on code in PR #10148: URL: https://github.com/apache/iceberg/pull/10148#discussion_r1572687325 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergWithHiveAutogatherEnable.java: ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

pvary commented on code in PR #10148: URL: https://github.com/apache/iceberg/pull/10148#discussion_r1572686815 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergWithHiveAutogatherEnable.java: ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] kafka-connect: correct partition transform support [iceberg]

amogh-jahagirdar commented on PR #10185: URL: https://github.com/apache/iceberg/pull/10185#issuecomment-2066942872 I don't really consider it a mistake to have the plural form. The exposed syntax and the actual metadata which is spec compliant are independent concerns. For the metadata we a

Re: [PR] Kafka-connect: Update config description [iceberg]

amogh-jahagirdar commented on PR #10184: URL: https://github.com/apache/iceberg/pull/10184#issuecomment-2066928555 I'll go ahead and merge, thanks @ajantha-bhat , thanks @Fokko @bryanck for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Kafka-connect: Update iceberg.hadoop-conf-dir config description [iceberg]

amogh-jahagirdar merged PR #10184: URL: https://github.com/apache/iceberg/pull/10184 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Kafka-connect: Handle namespace creation for auto table creation [iceberg]

Fokko commented on code in PR #10186: URL: https://github.com/apache/iceberg/pull/10186#discussion_r1572639039 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriterFactory.java: ## @@ -112,4 +117,18 @@ Table autoCreateTable(String tableName,

Re: [PR] Core: Lazily compute & cache hashCode in CharSequenceWrapper [iceberg]

amogh-jahagirdar commented on code in PR #10023: URL: https://github.com/apache/iceberg/pull/10023#discussion_r1572637783 ## api/src/main/java/org/apache/iceberg/util/CharSequenceWrapper.java: ## @@ -29,13 +29,16 @@ public static CharSequenceWrapper wrap(CharSequence seq) { }

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

stargrey102 commented on PR #10148: URL: https://github.com/apache/iceberg/pull/10148#issuecomment-2066908386 Hi @pvary, sure. I added a test with 2 cases: keep or not keep hive stats. Since the hive engine set hivestatsautogather to false by default: https://github.com/apache/iceberg/blo

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

stargrey102 commented on code in PR #10148: URL: https://github.com/apache/iceberg/pull/10148#discussion_r1572629527 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergWithHiveAutogatherEnable.java: ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] kafka-connect: correct partition transform support [iceberg]

ajantha-bhat commented on PR #10185: URL: https://github.com/apache/iceberg/pull/10185#issuecomment-2066858401 > I'd rather keep the plurals to avoid the trial-and-error approach of figuring out which works. As Fokko pointed out, Spark uses plurals for the transforms. I still think i

[I] Iceberg Hidden Partitioning and Spark SQL Wide Transformation Optimization [iceberg]

luca1x opened a new issue, #10187: URL: https://github.com/apache/iceberg/issues/10187 ### Query engine Spark 3.5 ### Question Hi, We are trying out Apache Iceberg on Glue/S3 in conjunction with Apache Spark. We are using version 3.5 As a small POC, we created

Re: [PR] Simplify expression when doing `{and,or}` operations [iceberg-rust]

liurenjie1024 merged PR #339: URL: https://github.com/apache/iceberg-rust/pull/339 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

liurenjie1024 commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1572472267 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,393 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more cont

Re: [PR] kafka-connect: correct partition transform support [iceberg]

bryanck commented on PR #10185: URL: https://github.com/apache/iceberg/pull/10185#issuecomment-2066699269 I'd rather keep the plurals to avoid the trial-and-error approach of figuring out which works. As Fokko pointed out, Spark uses plurals for the transforms. -- This is an automated me

Re: [PR] [WIP] Bump Iceberg to 1.5.0 on integration tests [iceberg-python]

ndrluis commented on PR #634: URL: https://github.com/apache/iceberg-python/pull/634#issuecomment-2066686189 Depends on #635 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: Implement the conversion from Iceberg Schema to Arrow Schema [iceberg-rust]

liurenjie1024 commented on code in PR #277: URL: https://github.com/apache/iceberg-rust/pull/277#discussion_r1572416327 ## crates/iceberg/src/arrow/schema.rs: ## @@ -385,25 +389,236 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter { } } +struct ToArrowSchemaConverter;

Re: [PR] feat: Implement the conversion from Iceberg Schema to Arrow Schema [iceberg-rust]

liurenjie1024 merged PR #277: URL: https://github.com/apache/iceberg-rust/pull/277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: Implement the conversion from Iceberg Schema to Arrow Schema [iceberg-rust]

liurenjie1024 commented on code in PR #277: URL: https://github.com/apache/iceberg-rust/pull/277#discussion_r1572414766 ## crates/iceberg/src/arrow/schema.rs: ## @@ -385,25 +389,236 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter { } } +struct ToArrowSchemaConverter;

Re: [PR] feat: Glue Catalog - table operations (3/3) [iceberg-rust]

liurenjie1024 commented on PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#issuecomment-2066617634 cc @Fokko @Xuanwo @sdd PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Kafka-connect: Handle namespace creation for auto table creation [iceberg]

ajantha-bhat commented on code in PR #10186: URL: https://github.com/apache/iceberg/pull/10186#discussion_r1572380923 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/IcebergWriterFactoryTest.java: ## @@ -83,4 +90,26 @@ public void testAutoCreateTable

Re: [PR] Kafka-connect: Handle namespace creation for auto table creation [iceberg]

ajantha-bhat commented on code in PR #10186: URL: https://github.com/apache/iceberg/pull/10186#discussion_r1572380923 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/IcebergWriterFactoryTest.java: ## @@ -83,4 +90,26 @@ public void testAutoCreateTable

Re: [PR] Kafka-connect: Handle namespace creation for auto table creation [iceberg]

ajantha-bhat commented on code in PR #10186: URL: https://github.com/apache/iceberg/pull/10186#discussion_r1572379348 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/IcebergWriterFactoryTest.java: ## @@ -47,7 +54,7 @@ public class IcebergWriterFactor

[I] Connect to multiple Azure accounts [iceberg-python]

cccs-jory opened a new issue, #636: URL: https://github.com/apache/iceberg-python/issues/636 ### Feature Request / Improvement I got PyIceberg working connecting to a SQL catalog (Postgres JDBC) and connecting to an ADLFS account using the `account-name` and `account-key` configurati

Re: [PR] Core: Improve size check in CatalogTests [iceberg]

Fokko merged PR #10182: URL: https://github.com/apache/iceberg/pull/10182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Core: Improve size check in CatalogTests [iceberg]

Fokko commented on PR #10182: URL: https://github.com/apache/iceberg/pull/10182#issuecomment-2066572996 Thanks for fixing this @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Flink 1.19: Run without Hadoop [iceberg]

Fokko commented on code in PR #7369: URL: https://github.com/apache/iceberg/pull/7369#discussion_r1572346493 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -69,7 +69,7 @@ private FlinkConfigOptions() {} public static final ConfigOpt

Re: [PR] chore: update roadmap [iceberg-rust]

liurenjie1024 commented on code in PR #336: URL: https://github.com/apache/iceberg-rust/pull/336#discussion_r1569808108 ## README.md: ## @@ -50,19 +50,19 @@ expand to other service. Reader | Feature| Status | -|--

Re: [PR] feat: Glue Catalog - table operations (3/3) [iceberg-rust]

liurenjie1024 commented on PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#issuecomment-2066513666 Let's wait a moment to see if others have comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1572265647 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,393 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1572263328 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,393 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Simplify expression when doing `{and,or}` operations [iceberg-rust]

marvinlanhenke commented on PR #339: URL: https://github.com/apache/iceberg-rust/pull/339#issuecomment-2066409941 > Everyone is invited to comment on my limited 🦀 knowledge I'm sorry I cannot comment on this - other than LGTM 😏 thanks @Fokko -- This is an automated message from the

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

marvinlanhenke commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1572246030 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,393 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more con

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

marvinlanhenke commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1572233768 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +239,27 @@ impl TableScan { .boxed()) } +fn create_manifest_eval_factory( +//&self, +

Re: [I] Compatibility issues with `org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0` [iceberg-rust]

martin-g commented on issue #338: URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2066328840 > @martin-g Do you have any ETA on Avro Rust 0.17? The Rust SDK is released with all other SDKs, i.e. when 1.12.0/1.11.4 is released. -- This is an automated message fro

Re: [I] Compatibility issues with `org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0` [iceberg-rust]

Fokko commented on issue #338: URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2066312852 @zeodtr Thanks for raising this issue. Looks like we need some proper Spark/Rust integration tests :) @martin-g Do you have any ETA on Avro Rust 0.17? -- This is an automat

[PR] Kafka-connect: Update config description [iceberg]

ajantha-bhat opened a new pull request, #10184: URL: https://github.com/apache/iceberg/pull/10184 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Infra: Track subtasks from Iceberg improvement proposal [iceberg]

ajantha-bhat commented on PR #10183: URL: https://github.com/apache/iceberg/pull/10183#issuecomment-2066295635 cc: @jbonofre, @liurenjie1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] Infra: Track subtasks from Iceberg improvement proposal [iceberg]

ajantha-bhat opened a new pull request, #10183: URL: https://github.com/apache/iceberg/pull/10183 As discussed in the mailing list: https://lists.apache.org/thread/ksgzw5wpqpoxvhlqo9xvn38j5tjb9nxs -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on PR #10151: URL: https://github.com/apache/iceberg/pull/10151#issuecomment-2066270577 > Quick question: now that flink 1.19 is available in the repo, do we still merge this to 1.18 and then later we port it to 1.19 and all the other versions? I usually try to keep th

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1572152325 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkInputFile.java: ## @@ -0,0 +1,179 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1572150806 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1572151051 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1572147961 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[PR] Simplify expression when doing `{and,or}` operations [iceberg-rust]

Fokko opened a new pull request, #339: URL: https://github.com/apache/iceberg-rust/pull/339 This will make sure that we nicely reduce the expression in the inclusive projection visitor: https://github.com/apache/iceberg-rust/blob/de80a2436bb2fbbd5b4ec6bcafd0bd041b263595/crates/iceberg/src/e

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1572124567 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

marvinlanhenke commented on PR #335: URL: https://github.com/apache/iceberg-rust/pull/335#issuecomment-2066180837 > This looks great! > > I think @marvinlanhenke has a valid concern on when to apply the rewrite-not. Let's defer that discussion when we start wiring everything together

Re: [I] Spark procedure to compute partition stats. [iceberg]

ShyamalaGowri commented on issue #10106: URL: https://github.com/apache/iceberg/issues/10106#issuecomment-2066148903 Hi @ajantha-bhat this is a widely required feature as it greatly affects the performance when spark executes queries on large scale data. Do we have any work in progress to

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

pvary commented on PR #10179: URL: https://github.com/apache/iceberg/pull/10179#issuecomment-2066090239 Please update the description of the PR. Also link the previous versions, docs, relevant stuff. So in the future it is easier to find them -- This is an automated message from the Apach

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1572022378 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/SinkV2Aggregator.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

pvary commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1572021707 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/DeltaManifests.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

SourabhBadhya commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1572010081 ## docs/docs/hive.md: ## @@ -431,12 +466,120 @@ ALTER TABLE t SET TBLPROPERTIES ('storage_handler'='org.apache.iceberg.mr.hive.H During the migration the data

Re: [PR] feat: Implement the conversion from Iceberg Schema to Arrow Schema [iceberg-rust]

ZENOTME commented on code in PR #277: URL: https://github.com/apache/iceberg-rust/pull/277#discussion_r1571986756 ## crates/iceberg/src/arrow/schema.rs: ## @@ -385,25 +389,236 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter { } } +struct ToArrowSchemaConverter; + +en

Re: [PR] feat: Implement the conversion from Iceberg Schema to Arrow Schema [iceberg-rust]

ZENOTME commented on code in PR #277: URL: https://github.com/apache/iceberg-rust/pull/277#discussion_r1571985492 ## crates/iceberg/src/arrow/schema.rs: ## @@ -385,25 +389,236 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter { } } +struct ToArrowSchemaConverter; + +en

[I] Compatibility issues with `org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0` [iceberg-rust]

zeodtr opened a new issue, #338: URL: https://github.com/apache/iceberg-rust/issues/338 Hi, I've been developing a query engine that uses `iceberg-rust` crate. Upon checking Iceberg compatibility with org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.4.3, I didn't encounter any issu

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#issuecomment-2065971864 There are intentionally a lot of `todo!`s in here. The aim is to get this PR merged so that more people can contribute implementations for the different visitor methods that need implement

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

sdd commented on PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#issuecomment-2065967273 @Fokko @liurenjie1024 @marvinlanhenke: I've rebased this on top of main now that the `InclusiveProjection` has been merged and it would be good to get some initial feedback. -- This is

Re: [PR] Build: Bump moto from 5.0.2 to 5.0.5 [iceberg-python]

HonahX merged PR #631: URL: https://github.com/apache/iceberg-python/pull/631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

Fokko merged PR #335: URL: https://github.com/apache/iceberg-rust/pull/335 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[PR] Core: Improve size check in CatalogTests [iceberg]

nastra opened a new pull request, #10182: URL: https://github.com/apache/iceberg/pull/10182 These assertions failed for me in a different context and the only info they would print is `0 != 1`. I've slightly updated the check so that when the assertion fails, you'll get more context:

Re: [PR] Build: Bump duckdb from 0.10.0 to 0.10.2 [iceberg-python]

HonahX merged PR #629: URL: https://github.com/apache/iceberg-python/pull/629 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Build: Bump typing-extensions from 4.9.0 to 4.11.0 [iceberg-python]

HonahX merged PR #630: URL: https://github.com/apache/iceberg-python/pull/630 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg