Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

2024-04-18 Thread via GitHub
sdd commented on code in PR #335: URL: https://github.com/apache/iceberg-rust/pull/335#discussion_r1571889478 ## crates/iceberg/src/expr/visitors/inclusive_projection.rs: ## @@ -0,0 +1,371 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-04-18 Thread via GitHub
SourabhBadhya commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1571882609 ## docs/docs/hive.md: ## @@ -34,6 +34,32 @@ Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following feat !!! warning DML operations w

Re: [PR] Build: Bump adlfs from 2024.2.0 to 2024.4.1 [iceberg-python]

2024-04-18 Thread via GitHub
HonahX merged PR #627: URL: https://github.com/apache/iceberg-python/pull/627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Build: Bump pyarrow from 15.0.0 to 15.0.2 [iceberg-python]

2024-04-18 Thread via GitHub
HonahX merged PR #628: URL: https://github.com/apache/iceberg-python/pull/628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Changes in describe behaviour of a table break partition info? [iceberg]

2024-04-18 Thread via GitHub
nastra commented on issue #10174: URL: https://github.com/apache/iceberg/issues/10174#issuecomment-2065833120 @brysd this is something that changed in Spark with https://github.com/apache/spark/commit/b581b1499abc1903bb742480bb8cac3659ebe185 -- This is an automated message from the Apache

Re: [I] spark.table() raises warn: Unclosed S3FileIO instance in NessieTableOperations [iceberg]

2024-04-18 Thread via GitHub
ajantha-bhat commented on issue #10144: URL: https://github.com/apache/iceberg/issues/10144#issuecomment-2065831049 the count from `buildIcebergCatalog` has to be closed for its fileIO to be closed. Now we are getting fileIO not closed warning because, catalog is not closed. --

Re: [I] spark.table() raises warn: Unclosed S3FileIO instance in NessieTableOperations [iceberg]

2024-04-18 Thread via GitHub
KingLommel commented on issue #10144: URL: https://github.com/apache/iceberg/issues/10144#issuecomment-2065825809 @ajantha-bhat This user defined function just calls the function ```org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:284)``` and puts in the right arguments a

[I] How to avoid performing partition key sorting when inserting into a partitioned Iceberg table? [iceberg]

2024-04-18 Thread via GitHub
(dt) stored as iceberg; ``` 2. Insert data with one partition key value into a partitioned table ``` insert into temp.partition_table select dt ,text as contents from temp.dataset where dt = '20240418' ``` 3. phy

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-18 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1571188458 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2332,119 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertT

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-04-18 Thread via GitHub
ajantha-bhat commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1571841196 ## docs/docs/hive.md: ## @@ -431,12 +466,120 @@ ALTER TABLE t SET TBLPROPERTIES ('storage_handler'='org.apache.iceberg.mr.hive.H During the migration the data

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-04-18 Thread via GitHub
nastra commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1571832123 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkV2Committer.java: ## @@ -0,0 +1,808 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-04-18 Thread via GitHub
nastra commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1571831392 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkV2Committer.java: ## @@ -0,0 +1,808 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-04-18 Thread via GitHub
nastra commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1571831722 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkV2Committer.java: ## @@ -0,0 +1,808 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] A new implementation of an Iceberg Sink [WIP] that will be used with upcoming Flink Compaction jobs [iceberg]

2024-04-18 Thread via GitHub
nastra commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1571830165 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestFlinkManifest.java: ## @@ -0,0 +1,295 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Verify release quality of life improvements [iceberg-python]

2024-04-18 Thread via GitHub
HonahX commented on code in PR #626: URL: https://github.com/apache/iceberg-python/pull/626#discussion_r1571811203 ## mkdocs/docs/verify-release.md: ## @@ -105,15 +105,17 @@ make test To run the full integration tests: ```sh -make test-s3 +make test-integration Review Comme

Re: [PR] add `InclusiveProjection` Visitor [iceberg-rust]

2024-04-18 Thread via GitHub
sdd commented on PR #335: URL: https://github.com/apache/iceberg-rust/pull/335#issuecomment-2065786584 I agree that rewrite-not should get applied. But it is not the responsibility of the `InclusiveProjection` itself. In this design, it happens already earlier on on the process, at the poin

Re: [PR] Kevinjqliu/poc parallelize tests [iceberg-python]

2024-04-18 Thread via GitHub
corleyma commented on code in PR #598: URL: https://github.com/apache/iceberg-python/pull/598#discussion_r1571723160 ## pyproject.toml: ## @@ -560,6 +561,474 @@ ignore_missing_imports = true module = "tenacity.*" ignore_missing_imports = true +[[tool.mypy.overrides]] Review

[I] Support for writing Parquet files from the Iceberg Java API without the Hadoop Configuration class [iceberg]

2024-04-18 Thread via GitHub
ms opened a new issue, #10180: URL: https://github.com/apache/iceberg/issues/10180 ### Feature Request / Improvement If the hadoop-common library is not present, trying to write a Parquet file: ```java DataWriter dataWriter = Parquet.writeData(file)

Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]

2024-04-18 Thread via GitHub
ms commented on issue #10127: URL: https://github.com/apache/iceberg/issues/10127#issuecomment-2065633615 Is there a difference in that wasbs is for blob storage, and abfss requires hierarchical storage (ADLSv2) to be enabled? If you try to read a blob storage account using the ADLS SDK

Re: [I] Add `to_file` with Python API [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #25: Add `to_file` with Python API URL: https://github.com/apache/iceberg-python/issues/25 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Add benchmark to the CI [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #27: Add benchmark to the CI URL: https://github.com/apache/iceberg-python/issues/27 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [I] Add support for Python 3.12 [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #28: URL: https://github.com/apache/iceberg-python/issues/28#issuecomment-2065518842 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Python: Add before- and after operations to `PreOrderSchemaVisitor` [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #32: Python: Add before- and after operations to `PreOrderSchemaVisitor` URL: https://github.com/apache/iceberg-python/issues/32 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Writing to S3 fails if the user is authenticated with `aws sso login` [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #39: URL: https://github.com/apache/iceberg-python/issues/39#issuecomment-2065518806 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] Add benchmark to the CI [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #27: URL: https://github.com/apache/iceberg-python/issues/27#issuecomment-2065518861 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Add `to_file` with Python API [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #25: URL: https://github.com/apache/iceberg-python/issues/25#issuecomment-2065518898 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Preserve `selected_fields` order in output [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #26: Preserve `selected_fields` order in output URL: https://github.com/apache/iceberg-python/issues/26 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Preserve `selected_fields` order in output [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #26: URL: https://github.com/apache/iceberg-python/issues/26#issuecomment-2065518884 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Add support for Python 3.12 [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #28: Add support for Python 3.12 URL: https://github.com/apache/iceberg-python/issues/28 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [I] Python: Add before- and after operations to `PreOrderSchemaVisitor` [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #32: URL: https://github.com/apache/iceberg-python/issues/32#issuecomment-2065518822 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Pass in the correct type for the VisitorWithParent [iceberg-python]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #58: URL: https://github.com/apache/iceberg-python/issues/58#issuecomment-2065518788 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] Partition Spec Evolution [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #2589: URL: https://github.com/apache/iceberg/issues/2589#issuecomment-2065517155 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] TableProperties 'write.format.default' not correct. [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #2440: URL: https://github.com/apache/iceberg/issues/2440#issuecomment-2065517004 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] TableProperties 'write.format.default' not correct. [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #2440: TableProperties 'write.format.default' not correct. URL: https://github.com/apache/iceberg/issues/2440 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Report Manifest/Metadata Files created During Migrate/Snapshot [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #2439: URL: https://github.com/apache/iceberg/issues/2439#issuecomment-2065516985 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Report Manifest/Metadata Files created During Migrate/Snapshot [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #2439: Report Manifest/Metadata Files created During Migrate/Snapshot URL: https://github.com/apache/iceberg/issues/2439 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] RepairManifestsAction [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #2435: URL: https://github.com/apache/iceberg/issues/2435#issuecomment-2065516969 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] RepairManifestsAction [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #2435: RepairManifestsAction URL: https://github.com/apache/iceberg/issues/2435 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Migrate table's "Rename to backup" may cause datafile movement [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #2425: Migrate table's "Rename to backup" may cause datafile movement URL: https://github.com/apache/iceberg/issues/2425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Migrate table's "Rename to backup" may cause datafile movement [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #2425: URL: https://github.com/apache/iceberg/issues/2425#issuecomment-2065516948 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Consider deprecating updating table state through ALTER TABLE ... SET TBLPROPERTIES [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #2424: URL: https://github.com/apache/iceberg/issues/2424#issuecomment-2065516919 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Consider deprecating updating table state through ALTER TABLE ... SET TBLPROPERTIES [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] closed issue #2424: Consider deprecating updating table state through ALTER TABLE ... SET TBLPROPERTIES URL: https://github.com/apache/iceberg/issues/2424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [I] Caching Tables in SparkCatalog via CachingCatalog by default leads to stale data [iceberg]

2024-04-18 Thread via GitHub
github-actions[bot] commented on issue #2319: URL: https://github.com/apache/iceberg/issues/2319#issuecomment-2065516839 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Verify release quality of life improvements [iceberg-python]

2024-04-18 Thread via GitHub
kevinjqliu commented on PR #626: URL: https://github.com/apache/iceberg-python/pull/626#issuecomment-2065491774 @Fokko @HonahX does it make sense to add the "verify release" link to the release email template? -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Verify release quality of life improvements [iceberg-python]

2024-04-18 Thread via GitHub
kevinjqliu commented on code in PR #626: URL: https://github.com/apache/iceberg-python/pull/626#discussion_r1571514702 ## mkdocs/docs/verify-release.md: ## @@ -105,15 +105,17 @@ make test To run the full integration tests: ```sh -make test-s3 +make test-integration Review C

Re: [PR] Verify release quality of life improvements [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #626: URL: https://github.com/apache/iceberg-python/pull/626#discussion_r1571468239 ## mkdocs/docs/verify-release.md: ## @@ -105,15 +105,17 @@ make test To run the full integration tests: ```sh -make test-s3 +make test-integration ``` -This w

Re: [PR] Verify release quality of life improvements [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #626: URL: https://github.com/apache/iceberg-python/pull/626#discussion_r1571466306 ## mkdocs/docs/verify-release.md: ## @@ -105,15 +105,17 @@ make test To run the full integration tests: ```sh -make test-s3 +make test-integration Review Commen

[PR] Build: Bump moto from 5.0.2 to 5.0.5 [iceberg-python]

2024-04-18 Thread via GitHub
dependabot[bot] opened a new pull request, #631: URL: https://github.com/apache/iceberg-python/pull/631 Bumps [moto](https://github.com/getmoto/moto) from 5.0.2 to 5.0.5. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 5.0.

[PR] Build: Bump typing-extensions from 4.9.0 to 4.11.0 [iceberg-python]

2024-04-18 Thread via GitHub
dependabot[bot] opened a new pull request, #630: URL: https://github.com/apache/iceberg-python/pull/630 Bumps [typing-extensions](https://github.com/python/typing_extensions) from 4.9.0 to 4.11.0. Release notes Sourced from https://github.com/python/typing_extensions/releases";>typ

[PR] Build: Bump duckdb from 0.10.0 to 0.10.2 [iceberg-python]

2024-04-18 Thread via GitHub
dependabot[bot] opened a new pull request, #629: URL: https://github.com/apache/iceberg-python/pull/629 Bumps [duckdb](https://github.com/duckdb/duckdb) from 0.10.0 to 0.10.2. Release notes Sourced from https://github.com/duckdb/duckdb/releases";>duckdb's releases. v0.10.2 Bu

[PR] Build: Bump pyarrow from 15.0.0 to 15.0.2 [iceberg-python]

2024-04-18 Thread via GitHub
dependabot[bot] opened a new pull request, #628: URL: https://github.com/apache/iceberg-python/pull/628 Bumps [pyarrow](https://github.com/apache/arrow) from 15.0.0 to 15.0.2. Commits https://github.com/apache/arrow/commit/e03105efc38edca4ca429bf967a17b4d0fbebe40";>e03105e MINO

[PR] Build: Bump adlfs from 2024.2.0 to 2024.4.1 [iceberg-python]

2024-04-18 Thread via GitHub
dependabot[bot] opened a new pull request, #627: URL: https://github.com/apache/iceberg-python/pull/627 Bumps [adlfs](https://github.com/fsspec/adlfs) from 2024.2.0 to 2024.4.1. Release notes Sourced from https://github.com/fsspec/adlfs/releases";>adlfs's releases. 2024.4.1

Re: [PR] Verify release quality of life improvements [iceberg-python]

2024-04-18 Thread via GitHub
kevinjqliu commented on code in PR #626: URL: https://github.com/apache/iceberg-python/pull/626#discussion_r1571442239 ## mkdocs/docs/verify-release.md: ## @@ -105,15 +105,17 @@ make test To run the full integration tests: Review Comment: I did not fully understand L87-89.

[PR] Verify release quality of life improvements [iceberg-python]

2024-04-18 Thread via GitHub
kevinjqliu opened a new pull request, #626: URL: https://github.com/apache/iceberg-python/pull/626 Minor edits to the "Verify Release" instructions. Added link to "verify release" in the release email. Here's what the "Cast the vote" section look like in Markdown ![Screenshot 20

Re: [PR] Flink: port #9464 to v1.17 and v1.19 [iceberg]

2024-04-18 Thread via GitHub
javrasya commented on PR #10177: URL: https://github.com/apache/iceberg/pull/10177#issuecomment-2065212817 Thank you for helping @elkhand 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Flink: backport PR #9464 for being able to serialize splits with bigger payload [iceberg]

2024-04-18 Thread via GitHub
javrasya commented on PR #10178: URL: https://github.com/apache/iceberg/pull/10178#issuecomment-2065211514 Closing since this was already done (Kudos to @elkhand ) https://github.com/apache/iceberg/pull/10177 -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] Flink: backport PR #9464 for being able to serialize splits with bigger payload [iceberg]

2024-04-18 Thread via GitHub
javrasya closed pull request #10178: Flink: backport PR #9464 for being able to serialize splits with bigger payload URL: https://github.com/apache/iceberg/pull/10178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-04-18 Thread via GitHub
RussellSpitzer commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1571301509 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/comet/CometColumnReader.java: ## @@ -0,0 +1,165 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-04-18 Thread via GitHub
RussellSpitzer commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1571295057 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/BaseColumnBatchLoader.java: ## @@ -0,0 +1,199 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-04-18 Thread via GitHub
RussellSpitzer commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1571293170 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/BaseColumnBatchLoader.java: ## @@ -0,0 +1,199 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Flink: port #9464 to v1.17 and v1.19 [iceberg]

2024-04-18 Thread via GitHub
stevenzwu merged PR #10177: URL: https://github.com/apache/iceberg/pull/10177 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-04-18 Thread via GitHub
pvary commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1571289437 ## docs/docs/hive.md: ## @@ -431,12 +466,120 @@ ALTER TABLE t SET TBLPROPERTIES ('storage_handler'='org.apache.iceberg.mr.hive.H During the migration the data files a

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-04-18 Thread via GitHub
RussellSpitzer commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1571286633 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkConfParser.java: ## @@ -196,6 +201,40 @@ private Duration toDuration(String time) { } }

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-04-18 Thread via GitHub
RussellSpitzer commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1571283635 ## spark/v3.4/build.gradle: ## @@ -70,8 +70,11 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") { exclude group: 'io.netty

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-04-18 Thread via GitHub
pvary commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1571271577 ## docs/docs/hive.md: ## @@ -34,6 +34,32 @@ Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following feat !!! warning DML operations work only

Re: [PR] Fix dependency with `deptry` [iceberg-python]

2024-04-18 Thread via GitHub
Fokko commented on code in PR #534: URL: https://github.com/apache/iceberg-python/pull/534#discussion_r1571256522 ## pyproject.toml: ## @@ -72,6 +72,10 @@ gcsfs = { version = ">=2023.1.0,<2024.1.0", optional = true } psycopg2-binary = { version = ">=2.9.6", optional = true } s

Re: [I] Iceberg may occur data duplication when use flink to write data to iceberg and commit failed [iceberg]

2024-04-18 Thread via GitHub
pvary commented on issue #10165: URL: https://github.com/apache/iceberg/issues/10165#issuecomment-2065036455 Could you please describe the exact si -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Iceberg may occur data duplication when use flink to write data to iceberg and commit failed [iceberg]

2024-04-18 Thread via GitHub
pvary closed issue #10165: Iceberg may occur data duplication when use flink to write data to iceberg and commit failed URL: https://github.com/apache/iceberg/issues/10165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Build: Bump aiohttp from 3.9.3 to 3.9.4 [iceberg-python]

2024-04-18 Thread via GitHub
Fokko merged PR #621: URL: https://github.com/apache/iceberg-python/pull/621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571245762 ## tests/integration/test_inspect_table.py: ## @@ -445,3 +445,65 @@ def check_pyiceberg_df_equals_spark_df(df: pa.Table, spark_df: DataFrame) -> Non df = t

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571245762 ## tests/integration/test_inspect_table.py: ## @@ -445,3 +445,65 @@ def check_pyiceberg_df_equals_spark_df(df: pa.Table, spark_df: DataFrame) -> Non df = t

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571245762 ## tests/integration/test_inspect_table.py: ## @@ -445,3 +445,65 @@ def check_pyiceberg_df_equals_spark_df(df: pa.Table, spark_df: DataFrame) -> Non df = t

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571245762 ## tests/integration/test_inspect_table.py: ## @@ -445,3 +445,65 @@ def check_pyiceberg_df_equals_spark_df(df: pa.Table, spark_df: DataFrame) -> Non df = t

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-18 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1571188458 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2332,119 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertT

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
Fokko commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571241435 ## tests/conftest.py: ## @@ -2060,7 +2060,7 @@ def spark() -> "SparkSession": .config("spark.sql.catalog.hive.warehouse", "s3://warehouse/hive/") .

Re: [PR] Build: Bump aiohttp from 3.9.3 to 3.9.4 [iceberg-python]

2024-04-18 Thread via GitHub
Fokko commented on PR #621: URL: https://github.com/apache/iceberg-python/pull/621#issuecomment-2064985759 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571237708 ## pyiceberg/table/__init__.py: ## @@ -3537,6 +3537,58 @@ def update_partitions_map( schema=table_schema, ) +def files(self) -> "pa.Table

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571235918 ## pyiceberg/table/__init__.py: ## @@ -3537,6 +3537,58 @@ def update_partitions_map( schema=table_schema, ) +def files(self) -> "pa.Table

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
geruh commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571235918 ## pyiceberg/table/__init__.py: ## @@ -3537,6 +3537,58 @@ def update_partitions_map( schema=table_schema, ) +def files(self) -> "pa.Table

Re: [PR] Flink: port #9464 to v1.17 and v1.19 [iceberg]

2024-04-18 Thread via GitHub
elkhand commented on PR #10177: URL: https://github.com/apache/iceberg/pull/10177#issuecomment-2064958765 cc: @stevenzwu @pvary will appreciate your review on https://github.com/apache/iceberg/pull/9464 backport into Flink 1.17 and Flink 1.19. cc: @javrasya -- This is an automated m

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-18 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1571188458 ## core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java: ## @@ -2329,6 +2332,119 @@ public void multipleDiffsAgainstMultipleTablesLastFails() { assertT

Re: [PR] Bump to Spark 3.4.3 [iceberg-python]

2024-04-18 Thread via GitHub
HonahX merged PR #622: URL: https://github.com/apache/iceberg-python/pull/622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Fix dependabot issue [iceberg-python]

2024-04-18 Thread via GitHub
Fokko closed issue #618: Fix dependabot issue URL: https://github.com/apache/iceberg-python/issues/618 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: i

Re: [I] Fix dependabot issue [iceberg-python]

2024-04-18 Thread via GitHub
Fokko commented on issue #618: URL: https://github.com/apache/iceberg-python/issues/618#issuecomment-2064568680 Closing this, dependabot is back! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-18 Thread via GitHub
elkhand commented on PR #9464: URL: https://github.com/apache/iceberg/pull/9464#issuecomment-2064558588 @javrasya Thanks for the PR. I think @mas-chen wanted to mention this PR: https://github.com/apache/iceberg/pull/9334 instead. -- This is an automated message from the Apache Git Servi

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-18 Thread via GitHub
rodmeneses commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1571094073 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/FlinkFileIOTest.java: ## @@ -0,0 +1,212 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-18 Thread via GitHub
rodmeneses commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1571092822 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkInputFile.java: ## @@ -0,0 +1,179 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-18 Thread via GitHub
rodmeneses commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1571089828 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-18 Thread via GitHub
rodmeneses commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1571086551 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-18 Thread via GitHub
rodmeneses commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1571083046 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-04-18 Thread via GitHub
rodmeneses commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1571082650 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
kevinjqliu commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571070645 ## tests/conftest.py: ## @@ -2060,7 +2060,7 @@ def spark() -> "SparkSession": .config("spark.sql.catalog.hive.warehouse", "s3://warehouse/hive/")

Re: [PR] Add Files metadata table [iceberg-python]

2024-04-18 Thread via GitHub
Gowthami03B commented on code in PR #614: URL: https://github.com/apache/iceberg-python/pull/614#discussion_r1571063843 ## tests/conftest.py: ## @@ -2060,7 +2060,7 @@ def spark() -> "SparkSession": .config("spark.sql.catalog.hive.warehouse", "s3://warehouse/hive/")

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-04-18 Thread via GitHub
mas-chen commented on PR #9464: URL: https://github.com/apache/iceberg/pull/9464#issuecomment-2064457196 @javrasya manually but you can use this an example https://github.com/apache/iceberg/pull/9464 (see PR description) -- This is an automated message from the Apache Git Service. To resp

[PR] Build: Bump aiohttp from 3.9.3 to 3.9.4 [iceberg-python]

2024-04-18 Thread via GitHub
dependabot[bot] opened a new pull request, #621: URL: https://github.com/apache/iceberg-python/pull/621 Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.3 to 3.9.4. Release notes Sourced from https://github.com/aio-libs/aiohttp/releases";>aiohttp's releases. 3.9

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-04-18 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571037034 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [I] Integration tests performance degradation [iceberg-python]

2024-04-18 Thread via GitHub
kevinjqliu commented on issue #604: URL: https://github.com/apache/iceberg-python/issues/604#issuecomment-2064388651 Here's what I'm blocked on specifically. Parallelize this test `test_query_filter_appended_null`, ``` PYTEST_ARGS="-n auto -k test_query_filter_appended_null" /us

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-04-18 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571033881 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-04-18 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1571032953 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Kevinjqliu/poc parallelize tests [iceberg-python]

2024-04-18 Thread via GitHub
Fokko commented on PR #598: URL: https://github.com/apache/iceberg-python/pull/598#issuecomment-2064376042 Why do I still have to approve your runs? :D -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

  1   2   >