[I] Rest ratalog hostname resolution fail when submiting a job [iceberg]

guitcastro opened a new issue, #9709: URL: https://github.com/apache/iceberg/issues/9709 ### Apache Iceberg version 1.4.2 ### Query engine Spark ### Please describe the bug 🐞 I am following the [quickstart](https://iceberg.apache.org/spark-quickstart/) guid

Re: [I] Tracking issues of Iceberg Rust 0.2.0 Release [iceberg-rust]

liurenjie1024 commented on issue #180: URL: https://github.com/apache/iceberg-rust/issues/180#issuecomment-1937420958 > I think we're ready for a release. Do we want to get in https://github.com/apache/iceberg-rust/pull/193 as well? Yes, it will make the env setup error more user fri

[PR] Build: Bump mkdocs-material from 9.5.7 to 9.5.9 [iceberg]

dependabot[bot] opened a new pull request, #9708: URL: https://github.com/apache/iceberg/pull/9708 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.7 to 9.5.9. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs-ma

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.37.0 [iceberg]

dependabot[bot] closed pull request #9637: Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.37.0 URL: https://github.com/apache/iceberg/pull/9637 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.37.0 [iceberg]

dependabot[bot] commented on PR #9637: URL: https://github.com/apache/iceberg/pull/9637#issuecomment-1937418003 Superseded by #9707. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.38.0 [iceberg]

dependabot[bot] opened a new pull request, #9707: URL: https://github.com/apache/iceberg/pull/9707 Bumps [com.palantir.baseline:gradle-baseline-java](https://github.com/palantir/gradle-baseline) from 4.42.0 to 5.38.0. Release notes Sourced from https://github.com/palantir/gradle-b

[PR] Build: Bump org.assertj:assertj-core from 3.25.2 to 3.25.3 [iceberg]

dependabot[bot] opened a new pull request, #9706: URL: https://github.com/apache/iceberg/pull/9706 Bumps [org.assertj:assertj-core](https://github.com/assertj/assertj) from 3.25.2 to 3.25.3. Release notes Sourced from https://github.com/assertj/assertj/releases";>org.assertj:assert

[PR] Build: Bump software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin from 2.0.0 to 2.0.1 [iceberg]

dependabot[bot] opened a new pull request, #9705: URL: https://github.com/apache/iceberg/pull/9705 Bumps [software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin](https://github.com/aws/aws-s3-accessgrants-plugin-java-v2) from 2.0.0 to 2.0.1. Commits See full diff in h

[PR] Build: Bump org.testcontainers:testcontainers from 1.19.4 to 1.19.5 [iceberg]

dependabot[bot] opened a new pull request, #9704: URL: https://github.com/apache/iceberg/pull/9704 Bumps [org.testcontainers:testcontainers](https://github.com/testcontainers/testcontainers-java) from 1.19.4 to 1.19.5. Release notes Sourced from https://github.com/testcontainers/t

[PR] Build: Bump io.airlift:aircompressor from 0.25 to 0.26 [iceberg]

dependabot[bot] opened a new pull request, #9700: URL: https://github.com/apache/iceberg/pull/9700 Bumps [io.airlift:aircompressor](https://github.com/airlift/aircompressor) from 0.25 to 0.26. Commits https://github.com/airlift/aircompressor/commit/8b9414d358e0a25a750445a4e17ca

[PR] Build: Bump org.openapitools:openapi-generator-gradle-plugin from 6.6.0 to 7.3.0 [iceberg]

dependabot[bot] opened a new pull request, #9703: URL: https://github.com/apache/iceberg/pull/9703 Bumps [org.openapitools:openapi-generator-gradle-plugin](https://github.com/OpenAPITools/openapi-generator) from 6.6.0 to 7.3.0. Release notes Sourced from https://github.com/OpenAPI

[PR] Build: Bump tez010 from 0.10.2 to 0.10.3 [iceberg]

dependabot[bot] opened a new pull request, #9702: URL: https://github.com/apache/iceberg/pull/9702 Bumps `tez010` from 0.10.2 to 0.10.3. Updates `org.apache.tez:tez-dag` from 0.10.2 to 0.10.3 Updates `org.apache.tez:tez-mapreduce` from 0.10.2 to 0.10.3 Dependabot will res

[PR] Build: Bump software.amazon.awssdk:bom from 2.23.17 to 2.24.0 [iceberg]

dependabot[bot] opened a new pull request, #9701: URL: https://github.com/apache/iceberg/pull/9701 Bumps software.amazon.awssdk:bom from 2.23.17 to 2.24.0. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softwar

[PR] Build: Bump junit from 5.10.1 to 5.10.2 [iceberg]

dependabot[bot] opened a new pull request, #9699: URL: https://github.com/apache/iceberg/pull/9699 Bumps `junit` from 5.10.1 to 5.10.2. Updates `org.junit.jupiter:junit-jupiter` from 5.10.1 to 5.10.2 Release notes Sourced from https://github.com/junit-team/junit5/releases";>org.j

Re: [I] Tracking issues of Iceberg Rust 0.2.0 Release [iceberg-rust]

Fokko commented on issue #180: URL: https://github.com/apache/iceberg-rust/issues/180#issuecomment-1937401499 I think we're ready for a release. Do we want to get in https://github.com/apache/iceberg-rust/pull/193 as well? -- This is an automated message from the Apache Git Service. To r

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485419930 ## tests/catalog/test_hive.py: ## @@ -294,6 +294,37 @@ def test_create_table(table_schema_simple: Schema, hive_database: HiveDatabase, assert meta

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485419930 ## tests/catalog/test_hive.py: ## @@ -294,6 +294,37 @@ def test_create_table(table_schema_simple: Schema, hive_database: HiveDatabase, assert meta

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

Fokko commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485413474 ## pyiceberg/table/metadata.py: ## @@ -260,8 +260,10 @@ def set_v2_compatible_defaults(cls, data: Dict[str, Any]) -> Dict[str, Any]: The TableMetadata

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485414210 ## tests/catalog/test_hive.py: ## @@ -294,6 +294,37 @@ def test_create_table(table_schema_simple: Schema, hive_database: HiveDatabase, assert meta

Re: [PR] Add Thrift and Hive to NOTICE [iceberg-python]

Fokko commented on PR #410: URL: https://github.com/apache/iceberg-python/pull/410#issuecomment-1937387259 @danielcweeks I just replied on the mailing list. We do ship the Python-generated Thrift definitions that are stored here: https://github.com/apache/iceberg-python/tree/main/vendor

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485409963 ## pyiceberg/table/metadata.py: ## @@ -260,8 +260,10 @@ def set_v2_compatible_defaults(cls, data: Dict[str, Any]) -> Dict[str, Any]: The Tab

Re: [PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

amogh-jahagirdar commented on code in PR #411: URL: https://github.com/apache/iceberg-python/pull/411#discussion_r1485409963 ## pyiceberg/table/metadata.py: ## @@ -260,8 +260,10 @@ def set_v2_compatible_defaults(cls, data: Dict[str, Any]) -> Dict[str, Any]: The Tab

[PR] Fix setting V1 format version for Non-REST catalogs [iceberg-python]

amogh-jahagirdar opened a new pull request, #411: URL: https://github.com/apache/iceberg-python/pull/411 Currently we always set V2 format for create table for non-rest Catalogs regardless of the specified version. This change addresses that. -- This is an automated message from the Apach

Re: [PR] Add Thrift and Hive to NOTICE [iceberg-python]

danielcweeks commented on PR #410: URL: https://github.com/apache/iceberg-python/pull/410#issuecomment-1937383779 @Fokko I'm not sure this is necessary since we don't actually bundle any thing in the project. Per the [ASF site](https://infra.apache.org/licensing-howto.html#bundled-vs-non-b

[PR] Add Thrift and Hive to NOTICE [iceberg-python]

Fokko opened a new pull request, #410: URL: https://github.com/apache/iceberg-python/pull/410 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [I] Slow parallel operations fail to commit [iceberg]

github-actions[bot] commented on issue #1286: URL: https://github.com/apache/iceberg/issues/1286#issuecomment-1937364310 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Iceberg Datasource Writer Should Automatically Prune Identity Transform Partition Columns [iceberg]

github-actions[bot] commented on issue #1281: URL: https://github.com/apache/iceberg/issues/1281#issuecomment-1937364304 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink: Implement Flink InputFormat and integrate it to FlinkCatalog [iceberg]

github-actions[bot] commented on issue #1275: URL: https://github.com/apache/iceberg/issues/1275#issuecomment-1937364293 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Add a github action to label PRs with the relevant subprojects that are affected [iceberg]

github-actions[bot] commented on issue #1277: URL: https://github.com/apache/iceberg/issues/1277#issuecomment-1937364299 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type ReferenceEquality. [iceberg]

github-actions[bot] commented on issue #1250: URL: https://github.com/apache/iceberg/issues/1250#issuecomment-1937364281 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] How to use `iceberg.mr.filter.expression` in the IcebergInputFormat? [iceberg]

github-actions[bot] commented on issue #1193: URL: https://github.com/apache/iceberg/issues/1193#issuecomment-1937364270 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Avoid rewriting big files in RewriteDataFilesAction [iceberg]

github-actions[bot] commented on issue #1159: URL: https://github.com/apache/iceberg/issues/1159#issuecomment-1937364247 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Error when creating table 'is not a directory or unable to create one' [iceberg]

github-actions[bot] commented on issue #1163: URL: https://github.com/apache/iceberg/issues/1163#issuecomment-1937364254 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[PR] Fix header links with underscores in title [iceberg]

raghits opened a new pull request, #9697: URL: https://github.com/apache/iceberg/pull/9697 Closes #9617 Changed all links to use `#expire_snapshots` instead of `#expire-snapshots` -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Fix header links with underscores in title. [iceberg]

raghits commented on issue #9617: URL: https://github.com/apache/iceberg/issues/9617#issuecomment-1937086864 Thanks. Sounds good. Also, looks like all procedure names have a `_` separator. So it would be consistent with that convention. -- This is an automated message from the Apache Git

Re: [I] Fix header links with underscores in title. [iceberg]

bitsondatadev commented on issue #9617: URL: https://github.com/apache/iceberg/issues/9617#issuecomment-1937083814 To keep compatibility with the old site, we should support both. The only time this becomes an issue is if you have a saved link (or Google in the short term) but using the `-`

Re: [I] rewrite_data_files procedure fails with Premature end of Content-Length when using S3 client [iceberg]

paulpaul1076 commented on issue #9679: URL: https://github.com/apache/iceberg/issues/9679#issuecomment-1937083012 @nastra The easiest way to reproduce it is just use my streaming job, just leave it running, maybe for a few days even. And also schedule in airflow the compaction job to run ev

Re: [I] Fix header links with underscores in title. [iceberg]

raghits commented on issue #9617: URL: https://github.com/apache/iceberg/issues/9617#issuecomment-1937082756 @bitsondatadev looks like `https://iceberg.apache.org/docs/latest/spark-procedures/#expire_snapshots` is valid URL. Should we use `_` underscore instead of hyphen `-` for these link

Re: [I] rewrite_data_files procedure fails with Premature end of Content-Length when using S3 client [iceberg]

paulpaul1076 commented on issue #9679: URL: https://github.com/apache/iceberg/issues/9679#issuecomment-1937082611 Let me know if you manage to do it or not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Arrow: Support Large Binary when using `to_arrow` [iceberg-python]

castedice commented on PR #409: URL: https://github.com/apache/iceberg-python/pull/409#issuecomment-1937052110 Thanks for review This change will require a few changes to the documentation. After checking the documentation, I'll create an additional PR. -- This is an automated me

Re: [PR] Arrow: Support Large Binary when using `to_arrow` [iceberg-python]

Fokko merged PR #409: URL: https://github.com/apache/iceberg-python/pull/409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Hive locking [iceberg-python]

danielcweeks merged PR #405: URL: https://github.com/apache/iceberg-python/pull/405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Hive locking [iceberg-python]

danielcweeks commented on code in PR #405: URL: https://github.com/apache/iceberg-python/pull/405#discussion_r1485173447 ## pyiceberg/catalog/hive.py: ## @@ -363,15 +381,23 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons self._write_

Re: [PR] Hive locking [iceberg-python]

danielcweeks commented on code in PR #405: URL: https://github.com/apache/iceberg-python/pull/405#discussion_r1485172192 ## tests/integration/test_reads.py: ## @@ -467,3 +469,26 @@ def test_null_list_and_map(catalog: Catalog) -> None: # assert arrow_table["col_list_with_str

Re: [PR] Hive locking [iceberg-python]

amogh-jahagirdar commented on code in PR #405: URL: https://github.com/apache/iceberg-python/pull/405#discussion_r1485171524 ## pyiceberg/catalog/hive.py: ## @@ -363,15 +381,23 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons self._wr

Re: [PR] Hive locking [iceberg-python]

Fokko commented on code in PR #405: URL: https://github.com/apache/iceberg-python/pull/405#discussion_r1485146614 ## tests/integration/test_reads.py: ## @@ -467,3 +469,26 @@ def test_null_list_and_map(catalog: Catalog) -> None: # assert arrow_table["col_list_with_struct"].t

[PR] Docs: Fix broken strike-through markup [iceberg]

munabedan opened a new pull request, #9696: URL: https://github.com/apache/iceberg/pull/9696 This commit should fix broken strike-through markup on https://iceberg.apache.org/spec/ page ![strikethroughfix](https://github.com/apache/iceberg/assets/45054928/2bcbe93d-2b7b-4cb9-84fb-a39e

[I] add support for DuckDB views as a valid data format [iceberg-python]

djouallah opened a new issue, #407: URL: https://github.com/apache/iceberg-python/issues/407 ### Feature Request / Improvement arrow table is taking a lot of memory and crash the system with any non trivial amount of data please add DuckDB views a valid data source format, I had the

Re: [I] rewrite_data_files procedure fails with Premature end of Content-Length when using S3 client [iceberg]

nastra commented on issue #9679: URL: https://github.com/apache/iceberg/issues/9679#issuecomment-1936929767 Thanks @paulpaul1076, I will try and reproduce this next week on my end -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Spark: Fix SparkTable to use name and effective snapshotID for comparing [iceberg]

nastra commented on code in PR #9455: URL: https://github.com/apache/iceberg/pull/9455#discussion_r1485011353 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -405,15 +407,18 @@ public boolean equals(Object other) { return false;