Re: [PR] mr：Fix issues 10639 [iceberg]

pvary commented on PR #10661: URL: https://github.com/apache/iceberg/pull/10661#issuecomment-2226801595 > compile jdk8, run jdk11, but we are actively working on jdk11 compile now, that should be in Hive-4.1.0 release When is Hive 4.1.0 planned? You might want to find and comment o

Re: [PR] mr：Fix issues 10639 [iceberg]

lurnagao-dahua commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1674923132 ## mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormats.java: ## @@ -381,6 +386,56 @@ public void testCustomCatalog() throws IOException { testI

Re: [PR] refine: move binary serialize in literal to datum [iceberg-rust]

ZENOTME commented on PR #456: URL: https://github.com/apache/iceberg-rust/pull/456#issuecomment-2226828202 cc @liurenjie1024 @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676794013 ## crates/iceberg/testdata/view_metadata/ViewMetadataV2Valid.json: ## @@ -0,0 +1,58 @@ +{ Review Comment: Ok, thanks Eduard for the Feedback. My preferred

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676794680 ## crates/iceberg/src/spec/view_metadata.rs: ## @@ -0,0 +1,682 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agre

Re: [PR] Core:support http lock-manager [iceberg]

snazy commented on PR #10688: URL: https://github.com/apache/iceberg/pull/10688#issuecomment-2226834838 It seems there are lot of individual points that should be discussed. @BsoBird do you mind adding a point to the "Discussions" section on the [agenda for the next community sync on July

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676795328 ## crates/iceberg/src/spec/view_metadata.rs: ## @@ -0,0 +1,682 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agre

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676795561 ## crates/iceberg/src/spec/view_metadata.rs: ## @@ -0,0 +1,682 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agre

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676794013 ## crates/iceberg/testdata/view_metadata/ViewMetadataV2Valid.json: ## @@ -0,0 +1,58 @@ +{ Review Comment: Ok, thanks Eduard for the Feedback. My preferred

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676796496 ## crates/iceberg/src/spec/view_version.rs: ## @@ -0,0 +1,347 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agree

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676800028 ## crates/iceberg/src/spec/view_version.rs: ## @@ -0,0 +1,347 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agree

Re: [PR] Core:support http lock-manager [iceberg]

BsoBird commented on PR #10688: URL: https://github.com/apache/iceberg/pull/10688#issuecomment-2226845799 @snazy Hello.I'm so glad you could get back to me. I'm just saying out loud the first thing that comes to mind: 1. Is everything done using rest-catalog? In many cases, use

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676801198 ## crates/iceberg/src/spec/view_version.rs: ## @@ -0,0 +1,347 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agree

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1676802744 ## crates/iceberg/src/spec/view_version.rs: ## @@ -0,0 +1,347 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agree

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#issuecomment-2226849477 @nastra I hope I addressed all your issues regarding functionality. If you are happy with the implementations, feel free to close our discussions :). @ZENOTME it would now be great t

Re: [PR] Core: Prevent dropping column which is referenced by active partition… [iceberg]

advancedxy commented on code in PR #10352: URL: https://github.com/apache/iceberg/pull/10352#discussion_r1676818347 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -533,6 +537,34 @@ private static Schema applyChanges( } } +Map> specToDeletes =

Re: [PR] Core: Prevent dropping column which is referenced by active partition… [iceberg]

advancedxy commented on PR #10352: URL: https://github.com/apache/iceberg/pull/10352#issuecomment-2226883163 > Sorry about the delay on this, got busy and forgot I had this open! I've seen more related issue reports to this, so I'm going to prioritize it. Well understood, and thanks f

[PR] Set boolean values in table properties to lowercase string [iceberg-python]

soumya-ghosh opened a new pull request, #924: URL: https://github.com/apache/iceberg-python/pull/924 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [I] Set properties boolean value to lowercase string [iceberg-python]

soumya-ghosh commented on issue #919: URL: https://github.com/apache/iceberg-python/issues/919#issuecomment-2226888124 @Fokko @kevinjqliu PR https://github.com/apache/iceberg-python/pull/924 is ready for review. -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] Add Snowflake catalog [iceberg-python]

prabodh1194 commented on PR #687: URL: https://github.com/apache/iceberg-python/pull/687#issuecomment-2226959566 @Fokko , considering that snowflake's rest catalog docs are still not released to public, we can only wait for it. However, I believe that this PR will continue to be relevant co

Re: [PR] Core:support http lock-manager [iceberg]

BsoBird commented on code in PR #10688: URL: https://github.com/apache/iceberg/pull/10688#discussion_r1675607925 ## core/src/main/java/org/apache/iceberg/lock/ServerSideHttpLockManager.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Core:support http lock-manager [iceberg]

BsoBird commented on code in PR #10688: URL: https://github.com/apache/iceberg/pull/10688#discussion_r1676850504 ## core/src/main/java/org/apache/iceberg/lock/ServerSideHttpLockManager.java: ## @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Standardize AWS credential names [iceberg-python]

syun64 commented on code in PR #922: URL: https://github.com/apache/iceberg-python/pull/922#discussion_r1676852897 ## pyiceberg/io/__init__.py: ## @@ -320,6 +331,13 @@ def _infer_file_io_from_scheme(path: str, properties: Properties) -> Optional[Fi return None +def _ge

Re: [PR] Standardize AWS credential names [iceberg-python]

syun64 commented on code in PR #922: URL: https://github.com/apache/iceberg-python/pull/922#discussion_r1676852897 ## pyiceberg/io/__init__.py: ## @@ -320,6 +331,13 @@ def _infer_file_io_from_scheme(path: str, properties: Properties) -> Optional[Fi return None +def _ge

Re: [PR] Build: don't include slf4j-api in bundled JARs [iceberg]

bryanck commented on PR #10665: URL: https://github.com/apache/iceberg/pull/10665#issuecomment-2226992307 Thanks for the contribution @devinrsmith ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Repair manifest action [iceberg]

szehon-ho commented on PR #10445: URL: https://github.com/apache/iceberg/pull/10445#issuecomment-2227007559 yea i think that makes sense, let's do it in a way that we can add more functionality later. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Repair manifest action [iceberg]

szehon-ho commented on PR #10445: URL: https://github.com/apache/iceberg/pull/10445#issuecomment-2227037227 @danielcweeks @tabmatfournier while we are discussing, do you guys think it makes sense later to integrate my functionality into this Repair action? its been awhile, but iirc it was

Re: [PR] Standardize AWS credential names [iceberg-python]

Fokko commented on code in PR #922: URL: https://github.com/apache/iceberg-python/pull/922#discussion_r1676882347 ## pyiceberg/io/__init__.py: ## @@ -320,6 +331,13 @@ def _infer_file_io_from_scheme(path: str, properties: Properties) -> Optional[Fi return None +def _get

Re: [PR] Standardize AWS credential names [iceberg-python]

Fokko commented on code in PR #922: URL: https://github.com/apache/iceberg-python/pull/922#discussion_r1676882383 ## pyiceberg/io/__init__.py: ## @@ -46,6 +48,10 @@ logger = logging.getLogger(__name__) +AWS_REGION = "client.region" Review Comment: Nice, that's a good fi

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

kevinjqliu commented on code in PR #921: URL: https://github.com/apache/iceberg-python/pull/921#discussion_r1676880923 ## pyiceberg/io/pyarrow.py: ## @@ -2079,36 +2082,63 @@ def _check_schema_compatible(table_schema: Schema, other_schema: pa.Schema, down Raises: V

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

kevinjqliu commented on PR #921: URL: https://github.com/apache/iceberg-python/pull/921#issuecomment-2227047760 @pdpark does this solution resolve your usecase? https://github.com/apache/iceberg-python/pull/829#issuecomment-2218116050 -- This is an automated message from the Apache

Re: [PR] Set boolean values in table properties to lowercase string [iceberg-python]

Fokko merged PR #924: URL: https://github.com/apache/iceberg-python/pull/924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Set properties boolean value to lowercase string [iceberg-python]

kevinjqliu commented on issue #919: URL: https://github.com/apache/iceberg-python/issues/919#issuecomment-2227049015 Resolved in #924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Create table properties does not support boolean value [iceberg-python]

kevinjqliu closed issue #895: Create table properties does not support boolean value URL: https://github.com/apache/iceberg-python/issues/895 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Set properties boolean value to lowercase string [iceberg-python]

kevinjqliu closed issue #919: Set properties boolean value to lowercase string URL: https://github.com/apache/iceberg-python/issues/919 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Create table properties does not support boolean value [iceberg-python]

kevinjqliu commented on issue #895: URL: https://github.com/apache/iceberg-python/issues/895#issuecomment-2227049076 Resolved in #924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[I] Calling load_table().scan().to_arrow() emits error on empty "names" field in name mapping [iceberg-python]

spock-abadai opened a new issue, #925: URL: https://github.com/apache/iceberg-python/issues/925 ### Apache Iceberg version main (development) ### Please describe the bug 🐞 On one of my iceberg tables, when I load a table and scan it, during the parsing of the name mappin

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

syun64 commented on PR #921: URL: https://github.com/apache/iceberg-python/pull/921#issuecomment-2227049610 Thank you for the review comments @kevinjqliu 🙏 > Porting over Fokko's comment from the other PR [#829 (review)](https://github.com/apache/iceberg-python/pull/829#pullrequestre

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

syun64 commented on code in PR #921: URL: https://github.com/apache/iceberg-python/pull/921#discussion_r1676884492 ## pyiceberg/io/pyarrow.py: ## @@ -2079,36 +2082,63 @@ def _check_schema_compatible(table_schema: Schema, other_schema: pa.Schema, down Raises: Value

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

syun64 commented on code in PR #921: URL: https://github.com/apache/iceberg-python/pull/921#discussion_r1676885325 ## pyiceberg/io/pyarrow.py: ## @@ -1450,14 +1451,17 @@ def field_partner(self, partner_struct: Optional[pa.Array], field_id: int, _: st except ValueEr

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

syun64 commented on code in PR #921: URL: https://github.com/apache/iceberg-python/pull/921#discussion_r1676885362 ## pyiceberg/io/pyarrow.py: ## @@ -2079,36 +2082,63 @@ def _check_schema_compatible(table_schema: Schema, other_schema: pa.Schema, down Raises: Value

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

kevinjqliu commented on PR #921: URL: https://github.com/apache/iceberg-python/pull/921#issuecomment-2227052294 My understanding of the "re-aligning" problem is the alignment between parquet and iceberg. similar to what Fokko mentioned here https://github.com/apache/iceberg-python

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

Fokko commented on code in PR #921: URL: https://github.com/apache/iceberg-python/pull/921#discussion_r1676911616 ## pyiceberg/io/pyarrow.py: ## @@ -2079,36 +2082,63 @@ def _check_schema_compatible(table_schema: Schema, other_schema: pa.Schema, down Raises: ValueE

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

Fokko commented on PR #921: URL: https://github.com/apache/iceberg-python/pull/921#issuecomment-2227093971 Just to expand on top of https://github.com/apache/iceberg-python/pull/921#issuecomment-2227052294 This is the situation where you already have two Iceberg schemas with the IDs

Re: [I] Calling load_table().scan().to_arrow() emits error on empty "names" field in name mapping [iceberg-python]

Fokko commented on issue #925: URL: https://github.com/apache/iceberg-python/issues/925#issuecomment-2227097367 @spock-abadai Thanks for reporting this. I agree that this seems to be incorrect. Are you interested in providing a PR? -- This is an automated message from the Apache Git Servi

Re: [I] Calling load_table().scan().to_arrow() emits error on empty "names" field in name mapping [iceberg-python]

spock-abadai commented on issue #925: URL: https://github.com/apache/iceberg-python/issues/925#issuecomment-2227109344 > @spock-abadai Thanks for reporting this. I agree that this seems to be incorrect. Are you interested in providing a PR? Sure, see #927 for review. -- This is an

Re: [PR] Concurrent table scans [iceberg-rust]

sdd commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1676960798 ## crates/iceberg/src/scan.rs: ## @@ -389,12 +333,158 @@ impl FileScanStreamContext { file_io, bound_filter, case_sensitive, +

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

syun64 commented on code in PR #921: URL: https://github.com/apache/iceberg-python/pull/921#discussion_r1676966897 ## tests/integration/test_add_files.py: ## @@ -501,14 +501,11 @@ def test_add_files_fails_on_schema_mismatch(spark: SparkSession, session_catalog )

Re: [PR] Allow writing `pa.Table` that are either a subset of table schema or in arbitrary order [iceberg-python]

syun64 commented on code in PR #921: URL: https://github.com/apache/iceberg-python/pull/921#discussion_r1676967257 ## pyiceberg/io/pyarrow.py: ## @@ -2079,36 +2083,63 @@ def _check_schema_compatible(table_schema: Schema, other_schema: pa.Schema, down Raises: Value

Re: [PR] Allow empty `names` in mapped field of Name Mapping [iceberg-python]

kevinjqliu commented on code in PR #927: URL: https://github.com/apache/iceberg-python/pull/927#discussion_r1676974835 ## tests/table/test_name_mapping.py: ## @@ -247,11 +247,6 @@ def test_mapping_lookup_by_name(table_name_mapping_nested: NameMapping) -> None: table_na

[PR] Core: Kryo serialization error with DataFile after copying [iceberg]

fengjiajie opened a new pull request, #10695: URL: https://github.com/apache/iceberg/pull/10695 In my use case with Iceberg 1.3, I have a Flink `function-1` that outputs a `DataStream`, which is then processed by the next function. The simplified code for `function-1` is as follows:

[PR] Build: Bump net.snowflake:snowflake-jdbc from 3.16.1 to 3.17.0 [iceberg]

dependabot[bot] opened a new pull request, #10696: URL: https://github.com/apache/iceberg/pull/10696 Bumps [net.snowflake:snowflake-jdbc](https://github.com/snowflakedb/snowflake-jdbc) from 3.16.1 to 3.17.0. Release notes Sourced from https://github.com/snowflakedb/snowflake-jdbc/

[PR] Build: Bump nessie from 0.92.0 to 0.92.1 [iceberg]

dependabot[bot] opened a new pull request, #10697: URL: https://github.com/apache/iceberg/pull/10697 Bumps `nessie` from 0.92.0 to 0.92.1. Updates `org.projectnessie.nessie:nessie-client` from 0.92.0 to 0.92.1 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.92.0

[PR] Build: Bump org.assertj:assertj-core from 3.26.0 to 3.26.3 [iceberg]

dependabot[bot] opened a new pull request, #10698: URL: https://github.com/apache/iceberg/pull/10698 Bumps [org.assertj:assertj-core](https://github.com/assertj/assertj) from 3.26.0 to 3.26.3. Release notes Sourced from https://github.com/assertj/assertj/releases";>org.assertj:asse

[PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.43.0 [iceberg]

dependabot[bot] opened a new pull request, #10699: URL: https://github.com/apache/iceberg/pull/10699 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.28.0 to 26.43.0. Release notes Sourced from https://github.com/googleapis/java-cloud-bo

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.42.0 [iceberg]

dependabot[bot] commented on PR #10556: URL: https://github.com/apache/iceberg/pull/10556#issuecomment-2227195410 Superseded by #10699. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.42.0 [iceberg]

dependabot[bot] closed pull request #10556: Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.42.0 URL: https://github.com/apache/iceberg/pull/10556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[PR] Build: Bump software.amazon.awssdk:bom from 2.26.16 to 2.26.20 [iceberg]

dependabot[bot] opened a new pull request, #10700: URL: https://github.com/apache/iceberg/pull/10700 Bumps software.amazon.awssdk:bom from 2.26.16 to 2.26.20. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=soft

Re: [PR] Allow empty `names` in mapped field of Name Mapping [iceberg-python]

spock-abadai commented on code in PR #927: URL: https://github.com/apache/iceberg-python/pull/927#discussion_r1677034401 ## tests/table/test_name_mapping.py: ## @@ -247,11 +247,6 @@ def test_mapping_lookup_by_name(table_name_mapping_nested: NameMapping) -> None: table_

Re: [PR] Allow empty `names` in mapped field of Name Mapping [iceberg-python]

spock-abadai commented on PR #927: URL: https://github.com/apache/iceberg-python/pull/927#issuecomment-2227208143 > Thanks for raising this PR @spock-abadai > > I think we should update line 40 as well: > > https://github.com/apache/iceberg-python/blob/3f44dfe711e96beda6aa8622c

Re: [PR] Allow empty `names` in mapped field of Name Mapping [iceberg-python]

Fokko commented on PR #927: URL: https://github.com/apache/iceberg-python/pull/927#issuecomment-2227217603 @spock-abadai Thanks again for working on this. I think we're almost there. Can you run `make lint` to fix the formatting issue? Thanks! -- This is an automated message from the Apac