[PR] Core: Add 'google' auth type to auth manager [iceberg]

2025-07-15 Thread via GitHub
ebyhr opened a new pull request, #13564: URL: https://github.com/apache/iceberg/pull/13564 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [I] Iceberg1.7.1 not support spark3.5.4 [iceberg]

2025-07-15 Thread via GitHub
wfxxh commented on issue #13047: URL: https://github.com/apache/iceberg/issues/13047#issuecomment-3077098825 https://github.com/user-attachments/assets/ab333154-af4e-456f-b304-6b6709f585e4"; /> https://github.com/user-attachments/assets/d4ab9a0d-3a84-40ea-9c30-16344c31bd97"; />

Re: [PR] feat: update pyiceberg/catalog/hive.py to support hive 4.x.x [iceberg-python]

2025-07-15 Thread via GitHub
Fokko commented on PR #2206: URL: https://github.com/apache/iceberg-python/pull/2206#issuecomment-3077069016 Great catch @kevinjqliu: > cant use `get_table_objects_by_name` its also removed in hive 4.0.1 To catch this, we probably want to regenerate the vendor package against 4

Re: [PR] feat: add schema conversion from avro `timestamp-millis` and `uuid` [iceberg-python]

2025-07-15 Thread via GitHub
Fokko commented on code in PR #2173: URL: https://github.com/apache/iceberg-python/pull/2173#discussion_r2209391896 ## pyiceberg/utils/schema_conversion.py: ## @@ -69,8 +69,10 @@ LOGICAL_FIELD_TYPE_MAPPING: Dict[Tuple[str, str], PrimitiveType] = { ("date", "int"): DateType

Re: [PR] feat: add timeout layer [iceberg-rust]

2025-07-15 Thread via GitHub
chenzl25 closed pull request #1513: feat: add timeout layer URL: https://github.com/apache/iceberg-rust/pull/1513 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[PR] feat: add timeout layer [iceberg-rust]

2025-07-15 Thread via GitHub
chenzl25 opened a new pull request, #1513: URL: https://github.com/apache/iceberg-rust/pull/1513 ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? ## Are these changes tested? -- This is an automated message

Re: [PR] feat: add schema conversion from avro `timestamp-millis` and `uuid` [iceberg-python]

2025-07-15 Thread via GitHub
matthias-Q commented on code in PR #2173: URL: https://github.com/apache/iceberg-python/pull/2173#discussion_r2209307665 ## pyiceberg/utils/schema_conversion.py: ## @@ -69,8 +69,10 @@ LOGICAL_FIELD_TYPE_MAPPING: Dict[Tuple[str, str], PrimitiveType] = { ("date", "int"): Dat

Re: [PR] feat: add schema conversion from avro `timestamp-millis` and `uuid` [iceberg-python]

2025-07-15 Thread via GitHub
matthias-Q commented on code in PR #2173: URL: https://github.com/apache/iceberg-python/pull/2173#discussion_r2209307665 ## pyiceberg/utils/schema_conversion.py: ## @@ -69,8 +69,10 @@ LOGICAL_FIELD_TYPE_MAPPING: Dict[Tuple[str, str], PrimitiveType] = { ("date", "int"): Dat

Re: [PR] feat: add schema conversion from avro `timestamp-millis` and `uuid` [iceberg-python]

2025-07-15 Thread via GitHub
Fokko commented on code in PR #2173: URL: https://github.com/apache/iceberg-python/pull/2173#discussion_r2209233326 ## pyiceberg/utils/schema_conversion.py: ## @@ -69,8 +69,10 @@ LOGICAL_FIELD_TYPE_MAPPING: Dict[Tuple[str, str], PrimitiveType] = { ("date", "int"): DateType

Re: [PR] feat: update pyiceberg/catalog/hive.py to support hive 4.x.x [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on PR #2206: URL: https://github.com/apache/iceberg-python/pull/2206#issuecomment-3076758106 ok this the expected replacement https://github.com/apache/hive/pull/3599/files#diff-2554b13fe8458e70912484d7e9c3f0c3403d0b27fe1affced872e576f2742ea9R268-R269 also feel

Re: [PR] Add schema conversion time [iceberg-python]

2025-07-15 Thread via GitHub
Fokko commented on code in PR #2215: URL: https://github.com/apache/iceberg-python/pull/2215#discussion_r2209225047 ## pyiceberg/utils/schema_conversion.py: ## @@ -68,8 +68,9 @@ LOGICAL_FIELD_TYPE_MAPPING: Dict[Tuple[str, str], PrimitiveType] = { ("date", "int"): DateTyp

Re: [PR] kafka-connect: resolve CVE-2025-48734 [iceberg]

2025-07-15 Thread via GitHub
ajantha-bhat commented on PR #13561: URL: https://github.com/apache/iceberg/pull/13561#issuecomment-3076749023 I locally tried by just adding one line under existing `resolutionStrategy` and everything worked `force 'commons-beanutils:commons-beanutils:1.11.0'` Trivy also ran after

Re: [PR] kafka-connect: resolve CVE-2025-48734 [iceberg]

2025-07-15 Thread via GitHub
ajantha-bhat commented on code in PR #13561: URL: https://github.com/apache/iceberg/pull/13561#discussion_r2209219898 ## kafka-connect/build.gradle: ## @@ -64,9 +64,14 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') { configurations { hive { ex

Re: [PR] kafka-connect: resolve CVE-2025-48734 [iceberg]

2025-07-15 Thread via GitHub
ajantha-bhat commented on code in PR #13561: URL: https://github.com/apache/iceberg/pull/13561#discussion_r2209219898 ## kafka-connect/build.gradle: ## @@ -64,9 +64,14 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') { configurations { hive { ex

Re: [PR] kafka-connect: resolve CVE-2025-48734 [iceberg]

2025-07-15 Thread via GitHub
ajantha-bhat commented on code in PR #13561: URL: https://github.com/apache/iceberg/pull/13561#discussion_r2209218886 ## kafka-connect/build.gradle: ## @@ -64,9 +64,14 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') { configurations { hive { ex

Re: [PR] Add schema conversion time [iceberg-python]

2025-07-15 Thread via GitHub
matthias-Q commented on code in PR #2215: URL: https://github.com/apache/iceberg-python/pull/2215#discussion_r2209205958 ## pyiceberg/utils/schema_conversion.py: ## @@ -68,8 +68,9 @@ LOGICAL_FIELD_TYPE_MAPPING: Dict[Tuple[str, str], PrimitiveType] = { ("date", "int"): Da

Re: [PR] feat: update pyiceberg/catalog/hive.py to support hive 4.x.x [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on PR #2206: URL: https://github.com/apache/iceberg-python/pull/2206#issuecomment-3076724802 Looks like `get_table_objects_by_name` was removed in 4.0.1 too https://github.com/apache/hive/pull/3599/files#diff-47ffee8549a256db9156ce4287f750674b1689362362db066010ff6031

Re: [PR] Add schema conversion time [iceberg-python]

2025-07-15 Thread via GitHub
matthias-Q commented on code in PR #2215: URL: https://github.com/apache/iceberg-python/pull/2215#discussion_r2209204616 ## tests/utils/test_schema_conversion.py: ## @@ -341,8 +342,20 @@ def test_convert_uuid_fixed_type() -> None: assert actual == UUIDType() +def test_c

[PR] try integration test with hive 4.0.1 and local pyiceberg [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu opened a new pull request, #2217: URL: https://github.com/apache/iceberg-python/pull/2217 # Rationale for this change # Are these changes tested? # Are there any user-facing changes? -- This is an automated message from the Apache Gi

Re: [PR] chore: upgrade nanoarrow dependency [iceberg-cpp]

2025-07-15 Thread via GitHub
wgtmac commented on PR #146: URL: https://github.com/apache/iceberg-cpp/pull/146#issuecomment-3076695781 > It might be better to update to 0.7.0 and stick with dlcdn? archive.apache may be flaky (as we've seen with other dependencies) Perhaps we need to add archive.apache as a fallbac

Re: [PR] chore: update nanoarrow download URL [iceberg-cpp]

2025-07-15 Thread via GitHub
gty404 commented on PR #146: URL: https://github.com/apache/iceberg-cpp/pull/146#issuecomment-3076658095 > It might be better to update to 0.7.0 and stick with dlcdn? archive.apache may be flaky (as we've seen with other dependencies) After upgrading to 0.7.0, there are other compilat

Re: [PR] kafka-connect: resolve CVE-2025-48734 [iceberg]

2025-07-15 Thread via GitHub
ajantha-bhat commented on code in PR #13561: URL: https://github.com/apache/iceberg/pull/13561#discussion_r2209112111 ## kafka-connect/build.gradle: ## @@ -262,4 +267,4 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-transforms') { test { useJUnitPlatform()

Re: [PR] chore: update nanoarrow download URL [iceberg-cpp]

2025-07-15 Thread via GitHub
zhjwpku commented on PR #146: URL: https://github.com/apache/iceberg-cpp/pull/146#issuecomment-3076528358 I got the following error with the old url, the new url WFM. ``` CMake Error at nanoarrow-subbuild/nanoarrow-populate-prefix/src/nanoarrow-populate-stamp/download-nanoarrow-pop

Re: [PR] chore: update nanoarrow download URL [iceberg-cpp]

2025-07-15 Thread via GitHub
zhjwpku commented on PR #146: URL: https://github.com/apache/iceberg-cpp/pull/146#issuecomment-3076527677 I got the following error with the old url, the new url WFM. ``` CMake Error at nanoarrow-subbuild/nanoarrow-populate-prefix/src/nanoarrow-populate-stamp/download-nanoarrow-pop

Re: [I] UUIDType with BucketTransform incorrectly converts int to str in PartitionKey [iceberg-python]

2025-07-15 Thread via GitHub
dingo4dev commented on issue #2002: URL: https://github.com/apache/iceberg-python/issues/2002#issuecomment-3076518946 @Fokko @kevinjqliu I also would like to raise a consensus question about the interoperability between python and java implementation. Because when I dig into the metadat

Re: [PR] feat: RegisterTable support for InMemoryCatalog [iceberg-cpp]

2025-07-15 Thread via GitHub
lishuxu commented on code in PR #142: URL: https://github.com/apache/iceberg-cpp/pull/142#discussion_r2209086861 ## test/in_memory_catalog_test.cc: ## @@ -58,6 +75,21 @@ TEST_F(InMemoryCatalogTest, TableExists) { EXPECT_THAT(result, HasValue(::testing::Eq(false))); } +TEST

Re: [PR] feat: RegisterTable support for InMemoryCatalog [iceberg-cpp]

2025-07-15 Thread via GitHub
lishuxu commented on code in PR #142: URL: https://github.com/apache/iceberg-cpp/pull/142#discussion_r2204242342 ## test/temp_file_test_base.h: ## @@ -31,6 +31,46 @@ namespace iceberg { +/// \brief Get the test name for inclusion in the filename Review Comment: I notice

Re: [I] UUIDType with BucketTransform incorrectly converts int to str in PartitionKey [iceberg-python]

2025-07-15 Thread via GitHub
dingo4dev commented on issue #2002: URL: https://github.com/apache/iceberg-python/issues/2002#issuecomment-3076513555 @kevinjqliu I think UUID's BucketTransform is not supported in spark iceberg with 1.9.1 version. Hope this fix https://github.com/apache/iceberg/pull/13324 will resol

Re: [PR] feat: RegisterTable support for InMemoryCatalog [iceberg-cpp]

2025-07-15 Thread via GitHub
lishuxu commented on code in PR #142: URL: https://github.com/apache/iceberg-cpp/pull/142#discussion_r2204247772 ## test/in_memory_catalog_test.cc: ## @@ -58,6 +75,21 @@ TEST_F(InMemoryCatalogTest, TableExists) { EXPECT_THAT(result, HasValue(::testing::Eq(false))); } +TEST

[PR] chore: upgrade nanoarrow dependency [iceberg-cpp]

2025-07-15 Thread via GitHub
gty404 opened a new pull request, #146: URL: https://github.com/apache/iceberg-cpp/pull/146 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [I] Auth Manager support [iceberg-rust]

2025-07-15 Thread via GitHub
CTTY commented on issue #1510: URL: https://github.com/apache/iceberg-rust/issues/1510#issuecomment-3076395429 Hi @rambleraptor , thanks for working on this! Should we include request signer in this issue? Right now there are catalogs like Glue that implement IRC, but will require a

Re: [PR] feat(datafusion): Support insert_into in IcebergTableProvider [iceberg-rust]

2025-07-15 Thread via GitHub
CTTY commented on code in PR #1511: URL: https://github.com/apache/iceberg-rust/pull/1511#discussion_r2208973814 ## crates/iceberg/src/spec/manifest/_serde.rs: ## @@ -96,9 +96,10 @@ impl ManifestEntryV1 { } } +/// todo doc #[serde_as] #[derive(Serialize, Deserialize)]

Re: [PR] feat(datafusion): Support insert_into in IcebergTableProvider [iceberg-rust]

2025-07-15 Thread via GitHub
CTTY commented on code in PR #1511: URL: https://github.com/apache/iceberg-rust/pull/1511#discussion_r2208973814 ## crates/iceberg/src/spec/manifest/_serde.rs: ## @@ -96,9 +96,10 @@ impl ManifestEntryV1 { } } +/// todo doc #[serde_as] #[derive(Serialize, Deserialize)]

Re: [PR] feat(datafusion): Support insert_into in IcebergTableProvider [iceberg-rust]

2025-07-15 Thread via GitHub
CTTY commented on code in PR #1511: URL: https://github.com/apache/iceberg-rust/pull/1511#discussion_r2208971455 ## crates/iceberg/src/arrow/value.rs: ## @@ -440,10 +440,12 @@ impl PartnerAccessor for ArrowArrayAccessor { Ok(schema_partner) } +// todo generat

Re: [PR] [docs] Tidy up left-hand navigation [iceberg]

2025-07-15 Thread via GitHub
manuzhang commented on code in PR #13491: URL: https://github.com/apache/iceberg/pull/13491#discussion_r2208969337 ## docs/mkdocs.yml: ## @@ -22,69 +22,79 @@ plugins: nav: - index.md - - Tables: -- branching.md -- configuration.md -- evolution.md -- mainte

[PR] Add initial AuthManager support [iceberg-rust]

2025-07-15 Thread via GitHub
rambleraptor opened a new pull request, #1512: URL: https://github.com/apache/iceberg-rust/pull/1512 ## Which issue does this PR close? This adds a definition for an user-selectable AuthManager class. It allows different authentication methods to be used with the REST Catalog.

Re: [I] how to grant s3 temp permissions when using pyiceberg? [iceberg-python]

2025-07-15 Thread via GitHub
github-actions[bot] commented on issue #1463: URL: https://github.com/apache/iceberg-python/issues/1463#issuecomment-3076331412 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the A

Re: [I] [BUG] ArrowTypeError: "Could not convert" Error in inspect._files method [iceberg-python]

2025-07-15 Thread via GitHub
github-actions[bot] closed issue #1477: [BUG] ArrowTypeError: "Could not convert" Error in inspect._files method URL: https://github.com/apache/iceberg-python/issues/1477 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] how to grant s3 temp permissions when using pyiceberg? [iceberg-python]

2025-07-15 Thread via GitHub
github-actions[bot] closed issue #1463: how to grant s3 temp permissions when using pyiceberg? URL: https://github.com/apache/iceberg-python/issues/1463 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] [BUG] ArrowTypeError: "Could not convert" Error in inspect._files method [iceberg-python]

2025-07-15 Thread via GitHub
github-actions[bot] commented on issue #1477: URL: https://github.com/apache/iceberg-python/issues/1477#issuecomment-3076331371 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the A

Re: [I] Column Names in REST calls [iceberg]

2025-07-15 Thread via GitHub
github-actions[bot] closed issue #11898: Column Names in REST calls URL: https://github.com/apache/iceberg/issues/11898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] Column Names in REST calls [iceberg]

2025-07-15 Thread via GitHub
github-actions[bot] commented on issue #11898: URL: https://github.com/apache/iceberg/issues/11898#issuecomment-3076325169 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Fields are out of order in equality delete files if equality fields are not together [iceberg]

2025-07-15 Thread via GitHub
github-actions[bot] commented on issue #11891: URL: https://github.com/apache/iceberg/issues/11891#issuecomment-3076325101 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Fields are out of order in equality delete files if equality fields are not together [iceberg]

2025-07-15 Thread via GitHub
github-actions[bot] closed issue #11891: Fields are out of order in equality delete files if equality fields are not together URL: https://github.com/apache/iceberg/issues/11891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [I] Parameter type is not org.apache.avro.Schema for AvroSchemaUtil.toIceberg() [iceberg]

2025-07-15 Thread via GitHub
github-actions[bot] commented on issue #11884: URL: https://github.com/apache/iceberg/issues/11884#issuecomment-3076325056 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] SparkValue converter Timestamp Issue [iceberg]

2025-07-15 Thread via GitHub
github-actions[bot] commented on issue #11840: URL: https://github.com/apache/iceberg/issues/11840#issuecomment-3076324992 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Parameter type is not org.apache.avro.Schema for AvroSchemaUtil.toIceberg() [iceberg]

2025-07-15 Thread via GitHub
github-actions[bot] closed issue #11884: Parameter type is not org.apache.avro.Schema for AvroSchemaUtil.toIceberg() URL: https://github.com/apache/iceberg/issues/11884 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] SparkValue converter Timestamp Issue [iceberg]

2025-07-15 Thread via GitHub
github-actions[bot] closed issue #11840: SparkValue converter Timestamp Issue URL: https://github.com/apache/iceberg/issues/11840 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] Feature/metadata deduplication 2130 [iceberg-python]

2025-07-15 Thread via GitHub
ForeverAngry opened a new pull request, #2216: URL: https://github.com/apache/iceberg-python/pull/2216 # Rationale for this change This PR addresses deduplicating snapshot metadata as outlined in #2130 # Are these changes tested? Yes. # Are there a

[PR] feat(datafusion): Support insert_into in IcebergTableProvider [iceberg-rust]

2025-07-15 Thread via GitHub
CTTY opened a new pull request, #1511: URL: https://github.com/apache/iceberg-rust/pull/1511 ## Which issue does this PR close? - A part of https://github.com/apache/iceberg-rust/issues/1382 ## What changes are included in this PR? ## Are these changes tes

Re: [PR] Add schema conversion time [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on PR #2215: URL: https://github.com/apache/iceberg-python/pull/2215#issuecomment-3075835601 looks like the linter failed, could you run `make lint`? ``` ruff (legacy alias)..Failed - hook id: ruff - exit c

Re: [PR] Spark 4.0: Preserve row lineage information on compaction [iceberg]

2025-07-15 Thread via GitHub
amogh-jahagirdar commented on code in PR #13555: URL: https://github.com/apache/iceberg/pull/13555#discussion_r2208766097 ## spark/v4.0/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java: ## @@ -976,19 +978,67 @@ public void tes

Re: [PR] Fix: Packed Parquet Int96 timestamps read incorrectly #13485 [iceberg]

2025-07-15 Thread via GitHub
Biman54 commented on PR #13486: URL: https://github.com/apache/iceberg/pull/13486#issuecomment-3075796132 I believe a similar issue is discussed in [#1138](https://github.com/apache/iceberg/issues/1138). -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Add schema conversion time [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on code in PR #2215: URL: https://github.com/apache/iceberg-python/pull/2215#discussion_r2208730847 ## pyiceberg/utils/schema_conversion.py: ## @@ -68,8 +68,9 @@ LOGICAL_FIELD_TYPE_MAPPING: Dict[Tuple[str, str], PrimitiveType] = { ("date", "int"): Da

Re: [PR] feat: update pyiceberg/catalog/hive.py to support hive 4.x.x [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on PR #2206: URL: https://github.com/apache/iceberg-python/pull/2206#issuecomment-3075704350 ah we cannot do that yet. kind of a chicken and egg problem we use pyiceberg 0.9.1 to provision the hive catalog https://github.com/apache/iceberg-python/blob/ad8263b1

[PR] Add schema conversion time [iceberg-python]

2025-07-15 Thread via GitHub
matthias-Q opened a new pull request, #2215: URL: https://github.com/apache/iceberg-python/pull/2215 # Rationale for this change * Fix schema_conversion for `timestamp-millis` * add conversion for `time-millis` Following up on #2173 the `timestamp-millis` is actually stored are `long

Re: [PR] feat: update pyiceberg/catalog/hive.py to support hive 4.x.x [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on PR #2206: URL: https://github.com/apache/iceberg-python/pull/2206#issuecomment-3075618914 @igorvoltaic lets run the integration tests against 4.0.1! https://github.com/apache/iceberg-python/blob/ad8263b1be048c8cb67d40efe70f494a4f1cb374/dev/hive/Dockerfile#L26

Re: [PR] kafka-connect: resolve CVE-2025-48734 [iceberg]

2025-07-15 Thread via GitHub
liko9 commented on PR #13561: URL: https://github.com/apache/iceberg/pull/13561#issuecomment-3075546768 resolves https://github.com/apache/iceberg/issues/13563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[I] kafka-connect: subject to CVE-2025-48734 [iceberg]

2025-07-15 Thread via GitHub
liko9 opened a new issue, #13563: URL: https://github.com/apache/iceberg/issues/13563 ### Apache Iceberg version main (development) ### Query engine Kafka Connect ### Please describe the bug 🐞 When building the Kafka Connect with Hive from main (just in adva

Re: [PR] feat: update pyiceberg/catalog/hive.py to support hive 4.x.x [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on PR #2206: URL: https://github.com/apache/iceberg-python/pull/2206#issuecomment-3075543308 oh interesting. so `get_table` was not deprecated in `4.0.0`, but rather in `4.0.1` 🤔 see the [4.0.1 changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?ve

Re: [I] Can't partition by nested field [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu closed issue #2095: Can't partition by nested field URL: https://github.com/apache/iceberg-python/issues/2095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Fix support for writing to nested field partition [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu merged PR #2204: URL: https://github.com/apache/iceberg-python/pull/2204 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] Spark: Doing a Coalesce and foreachpartitions in spark directly on an iceberg table is leaking memory heavy iterators [iceberg]

2025-07-15 Thread via GitHub
viirya commented on issue #13297: URL: https://github.com/apache/iceberg/issues/13297#issuecomment-3075399684 I proposed a fix at https://github.com/apache/spark/pull/51503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Update the range in format-version (for v3/v4) [iceberg]

2025-07-15 Thread via GitHub
amogh-jahagirdar commented on code in PR #13505: URL: https://github.com/apache/iceberg/pull/13505#discussion_r2208550568 ## open-api/rest-catalog-open-api.py: ## @@ -1147,7 +1147,7 @@ class NotExpression(BaseModel): class TableMetadata(BaseModel): -format_version: int

[I] Support for Write Positional Delete files [iceberg-go]

2025-07-15 Thread via GitHub
alexandre-normand opened a new issue, #487: URL: https://github.com/apache/iceberg-go/issues/487 ### Feature Request / Improvement First, thanks for iterating quickly on the iceberg-go implementation. This is going to be very useful for us and make integrating with Iceberg easier. We

Re: [I] Spark: Doing a Coalesce and foreachpartitions in spark directly on an iceberg table is leaking memory heavy iterators [iceberg]

2025-07-15 Thread via GitHub
viirya commented on issue #13297: URL: https://github.com/apache/iceberg/issues/13297#issuecomment-3075297885 Just read through early discussion. So the iterator held by task completion listener is heavy for some Iceberg tables. Once iterator is exhausted before the task finishes, the itera

[I] Auth Manager support [iceberg-rust]

2025-07-15 Thread via GitHub
rambleraptor opened a new issue, #1510: URL: https://github.com/apache/iceberg-rust/issues/1510 ### What's the feature are you trying to implement? Java and Python have [AuthManager](https://iceberg.apache.org/javadoc/1.8.0/org/apache/iceberg/rest/auth/AuthManager.html) support for s

[PR] Arrow: Add nanosec precision timestamp [iceberg]

2025-07-15 Thread via GitHub
nandorKollar opened a new pull request, #13562: URL: https://github.com/apache/iceberg/pull/13562 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] [Feature Request] Supports StageOnly metadataUpdate [iceberg-python]

2025-07-15 Thread via GitHub
stevie9868 commented on issue #2214: URL: https://github.com/apache/iceberg-python/issues/2214#issuecomment-3075050481 We have some internal used case for this feature and If people think this is a useful feature, I am willing to contribute. -- This is an automated message from the Apach

[I] [Feature Request] Supports StageOnly metadataUpdate [iceberg-python]

2025-07-15 Thread via GitHub
stevie9868 opened a new issue, #2214: URL: https://github.com/apache/iceberg-python/issues/2214 ### Feature Request / Improvement Currently in java, there is a way to set stageOnly for [snapshotUpdate](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/S

Re: [I] Spark: Doing a Coalesce and foreachpartitions in spark directly on an iceberg table is leaking memory heavy iterators [iceberg]

2025-07-15 Thread via GitHub
szehon-ho commented on issue #13297: URL: https://github.com/apache/iceberg/issues/13297#issuecomment-3074751461 sorry i am a bit behind on this thread, so the MetricsRowIterator and ParquetPartitionReaderFactory fix together did not fix the issue? cc @viirya as well -- This is an autom

Re: [PR] AWS: feat Turning AAL default On [iceberg]

2025-07-15 Thread via GitHub
sullis commented on PR #13527: URL: https://github.com/apache/iceberg/pull/13527#issuecomment-3074721785 > > > fyi: I mentioned S3 Analytics Accelerator in my NYC Iceberg Meetup talk on July 10th > > > https://docs.google.com/presentation/d/1GgBWJwxP_rZLMt4Kixv697CCBO_7vm1VM9YENs2MwdA/ed

Re: [PR] AWS: feat Turning AAL default On [iceberg]

2025-07-15 Thread via GitHub
sullis commented on PR #13527: URL: https://github.com/apache/iceberg/pull/13527#issuecomment-3074690773 > > fyi: I mentioned S3 Analytics Accelerator in my NYC Iceberg Meetup talk on July 10th > > https://docs.google.com/presentation/d/1GgBWJwxP_rZLMt4Kixv697CCBO_7vm1VM9YENs2MwdA/edit

Re: [I] Publish Iceberg kafka connect runtime to Confluent hub [iceberg]

2025-07-15 Thread via GitHub
liko9 commented on issue #10745: URL: https://github.com/apache/iceberg/issues/10745#issuecomment-3074579462 https://github.com/apache/iceberg/pull/13561 - probably should create a separate issue for this CVE which could also be associated. -- This is an automated message from the Apache

[PR] kafka-connect: resolve CVE-2025-48734 [iceberg]

2025-07-15 Thread via GitHub
liko9 opened a new pull request, #13561: URL: https://github.com/apache/iceberg/pull/13561 removes commons-beanutils from non-hive deps, bump version to 1.11.0 to resolve CVE when using hive -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Schema evolution tests with initial defaults [iceberg]

2025-07-15 Thread via GitHub
anoopj commented on code in PR #13537: URL: https://github.com/apache/iceberg/pull/13537#discussion_r2208090103 ## core/src/test/java/org/apache/iceberg/TestScansAndSchemaEvolution.java: ## @@ -113,4 +116,79 @@ public void testPartitionSourceRename() throws IOException {

Re: [I] Publish Iceberg kafka connect runtime to Confluent hub [iceberg]

2025-07-15 Thread via GitHub
liko9 commented on issue #10745: URL: https://github.com/apache/iceberg/issues/10745#issuecomment-3074463048 > │ commons-beanutils:commons-beanutils │ CVE-2025-48734 │ HIGH │ fixed │ 1.9.4 │ 1.11.0│ commons-beanutils: Apache Commons BeanUtils:│ > │ (commons-b

Re: [PR] Schema evolution tests with initial defaults [iceberg]

2025-07-15 Thread via GitHub
nastra commented on code in PR #13537: URL: https://github.com/apache/iceberg/pull/13537#discussion_r2208028442 ## core/src/test/java/org/apache/iceberg/TestScansAndSchemaEvolution.java: ## @@ -113,4 +116,79 @@ public void testPartitionSourceRename() throws IOException {

Re: [I] [discussion] dealing with multiple pyarrow versions [iceberg-python]

2025-07-15 Thread via GitHub
gabeiglio commented on issue #2209: URL: https://github.com/apache/iceberg-python/issues/2209#issuecomment-3074406507 Pyarrow versioning is a pain-point internally since it prevents us from integrating to existing services that depend on a specific (old) pyarrow version. I would be very in

Re: [PR] Schema evolution tests with initial defaults [iceberg]

2025-07-15 Thread via GitHub
nastra commented on code in PR #13537: URL: https://github.com/apache/iceberg/pull/13537#discussion_r2208023529 ## core/src/test/java/org/apache/iceberg/TestScansAndSchemaEvolution.java: ## @@ -113,4 +116,79 @@ public void testPartitionSourceRename() throws IOException {

Re: [PR] Add support for Bodo DataFrame [iceberg-python]

2025-07-15 Thread via GitHub
ehsantn commented on PR #2167: URL: https://github.com/apache/iceberg-python/pull/2167#issuecomment-3074384043 Thanks for the review and help @kevinjqliu ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] feat: update pyiceberg/catalog/hive.py to support hive 4.x.x [iceberg-python]

2025-07-15 Thread via GitHub
igorvoltaic commented on PR #2206: URL: https://github.com/apache/iceberg-python/pull/2206#issuecomment-3074381672 > ah looks like CI failed because we mock `.get_table` in tests https://grep.app/search?f.path=tests%2F&f.path.pattern=tests&f.repo.pattern=iceberg-python&q=.get_table >

Re: [PR] Build: Bump aiohttp from 3.12.13 to 3.12.14 [iceberg-python]

2025-07-15 Thread via GitHub
dependabot[bot] commented on PR #2213: URL: https://github.com/apache/iceberg-python/pull/2213#issuecomment-3074343099 Looks like aiohttp is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Build: Bump aiohttp from 3.12.13 to 3.12.14 [iceberg-python]

2025-07-15 Thread via GitHub
dependabot[bot] closed pull request #2213: Build: Bump aiohttp from 3.12.13 to 3.12.14 URL: https://github.com/apache/iceberg-python/pull/2213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Build: Bump aiohttp from 3.12.13 to 3.12.14 [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on PR #2213: URL: https://github.com/apache/iceberg-python/pull/2213#issuecomment-3074341995 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Add support for Bodo DataFrame [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu commented on PR #2167: URL: https://github.com/apache/iceberg-python/pull/2167#issuecomment-3074340735 Thanks for adding this @ehsantn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Add support for Bodo DataFrame [iceberg-python]

2025-07-15 Thread via GitHub
kevinjqliu merged PR #2167: URL: https://github.com/apache/iceberg-python/pull/2167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] [docs] Tidy up left-hand navigation [iceberg]

2025-07-15 Thread via GitHub
stevenzwu commented on code in PR #13491: URL: https://github.com/apache/iceberg/pull/13491#discussion_r2207948784 ## docs/mkdocs.yml: ## @@ -22,69 +22,79 @@ plugins: nav: - index.md - - Tables: -- branching.md -- configuration.md -- evolution.md -- mainte

Re: [I] Publish Iceberg kafka connect runtime to Confluent hub [iceberg]

2025-07-15 Thread via GitHub
rmoff commented on issue #10745: URL: https://github.com/apache/iceberg/issues/10745#issuecomment-3074300861 Here's the build & scan with Trivy against main: ``` $ git rev-parse HEAD 061ae58986db3495ff3af6f1932a96dd086e5fbd $ sdk use java 11.0.26-tem $ ./gradlew

Re: [PR] [docs] Tidy up left-hand navigation [iceberg]

2025-07-15 Thread via GitHub
stevenzwu commented on code in PR #13491: URL: https://github.com/apache/iceberg/pull/13491#discussion_r2207948784 ## docs/mkdocs.yml: ## @@ -22,69 +22,79 @@ plugins: nav: - index.md - - Tables: -- branching.md -- configuration.md -- evolution.md -- mainte

Re: [PR] Spark: Add Variant read support for Spark Iceberg tables [iceberg]

2025-07-15 Thread via GitHub
aihuaxu commented on code in PR #13219: URL: https://github.com/apache/iceberg/pull/13219#discussion_r2207930519 ## parquet/src/main/java/org/apache/iceberg/parquet/TripleIterator.java: ## @@ -21,7 +21,7 @@ import java.util.Iterator; import org.apache.parquet.io.api.Binary;

Re: [I] org.apache.thrift.TApplicationException: Invalid method name: 'get_table' [iceberg]

2025-07-15 Thread via GitHub
deniskuzZ commented on issue #12878: URL: https://github.com/apache/iceberg/issues/12878#issuecomment-3074232913 @shorrocka sorry, I though `HiveMetaStoreClient` is packaged into iceberg-spark-runtime. Can you check where is that class comes from in Spark classpath, maybe `hive-standalone-

Re: [I] org.apache.thrift.TApplicationException: Invalid method name: 'get_table' [iceberg]

2025-07-15 Thread via GitHub
shorrocka commented on issue #12878: URL: https://github.com/apache/iceberg/issues/12878#issuecomment-3073627520 I think the error I am seeing is different than the one that the PR you linked fixes. When I try and query an iceberg table in the hive 4.0.1 metastore I get: `Caused by: org.

Re: [I] iceberg.EqualTo row filter fails on LargeStringType [iceberg-go]

2025-07-15 Thread via GitHub
hamilton-earthscope commented on issue #486: URL: https://github.com/apache/iceberg-go/issues/486#issuecomment-3074226022 One important note is that our iceberg table (written by pyiceberg) is being compacted by AWS Glue table optimizations. The resulting parquet files written by Glue

Re: [PR] Core: Add Catalog Transactions API [iceberg]

2025-07-15 Thread via GitHub
hantangwangd commented on code in PR #6948: URL: https://github.com/apache/iceberg/pull/6948#discussion_r2207898401 ## core/src/main/java/org/apache/iceberg/catalog/BaseCatalogTransaction.java: ## @@ -0,0 +1,412 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [I] Publish Iceberg kafka connect runtime to Confluent hub [iceberg]

2025-07-15 Thread via GitHub
ajantha-bhat commented on issue #10745: URL: https://github.com/apache/iceberg/issues/10745#issuecomment-3074194064 Also [this PR](https://github.com/apache/iceberg/pull/10739#issue-2421613146) mentions that there are two distributions. We should publish the one WITHOUT HMS. -- This is

[I] iceberg.EqualTo row filter fails on LargeStringType [iceberg-go]

2025-07-15 Thread via GitHub
hamilton-earthscope opened a new issue, #486: URL: https://github.com/apache/iceberg-go/issues/486 ### Apache Iceberg version None ### Please describe the bug 🐞 As a developer using `iceberg-go` to query an Iceberg table written by `pyiceberg`, I'm experiencing errors wh

Re: [PR] Core: Expose the stats of the manifest file content cache [iceberg]

2025-07-15 Thread via GitHub
gaborkaszab commented on PR #13560: URL: https://github.com/apache/iceberg/pull/13560#issuecomment-3074096522 As described in the commit message, the purpose here is to expose the stats of the manifest file content cache. This would be useful to see the cache hot/miss ratio and to have some

[PR] Core: Expose the stats of the manifest file content cache [iceberg]

2025-07-15 Thread via GitHub
gaborkaszab opened a new pull request, #13560: URL: https://github.com/apache/iceberg/pull/13560 For observability purposes clients could use the stats of the manifest file content cache to see for instance the cache hit/miss ratio so that users can fine tune the configuration of the cache.

Re: [PR] Fix: Packed Parquet Int96 timestamps read incorrectly #13485 [iceberg]

2025-07-15 Thread via GitHub
xajohnson commented on PR #13486: URL: https://github.com/apache/iceberg/pull/13486#issuecomment-3073965290 I've added a test case and a sample parquet file that reproduces the problem. Unfortunately, I'm unable to create this parquet file using the existing Iceberg test code. If this is

  1   2   >