Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-01 Thread via GitHub
lidavidm commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2024138415 ## src/iceberg/expression.h: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See t

Re: [D] Which Iceberg versions does this crate support? [iceberg-rust]

2025-04-01 Thread via GitHub
GitHub user liurenjie1024 added a comment to the discussion: Which Iceberg versions does this crate support? Hi, @k-bx currently we are requiring rust `1.77.1`. You can find our msrv policy here: https://github.com/apache/iceberg-rust?tab=readme-ov-file#supported-rust-version GitHub link: h

Re: [PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.4.2 to 5.4.3 [iceberg]

2025-04-01 Thread via GitHub
Fokko merged PR #12685: URL: https://github.com/apache/iceberg/pull/12685 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Spark 3.4: Fix RewriteDataFiles with partial progress enabled and max-failed-commits larger than total-file-group [iceberg]

2025-04-01 Thread via GitHub
nastra merged PR #12701: URL: https://github.com/apache/iceberg/pull/12701 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Pass storage credentials from LoadTableResponse to FileIO [iceberg]

2025-04-01 Thread via GitHub
nastra commented on code in PR #12591: URL: https://github.com/apache/iceberg/pull/12591#discussion_r2024132740 ## core/src/main/java/org/apache/iceberg/io/SupportsStorageCredentials.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] feat: Infer partition values from statistics [iceberg-rust]

2025-04-01 Thread via GitHub
ZENOTME commented on code in PR #1079: URL: https://github.com/apache/iceberg-rust/pull/1079#discussion_r2024083268 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -458,6 +459,54 @@ impl ParquetWriter { Ok(builder) } + +#[allow(dead_code)] +

Re: [PR] feat:add init expression interface. [iceberg-cpp]

2025-04-01 Thread via GitHub
lidavidm commented on code in PR #58: URL: https://github.com/apache/iceberg-cpp/pull/58#discussion_r2024079986 ## src/iceberg/expression.h: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See t

[PR] Build: Bump sqlalchemy from 2.0.39 to 2.0.40 [iceberg-python]

2025-04-01 Thread via GitHub
dependabot[bot] opened a new pull request, #1874: URL: https://github.com/apache/iceberg-python/pull/1874 Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.39 to 2.0.40. Release notes Sourced from https://github.com/sqlalchemy/sqlalchemy/releases";>sqlalchemy's

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2024040930 ## dev/docker-compose.yml: ## @@ -65,6 +65,13 @@ services: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] + az

Re: [PR] doc: Add implementation status to `README` [iceberg-rust]

2025-04-01 Thread via GitHub
jonathanc-n commented on PR #1152: URL: https://github.com/apache/iceberg-rust/pull/1152#issuecomment-2771280147 > > Should this be put on the website instead? > > which website are you referring to? @kevinjqliu This one here: https://rust.iceberg.apache.org/ -- This is an au

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't respect nested columns [iceberg]

2025-04-01 Thread via GitHub
github-actions[bot] commented on issue #11253: URL: https://github.com/apache/iceberg/issues/11253#issuecomment-2770956706 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Add User Interface to Iceberg based lakehouse [iceberg]

2025-04-01 Thread via GitHub
github-actions[bot] commented on issue #10980: URL: https://github.com/apache/iceberg/issues/10980#issuecomment-2770956536 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] feat: convert arrow schema to iceberg schema [iceberg-cpp]

2025-04-01 Thread via GitHub
wgtmac commented on PR #57: URL: https://github.com/apache/iceberg-cpp/pull/57#issuecomment-2771198593 Thanks @lidavidm @zhjwpku! @Fokko @Xuanwo Could you help review and merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]

2025-04-01 Thread via GitHub
liurenjie1024 merged PR #1150: URL: https://github.com/apache/iceberg-rust/pull/1150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Introduce TimestampNs, TimestampTzNs support for Bucket transforms [iceberg-rust]

2025-04-01 Thread via GitHub
liurenjie1024 closed issue #1110: Introduce TimestampNs, TimestampTzNs support for Bucket transforms URL: https://github.com/apache/iceberg-rust/issues/1110 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Introduce TimestampNs, TimestampTzNs support for Bucket transforms [iceberg-rust]

2025-04-01 Thread via GitHub
liurenjie1024 closed issue #1110: Introduce TimestampNs, TimestampTzNs support for Bucket transforms URL: https://github.com/apache/iceberg-rust/issues/1110 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] doc: Add implementation status to `README` [iceberg-rust]

2025-04-01 Thread via GitHub
liurenjie1024 merged PR #1152: URL: https://github.com/apache/iceberg-rust/pull/1152 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] doc: Add implementation status to `README` [iceberg-rust]

2025-04-01 Thread via GitHub
kevinjqliu commented on PR #1152: URL: https://github.com/apache/iceberg-rust/pull/1152#issuecomment-2771254423 > Should this be put on the website instead? which website are you referring to? -- This is an automated message from the Apache Git Service. To respond to the message, p

[PR] feat: Add basic conflict detection [iceberg-rust]

2025-04-01 Thread via GitHub
jonathanc-n opened a new pull request, #1154: URL: https://github.com/apache/iceberg-rust/pull/1154 ## Which issue does this PR close? - Mentioned [here](https://github.com/apache/iceberg-rust/issues/964) ## What changes are included in this PR? Added basic confli

Re: [PR] feat: Add basic conflict detection [iceberg-rust]

2025-04-01 Thread via GitHub
jonathanc-n commented on PR #1154: URL: https://github.com/apache/iceberg-rust/pull/1154#issuecomment-2771198141 Is this on track with what we want? cc @liurenjie1024 @Fokko @ZENOTME -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2023948645 ## dev/docker-compose.yml: ## @@ -65,6 +65,13 @@ services: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] + az

[PR] feat(io): add OSS storage implementation and update dependencies [iceberg-rust]

2025-04-01 Thread via GitHub
divinerapier opened a new pull request, #1153: URL: https://github.com/apache/iceberg-rust/pull/1153 ## What changes are included in this PR? - [x] Support AliyunOSS backend by OpenDAL - [x] Fix typo ## Are these changes tested? The support for AliyunO

Re: [PR] doc: Add implementation status to `README` [iceberg-rust]

2025-04-01 Thread via GitHub
jonathanc-n commented on PR #1152: URL: https://github.com/apache/iceberg-rust/pull/1152#issuecomment-2771046503 Should this be put on the website instead? cc @k-bx @liurenjie1024 @Fokko @kevinjqliu -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Doc: Remove Hive 2.x/3.x references from hive.md [iceberg]

2025-04-01 Thread via GitHub
jackylee-ch commented on PR #12700: URL: https://github.com/apache/iceberg/pull/12700#issuecomment-2771027185 > Hive 2/3 runtime still exists on 1.7.x branch, so it's marked as deprecated in https://iceberg.apache.org/multi-engine-support/#apache-hive. I don't think we can remove related do

Re: [I] Incorrect schema used when using time-travel [iceberg]

2025-04-01 Thread via GitHub
github-actions[bot] closed issue #11162: Incorrect schema used when using time-travel URL: https://github.com/apache/iceberg/issues/11162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] feat: add file_io and local impl by adapting arrow::filesystem [iceberg-cpp]

2025-04-01 Thread via GitHub
wgtmac commented on code in PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30#discussion_r2022045457 ## src/iceberg/arrow/io/arrow_fs_file_io.cc: ## @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agree

Re: [PR] allow dashes in glue database and table names [iceberg]

2025-04-01 Thread via GitHub
amogh-jahagirdar commented on PR #12677: URL: https://github.com/apache/iceberg/pull/12677#issuecomment-2766347114 cc @geruh @xiaoxuandev @lliangyu-lin could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] Inexplainable behavior for SQLCatalog with Postgres and MinIO [iceberg]

2025-04-01 Thread via GitHub
github-actions[bot] commented on issue #11250: URL: https://github.com/apache/iceberg/issues/11250#issuecomment-2770956670 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Incorrect schema used when using time-travel [iceberg]

2025-04-01 Thread via GitHub
github-actions[bot] commented on issue #11162: URL: https://github.com/apache/iceberg/issues/11162#issuecomment-2770956596 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-04-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #12670: URL: https://github.com/apache/iceberg/pull/12670#discussion_r2023796463 ## core/src/main/java/org/apache/iceberg/MetadataUpdateParser.java: ## @@ -150,7 +148,7 @@ private MetadataUpdateParser() {} .put(MetadataUpdate.S

Re: [I] Disaster Recovery Options for AWS Athena/Iceberg Integration [iceberg]

2025-04-01 Thread via GitHub
github-actions[bot] commented on issue #6619: URL: https://github.com/apache/iceberg/issues/6619#issuecomment-2770956423 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-04-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #12670: URL: https://github.com/apache/iceberg/pull/12670#discussion_r2023796463 ## core/src/main/java/org/apache/iceberg/MetadataUpdateParser.java: ## @@ -150,7 +148,7 @@ private MetadataUpdateParser() {} .put(MetadataUpdate.S

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-04-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #12670: URL: https://github.com/apache/iceberg/pull/12670#discussion_r2023796463 ## core/src/main/java/org/apache/iceberg/MetadataUpdateParser.java: ## @@ -150,7 +148,7 @@ private MetadataUpdateParser() {} .put(MetadataUpdate.S

Re: [PR] Core: Pass storage credentials from LoadTableResponse to FileIO [iceberg]

2025-04-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #12591: URL: https://github.com/apache/iceberg/pull/12591#discussion_r2023775261 ## core/src/main/java/org/apache/iceberg/io/SupportsStorageCredentials.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [I] Hive Metastore conf for the catalog [iceberg]

2025-04-01 Thread via GitHub
pvary commented on issue #12699: URL: https://github.com/apache/iceberg/issues/12699#issuecomment-2770841366 @Shekharrajak: It is possible to provide a hiveConf during catalog creation (`CatalogUtil.buildIcebergCatalog`): ``` /** * Build an Iceberg {@link Catalog} based on a ma

Re: [PR] Flink: Backport support source watermark for flink sql windows [iceberg]

2025-04-01 Thread via GitHub
pvary commented on PR #12697: URL: https://github.com/apache/iceberg/pull/12697#issuecomment-2770827130 Merged to main. Thanks for the PR @swapna267! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Flink: Backport support source watermark for flink sql windows [iceberg]

2025-04-01 Thread via GitHub
pvary merged PR #12697: URL: https://github.com/apache/iceberg/pull/12697 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Doc: Remove Hive 2.x/3.x references from hive.md [iceberg]

2025-04-01 Thread via GitHub
jackylee-ch commented on PR #12700: URL: https://github.com/apache/iceberg/pull/12700#issuecomment-2769390132 @manuzhang @nastra PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Add MetricsReporter for SnapshotManager [iceberg]

2025-04-01 Thread via GitHub
nastra merged PR #12665: URL: https://github.com/apache/iceberg/pull/12665 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] fix(metadata): export iceberg schema in snapshots table [iceberg-rust]

2025-04-01 Thread via GitHub
liurenjie1024 merged PR #1135: URL: https://github.com/apache/iceberg-rust/pull/1135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] feat: add partition field/partition spec [iceberg-cpp]

2025-04-01 Thread via GitHub
wgtmac commented on PR #54: URL: https://github.com/apache/iceberg-cpp/pull/54#issuecomment-2768372630 @lidavidm @zhjwpku @yingcai-cy Do you want to take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]

2025-04-01 Thread via GitHub
jonathanc-n commented on code in PR #1150: URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023707366 ## crates/iceberg/src/transform/bucket.rs: ## @@ -888,4 +904,66 @@ mod test { Datum::int(32) ); } + +#[test] +fn test_timest

Re: [PR] feat(table): write an Arrow Table or stream of Arrow records [iceberg-go]

2025-04-01 Thread via GitHub
zeroshade commented on PR #369: URL: https://github.com/apache/iceberg-go/pull/369#issuecomment-2770712452 @Fokko Updated from your feedback and added to the `TestMergeManifests` test so that it verifies the snapshot IDs and sequence numbers after the merging occurs -- This is an automat

[PR] Materialized View Spec [iceberg]

2025-04-01 Thread via GitHub
JanKaul opened a new pull request, #11041: URL: https://github.com/apache/iceberg/pull/11041 This PR implements the Iceberg Materialized View Proposal #10043 by adding a section for Materialized Views to the View spec. It follows the design of the [proposal document](https://docs.google.co

Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]

2025-04-01 Thread via GitHub
kevinjqliu commented on code in PR #1150: URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023649923 ## crates/iceberg/src/transform/bucket.rs: ## @@ -888,4 +904,66 @@ mod test { Datum::int(32) ); } + +#[test] +fn test_timesta

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
xuhui-lu commented on PR #313: URL: https://github.com/apache/iceberg-go/pull/313#issuecomment-2770020467 removed extra environment variable. The test could pass now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
deniskuzZ commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023418469 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception

Re: [PR] Docs: Update link for User-Defined Tag Restrictions in AWS documentation [iceberg]

2025-04-01 Thread via GitHub
XBaith commented on PR #12698: URL: https://github.com/apache/iceberg/pull/12698#issuecomment-2768162636 cc @manuzhang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] AWS: Delegate part of AWS integration tests to using mock aws services and enable tests in check task [iceberg]

2025-04-01 Thread via GitHub
lliangyu-lin commented on PR #12671: URL: https://github.com/apache/iceberg/pull/12671#issuecomment-2770585949 @xiaoxuandev @nastra There are some concerns on moving the tests to moto since we need to add additional configs to make it work correctly. So I revert those changes in this PR and

Re: [I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2025-04-01 Thread via GitHub
arvind-dhariwal commented on issue #10808: URL: https://github.com/apache/iceberg/issues/10808#issuecomment-2769637038 I tried with DBRT 16.3 Apache Spark 3.5.2, Scala 2.12 and got same error while reading iceberg table. -- This is an automated message from the Apache Git Service. To resp

Re: [PR] feat(table): write an Arrow Table or stream of Arrow records [iceberg-go]

2025-04-01 Thread via GitHub
Fokko commented on PR #369: URL: https://github.com/apache/iceberg-go/pull/369#issuecomment-2769909720 Ideally something that uses the Java SDK since that's the most feature complete one :) -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
kevinjqliu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2023572413 ## dev/docker-compose.yml: ## @@ -65,6 +65,13 @@ services: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] +

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
kevinjqliu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2023572413 ## dev/docker-compose.yml: ## @@ -65,6 +65,13 @@ services: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] +

Re: [PR] AWS: Delegate part of AWS integration tests to using mock aws services and enable tests in check task [iceberg]

2025-04-01 Thread via GitHub
lliangyu-lin commented on code in PR #12671: URL: https://github.com/apache/iceberg/pull/12671#discussion_r2023216967 ## aws/src/integration/java/org/apache/iceberg/aws/dynamodb/TestDynamoDbCatalog.java: ## @@ -56,8 +58,8 @@ import software.amazon.awssdk.services.s3.model.HeadO

Re: [PR] feat(table): write an Arrow Table or stream of Arrow records [iceberg-go]

2025-04-01 Thread via GitHub
Fokko commented on PR #369: URL: https://github.com/apache/iceberg-go/pull/369#issuecomment-2770500071 Also, V2 tables need a bit of TLC. I've added the comments using a `#`: ```json { "status": 1, "snapshot_id": { "long": 3987426542168867673 # The snapshot should

Re: [PR] feat: add file_io and local impl by adapting arrow::filesystem [iceberg-cpp]

2025-04-01 Thread via GitHub
Fokko merged PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] feat(table): write an Arrow Table or stream of Arrow records [iceberg-go]

2025-04-01 Thread via GitHub
Fokko commented on PR #369: URL: https://github.com/apache/iceberg-go/pull/369#issuecomment-2770464604 @zeroshade found an issue with the `snapshot_id` being updated: ```json "snapshots": [ { "snapshot-id": 3510441037372461000, "sequence-number": 0, "timest

[PR] [WIP] feat: Commit Table glue [iceberg-go]

2025-04-01 Thread via GitHub
piyushsingariya opened a new pull request, #370: URL: https://github.com/apache/iceberg-go/pull/370 - This PR is an experiment, adding Commit Table in Glue Catalog. - Opening a draft WIP PR, as a sign of commitment and progress tracking -- This is an automated message from the Apache Gi

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2023478185 ## io/azure_integration_test.go: ## @@ -0,0 +1,159 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2023469398 ## io/azure.go: ## @@ -0,0 +1,120 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
xuhui-lu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2023459346 ## io/azure_integration_test.go: ## @@ -0,0 +1,148 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
deniskuzZ commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023418469 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception

Re: [I] Feature: S3 Remote Signing [iceberg-rust]

2025-04-01 Thread via GitHub
delta003 commented on issue #506: URL: https://github.com/apache/iceberg-rust/issues/506#issuecomment-2769991914 hii, can I help with this somehow? If I understand the state currently: reqsign is going through a refactor which makes it possible to configure a signer (based on https:/

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
deniskuzZ commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023418469 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
deniskuzZ commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023418469 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception

[I] Hive Metastore conf for the catalog [iceberg]

2025-04-01 Thread via GitHub
Shekharrajak opened a new issue, #12699: URL: https://github.com/apache/iceberg/issues/12699 ### Query engine Java ### Question Where can we configure the `metastore.thrift.uris` and other metastore like `metastore.kerberos.principal` ? -- This is an automated messag

Re: [PR] feat(table): write an Arrow Table or stream of Arrow records [iceberg-go]

2025-04-01 Thread via GitHub
zeroshade commented on code in PR #369: URL: https://github.com/apache/iceberg-go/pull/369#discussion_r2023420124 ## table/internal/parquet_files.go: ## @@ -170,6 +172,86 @@ func (parquetFormat) PrimitiveTypeToPhysicalType(typ iceberg.PrimitiveType) stri } } +func (p

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
deniskuzZ commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023418469 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
deniskuzZ commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023396642 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception

Re: [PR] feat: add support for azure blob with connection string/sas token/account key [iceberg-go]

2025-04-01 Thread via GitHub
kevinjqliu commented on code in PR #313: URL: https://github.com/apache/iceberg-go/pull/313#discussion_r2023389280 ## dev/docker-compose.yml: ## @@ -65,6 +65,13 @@ services: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] +

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
deniskuzZ commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023379656 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
deniskuzZ commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023379656 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-04-01 Thread via GitHub
flyrain commented on PR #12418: URL: https://github.com/apache/iceberg/pull/12418#issuecomment-2770212013 Some CI pipelines failed, but it seems not related to this PR. ``` > Task :iceberg-core:test TestHadoopCommits > testConcurrentFastAppends(File) FAILED org.awaitility.

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-04-01 Thread via GitHub
dramaticlly commented on PR #12418: URL: https://github.com/apache/iceberg/pull/12418#issuecomment-2770209266 thank you @andyglow for your first contribution to apache iceberg and @flyrain @nastra for the review! -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] feat: add file_io and local impl by adapting arrow::filesystem [iceberg-cpp]

2025-04-01 Thread via GitHub
zhjwpku commented on code in PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30#discussion_r2020856602 ## src/iceberg/file_io.h: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NO

Re: [PR] Build and test hive-metastore with Hive 3 and Hive 4 [iceberg]

2025-04-01 Thread via GitHub
wypoon commented on code in PR #12681: URL: https://github.com/apache/iceberg/pull/12681#discussion_r2023308361 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -445,7 +444,7 @@ public void testCreateTableCustomSortOrder() throws Exception {

Re: [PR] doc: Clarify `arrow_schema_to_schema` requires fields with field id [iceberg-rust]

2025-04-01 Thread via GitHub
liurenjie1024 merged PR #1151: URL: https://github.com/apache/iceberg-rust/pull/1151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Use delimited column names in CreateChangelogViewProcedure [iceberg]

2025-04-01 Thread via GitHub
flyrain merged PR #12418: URL: https://github.com/apache/iceberg/pull/12418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [D] Which Iceberg versions does this crate support? [iceberg-rust]

2025-04-01 Thread via GitHub
GitHub user jonathanc-n added a comment to the discussion: Which Iceberg versions does this crate support? @k-bx Current status is a bit behind. There should be updates for it after the 0.5.0 release which is coming soon. I'll try to add a pr for having this status on the iceberg-rust page by

Re: [PR] Core: ability to inject an AuthManager in RESTCatalog [iceberg]

2025-04-01 Thread via GitHub
danielcweeks commented on PR #12655: URL: https://github.com/apache/iceberg/pull/12655#issuecomment-2769935063 @adutra, @varpa89, @gh-yzou I'm not opposed adding the flexibility to reuse the auth manager, but what I'd really like to avoid is everyone going off and rebuilding auth manager im

Re: [PR] feat(table): write an Arrow Table or stream of Arrow records [iceberg-go]

2025-04-01 Thread via GitHub
Fokko commented on code in PR #369: URL: https://github.com/apache/iceberg-go/pull/369#discussion_r2021749556 ## table/internal/parquet_files.go: ## @@ -170,6 +172,86 @@ func (parquetFormat) PrimitiveTypeToPhysicalType(typ iceberg.PrimitiveType) stri } } +func (parqu

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-04-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #12670: URL: https://github.com/apache/iceberg/pull/12670#discussion_r2019189952 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1436,12 +1436,14 @@ public Builder removeSnapshots(Collection idsToRemove) { privat

Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]

2025-04-01 Thread via GitHub
kevinjqliu commented on code in PR #1150: URL: https://github.com/apache/iceberg-rust/pull/1150#discussion_r2023158926 ## crates/iceberg/src/transform/bucket.rs: ## @@ -167,6 +167,16 @@ impl TransformFunction for Bucket { .downcast_ref::() .unwr

Re: [PR] feat: Support `TimestampNs` and TimestampTzNs` in bucket transform [iceberg-rust]

2025-04-01 Thread via GitHub
kevinjqliu commented on PR #1150: URL: https://github.com/apache/iceberg-rust/pull/1150#issuecomment-2769870847 @Fokko FYI new transform function we can use for https://github.com/apache/iceberg-python/pull/1833 -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] Build: Bump rich from 13.9.4 to 14.0.0 [iceberg-python]

2025-04-01 Thread via GitHub
Fokko merged PR #1868: URL: https://github.com/apache/iceberg-python/pull/1868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] feat: add partition field/partition spec [iceberg-cpp]

2025-04-01 Thread via GitHub
Fokko commented on code in PR #54: URL: https://github.com/apache/iceberg-cpp/pull/54#discussion_r2022929047 ## src/iceberg/transform.h: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the N

Re: [PR] Set field-id when needed [iceberg-python]

2025-04-01 Thread via GitHub
Fokko merged PR #1867: URL: https://github.com/apache/iceberg-python/pull/1867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

2025-04-01 Thread via GitHub
Fokko closed issue #1798: Error reading table after appending pyarrow table URL: https://github.com/apache/iceberg-python/issues/1798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Build: Bump mkdocstrings from 0.29.0 to 0.29.1 [iceberg-python]

2025-04-01 Thread via GitHub
Fokko merged PR #1871: URL: https://github.com/apache/iceberg-python/pull/1871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2025-04-01 Thread via GitHub
arvind-dhariwal commented on issue #10808: URL: https://github.com/apache/iceberg/issues/10808#issuecomment-2769663429 Py4JJavaError: An error occurred while calling o457.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, m

Re: [PR] Spark 3.5: Fix RewriteDataFiles with partial progress enabled and max-failed-commits larger than total-file-group [iceberg]

2025-04-01 Thread via GitHub
RussellSpitzer merged PR #12120: URL: https://github.com/apache/iceberg/pull/12120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] `partial-progress.max-failed-commits` Incorrectly compare the failureCommit value [iceberg]

2025-04-01 Thread via GitHub
RussellSpitzer closed issue #12076: `partial-progress.max-failed-commits` Incorrectly compare the failureCommit value URL: https://github.com/apache/iceberg/issues/12076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] AWS: Add parameter of excluding non-current fields in Glue [iceberg]

2025-04-01 Thread via GitHub
duoxoud commented on PR #12664: URL: https://github.com/apache/iceberg/pull/12664#issuecomment-2769595436 > Displaying non-current columns is intentional in Glue, as users may use LakeFormation and need to access dropped columns. Users should not rely on Glue for the latest table status, Ic

Re: [PR] feat(table): write an Arrow Table or stream of Arrow records [iceberg-go]

2025-04-01 Thread via GitHub
zeroshade commented on PR #369: URL: https://github.com/apache/iceberg-go/pull/369#issuecomment-2769619836 @laskoviymishka Yup! that's on my list of things to do. So far I've only tested it locally but I plan on doing a more thorough test suite for duckdb/clickhouse testing etc. -- This

Re: [PR] Core: Add MetricsReporter for SnapshotManager [iceberg]

2025-04-01 Thread via GitHub
wangyinsheng commented on PR #12665: URL: https://github.com/apache/iceberg/pull/12665#issuecomment-2769600037 @nastra PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: add partition field/partition spec [iceberg-cpp]

2025-04-01 Thread via GitHub
Fokko merged PR #54: URL: https://github.com/apache/iceberg-cpp/pull/54 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] feat: add file_io and local impl by adapting arrow::filesystem [iceberg-cpp]

2025-04-01 Thread via GitHub
zhjwpku commented on PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30#issuecomment-2769414976 @Fokko @Xuanwo I'd appreciate if you can review this PR when you got some time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] feat: add file_io and local impl by adapting arrow::filesystem [iceberg-cpp]

2025-04-01 Thread via GitHub
zhjwpku commented on code in PR #30: URL: https://github.com/apache/iceberg-cpp/pull/30#discussion_r2022213768 ## src/iceberg/file_io.h: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NO

Re: [PR] feat: add partition field/partition spec [iceberg-cpp]

2025-04-01 Thread via GitHub
gty404 commented on code in PR #54: URL: https://github.com/apache/iceberg-cpp/pull/54#discussion_r2022757545 ## src/iceberg/transform.h: ## @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the

Re: [I] Cannot scan empty table [iceberg-rust]

2025-04-01 Thread via GitHub
liurenjie1024 commented on issue #1145: URL: https://github.com/apache/iceberg-rust/issues/1145#issuecomment-2769388664 Thanks @danking for reporting this, IIRC this has been fixed in latest main branch, do you mind to test in latest main branch? -- This is an automated message from the A

  1   2   >