[I] Support for Identity Columns in Apache Iceberg [iceberg]

2025-02-16 Thread via GitHub
nqvuong1998 opened a new issue, #12297: URL: https://github.com/apache/iceberg/issues/12297 ### Feature Request / Improvement **Summary**: Apache Iceberg should support identity columns similar to Delta Lake. This feature would allow users to define identity columns in Iceberg tabl

Re: [PR] Spark 3.5: Fix job description of RewriteTablePathSparkAction [iceberg]

2025-02-16 Thread via GitHub
ebyhr commented on code in PR #12282: URL: https://github.com/apache/iceberg/pull/12282#discussion_r1957755453 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -239,7 +239,7 @@ private boolean versionInFilePath(String pat

Re: [PR] Core,REST: extend httpClient builder to support tls factory [iceberg]

2025-02-16 Thread via GitHub
munendrasn commented on PR #11979: URL: https://github.com/apache/iceberg/pull/11979#issuecomment-2662294516 @nastra could you please review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Spark 3.5: Fix job description of RewriteTablePathSparkAction [iceberg]

2025-02-16 Thread via GitHub
manuzhang commented on code in PR #12282: URL: https://github.com/apache/iceberg/pull/12282#discussion_r1957749447 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -239,7 +239,7 @@ private boolean versionInFilePath(String

[PR] Add properties support for HadoopTables.load() (#12251) [iceberg]

2025-02-16 Thread via GitHub
qqchang2nd opened a new pull request, #12296: URL: https://github.com/apache/iceberg/pull/12296 Description: Currently, HadoopTables.load() doesn't support passing custom properties when loading tables. While HiveCatalog and HadoopCatalog support manifest caching through their initialize

Re: [PR] AWS, AZURE: Move docker-based tests to integration test source [iceberg]

2025-02-16 Thread via GitHub
nastra commented on code in PR #12274: URL: https://github.com/apache/iceberg/pull/12274#discussion_r1957734542 ## azure/src/integration/java/org/apache/iceberg/azure/adlsv2/ADLSLocationTest.java: ## Review Comment: this file is a normal unit test so it can stay -- This

Re: [PR] AWS, AZURE: Move docker-based tests to integration test source [iceberg]

2025-02-16 Thread via GitHub
nastra commented on code in PR #12274: URL: https://github.com/apache/iceberg/pull/12274#discussion_r1957732904 ## build.gradle: ## @@ -546,6 +546,27 @@ project(':iceberg-azure') { testImplementation libs.esotericsoftware.kryo testImplementation libs.testcontainers

Re: [I] Add properties support for HadoopTables.load() [iceberg]

2025-02-16 Thread via GitHub
qqchang2nd commented on issue #12251: URL: https://github.com/apache/iceberg/issues/12251#issuecomment-2662259040 Let me share our use case at Sensors Data. We use HadoopTables to manage Iceberg tables with Impala as our query engine. In one of our customer environments, we encounter

Re: [PR] REST: Fallback to older logic for existence check in RESTSessionCatalog [iceberg]

2025-02-16 Thread via GitHub
nastra commented on PR #12294: URL: https://github.com/apache/iceberg/pull/12294#issuecomment-2662250422 @ebyhr just FYI that this was already proposed in https://github.com/apache/iceberg/pull/12259#discussion_r1955661232 -- This is an automated message from the Apache Git Service. To re

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-16 Thread via GitHub
dramaticlly commented on PR #12115: URL: https://github.com/apache/iceberg/pull/12115#issuecomment-2662241789 > I think it almost looks ready, I just had one last comment, sorry about not putting it correctly before ! > > I think when I looked last night at for [#12115 (comment)](htt

Re: [PR] Core: Fix failure when reading files table with branch [iceberg]

2025-02-16 Thread via GitHub
szehon-ho commented on PR #11719: URL: https://github.com/apache/iceberg/pull/11719#issuecomment-2662237508 @ebyhr can you mark as non-draft if its ready to review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] List data and metadata directories instead of table root [iceberg]

2025-02-16 Thread via GitHub
szehon-ho commented on PR #12278: URL: https://github.com/apache/iceberg/pull/12278#issuecomment-2662233970 I think it makes some sense to me, but maybe better to put it behind a flag as its a behavior change? Something like -- listCurrentPaths? Curious what others think? -- This is

Re: [PR] Docs: Add warning about `snapshot_ids` arg in `expired_snapshots` procedure [iceberg]

2025-02-16 Thread via GitHub
manuzhang commented on code in PR #12291: URL: https://github.com/apache/iceberg/pull/12291#discussion_r1957714962 ## docs/docs/spark-procedures.md: ## @@ -270,7 +270,8 @@ the `expire_snapshots` procedure will never remove files which are still require | `retain_last` || i

Re: [I] previous eq deletes handling on new write [iceberg]

2025-02-16 Thread via GitHub
pvary commented on issue #12280: URL: https://github.com/apache/iceberg/issues/12280#issuecomment-2662219496 Be careful about rewriting equality deletes to new equality deletes. The equality delete will remove every occurrence of the previous row in previous commits. For example: - Co

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-16 Thread via GitHub
szehon-ho commented on PR #12115: URL: https://github.com/apache/iceberg/pull/12115#issuecomment-2662217136 I think it almost looks ready, I just had one last comment, sorry about that! I think when I looked last night at for https://github.com/apache/iceberg/pull/12115#discussion_r19

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1957681443 ## bindings/c/Cargo.toml: ## @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-16 Thread via GitHub
Xuanwo commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r195768 ## bindings/c/CMakeLists.txt: ## @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NO

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-16 Thread via GitHub
Xuanwo commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1957670325 ## bindings/c/src/specs.rs: ## @@ -0,0 +1,153 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the N

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-16 Thread via GitHub
Xuanwo commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1957669757 ## bindings/c/Cargo.toml: ## @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE

Re: [I] API to find out the number of datafiles deleted [iceberg]

2025-02-16 Thread via GitHub
Shekharrajak commented on issue #12288: URL: https://github.com/apache/iceberg/issues/12288#issuecomment-2662128376 I found a testcases : ``` Map summary = table.currentSnapshot().summary(); assertThat(summary.get("deleted-data-files")) .as("Deleted files count mu

Re: [I] Java doc link is not working [iceberg]

2025-02-16 Thread via GitHub
manuzhang commented on issue #12166: URL: https://github.com/apache/iceberg/issues/12166#issuecomment-2661898504 This has been fixed now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-02-16 Thread via GitHub
connortsui20 commented on code in PR #965: URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957524251 ## crates/catalog/rest/src/catalog.rs: ## @@ -670,21 +719,47 @@ impl Catalog for RestCatalog { }) .build()?; -let resp = sel

Re: [I] Java doc link is not working [iceberg]

2025-02-16 Thread via GitHub
manuzhang closed issue #12166: Java doc link is not working URL: https://github.com/apache/iceberg/issues/12166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Flaky test `TestFlinkTableSink > testInsertFromSourceTable` [iceberg]

2025-02-16 Thread via GitHub
manuzhang commented on issue #11833: URL: https://github.com/apache/iceberg/issues/11833#issuecomment-2661897564 @stevenzwu do you know about these failures? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-02-16 Thread via GitHub
connortsui20 commented on code in PR #965: URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957528321 ## crates/catalog/rest/src/catalog.rs: ## @@ -670,21 +719,47 @@ impl Catalog for RestCatalog { }) .build()?; -let resp = sel

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-02-16 Thread via GitHub
connortsui20 commented on code in PR #965: URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957528321 ## crates/catalog/rest/src/catalog.rs: ## @@ -670,21 +719,47 @@ impl Catalog for RestCatalog { }) .build()?; -let resp = sel

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-02-16 Thread via GitHub
connortsui20 commented on code in PR #965: URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957528321 ## crates/catalog/rest/src/catalog.rs: ## @@ -670,21 +719,47 @@ impl Catalog for RestCatalog { }) .build()?; -let resp = sel

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-02-16 Thread via GitHub
connortsui20 commented on code in PR #965: URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957526271 ## crates/catalog/rest/src/catalog.rs: ## @@ -507,78 +530,98 @@ impl Catalog for RestCatalog { }) .build()?; -let resp = sel

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-02-16 Thread via GitHub
connortsui20 commented on code in PR #965: URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957521755 ## crates/catalog/rest/src/catalog.rs: ## @@ -312,90 +317,107 @@ impl RestCatalog { } } +/// All requests and expected responses are derived from the RE

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-02-16 Thread via GitHub
connortsui20 commented on code in PR #965: URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957521755 ## crates/catalog/rest/src/catalog.rs: ## @@ -312,90 +317,107 @@ impl RestCatalog { } } +/// All requests and expected responses are derived from the RE

Re: [I] Flaky test `TestFlinkTableSink > testInsertFromSourceTable` [iceberg]

2025-02-16 Thread via GitHub
manuzhang commented on issue #11833: URL: https://github.com/apache/iceberg/issues/11833#issuecomment-2661837910 Same as above from https://github.com/apache/iceberg/actions/runs/13356805548/job/37300702080, but with more stacktraces. ``` TestFlinkTableSink > testReplacePartitions

Re: [PR] Build: Bump datamodel-code-generator from 0.27.2 to 0.28.1 [iceberg]

2025-02-16 Thread via GitHub
manuzhang commented on PR #12290: URL: https://github.com/apache/iceberg/pull/12290#issuecomment-2661831655 Thanks @Fokko for quick review and merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] refactor: REST `Catalog` implementation [iceberg-rust]

2025-02-16 Thread via GitHub
Xuanwo commented on code in PR #965: URL: https://github.com/apache/iceberg-rust/pull/965#discussion_r1957506030 ## crates/catalog/rest/src/catalog.rs: ## @@ -312,90 +317,107 @@ impl RestCatalog { } } +/// All requests and expected responses are derived from the REST cat

Re: [PR] chore(deps): Bump apache/skywalking-eyes from 0.6.0 to 0.7.0 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 merged PR #969: URL: https://github.com/apache/iceberg-rust/pull/969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

2025-02-16 Thread via GitHub
joshua-yeung-mox commented on issue #10524: URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2661813598 @salah-djb Thanks for your explanation. I also want to know which spark.hadoop.aws.catalog.credentials.provider.factory.class should I use if I want to use assumed role fo

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1957494635 ## crates/iceberg/src/transaction.rs: ## @@ -169,6 +177,180 @@ impl<'a> Transaction<'a> { catalog.update_table(table_commit).await } + +///

[PR] [docs] add docs about nightly build [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu opened a new pull request, #1670: URL: https://github.com/apache/iceberg-python/pull/1670 Nightly testpypi build was added in #1601 This PR adds a subpage under "Releases" about the nightly build. Running `make docs-serve` locally, ![Screenshot 2025-02-16 at 5 52 45  P

Re: [PR] chore(deps): Bump aws-sdk-s3tables from 1.9.0 to 1.10.0 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 merged PR #970: URL: https://github.com/apache/iceberg-rust/pull/970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat(datafusion): Treat timestamp conversion functions like a cast. [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on code in PR #945: URL: https://github.com/apache/iceberg-rust/pull/945#discussion_r1957488879 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -119,7 +122,53 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult {

Re: [PR] feat(datafusion): Treat timestamp conversion functions like a cast. [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on code in PR #945: URL: https://github.com/apache/iceberg-rust/pull/945#discussion_r1957487320 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -119,7 +122,53 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult {

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1957485800 ## bindings/c/Cargo.toml: ## @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] REST: Fallback to older logic for existence check in RESTSessionCatalog [iceberg]

2025-02-16 Thread via GitHub
ebyhr commented on PR #12294: URL: https://github.com/apache/iceberg/pull/12294#issuecomment-2661769714 cc: @nastra @danielcweeks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[PR] REST:Fallback to older logic for existence check in RESTSessionCatalog [iceberg]

2025-02-16 Thread via GitHub
ebyhr opened a new pull request, #12294: URL: https://github.com/apache/iceberg/pull/12294 The following PRs have prevented Trino from upgrading to Iceberg 1.8.0: * https://github.com/apache/iceberg/pull/11761 * https://github.com/apache/iceberg/pull/11760 * https://github.com/apache

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1957483479 ## bindings/c/CMakeLists.txt: ## Review Comment: Oh, I forgot what we need to use cmake to build static c library. -- This is an automated message fro

Re: [PR] chore(deps): Bump volo-thrift from 0.10.5 to 0.10.6 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 closed pull request #972: chore(deps): Bump volo-thrift from 0.10.5 to 0.10.6 URL: https://github.com/apache/iceberg-rust/pull/972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] chore(deps): Bump volo-thrift from 0.10.5 to 0.10.6 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on PR #972: URL: https://github.com/apache/iceberg-rust/pull/972#issuecomment-2661764273 Close this as only cargo.lock changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] chore(deps): Bump volo-thrift from 0.10.5 to 0.10.6 [iceberg-rust]

2025-02-16 Thread via GitHub
dependabot[bot] commented on PR #972: URL: https://github.com/apache/iceberg-rust/pull/972#issuecomment-2661764305 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] chore(deps): Bump aws-sdk-glue from 1.76.0 to 1.82.0 [iceberg-rust]

2025-02-16 Thread via GitHub
dependabot[bot] commented on PR #971: URL: https://github.com/apache/iceberg-rust/pull/971#issuecomment-2661760411 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] chore(deps): Bump aws-sdk-glue from 1.76.0 to 1.82.0 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on PR #971: URL: https://github.com/apache/iceberg-rust/pull/971#issuecomment-2661760383 Conflict MSRV. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] chore(deps): Bump aws-sdk-glue from 1.76.0 to 1.82.0 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 closed pull request #971: chore(deps): Bump aws-sdk-glue from 1.76.0 to 1.82.0 URL: https://github.com/apache/iceberg-rust/pull/971 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] chore(deps): Bump aws-config from 1.5.15 to 1.5.16 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 merged PR #973: URL: https://github.com/apache/iceberg-rust/pull/973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] fix: Fix typos upgrade to 1.29.7 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 merged PR #974: URL: https://github.com/apache/iceberg-rust/pull/974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] chore(deps): Bump crate-ci/typos from 1.29.5 to 1.29.7 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 closed pull request #968: chore(deps): Bump crate-ci/typos from 1.29.5 to 1.29.7 URL: https://github.com/apache/iceberg-rust/pull/968 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] chore(deps): Bump crate-ci/typos from 1.29.5 to 1.29.7 [iceberg-rust]

2025-02-16 Thread via GitHub
dependabot[bot] commented on PR #968: URL: https://github.com/apache/iceberg-rust/pull/968#issuecomment-2661758605 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] fix: Fix typos upgrade to 1.29.7 [iceberg-rust]

2025-02-16 Thread via GitHub
liurenjie1024 commented on PR #974: URL: https://github.com/apache/iceberg-rust/pull/974#issuecomment-2661758482 Thanks @jonathanc-n for fixing this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] REST: Expose endpoints in RESTSessionCatalog [iceberg]

2025-02-16 Thread via GitHub
ebyhr closed pull request #12293: REST: Expose endpoints in RESTSessionCatalog URL: https://github.com/apache/iceberg/pull/12293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1957464703 ## pyiceberg/catalog/__init__.py: ## @@ -945,8 +956,8 @@ def _get_default_warehouse_location(self, database_name: str, table_name: str) - raise Valu

Re: [I] Field comments are not written for timestamp field [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] closed issue #4212: Field comments are not written for timestamp field URL: https://github.com/apache/iceberg/issues/4212 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Field comments are not written for timestamp field [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] commented on issue #4212: URL: https://github.com/apache/iceberg/issues/4212#issuecomment-2661697989 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Cannot specify "file://" uri or direct local file location for warehouse in Windows [iceberg-python]

2025-02-16 Thread via GitHub
github-actions[bot] commented on issue #1005: URL: https://github.com/apache/iceberg-python/issues/1005#issuecomment-266174 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the A

Re: [I] [Feat] Support Merge-on-Read mode for Deletes [iceberg-python]

2025-02-16 Thread via GitHub
github-actions[bot] commented on issue #1078: URL: https://github.com/apache/iceberg-python/issues/1078#issuecomment-2661699969 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Cannot specify "file://" uri or direct local file location for warehouse in Windows [iceberg-python]

2025-02-16 Thread via GitHub
github-actions[bot] closed issue #1005: Cannot specify "file://" uri or direct local file location for warehouse in Windows URL: https://github.com/apache/iceberg-python/issues/1005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Core: Fix failure when reading files table with branch [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] commented on PR #11719: URL: https://github.com/apache/iceberg/pull/11719#issuecomment-2661698196 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Define behavior of gc.enabled and location ownership [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] commented on issue #4159: URL: https://github.com/apache/iceberg/issues/4159#issuecomment-2661697965 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Core: Properly detect metadata tables [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] commented on PR #11963: URL: https://github.com/apache/iceberg/pull/11963#issuecomment-2661698243 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Set missing table-default property in RESTSessionCatalog [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] commented on PR #11646: URL: https://github.com/apache/iceberg/pull/11646#issuecomment-2661698137 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Expose `added_rows_count`, `existing_rows_count` and `deleted_rows_count` fields in all_manifests and manifests tables [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] commented on PR #11679: URL: https://github.com/apache/iceberg/pull/11679#issuecomment-2661698175 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Map methods should return immutable collections [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] closed pull request #11304: Core: Map methods should return immutable collections URL: https://github.com/apache/iceberg/pull/11304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Add aliyun-bundle jar [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] commented on issue #10970: URL: https://github.com/apache/iceberg/issues/10970#issuecomment-2661698070 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Core: Map methods should return immutable collections [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] commented on PR #11304: URL: https://github.com/apache/iceberg/pull/11304#issuecomment-2661698107 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Define behavior of gc.enabled and location ownership [iceberg]

2025-02-16 Thread via GitHub
github-actions[bot] closed issue #4159: Define behavior of gc.enabled and location ownership URL: https://github.com/apache/iceberg/issues/4159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Add upsert docs [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu merged PR #1665: URL: https://github.com/apache/iceberg-python/pull/1665 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] [feat] add missing metadata tables [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on issue #1053: URL: https://github.com/apache/iceberg-python/issues/1053#issuecomment-2661633989 hey folks, sorry for the late response here. I think there are a couple of different things here. 1. time travel for metadata tables. this is a feature for metad

Re: [PR] position_deletes metadata table [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on code in PR #1615: URL: https://github.com/apache/iceberg-python/pull/1615#discussion_r1948161771 ## pyiceberg/manifest.py: ## @@ -320,6 +320,34 @@ def data_file_with_partition(partition_type: StructType, format_version: TableVe ) +class Position

Re: [PR] support all_entries in pyiceberg [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on code in PR #1608: URL: https://github.com/apache/iceberg-python/pull/1608#discussion_r1957418226 ## pyiceberg/table/inspect.py: ## @@ -157,74 +158,96 @@ def _readable_metrics_struct(bound_type: PrimitiveType) -> pa.StructType: pa.field("

[PR] Add pagination for list table operation across different catalog types [iceberg-go]

2025-02-16 Thread via GitHub
dttung2905 opened a new pull request, #306: URL: https://github.com/apache/iceberg-go/pull/306 ## **Goal** To support pagination for ListTables method. Similar to `ListViews` operation in PR https://github.com/apache/iceberg-go/pull/290. We need to change the method interface in `catalog

Re: [PR] Feat: replace sort order [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on code in PR #1500: URL: https://github.com/apache/iceberg-python/pull/1500#discussion_r1957416511 ## pyiceberg/table/update/sorting.py: ## @@ -0,0 +1,117 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreem

Re: [PR] Implement update for `remove-snapshots` action [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on PR #1561: URL: https://github.com/apache/iceberg-python/pull/1561#issuecomment-2661608880 now that #1598 is merged, @grihabor could you rebase main? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Fix: `SqlCatalog` list_namespaces() should return only sub-namespaces [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on code in PR #1629: URL: https://github.com/apache/iceberg-python/pull/1629#discussion_r1957407215 ## tests/catalog/test_sql.py: ## @@ -1117,17 +1117,30 @@ def test_create_namespace_with_empty_identifier(catalog: SqlCatalog, empty_names lazy_fixtu

[PR] Docs: Add warning about `snapshot_ids` arg in `expired_snapshots` procedure [iceberg]

2025-02-16 Thread via GitHub
estherbester opened a new pull request, #12291: URL: https://github.com/apache/iceberg/pull/12291 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Fix: `SqlCatalog` list_namespaces() should return only sub-namespaces [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on code in PR #1629: URL: https://github.com/apache/iceberg-python/pull/1629#discussion_r1957158183 ## pyiceberg/catalog/sql.py: ## @@ -610,15 +610,26 @@ def list_namespaces(self, namespace: Union[str, Identifier] = ()) -> List[Identi table_stmt =

Re: [I] previous eq deletes handling on new write [iceberg]

2025-02-16 Thread via GitHub
eshishki commented on issue #12280: URL: https://github.com/apache/iceberg/issues/12280#issuecomment-2661592485 in our scenario each commit adds 1 eq delete file, every 5 minutes, 12 times an hour we run compaction say every hour, and the number of eq delete files stays within reason

Re: [I] [info] pyiceberg integration tests no longer works with java 8 [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu closed issue #1668: [info] pyiceberg integration tests no longer works with java 8 URL: https://github.com/apache/iceberg-python/issues/1668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[I] [info] pyiceberg integration tests no longer works with java 8 [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu opened a new issue, #1668: URL: https://github.com/apache/iceberg-python/issues/1668 ### Apache Iceberg version None ### Please describe the bug 🐞 Opening this ticket so the error message becomes searchable. Running integration tests (`make test-integra

Re: [PR] Add upsert docs [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on PR #1665: URL: https://github.com/apache/iceberg-python/pull/1665#issuecomment-2661577715 LGTM! Thanks @Fokko and thanks @mattmartin14 for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Add upsert docs [iceberg-python]

2025-02-16 Thread via GitHub
Fokko commented on PR #1665: URL: https://github.com/apache/iceberg-python/pull/1665#issuecomment-2661563124 > I honestly didn't even know about the iceberg specific identifier fields until you had recently mentioned it. I can't imagine many have. I see situations where teams have already b

Re: [I] [feature] for `upsert`, add uniqueness check when `identifier_field_ids` is set [iceberg-python]

2025-02-16 Thread via GitHub
Fokko commented on issue #1667: URL: https://github.com/apache/iceberg-python/issues/1667#issuecomment-2661562268 Thinking of it, I believe Trino fails when it finds duplicate fields on the read-path, maybe that's a better check to set. That would also cover https://github.com/apache/icebe

Re: [I] [feature] for `upsert`, add uniqueness check when `identifier_field_ids` is set [iceberg-python]

2025-02-16 Thread via GitHub
Fokko commented on issue #1667: URL: https://github.com/apache/iceberg-python/issues/1667#issuecomment-2661560018 I think this should be the other way around. If you use `join_cols`, and you have a `identifier_field_ids` set, you want to make sure that the uniqueness constraint still holds

Re: [I] previous eq deletes handling on new write [iceberg]

2025-02-16 Thread via GitHub
singhpk234 commented on issue #12280: URL: https://github.com/apache/iceberg/issues/12280#issuecomment-2661558013 I see if it written this way i.e join, each eq delete would be scanned only once right (same is what Impala does). Is there a configuration to read multiple eq deletes in a sing

Re: [I] Print un-pretty metadata JSON files without whitespace [iceberg]

2025-02-16 Thread via GitHub
singhpk234 commented on issue #12281: URL: https://github.com/apache/iceberg/issues/12281#issuecomment-2661555646 This is interesting, does removing the pretty causes any issues for rendering in common ui tools or any other reader ? > I have seen a metadata files which was 53.6MB. Af

Re: [PR] feat: Add `StrictMetricsEvaluator` [iceberg-rust]

2025-02-16 Thread via GitHub
jonathanc-n commented on code in PR #963: URL: https://github.com/apache/iceberg-rust/pull/963#discussion_r1955029747 ## crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs: ## @@ -0,0 +1,1854 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [I] API to find out the number of datafiles deleted [iceberg]

2025-02-16 Thread via GitHub
singhpk234 commented on issue #12288: URL: https://github.com/apache/iceberg/issues/12288#issuecomment-2661554039 If it's just the count you can use snapshot summary : https://iceberg.apache.org/spec/?h=spec#metrics people also use partition summary to have partition level break down !

Re: [I] [feature] On the write path, take into account `identifier_field_ids` [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on issue #1666: URL: https://github.com/apache/iceberg-python/issues/1666#issuecomment-2661528784 related: #1667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Add upsert docs [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on PR #1665: URL: https://github.com/apache/iceberg-python/pull/1665#issuecomment-2661524962 > Not to sound blunt but the example above seems odd TBH @mattmartin14 it is an odd example! I had a feeling this can break the uniqueness constraint so I crafted an examp

Re: [PR] Add upsert docs [iceberg-python]

2025-02-16 Thread via GitHub
kevinjqliu commented on PR #1665: URL: https://github.com/apache/iceberg-python/pull/1665#issuecomment-2661523189 > Yes, that is an issue, but we don't respect this for any of the operations (append, etc). Doing this would make the operations expensive so we could leave this up to the user.

Re: [PR] Build: Bump datamodel-code-generator from 0.27.2 to 0.28.1 [iceberg]

2025-02-16 Thread via GitHub
Fokko merged PR #12290: URL: https://github.com/apache/iceberg/pull/12290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump datamodel-code-generator from 0.27.2 to 0.28.1 [iceberg]

2025-02-16 Thread via GitHub
dependabot[bot] commented on PR #12285: URL: https://github.com/apache/iceberg/pull/12285#issuecomment-2661503152 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, l

Re: [PR] Build: Bump datamodel-code-generator from 0.27.2 to 0.28.1 [iceberg]

2025-02-16 Thread via GitHub
manuzhang closed pull request #12285: Build: Bump datamodel-code-generator from 0.27.2 to 0.28.1 URL: https://github.com/apache/iceberg/pull/12285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Build: Bump datamodel-code-generator from 0.27.2 to 0.28.1 [iceberg]

2025-02-16 Thread via GitHub
manuzhang commented on PR #12285: URL: https://github.com/apache/iceberg/pull/12285#issuecomment-2661503059 datamodel-code-generator 0.28.0 drops python <3.9 support. I opened #12290 to update `target-python-version` to 3.9 as well. -- This is an automated message from the Apache Git Serv

[PR] Docs: Refactor site navigation bar [iceberg]

2025-02-16 Thread via GitHub
manuzhang opened a new pull request, #12289: URL: https://github.com/apache/iceberg/pull/12289 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

  1   2   >