Re: [PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-28 Thread via GitHub
chinmay-bhat commented on code in PR #748: URL: https://github.com/apache/iceberg-python/pull/748#discussion_r1616712540 ## pyiceberg/table/__init__.py: ## @@ -1290,6 +1291,19 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: return self.snapshot_by_i

Re: [PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-28 Thread via GitHub
chinmay-bhat commented on code in PR #748: URL: https://github.com/apache/iceberg-python/pull/748#discussion_r1616717181 ## pyiceberg/table/__init__.py: ## @@ -1290,6 +1291,19 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: return self.snapshot_by_i

Re: [PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-28 Thread via GitHub
chinmay-bhat commented on code in PR #748: URL: https://github.com/apache/iceberg-python/pull/748#discussion_r1616717556 ## pyiceberg/table/snapshots.py: ## @@ -412,3 +417,12 @@ def _update_totals(total_property: str, added_property: str, removed_property: s def set_when_posit

Re: [PR] Infra: Track subtasks from Iceberg improvement proposal [iceberg]

2024-05-28 Thread via GitHub
ajantha-bhat commented on PR #10183: URL: https://github.com/apache/iceberg/pull/10183#issuecomment-2134518588 If not this, we need other ways to solve this problem. Currently a proposal can be split into multiple tasks. We don't know how many tasks completed and how many pending.

Re: [PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-28 Thread via GitHub
chinmay-bhat commented on code in PR #748: URL: https://github.com/apache/iceberg-python/pull/748#discussion_r1616717181 ## pyiceberg/table/__init__.py: ## @@ -1290,6 +1291,19 @@ def snapshot_by_name(self, name: str) -> Optional[Snapshot]: return self.snapshot_by_i

Re: [PR] Introduce hierarchical namespaces into SqlCatalog [iceberg-python]

2024-05-28 Thread via GitHub
Fokko merged PR #591: URL: https://github.com/apache/iceberg-python/pull/591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Copy iceberg table from hdfs to GCS and register table to BLMS [iceberg]

2024-05-28 Thread via GitHub
Fokko commented on issue #10389: URL: https://github.com/apache/iceberg/issues/10389#issuecomment-2134571953 Hey @Manoranjan93 Thanks for reaching out! Iceberg has absolute paths, so you cannot just copy the files as the paths in the metadata still point to S3. The best way to copy th

Re: [PR] Build: Bump net.snowflake:snowflake-jdbc from 3.15.1 to 3.16.0 [iceberg]

2024-05-28 Thread via GitHub
Fokko merged PR #10269: URL: https://github.com/apache/iceberg/pull/10269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] Javadoc issues [iceberg]

2024-05-28 Thread via GitHub
Fokko commented on issue #10378: URL: https://github.com/apache/iceberg/issues/10378#issuecomment-2134575602 Thanks for caching these @manuzhang. Let me know if you need any help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] [Build] Add a script to execute revapi without gradle plugin [iceberg]

2024-05-28 Thread via GitHub
Fokko commented on code in PR #10386: URL: https://github.com/apache/iceberg/pull/10386#discussion_r1616786126 ## dev/revapi: ## @@ -0,0 +1,86 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. S

Re: [PR] [Build] Add a script to execute revapi without gradle plugin [iceberg]

2024-05-28 Thread via GitHub
jbonofre commented on PR #10386: URL: https://github.com/apache/iceberg/pull/10386#issuecomment-2134605932 You can see it's not source breaking change. We just check source breaks (not semantic or binary). -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] [Build] Add a script to execute revapi without gradle plugin [iceberg]

2024-05-28 Thread via GitHub
jbonofre commented on code in PR #10386: URL: https://github.com/apache/iceberg/pull/10386#discussion_r1616817365 ## dev/revapi: ## @@ -0,0 +1,86 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements.

Re: [PR] AWS: Retain Glue Catalog column comment after updating Iceberg table [iceberg]

2024-05-28 Thread via GitHub
aajisaka commented on code in PR #10276: URL: https://github.com/apache/iceberg/pull/10276#discussion_r1616950141 ## aws/src/main/java/org/apache/iceberg/aws/glue/IcebergToGlueConverter.java: ## @@ -234,8 +258,26 @@ static void setTableInputInformation( Optional.ofNullabl

Re: [PR] Manifest list encryption [iceberg]

2024-05-28 Thread via GitHub
ggershinsky closed pull request #7770: Manifest list encryption URL: https://github.com/apache/iceberg/pull/7770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[PR] Manifest list encryption [iceberg]

2024-05-28 Thread via GitHub
ggershinsky opened a new pull request, #7770: URL: https://github.com/apache/iceberg/pull/7770 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[I] Iceberg Roadmap is 404 [iceberg]

2024-05-28 Thread via GitHub
gphilipp opened a new issue, #10390: URL: https://github.com/apache/iceberg/issues/10390 ### Query engine _No response_ ### Question This page yields a 404: https://iceberg.apache.org/roadmap/ Is this where the roadmap is supposed to be? -- This is an autom

Re: [PR] Core: Fix create v1 table on REST Catalog [iceberg]

2024-05-28 Thread via GitHub
hantangwangd commented on PR #10369: URL: https://github.com/apache/iceberg/pull/10369#issuecomment-2135076025 Hi @nastra, can you please take a look at this fix when available? Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-05-28 Thread via GitHub
ajantha-bhat commented on issue #10390: URL: https://github.com/apache/iceberg/issues/10390#issuecomment-2135110463 It was removed after discussion in https://github.com/apache/iceberg/pull/9941 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-05-28 Thread via GitHub
gphilipp commented on issue #10390: URL: https://github.com/apache/iceberg/issues/10390#issuecomment-2135116057 Thanks, I also found out. It'd be nice to point to _something_. For example, Delta Lake has a placeholder page which points to https://github.com/orgs/delta-io/projects/10/views/1

Re: [PR] Docs: Add archive for documentations older than 1.4.0 [iceberg]

2024-05-28 Thread via GitHub
ajantha-bhat commented on code in PR #10374: URL: https://github.com/apache/iceberg/pull/10374#discussion_r1617179538 ## site/docs/archive.md: ## @@ -0,0 +1,34 @@ +--- +title: "Archive" +--- + + +Documentations of Iceberg versions older than 1.4.0 are no longer maintained. Here

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-05-28 Thread via GitHub
gphilipp commented on issue #10390: URL: https://github.com/apache/iceberg/issues/10390#issuecomment-2135144837 Maybe just add a pointer to https://github.com/apache/iceberg/projects?query=is%3Aopen&type=classic in https://iceberg.apache.org/roadmap/? -- This is an automated message from

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-05-28 Thread via GitHub
ajantha-bhat commented on issue #10390: URL: https://github.com/apache/iceberg/issues/10390#issuecomment-2135151506 https://github.com/apache/iceberg/projects?query=is%3Aopen&type=classic is not well maintained as of now. Also It only captures ongoing projects. Roadmaps usually will also in

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-05-28 Thread via GitHub
jbonofre commented on issue #10390: URL: https://github.com/apache/iceberg/issues/10390#issuecomment-2135157480 roadmap page has been removed to avoid confusion as it was not up to date. We proposed/discussed to replace with a link on GitHub issues using milestone tag. That said, I'm not

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-05-28 Thread via GitHub
Fokko commented on issue #10390: URL: https://github.com/apache/iceberg/issues/10390#issuecomment-2135357197 Removing the roadmap was an attempt to have fewer places where we track things, so it would become more likely to keep things in sync. I think we should keep the milestones a t

Re: [PR] Docs: Ignore link-checking for Apache mailing list and GNU website [iceberg]

2024-05-28 Thread via GitHub
Fokko commented on PR #10388: URL: https://github.com/apache/iceberg/pull/10388#issuecomment-2135363789 > but the link checker is very-very flaky Yes, I agree, it is creating more frustration than actually helping right now :( I'm okay with disabling it until we find something better

Re: [PR] feat: Adding literals [iceberg-go]

2024-05-28 Thread via GitHub
zeroshade commented on PR #76: URL: https://github.com/apache/iceberg-go/pull/76#issuecomment-2135416145 updated and fixed the conflicts @Fokko! :smile: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] feat: Adding literals [iceberg-go]

2024-05-28 Thread via GitHub
Fokko merged PR #76: URL: https://github.com/apache/iceberg-go/pull/76 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache

Re: [PR] feat: Adding literals [iceberg-go]

2024-05-28 Thread via GitHub
Fokko commented on PR #76: URL: https://github.com/apache/iceberg-go/pull/76#issuecomment-2135444853 @zeroshade Awesome, let's get this in! 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Docs: Add archive for documentations older than 1.4.0 [iceberg]

2024-05-28 Thread via GitHub
manuzhang commented on code in PR #10374: URL: https://github.com/apache/iceberg/pull/10374#discussion_r1617496387 ## site/docs/archive.md: ## @@ -0,0 +1,34 @@ +--- +title: "Archive" +--- + + +Documentations of Iceberg versions older than 1.4.0 are no longer maintained. Here ar

[PR] Add "services-s3" feature for opendal [iceberg-rust]

2024-05-28 Thread via GitHub
c-thiel opened a new pull request, #384: URL: https://github.com/apache/iceberg-rust/pull/384 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-05-28 Thread via GitHub
jacobmarble commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2135579590 @nastra or @rdblue - friendly reminder to review this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add "services-s3" feature for opendal [iceberg-rust]

2024-05-28 Thread via GitHub
c-thiel closed pull request #384: Add "services-s3" feature for opendal URL: https://github.com/apache/iceberg-rust/pull/384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] bug: Iceberg opendal missing s3 support [iceberg-rust]

2024-05-28 Thread via GitHub
Xuanwo commented on issue #385: URL: https://github.com/apache/iceberg-rust/issues/385#issuecomment-2135636174 I believe we should enable the storage we want in `iceberg` crate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Core: Throw CommitStateUnknownException if RuntimeException that is not marked as cleanable is thrown [iceberg]

2024-05-28 Thread via GitHub
stevenzwu commented on PR #10373: URL: https://github.com/apache/iceberg/pull/10373#issuecomment-2135677546 > Upstream callers of the transaction/snapshot producer APIs such as engines like Spark currently handle CommitStateUnknown exceptions to avoid cleaning up data files when it's unclea

Re: [PR] Docs: Add archive for documentations older than 1.4.0 [iceberg]

2024-05-28 Thread via GitHub
ajantha-bhat commented on code in PR #10374: URL: https://github.com/apache/iceberg/pull/10374#discussion_r1617622341 ## site/docs/archive.md: ## @@ -0,0 +1,34 @@ +--- +title: "Archive" +--- + + +Documentations of Iceberg versions older than 1.4.0 are no longer maintained. Here

Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]

2024-05-28 Thread via GitHub
pvary commented on code in PR #10370: URL: https://github.com/apache/iceberg/pull/10370#discussion_r1617641116 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws

Re: [PR] Add Partitions Metadata Table [iceberg-python]

2024-05-28 Thread via GitHub
jlhester commented on PR #603: URL: https://github.com/apache/iceberg-python/pull/603#issuecomment-2136012377 Thanks @syun64 for implementing this! It would be pretty useful for some work I'm doing. When do you think this will be released? I see it's on the [near term roadmap](https://githu

Re: [PR] Spark Action to Analyze table [iceberg]

2024-05-28 Thread via GitHub
szehon-ho commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1617834879 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] AWS: add retry logic to S3InputStream [iceberg]

2024-05-28 Thread via GitHub
puchengy commented on PR #4912: URL: https://github.com/apache/iceberg/pull/4912#issuecomment-2136041865 @amogh-jahagirdar Hello, i wonder is there any update on this? Our clients from time to time hit the issue and cause some interruptions to their workload. Thanks! -- This is an automa

Re: [PR] AWS: add retry logic to S3InputStream [iceberg]

2024-05-28 Thread via GitHub
amogh-jahagirdar commented on PR #4912: URL: https://github.com/apache/iceberg/pull/4912#issuecomment-2136044924 Hey @puchengy I'm wokring on it! I am aiming to have a PR out sometime this week to address this. -- This is an automated message from the Apache Git Service. To respond to th

[PR] Bump pydantic from 2.7.1 to 2.7.2 [iceberg-python]

2024-05-28 Thread via GitHub
dependabot[bot] opened a new pull request, #775: URL: https://github.com/apache/iceberg-python/pull/775 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.7.1 to 2.7.2. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

[PR] Bump coverage from 7.5.2 to 7.5.3 [iceberg-python]

2024-05-28 Thread via GitHub
dependabot[bot] opened a new pull request, #776: URL: https://github.com/apache/iceberg-python/pull/776 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.5.2 to 7.5.3. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's chang

[PR] feat(exprs): Adding BooleanExpressions and Predicates [iceberg-go]

2024-05-28 Thread via GitHub
zeroshade opened a new pull request, #91: URL: https://github.com/apache/iceberg-go/pull/91 Building on #76, this PR implements the interfaces and basic implementations of BooleanExpressions, bound and unbound predicates, and a series of wrapper functions for convenience. It also adds tests

Re: [PR] feat(exprs): Adding BooleanExpressions and Predicates [iceberg-go]

2024-05-28 Thread via GitHub
zeroshade commented on PR #91: URL: https://github.com/apache/iceberg-go/pull/91#issuecomment-2136258091 CC @Fokko @wolfeidau @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] [feat] Ability to read/write table using `version-hint.txt` [iceberg-python]

2024-05-28 Thread via GitHub
kevinjqliu commented on issue #763: URL: https://github.com/apache/iceberg-python/issues/763#issuecomment-2136335963 We discussed this issue in the monthly sync and agreed that this is a useful feature. We'll first implement the read side in `pyiceberg`. The write side is complicated due to

Re: [I] Improve readme for pyiceberg [iceberg-python]

2024-05-28 Thread via GitHub
kevinjqliu commented on issue #774: URL: https://github.com/apache/iceberg-python/issues/774#issuecomment-2136348284 Great suggestion! It'll be good to balance contents between the README and the pyiceberg website (https://py.iceberg.apache.org/) -- This is an automated message from t

Re: [PR] Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan [iceberg]

2024-05-28 Thread via GitHub
manuzhang closed pull request #10252: Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan URL: https://github.com/apache/iceberg/pull/10252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan [iceberg]

2024-05-28 Thread via GitHub
manuzhang closed pull request #10252: Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan URL: https://github.com/apache/iceberg/pull/10252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan [iceberg]

2024-05-28 Thread via GitHub
flyrain commented on PR #10252: URL: https://github.com/apache/iceberg/pull/10252#issuecomment-2136479142 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-05-28 Thread via GitHub
amogh-jahagirdar commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1618157387 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2210,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFil

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-05-28 Thread via GitHub
amogh-jahagirdar commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1618157387 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2210,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFil

Re: [PR] Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan [iceberg]

2024-05-28 Thread via GitHub
manuzhang commented on PR #10252: URL: https://github.com/apache/iceberg/pull/10252#issuecomment-2136502711 Not sure why some CI runs get cancelled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Bump coverage from 7.5.2 to 7.5.3 [iceberg-python]

2024-05-28 Thread via GitHub
Fokko merged PR #776: URL: https://github.com/apache/iceberg-python/pull/776 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan [iceberg]

2024-05-28 Thread via GitHub
manuzhang closed pull request #10252: Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan URL: https://github.com/apache/iceberg/pull/10252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan [iceberg]

2024-05-28 Thread via GitHub
manuzhang commented on PR #10252: URL: https://github.com/apache/iceberg/pull/10252#issuecomment-2136545206 It looks to be related to runner ![CleanShot 2024-05-29 at 13 31 22@2x](https://github.com/apache/iceberg/assets/1191767/81d27b56-0b85-4d29-a1cc-bda5f2089714) -- This is an a

Re: [PR] Bump pydantic from 2.7.1 to 2.7.2 [iceberg-python]

2024-05-28 Thread via GitHub
Fokko merged PR #775: URL: https://github.com/apache/iceberg-python/pull/775 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Iceberg TTL setting [iceberg]

2024-05-28 Thread via GitHub
smileyboy2019 commented on issue #10372: URL: https://github.com/apache/iceberg/issues/10372#issuecomment-2136665243 是支持生命周期的TTL,同时设置归档目录,到期的数据自动归档历史数据路径里面 //TTL时间设置,超过7天数据自动删除 CREATE TABLE my_table ( id INT, name STRING ) PARTITIONED BY (dt STRING, city STRING) TBLPROPERTIES (