Re: [PR] Spark partial limit push down [iceberg]

2024-11-12 Thread via GitHub
aokolnychyi commented on PR #10943: URL: https://github.com/apache/iceberg/pull/10943#issuecomment-2471409893 Ack, will try to get to this asap. I am partially off this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
kevinjqliu opened a new pull request, #200: URL: https://github.com/apache/iceberg-go/pull/200 Deprecate the use of - https://dist.apache.org/repos/dist/dev/iceberg/KEYS in favor of - https://downloads.apache.org/iceberg/KEYS which is an alias for https://dist.apache.org/repos/dis

[PR] use `dist/release` KEYS [iceberg]

2024-11-12 Thread via GitHub
kevinjqliu opened a new pull request, #11526: URL: https://github.com/apache/iceberg/pull/11526 Deprecate the use of - https://dist.apache.org/repos/dist/dev/iceberg/KEYS in favor of - https://downloads.apache.org/iceberg/KEYS which is an alias for https://dist.apache.org/repos/di

Re: [PR] Add @override [iceberg-python]

2024-11-12 Thread via GitHub
cosmastech commented on PR #1312: URL: https://github.com/apache/iceberg-python/pull/1312#issuecomment-2471110050 @kevinjqliu Would you mind taking a look at what I've implemented thus far? It looks like this is failing in the pipeline, since typing.override doesn't exist prior to 3.13. Did

Re: [PR] Add @override [iceberg-python]

2024-11-12 Thread via GitHub
kevinjqliu commented on PR #1312: URL: https://github.com/apache/iceberg-python/pull/1312#issuecomment-2471317184 thanks for the PR! Looks like i was wrong and we need to do something like this in order to make the other python versions happy https://github.com/apache/iceberg-python/i

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-12 Thread via GitHub
wypoon commented on code in PR #11520: URL: https://github.com/apache/iceberg/pull/11520#discussion_r1838526411 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReader.java: ## @@ -28,5 +28,14 @@ public interface ParquetValueReader { List> columns(); + /**

Re: [I] Incorrect Metrics Calculation for Iceberg Table Due to Column Name Transformation with Special Characters [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed issue #10115: Incorrect Metrics Calculation for Iceberg Table Due to Column Name Transformation with Special Characters URL: https://github.com/apache/iceberg/issues/10115 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #11064: URL: https://github.com/apache/iceberg/pull/11064#issuecomment-2471994274 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Fix `ADLSFileIO.listPrefix()` behavior [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10403: Fix `ADLSFileIO.listPrefix()` behavior URL: https://github.com/apache/iceberg/pull/10403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] docs: Broken link from catalog concepts to configuration page [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10438: docs: Broken link from catalog concepts to configuration page URL: https://github.com/apache/iceberg/pull/10438 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Incorrect Metrics Calculation for Iceberg Table Due to Column Name Transformation with Special Characters [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on issue #10115: URL: https://github.com/apache/iceberg/issues/10115#issuecomment-2471993321 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Table Scan Delete File Handling: Positional and Equality Delete Support [iceberg-rust]

2024-11-12 Thread via GitHub
liurenjie1024 commented on PR #652: URL: https://github.com/apache/iceberg-rust/pull/652#issuecomment-2472270721 > Thanks so much for the review on this @liurenjie1024 - I've been ill for the past week or two so I've not had chance to work through your review in detail yet. I just wanted to

Re: [PR] Core: Change Delete granularity to file for new tables [iceberg]

2024-11-12 Thread via GitHub
manuzhang commented on PR #11478: URL: https://github.com/apache/iceberg/pull/11478#issuecomment-2472286633 I don't see #11273 being back-ported to Spark 3.3, 3.4 yet. Shall we skip changes to Spark 3.3, 3.4 until that is done? -- This is an automated message from the Apache Git Service.

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2024-11-12 Thread via GitHub
amogh-jahagirdar commented on code in PR #11525: URL: https://github.com/apache/iceberg/pull/11525#discussion_r1838502237 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/RewritableDeletes.java: ## @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Founda

Re: [I] Consider Using object_store as IO Abstraction [iceberg-rust]

2024-11-12 Thread via GitHub
BlakeOrth commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2471199556 Hi, I'm a relatively new user of the `icberg-rust` crate(s) and was hoping I could bring this discussion back to get some movement here. While Iceberg is a relatively new ecosys

[PR] ci: Quick update for release_rc.sh and the LICENSE [iceberg-go]

2024-11-12 Thread via GitHub
zeroshade opened a new pull request, #201: URL: https://github.com/apache/iceberg-go/pull/201 As we don't package any binaries for distribution, it should be sufficient for us to have the LICENSE simply cover the Apache License that the code is covered under. Also updating the `relea

Re: [PR] ci: Quick update for release_rc.sh and the LICENSE [iceberg-go]

2024-11-12 Thread via GitHub
zeroshade commented on PR #201: URL: https://github.com/apache/iceberg-go/pull/201#issuecomment-2471652518 @kevinjqliu @Fokko @nastra can you please take a look at this? Once this and https://github.com/apache/iceberg-go/pull/199 are merged we can create a new RC -- This is an automated m

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-11-12 Thread via GitHub
loicalleyne commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2471667064 hi @dwilson1988 yes, I'm wrapping up some work on another project and will be jumping back on this in a day or two. -- This is an automated message from the Apache Git Service.

Re: [PR] Add option to set max concurrency for table scan operations [iceberg-go]

2024-11-12 Thread via GitHub
zeroshade commented on code in PR #198: URL: https://github.com/apache/iceberg-go/pull/198#discussion_r1838822807 ## table/scanner.go: ## @@ -22,12 +22,12 @@ import ( "context" "fmt" "iter" - "runtime" "slices" "sync" "gith

Re: [PR] AWS: Enable RetryMode for AWS KMS client [iceberg]

2024-11-12 Thread via GitHub
amogh-jahagirdar commented on code in PR #11420: URL: https://github.com/apache/iceberg/pull/11420#discussion_r1839219803 ## aws/src/main/java/org/apache/iceberg/aws/kms/KmsClientProperties.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [I] Iceberg Spark Extensions conflict with Paimon [iceberg]

2024-11-12 Thread via GitHub
wForget commented on issue #10143: URL: https://github.com/apache/iceberg/issues/10143#issuecomment-2472216246 Fixed by https://github.com/apache/iceberg/pull/11480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] [WIP] Core: Prototype for DVs in V3 [iceberg]

2024-11-12 Thread via GitHub
aokolnychyi closed pull request #11302: [WIP] Core: Prototype for DVs in V3 URL: https://github.com/apache/iceberg/pull/11302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [WIP] Core: Prototype for DVs in V3 [iceberg]

2024-11-12 Thread via GitHub
aokolnychyi commented on PR #11302: URL: https://github.com/apache/iceberg/pull/11302#issuecomment-2471377793 Closing this in favor of other smaller PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] use `dist/release` KEYS [iceberg]

2024-11-12 Thread via GitHub
Fokko commented on PR #11526: URL: https://github.com/apache/iceberg/pull/11526#issuecomment-2471393990 Thanks for fixing this right away @kevinjqliu and thanks for the review @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] use `dist/release` KEYS [iceberg]

2024-11-12 Thread via GitHub
Fokko merged PR #11526: URL: https://github.com/apache/iceberg/pull/11526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
RussellSpitzer merged PR #200: URL: https://github.com/apache/iceberg-go/pull/200 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ice

Re: [PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
RussellSpitzer commented on PR #200: URL: https://github.com/apache/iceberg-go/pull/200#issuecomment-2471423776 Thanks @kevinjqliu for the PR and @zeroshade, @Fokko for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Spark partial limit push down [iceberg]

2024-11-12 Thread via GitHub
aokolnychyi commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1838670199 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan buildBat

[PR] Spark 3.5: Adapt PlanningBenchmark for DVs [iceberg]

2024-11-12 Thread via GitHub
aokolnychyi opened a new pull request, #11531: URL: https://github.com/apache/iceberg/pull/11531 This PR adapts our `PlanningBenchmark` for DVs. ``` Benchmark (type) Mode CntScore Er

[I] How to run streaming upserts and maintenance simultaneously? [iceberg]

2024-11-12 Thread via GitHub
meatheadmike opened a new issue, #11530: URL: https://github.com/apache/iceberg/issues/11530 ### Query engine Spark ### Question I'm trying to build a streaming upsert process using the latest version of spark (3.5.3) and iceberg (1.7.0). So far I've managed to get the s

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2024-11-12 Thread via GitHub
amogh-jahagirdar commented on PR #11525: URL: https://github.com/apache/iceberg/pull/11525#issuecomment-2472056218 >Just as a gut comment, if we just compressed them shouldn't we get almost all the benefits we are looking for? They are just a bunch of strings so the binary representation of

Re: [I] Support for snowflake catalog in apache iceberg [iceberg-python]

2024-11-12 Thread via GitHub
github-actions[bot] commented on issue #685: URL: https://github.com/apache/iceberg-python/issues/685#issuecomment-2471996089 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [PR] Improve MetricsReporter loading with class loader fallback [iceberg]

2024-11-12 Thread via GitHub
nastra commented on PR #10459: URL: https://github.com/apache/iceberg/pull/10459#issuecomment-2472584940 @bk-mz feel free to re-open the PR to mark it not stale -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] Consider Using object_store as IO Abstraction [iceberg-rust]

2024-11-12 Thread via GitHub
Xuanwo commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2472488509 Hi, thank you, @BlakeOrth, for bringing this up. It's part of our community's philosophy not to choose a winner. All implementations of Iceberg don't directly expose the und

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-12 Thread via GitHub
nastra commented on PR #11064: URL: https://github.com/apache/iceberg/pull/11064#issuecomment-2472590892 @ebyhr are you still working on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] ci: Quick update for release_rc.sh and the LICENSE [iceberg-go]

2024-11-12 Thread via GitHub
nastra merged PR #201: URL: https://github.com/apache/iceberg-go/pull/201 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] [AWS] S3FileIO - Add Cross-Region Bucket Access [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #9804: [AWS] S3FileIO - Add Cross-Region Bucket Access URL: https://github.com/apache/iceberg/pull/9804 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] [AWS] S3FileIO - Add Cross-Region Bucket Access [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #9804: URL: https://github.com/apache/iceberg/pull/9804#issuecomment-2471993254 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [I] How tracke authors of iceberg snapshots? [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on issue #9928: URL: https://github.com/apache/iceberg/issues/9928#issuecomment-2471993275 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Spark: readStream from Iceberg doesn't progress anymore after running Maintenance (rewrite_data_files and rewrite_manifests) [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on issue #10117: URL: https://github.com/apache/iceberg/issues/10117#issuecomment-2471993343 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Spark: readStream from Iceberg doesn't progress anymore after running Maintenance (rewrite_data_files and rewrite_manifests) [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed issue #10117: Spark: readStream from Iceberg doesn't progress anymore after running Maintenance (rewrite_data_files and rewrite_manifests) URL: https://github.com/apache/iceberg/issues/10117 -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Repair manifest action [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #10445: URL: https://github.com/apache/iceberg/pull/10445#issuecomment-2471993606 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] How tracke authors of iceberg snapshots? [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed issue #9928: How tracke authors of iceberg snapshots? URL: https://github.com/apache/iceberg/issues/9928 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Build: Bump Hive 2.3.10 [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10447: Build: Bump Hive 2.3.10 URL: https://github.com/apache/iceberg/pull/10447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Test: JdbcCatalog should not drop child namespaces [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #11063: URL: https://github.com/apache/iceberg/pull/11063#issuecomment-2471994233 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] DO-NOT-MERGE: Jackson access issue [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10460: DO-NOT-MERGE: Jackson access issue URL: https://github.com/apache/iceberg/pull/10460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Fix `ADLSFileIO.listPrefix()` behavior [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #10403: URL: https://github.com/apache/iceberg/pull/10403#issuecomment-2471993517 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] [WIP]Spark: Asynchronous Spark Micro Batch Planner [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #10444: URL: https://github.com/apache/iceberg/pull/10444#issuecomment-2471993578 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Build: Bump Hive 2.3.10 [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #10447: URL: https://github.com/apache/iceberg/pull/10447#issuecomment-2471993635 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] [WIP]Spark: Asynchronous Spark Micro Batch Planner [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10444: [WIP]Spark: Asynchronous Spark Micro Batch Planner URL: https://github.com/apache/iceberg/pull/10444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Repair manifest action [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10445: Repair manifest action URL: https://github.com/apache/iceberg/pull/10445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Core: Use Failsafe in ClientPoolImpl retry logic [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10458: Core: Use Failsafe in ClientPoolImpl retry logic URL: https://github.com/apache/iceberg/pull/10458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Hive: Return new scan after applying column project parameter [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #10449: URL: https://github.com/apache/iceberg/pull/10449#issuecomment-2471993676 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Improve MetricsReporter loading with class loader fallback [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #10459: URL: https://github.com/apache/iceberg/pull/10459#issuecomment-2471993741 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Improve MetricsReporter loading with class loader fallback [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10459: Improve MetricsReporter loading with class loader fallback URL: https://github.com/apache/iceberg/pull/10459 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] docs: Broken link from catalog concepts to configuration page [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #10438: URL: https://github.com/apache/iceberg/pull/10438#issuecomment-2471993552 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Hive: Return new scan after applying column project parameter [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] closed pull request #10449: Hive: Return new scan after applying column project parameter URL: https://github.com/apache/iceberg/pull/10449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Core: Use Failsafe in ClientPoolImpl retry logic [iceberg]

2024-11-12 Thread via GitHub
github-actions[bot] commented on PR #10458: URL: https://github.com/apache/iceberg/pull/10458#issuecomment-2471993703 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Support for snowflake catalog in apache iceberg [iceberg-python]

2024-11-12 Thread via GitHub
github-actions[bot] closed issue #685: Support for snowflake catalog in apache iceberg URL: https://github.com/apache/iceberg-python/issues/685 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[PR] Bump getdaft from 0.3.11 to 0.3.12 [iceberg-python]

2024-11-12 Thread via GitHub
dependabot[bot] opened a new pull request, #1316: URL: https://github.com/apache/iceberg-python/pull/1316 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.3.11 to 0.3.12. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2024-11-12 Thread via GitHub
amogh-jahagirdar commented on PR #11525: URL: https://github.com/apache/iceberg/pull/11525#issuecomment-2471722518 cc @singhpk234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2024-11-12 Thread via GitHub
RussellSpitzer commented on PR #11525: URL: https://github.com/apache/iceberg/pull/11525#issuecomment-2471741365 Just as a gut comment, if we just compressed them shouldn't we get almost all the benefits we are looking for? They are just a bunch of strings so the binary representation of al

Re: [I] Adjust the "table_exists" behavior in the REST Catalog [iceberg-python]

2024-11-12 Thread via GitHub
djouallah commented on issue #1018: URL: https://github.com/apache/iceberg-python/issues/1018#issuecomment-2471827257 catalog.url(Endpoints.load_table, prefixed=True, **catalog._split_identifier_for_path(identifier_tuple)) https://x-polaris.snowflakecomputing.com/polaris/api/cat

Re: [PR] feat: Add ViewUpdate to catalog [iceberg-rust]

2024-11-12 Thread via GitHub
c-thiel commented on PR #690: URL: https://github.com/apache/iceberg-rust/pull/690#issuecomment-2472609251 @liurenjie1024 it is rebased -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Spec: Add expiry time config to REST table load [iceberg]

2024-11-12 Thread via GitHub
munendrasn closed pull request #10873: Spec: Add expiry time config to REST table load URL: https://github.com/apache/iceberg/pull/10873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Spec: Add expiry time config to REST table load [iceberg]

2024-11-12 Thread via GitHub
munendrasn commented on PR #10873: URL: https://github.com/apache/iceberg/pull/10873#issuecomment-2472601833 Closing this as Store specific expiry time config is being introduced cc @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-12 Thread via GitHub
ebyhr commented on PR #11064: URL: https://github.com/apache/iceberg/pull/11064#issuecomment-2472604215 @nastra Yes, I was actually waiting for next review round. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Core: Use encoding/decoding methods for namespaces and deprecate Splitter/Joiner [iceberg]

2024-11-12 Thread via GitHub
nastra commented on PR #10858: URL: https://github.com/apache/iceberg/pull/10858#issuecomment-2472617654 @mayankvadariya can you please open a separate issue for this? Also do you have a reproducible example where this fails? I'm guessing this is in the Trino PR you linked? -- This is an

Re: [PR] Spark 3.5: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-12 Thread via GitHub
RussellSpitzer merged PR #11480: URL: https://github.com/apache/iceberg/pull/11480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Spark 3.5: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-12 Thread via GitHub
RussellSpitzer commented on PR #11480: URL: https://github.com/apache/iceberg/pull/11480#issuecomment-2470995100 Thanks @pan3793 for making sure things remain more compatible across the ecosystem. Thanks @huaxingao for reviewing! -- This is an automated message from the Apache Git Service

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-12 Thread via GitHub
rocco408 commented on PR #11419: URL: https://github.com/apache/iceberg/pull/11419#issuecomment-2471220679 Thanks again Russ, and Huaxin. I see where my previous attempt failed. In my last attempt I had swapped the placement of params to this `isPromotionAllowed(existingType, newType.asPrim

Re: [PR] Core: Use encoding/decoding methods for namespaces and deprecate Splitter/Joiner [iceberg]

2024-11-12 Thread via GitHub
mayankvadariya commented on PR #10858: URL: https://github.com/apache/iceberg/pull/10858#issuecomment-2471482802 This change seem to have affected nested namespace support. With encoded params, `queryParams` are further encoded in `URIBuilder`. https://github.com/apache/iceberg

[I] write.wap.enabled / spark.wap.branch behavior isn't really documented [iceberg]

2024-11-12 Thread via GitHub
wimlewis-amazon opened a new issue, #11528: URL: https://github.com/apache/iceberg/issues/11528 ### Feature Request / Improvement It's unclear from reading the doc what the effect of setting `write.wap.enabled` actually is. I can make some assumptions based on the write-audit-publish

[PR] Spark 3.5: Adapt DeleteFileIndexBenchmark for DVs [iceberg]

2024-11-12 Thread via GitHub
aokolnychyi opened a new pull request, #11529: URL: https://github.com/apache/iceberg/pull/11529 This PR adapts our `DeleteFileIndexBenchmark` for DVs. ``` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-12 Thread via GitHub
wypoon commented on PR #11520: URL: https://github.com/apache/iceberg/pull/11520#issuecomment-2471180832 @Fokko thanks for reviewing! I'd actually updated the deprecation comments locally but failed to commit the change before I pushed. -- This is an automated message from the Apache G

Re: [PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
kevinjqliu commented on code in PR #200: URL: https://github.com/apache/iceberg-go/pull/200#discussion_r1838528172 ## dev/release/verify_rc.sh: ## @@ -34,7 +34,7 @@ set -x VERSION="$1" RC="$2" -ICEBERG_DIST_BASE_URL="https://dist.apache.org/repos/dist/dev/iceberg"; +ICEBERG_

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-12 Thread via GitHub
wypoon commented on code in PR #11520: URL: https://github.com/apache/iceberg/pull/11520#discussion_r1838526411 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReader.java: ## @@ -28,5 +28,14 @@ public interface ParquetValueReader { List> columns(); + /**

Re: [PR] use `dist/release` KEYS [iceberg]

2024-11-12 Thread via GitHub
kevinjqliu commented on PR #11526: URL: https://github.com/apache/iceberg/pull/11526#issuecomment-2471280056 @RussellSpitzer used `downloads` whenever possible, except when uploading new keys -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [I] iceberg mor table execute merge very very slow [iceberg]

2024-11-12 Thread via GitHub
wimlewis-amazon commented on issue #7431: URL: https://github.com/apache/iceberg/issues/7431#issuecomment-2471204881 I'm having what looks like the same or similar problem. 1.6.1 seems a little bit faster than 1.0.0 or 1.4.x, but still incredibly slow — far too slow for MOR to be a usable m

Re: [PR] feat: more builders and writing manifests [iceberg-go]

2024-11-12 Thread via GitHub
dwilson1988 commented on PR #177: URL: https://github.com/apache/iceberg-go/pull/177#issuecomment-2470832115 > @dwilson1988 can you please rebase/fix the merge conflicts? @nastra done! -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] use `dist/release` KEYS [iceberg]

2024-11-12 Thread via GitHub
RussellSpitzer commented on code in PR #11526: URL: https://github.com/apache/iceberg/pull/11526#discussion_r1838577662 ## site/docs/how-to-release.md: ## @@ -204,7 +204,7 @@ The release tarball, signature, and checksums are here: * https://dist.apache.org/repos/dist/dev/icebe

Re: [PR] Spark partial limit push down [iceberg]

2024-11-12 Thread via GitHub
RussellSpitzer commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1838675045 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -405,14 +420,35 @@ public Scan build() { private Scan build

Re: [PR] Spark partial limit push down [iceberg]

2024-11-12 Thread via GitHub
aokolnychyi commented on code in PR #10943: URL: https://github.com/apache/iceberg/pull/10943#discussion_r1838676561 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseRowReader.java: ## @@ -48,6 +50,17 @@ abstract class BaseRowReader extends BaseReader taskG

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2024-11-12 Thread via GitHub
amogh-jahagirdar commented on code in PR #11525: URL: https://github.com/apache/iceberg/pull/11525#discussion_r1838499433 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/RewritableDeletes.java: ## @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2024-11-12 Thread via GitHub
amogh-jahagirdar commented on code in PR #11525: URL: https://github.com/apache/iceberg/pull/11525#discussion_r1838499433 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/RewritableDeletes.java: ## @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
RussellSpitzer commented on code in PR #200: URL: https://github.com/apache/iceberg-go/pull/200#discussion_r1838503234 ## dev/release/verify_rc.sh: ## @@ -34,7 +34,7 @@ set -x VERSION="$1" RC="$2" -ICEBERG_DIST_BASE_URL="https://dist.apache.org/repos/dist/dev/iceberg"; +ICEB

Re: [PR] Ignore schema merge updates from long -> int [iceberg]

2024-11-12 Thread via GitHub
rocco408 commented on code in PR #11419: URL: https://github.com/apache/iceberg/pull/11419#discussion_r1838575824 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWriterV2.java: ## @@ -244,4 +247,132 @@ public void testMergeSchemaSparkConfiguration(

Re: [PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
kevinjqliu commented on PR #200: URL: https://github.com/apache/iceberg-go/pull/200#issuecomment-2471284230 Updated to use `downloads` whenever possible, except when uploading new keys -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] Fix verification script, `verify_rc.sh` [iceberg-go]

2024-11-12 Thread via GitHub
kevinjqliu commented on PR #199: URL: https://github.com/apache/iceberg-go/pull/199#issuecomment-2471312299 verified change by running `dev/release/verify_rc.sh 0.1.0 0` ``` + VERIFY_SUCCESS=yes + echo 'RC looks good!' RC looks good! ``` -- This is an automated message

Re: [PR] use `dist/release` KEYS [iceberg]

2024-11-12 Thread via GitHub
RussellSpitzer commented on code in PR #11526: URL: https://github.com/apache/iceberg/pull/11526#discussion_r1838578205 ## dev/source-release.sh: ## @@ -149,7 +149,7 @@ The release tarball, signature, and checksums are here: * https://dist.apache.org/repos/dist/dev/iceberg/apa

Re: [PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
kevinjqliu commented on code in PR #200: URL: https://github.com/apache/iceberg-go/pull/200#discussion_r1838512881 ## dev/release/verify_rc.sh: ## @@ -34,7 +34,7 @@ set -x VERSION="$1" RC="$2" -ICEBERG_DIST_BASE_URL="https://dist.apache.org/repos/dist/dev/iceberg"; +ICEBERG_

Re: [PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
RussellSpitzer commented on code in PR #200: URL: https://github.com/apache/iceberg-go/pull/200#discussion_r1838500357 ## dev/release/verify_rc.sh: ## @@ -34,7 +34,7 @@ set -x VERSION="$1" RC="$2" -ICEBERG_DIST_BASE_URL="https://dist.apache.org/repos/dist/dev/iceberg"; +ICEB

Re: [PR] use `dist/release` KEYS [iceberg-go]

2024-11-12 Thread via GitHub
kevinjqliu commented on code in PR #200: URL: https://github.com/apache/iceberg-go/pull/200#discussion_r1838576468 ## dev/release/verify_rc.sh: ## @@ -34,7 +34,7 @@ set -x VERSION="$1" RC="$2" -ICEBERG_DIST_BASE_URL="https://dist.apache.org/repos/dist/dev/iceberg"; +ICEBERG_

Re: [PR] Fix verification script, `verify_rc.sh` [iceberg-go]

2024-11-12 Thread via GitHub
kevinjqliu commented on PR #199: URL: https://github.com/apache/iceberg-go/pull/199#issuecomment-2471457974 @Fokko do you mind taking a look? this fixes the verification script. Would be good to have merged for the new RC -- This is an automated message from the Apache Git Service. To re

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2024-11-12 Thread via GitHub
amogh-jahagirdar commented on code in PR #11525: URL: https://github.com/apache/iceberg/pull/11525#discussion_r1838496080 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java: ## @@ -500,33 +500,32 @@ protected PartitioningWriter, Delet

Re: [PR] Spark: Relativize in-memory paths for data file and rewritable delete file locations [iceberg]

2024-11-12 Thread via GitHub
amogh-jahagirdar commented on code in PR #11525: URL: https://github.com/apache/iceberg/pull/11525#discussion_r1838496080 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java: ## @@ -500,33 +500,32 @@ protected PartitioningWriter, Delet

Re: [I] Update `KEYS` file reference [iceberg-python]

2024-11-12 Thread via GitHub
Fokko closed issue #1311: Update `KEYS` file reference URL: https://github.com/apache/iceberg-python/issues/1311 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] use `dist/release` KEYS [iceberg-python]

2024-11-12 Thread via GitHub
Fokko merged PR #1315: URL: https://github.com/apache/iceberg-python/pull/1315 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Table Scan Delete File Handling: Positional and Equality Delete Support [iceberg-rust]

2024-11-12 Thread via GitHub
sdd commented on PR #652: URL: https://github.com/apache/iceberg-rust/pull/652#issuecomment-2470127741 Thanks so much for the review on this @liurenjie1024 - I've been ill for the past week or two so I've not had chance to work through your review in detail yet. I just wanted to let you kno

Re: [PR] API: Support removeUnusedSpecs in ExpireSnapshots [iceberg]

2024-11-12 Thread via GitHub
nastra commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1837945532 ## core/src/main/java/org/apache/iceberg/MetadataUpdateParser.java: ## @@ -126,6 +127,9 @@ private MetadataUpdateParser() {} // SetCurrentViewVersion private sta

  1   2   >