Re: [I] Inconsistent initial field ids with REST catalog [iceberg]

2024-04-03 Thread via GitHub
nastra closed issue #10084: Inconsistent initial field ids with REST catalog URL: https://github.com/apache/iceberg/issues/10084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Spark configuration for amazon access key and secret key with glue catalog for apache Iceberg is not honoring [iceberg]

2024-04-03 Thread via GitHub
nastra commented on issue #10078: URL: https://github.com/apache/iceberg/issues/10078#issuecomment-2036229721 @AwasthiSomesh you might want to go through the link that I posted (and also through the Glue docs on how to connect to Iceberg). The settings you need could depend on your local AW

Re: [I] Spark configuration for amazon access key and secret key with glue catalog for apache Iceberg is not honoring [iceberg]

2024-04-03 Thread via GitHub
AwasthiSomesh commented on issue #10078: URL: https://github.com/apache/iceberg/issues/10078#issuecomment-2036225160 @lxs360 were you able resolve this issue ? I'm also facing the same issue using same spark configuration? I see you already reported this issue long back -https://gith

Re: [I] apache iceberg对接s3存储,创建存储位置为s3的表格,java代码操作遇到了问题Unable to load region [iceberg]

2024-04-03 Thread via GitHub
AwasthiSomesh commented on issue #4739: URL: https://github.com/apache/iceberg/issues/4739#issuecomment-2036212994 @lxs360 were you able resolve this issue ? I'm also facing the same issue using same spark configuration? -- This is an automated message from the Apache Git Service. To resp

Re: [I] Inconsistent initial field ids with REST catalog [iceberg]

2024-04-03 Thread via GitHub
nastra commented on issue #10084: URL: https://github.com/apache/iceberg/issues/10084#issuecomment-2036212500 > ``` > spark.sql("CREATE TABLE rest.test.table1(id bigint, data string)").show(false) > ``` > > And you can see the request body sent to the createTable endpoint will

Re: [I] Support CreateTableTransaction [iceberg-python]

2024-04-03 Thread via GitHub
HonahX closed issue #483: Support CreateTableTransaction URL: https://github.com/apache/iceberg-python/issues/483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

2024-04-03 Thread via GitHub
HonahX merged PR #498: URL: https://github.com/apache/iceberg-python/pull/498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

2024-04-03 Thread via GitHub
HonahX commented on PR #498: URL: https://github.com/apache/iceberg-python/pull/498#issuecomment-2036194320 Thanks @Fokko @syun64 for detailed reviewing! Since this PR changes/refactors lots of code, I'll merge this first and leave other works (such as supporting createTableTrans in Hive an

Re: [I] Spark configuration for amazon access key and secret key with glue catalog for apache Iceberg is not honoring [iceberg]

2024-04-03 Thread via GitHub
AwasthiSomesh commented on issue #10078: URL: https://github.com/apache/iceberg/issues/10078#issuecomment-2036193452 @nastra Thanks a lot for your reply. I am facing credentials load issue from spark .. could you please help how to add access-key and secret key with region on spark

Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

2024-04-03 Thread via GitHub
HonahX commented on code in PR #498: URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1550918541 ## pyiceberg/table/__init__.py: ## @@ -760,7 +852,10 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda for update in updates

Re: [I] Support relative paths in Table Metadata [iceberg]

2024-04-03 Thread via GitHub
ksmatharoo commented on issue #1617: URL: https://github.com/apache/iceberg/issues/1617#issuecomment-2036181086 @jotarada We encountered a similar issue when we switched S3 vendors, necessitating the copying of all data to another bucket, resulting in unreadable iceberg tables. Currently, w

Re: [I] How to move Iceberg table from one location to another [iceberg]

2024-04-03 Thread via GitHub
ksmatharoo commented on issue #3142: URL: https://github.com/apache/iceberg/issues/3142#issuecomment-2036179649 @atifiu @meicanhong We encountered a similar issue when we switched S3 vendors, necessitating the copying of all data to another bucket, resulting in unreadable iceberg tables. Cu

Re: [I] Spark configuration for amazon access key and secret key with glue catalog for apache Iceberg is not honoring [iceberg]

2024-04-03 Thread via GitHub
nastra commented on issue #10078: URL: https://github.com/apache/iceberg/issues/10078#issuecomment-2036175205 I see you have ``` .config("spark.sql.catalog.AwsDataCatalog","org.apache.iceberg.spark.SparkSessionCatalog") .config("spark.sql.catalog.AwsDataCatalog", "org.apache.icebe

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-04-03 Thread via GitHub
marvinlanhenke commented on code in PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#discussion_r1550859891 ## crates/iceberg/src/expr/visitors/bound_predicate_evaluator.rs: ## @@ -0,0 +1,339 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or m

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-04-03 Thread via GitHub
marvinlanhenke commented on code in PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#discussion_r1550844497 ## crates/iceberg/src/expr/visitors/bound_predicate_evaluator.rs: ## @@ -0,0 +1,339 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or m

Re: [PR] Docs: Document support for binary in truncate transform [iceberg]

2024-04-03 Thread via GitHub
amogh-jahagirdar commented on code in PR #10079: URL: https://github.com/apache/iceberg/pull/10079#discussion_r1550841639 ## format/spec.md: ## @@ -314,7 +314,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ |---|--

Re: [PR] Spark 3.5: Spark action to compute the partition stats [iceberg]

2024-04-03 Thread via GitHub
aokolnychyi commented on PR #9437: URL: https://github.com/apache/iceberg/pull/9437#issuecomment-2036115472 I cloned this change and played with it locally. Here are my thoughts. 1. We should focus on the local implementation for now. I think it is going to perform OK for most use cas

Re: [PR] Docs: Document support for binary in truncate transform [iceberg]

2024-04-03 Thread via GitHub
amogh-jahagirdar commented on code in PR #10079: URL: https://github.com/apache/iceberg/pull/10079#discussion_r1550831622 ## format/spec.md: ## @@ -314,7 +314,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ |---|--

Re: [I] iceberg-aws: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-04-03 Thread via GitHub
tomtongue commented on issue #9080: URL: https://github.com/apache/iceberg/issues/9080#issuecomment-2036003100 @nastra If no one is assigned with this project, could you assign this with me? I'll work on this. -- This is an automated message from the Apache Git Service. To respond to the

[I] iceberg does not allow selection of AWS profiles other than "default" [iceberg]

2024-04-03 Thread via GitHub
hgschmie opened a new issue, #10085: URL: https://github.com/apache/iceberg/issues/10085 ### Apache Iceberg version 1.5.0 (latest release) ### Query engine Flink ### Please describe the bug 🐞 We have a test setup where multiple AWS profiles exist in the usua

Re: [PR] Parquet: Make row-group filters cooperate to filter [iceberg]

2024-04-03 Thread via GitHub
zhongyujiang commented on PR #6893: URL: https://github.com/apache/iceberg/pull/6893#issuecomment-2035906920 > any reason why you stopped pursuing it? There was relatively little feedback from the community after openning this PR, so I did not proceed with it further. Since more peo

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-03 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1550695864 ## core/src/main/java/org/apache/iceberg/rest/PaginatedList.java: ## @@ -0,0 +1,269 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-03 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1550695864 ## core/src/main/java/org/apache/iceberg/rest/PaginatedList.java: ## @@ -0,0 +1,269 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-03 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1550695864 ## core/src/main/java/org/apache/iceberg/rest/PaginatedList.java: ## @@ -0,0 +1,269 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-03 Thread via GitHub
rahil-c commented on PR #9782: URL: https://github.com/apache/iceberg/pull/9782#issuecomment-2035897279 > I just want to add a few high-level comments here: > > 1. It looks like `pageSize` is currently required, but I believe the default should be that if it is not set, no pagination

Re: [I] TableProperties 'write.format.default' not correct. [iceberg]

2024-04-03 Thread via GitHub
github-actions[bot] commented on issue #2440: URL: https://github.com/apache/iceberg/issues/2440#issuecomment-2035836502 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Report Manifest/Metadata Files created During Migrate/Snapshot [iceberg]

2024-04-03 Thread via GitHub
github-actions[bot] commented on issue #2439: URL: https://github.com/apache/iceberg/issues/2439#issuecomment-2035836481 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] RepairManifestsAction [iceberg]

2024-04-03 Thread via GitHub
github-actions[bot] commented on issue #2435: URL: https://github.com/apache/iceberg/issues/2435#issuecomment-2035836465 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Consider deprecating updating table state through ALTER TABLE ... SET TBLPROPERTIES [iceberg]

2024-04-03 Thread via GitHub
github-actions[bot] commented on issue #2424: URL: https://github.com/apache/iceberg/issues/2424#issuecomment-2035836434 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Migrate table's "Rename to backup" may cause datafile movement [iceberg]

2024-04-03 Thread via GitHub
github-actions[bot] commented on issue #2425: URL: https://github.com/apache/iceberg/issues/2425#issuecomment-2035836450 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Extend HTTPClient Builder to allow setting a proxy server [iceberg]

2024-04-03 Thread via GitHub
amogh-jahagirdar commented on code in PR #10052: URL: https://github.com/apache/iceberg/pull/10052#discussion_r1550644916 ## core/src/test/java/org/apache/iceberg/rest/TestHTTPClient.java: ## @@ -121,6 +128,92 @@ public void testHeadFailure() throws JsonProcessingException {

Re: [I] Inconsistent initial field ids with REST catalog [iceberg]

2024-04-03 Thread via GitHub
RussellSpitzer commented on issue #10084: URL: https://github.com/apache/iceberg/issues/10084#issuecomment-2035737989 The core lib defined this behavior a while back. Any table that goes through the create table logic has its fields reordered starting a 1 regardless of the schemas field ids

Re: [PR] Move writes to Transaction [iceberg-python]

2024-04-03 Thread via GitHub
syun64 commented on code in PR #571: URL: https://github.com/apache/iceberg-python/pull/571#discussion_r1550573900 ## tests/integration/test_writes.py: ## @@ -778,3 +778,31 @@ def test_inspect_snapshots( continue assert left == right, f"Difference

[I] Inconsistent initial field ids with REST catalog [iceberg]

2024-04-03 Thread via GitHub
mrcnc opened a new issue, #10084: URL: https://github.com/apache/iceberg/issues/10084 ### Query engine Using Spark 3.4.0 with Iceberg 1.4.3 ### Question To reproduce this behavior you can start a spark shell configured with 2 catalogs ``` SPARK_VERSION=3.4 I

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-03 Thread via GitHub
viirya commented on PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#issuecomment-2035635200 I've addressed some of above reviews. I will resolve other reviews soon. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-03 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1550524484 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +fn get_

[I] Add samples for listing table.files and table.partitions [iceberg-python]

2024-04-03 Thread via GitHub
moryachok opened a new issue, #578: URL: https://github.com/apache/iceberg-python/issues/578 ### Question After examining your documentation, I couldn't locate the methods to list a table's partitions or its data files. Could you please provide relevant examples or consider adding th

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-04-03 Thread via GitHub
Fokko commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1550332689 ## pyiceberg/typedef.py: ## @@ -199,3 +199,7 @@ def __repr__(self) -> str: def record_fields(self) -> List[str]: """Return values of all the fields of

Re: [PR] [Doc] Update how-to-release.md [iceberg-python]

2024-04-03 Thread via GitHub
Fokko merged PR #576: URL: https://github.com/apache/iceberg-python/pull/576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Disallow default header to be overwritten [iceberg-python]

2024-04-03 Thread via GitHub
Fokko merged PR #577: URL: https://github.com/apache/iceberg-python/pull/577 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Custom header should not overwrite default header in RESTCatalog [iceberg-python]

2024-04-03 Thread via GitHub
Fokko closed issue #550: Custom header should not overwrite default header in RESTCatalog URL: https://github.com/apache/iceberg-python/issues/550 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-03 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1550312388 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +fn get_

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-03 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1550310964 ## crates/iceberg/src/arrow.rs: ## @@ -113,6 +143,405 @@ impl ArrowReader { // TODO: full implementation ProjectionMask::all() } + +fn get_

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-04-03 Thread via GitHub
sdd commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-2035371003 Hi @liurenjie1024 and @marvinlanhenke. I've updated this PR to align with the direction suggested above - thanks a lot for all the feedback! This PR is clearly far too big to easily

Re: [PR] chore: upgrade to rust-version 1.77.1 [iceberg-rust]

2024-04-03 Thread via GitHub
Fokko commented on PR #316: URL: https://github.com/apache/iceberg-rust/pull/316#issuecomment-2035331577 Merging this to fix the CI, thanks @marvinlanhenke 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] CI-Build: Breaks due to `motore` update [iceberg-rust]

2024-04-03 Thread via GitHub
Fokko closed issue #315: CI-Build: Breaks due to `motore` update URL: https://github.com/apache/iceberg-rust/issues/315 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] chore: upgrade to rust-version 1.77.1 [iceberg-rust]

2024-04-03 Thread via GitHub
Fokko merged PR #316: URL: https://github.com/apache/iceberg-rust/pull/316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] GCS support? [iceberg-go]

2024-04-03 Thread via GitHub
thorfour closed issue #60: GCS support? URL: https://github.com/apache/iceberg-go/issues/60 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsub

Re: [I] Why Iceberg do not support column with default value? [iceberg]

2024-04-03 Thread via GitHub
Pilipets commented on issue #8527: URL: https://github.com/apache/iceberg/issues/8527#issuecomment-2035285224 Is there any examples for Iceberg defaults with V3? The description makes sense, but not sure what's the format or how to set initial-default from the given link. -- This is an a

[PR] chore: upgrade to rust-version 1.77.1 [iceberg-rust]

2024-04-03 Thread via GitHub
marvinlanhenke opened a new pull request, #316: URL: https://github.com/apache/iceberg-rust/pull/316 closes #315 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Parquet: Make row-group filters cooperate to filter [iceberg]

2024-04-03 Thread via GitHub
cccs-jc commented on PR #6893: URL: https://github.com/apache/iceberg/pull/6893#issuecomment-2035231045 I'm interested in getting that PR into the upstream Iceberg. @zhongyujiang any reason why you stopped pursuing it? Are you using it in production? -- This is an automated message from t

Re: [PR] Move writes to Transaction [iceberg-python]

2024-04-03 Thread via GitHub
jqin61 commented on code in PR #571: URL: https://github.com/apache/iceberg-python/pull/571#discussion_r1550202992 ## tests/integration/test_writes.py: ## @@ -778,3 +778,31 @@ def test_inspect_snapshots( continue assert left == right, f"Difference

Re: [PR] Disallow default header to be overwritten [iceberg-python]

2024-04-03 Thread via GitHub
flyrain commented on PR #577: URL: https://github.com/apache/iceberg-python/pull/577#issuecomment-2035208510 @Fokko, do you wanna take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Unable to load an iceberg table from aws glue catalog [iceberg-python]

2024-04-03 Thread via GitHub
kevinjqliu commented on issue #515: URL: https://github.com/apache/iceberg-python/issues/515#issuecomment-2035144573 thank you! should we close this in favor of #570? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] PyArrow S3FileSystem doesn't honor the AWS profile config [iceberg-python]

2024-04-03 Thread via GitHub
kevinjqliu commented on issue #570: URL: https://github.com/apache/iceberg-python/issues/570#issuecomment-2035130535 I think it makes sense to have both a "catalog level" configuration and a "file level" configuration. A catalog might have a different set of permissions from when reading sp

Re: [PR] Disallow default header to be overwritten [iceberg-python]

2024-04-03 Thread via GitHub
whynick1 commented on PR #577: URL: https://github.com/apache/iceberg-python/pull/577#issuecomment-2035081299 @flyrain @liurenjie1024 Can you take a look please. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Disallow default header to be overwritten [iceberg-python]

2024-04-03 Thread via GitHub
whynick1 commented on code in PR #577: URL: https://github.com/apache/iceberg-python/pull/577#discussion_r1550080729 ## tests/catalog/test_rest.py: ## @@ -277,36 +277,55 @@ def test_properties_sets_headers(requests_mock: Mocker) -> None: ) catalog = RestCatalog( -

Re: [PR] Disallow default header to be overwritten [iceberg-python]

2024-04-03 Thread via GitHub
whynick1 commented on code in PR #577: URL: https://github.com/apache/iceberg-python/pull/577#discussion_r1550080729 ## tests/catalog/test_rest.py: ## @@ -277,36 +277,55 @@ def test_properties_sets_headers(requests_mock: Mocker) -> None: ) catalog = RestCatalog( -

Re: [I] Spark configuration for amazon access key and secret key with glue catalog for apache Iceberg is not honoring [iceberg]

2024-04-03 Thread via GitHub
AwasthiSomesh commented on issue #10078: URL: https://github.com/apache/iceberg/issues/10078#issuecomment-2035029132 @nastra Even I see we are getting error for region unable to load. Error:- Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Una

Re: [PR] Add option to delete datafiles [iceberg-python]

2024-04-03 Thread via GitHub
jqin61 commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1548568425 ## pyiceberg/table/__init__.py: ## @@ -2726,6 +2731,112 @@ def _commit(self) -> UpdatesAndRequirements: ) +class DeleteFiles(_MergingSnapshotProducer):

Re: [PR] Spark 3.5: Spark action to compute the partition stats [iceberg]

2024-04-03 Thread via GitHub
aokolnychyi commented on PR #9437: URL: https://github.com/apache/iceberg/pull/9437#issuecomment-203504 Will take a look this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] Spark configuration for amazon access key and secret key with glue catalog for apache Iceberg is not honoring [iceberg]

2024-04-03 Thread via GitHub
AwasthiSomesh commented on issue #10078: URL: https://github.com/apache/iceberg/issues/10078#issuecomment-2034985982 @nastra Thanks for your reply After adding this we see the same error. Could you please check is there any mistake from my end .config("spark.sql.catalog.Aw

Re: [I] CI-Build: Breaks due to `motore` update [iceberg-rust]

2024-04-03 Thread via GitHub
liurenjie1024 commented on issue #315: URL: https://github.com/apache/iceberg-rust/issues/315#issuecomment-2034932066 I'm in favor of updating rustc since we want to use latest `volo-thrift`, the only cons is that it requires users to upgrade rustc to 1.77.0+. > Running cargo update m

Re: [PR] objstore: Replace IO interface with objstore.Bucket [iceberg-go]

2024-04-03 Thread via GitHub
zeroshade commented on PR #66: URL: https://github.com/apache/iceberg-go/pull/66#issuecomment-2034881770 That's my fault for not responding here, it's been a busy month. My apologies. I would absolutely be willing to approve a cleaned up PR. -- This is an automated message from the

Re: [PR] objstore: Replace IO interface with objstore.Bucket [iceberg-go]

2024-04-03 Thread via GitHub
thorfour commented on PR #66: URL: https://github.com/apache/iceberg-go/pull/66#issuecomment-2034863026 Seems like this repo is relatively inactive. So I'm going to close this for now, and continue in our fork. Thanks! -- This is an automated message from the Apache Git Service. To respon

Re: [PR] objstore: Replace IO interface with objstore.Bucket [iceberg-go]

2024-04-03 Thread via GitHub
thorfour closed pull request #66: objstore: Replace IO interface with objstore.Bucket URL: https://github.com/apache/iceberg-go/pull/66 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Support identifier warehouses [iceberg-rust]

2024-04-03 Thread via GitHub
Fokko commented on code in PR #308: URL: https://github.com/apache/iceberg-rust/pull/308#discussion_r1549860927 ## crates/catalog/rest/src/catalog.rs: ## @@ -617,7 +617,13 @@ impl RestCatalog { props.extend(config); } -let file_io = match self.co

Re: [PR] 5 dremio blog march 2024 [iceberg]

2024-04-03 Thread via GitHub
nastra merged PR #10067: URL: https://github.com/apache/iceberg/pull/10067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] CI-Build: Breaks due to `motore` update [iceberg-rust]

2024-04-03 Thread via GitHub
marvinlanhenke commented on issue #315: URL: https://github.com/apache/iceberg-rust/issues/315#issuecomment-2034703376 cc @liurenjie1024 @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] refine: seperate parquet reader and arrow convert [iceberg-rust]

2024-04-03 Thread via GitHub
liurenjie1024 merged PR #313: URL: https://github.com/apache/iceberg-rust/pull/313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: Glue Catalog - namespace operations (3/3) [iceberg-rust]

2024-04-03 Thread via GitHub
marvinlanhenke commented on PR #314: URL: https://github.com/apache/iceberg-rust/pull/314#issuecomment-2034723238 @liurenjie1024 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[I] Add capability to support an external index for additional file filtering [iceberg]

2024-04-03 Thread via GitHub
asuresh8 opened a new issue, #10083: URL: https://github.com/apache/iceberg/issues/10083 ### Feature Request / Improvement Iceberg supports skip indexes natively, but these skip indexes are limited in scope. Users should be able to create additional indices that can be applied at que

Re: [PR] 5 dremio blog march 2024 [iceberg]

2024-04-03 Thread via GitHub
AlexMercedCoder commented on code in PR #10067: URL: https://github.com/apache/iceberg/pull/10067#discussion_r1549766534 ## site/docs/blogs.md: ## @@ -77,26 +119,31 @@ Here is a list of company blogs that talk about Iceberg. The blogs are ordered f **Authors**: [Ayush Saxena

[PR] feat: Glue Catalog - namespace operations (3/3) [iceberg-rust]

2024-04-03 Thread via GitHub
marvinlanhenke opened a new pull request, #314: URL: https://github.com/apache/iceberg-rust/pull/314 ### Which issue does this PR close? Closes #249 (Task 3/3) ### Rationale for this change Add support for Glue Catalog, to reach feature parity with other implementations.

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-03 Thread via GitHub
nastra merged PR #10069: URL: https://github.com/apache/iceberg/pull/10069 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on PR #9793: URL: https://github.com/apache/iceberg/pull/9793#issuecomment-2034605250 > @nastra yes, I'm gonna made changes you proposed. In majority changes I switched Junit4 to Junit5 not AssertJ because it was said to switch to Junit5. Anyway, thank you for your comments

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
igoradulian commented on PR #9793: URL: https://github.com/apache/iceberg/pull/9793#issuecomment-2034574140 @nastra yes, I'm gonna made changes you proposed. In majority changes I switched Junit4 to Junit5 not AssertJ because it was said to switch to Junit5. Anyway, thank you for your comme

Re: [PR] Fix `rewrite_position_delete_files` result file set [iceberg]

2024-04-03 Thread via GitHub
nastra commented on PR #9945: URL: https://github.com/apache/iceberg/pull/9945#issuecomment-2034532286 @bk-mz I don't think the test actually reproduces the underlying issue. See my comment in https://github.com/apache/iceberg/pull/9945#discussion_r1526415663 -- This is an automated messa

Re: [PR] Build: add gradle wrapper validation action [iceberg]

2024-04-03 Thread via GitHub
nastra closed pull request #10073: Build: add gradle wrapper validation action URL: https://github.com/apache/iceberg/pull/10073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] typealias for table version [iceberg-python]

2024-04-03 Thread via GitHub
MehulBatra commented on PR #566: URL: https://github.com/apache/iceberg-python/pull/566#issuecomment-2034449315 > Thanks for working on this @MehulBatra 🙌 Always great to work on Iceberg and learn from you and the team! -- This is an automated message from the Apache Git Service. To

Re: [PR] 5 dremio blog march 2024 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on code in PR #10067: URL: https://github.com/apache/iceberg/pull/10067#discussion_r1549628699 ## site/docs/blogs.md: ## @@ -77,26 +119,31 @@ Here is a list of company blogs that talk about Iceberg. The blogs are ordered f **Authors**: [Ayush Saxena](https:/

Re: [PR] typealias for table version [iceberg-python]

2024-04-03 Thread via GitHub
Fokko merged PR #566: URL: https://github.com/apache/iceberg-python/pull/566 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-04-03 Thread via GitHub
nk1506 commented on PR #9852: URL: https://github.com/apache/iceberg/pull/9852#issuecomment-2034389989 @szehon-ho / @nastra , Please let me know if there are any comments left . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

2024-04-03 Thread via GitHub
Fokko commented on code in PR #498: URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1549581440 ## mkdocs/docs/api.md: ## @@ -165,6 +165,25 @@ catalog.create_table( ) ``` +To create a table with some subsequent changes in a transaction: Review Comment:

Re: [PR] Support CreateTableTransaction in Glue and Rest [iceberg-python]

2024-04-03 Thread via GitHub
Fokko commented on code in PR #498: URL: https://github.com/apache/iceberg-python/pull/498#discussion_r1549588018 ## pyiceberg/table/__init__.py: ## @@ -760,7 +852,10 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda for update in updates:

Re: [I] SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog [iceberg]

2024-04-03 Thread via GitHub
ajantha-bhat commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2034277214 Namespace has to be created explicitly in Nessie as described in https://projectnessie.org/blog/namespace-enforcement/ -- This is an automated message from the Apache Git Se

Re: [I] SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog [iceberg]

2024-04-03 Thread via GitHub
nastra commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2034239933 @matepek fundamentally that issue is the same as I described in https://github.com/apache/iceberg/issues/10003#issuecomment-2007780751. `SparkSessionCatalog` doesn't create a namesp

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1549453971 ## data/src/test/java/org/apache/iceberg/data/avro/TestSingleMessageEncoding.java: ## @@ -122,7 +123,7 @@ public void testSchemaEvolution() throws Exception { dec

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1549450840 ## data/src/test/java/org/apache/iceberg/data/DeleteReadTests.java: ## @@ -108,11 +112,8 @@ public void writeTestDataFile() throws IOException { records.add(record.

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1549452602 ## data/src/test/java/org/apache/iceberg/data/TestGenericRecord.java: ## @@ -19,12 +19,12 @@ package org.apache.iceberg.data; import static org.apache.iceberg.types.

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1549452056 ## data/src/test/java/org/apache/iceberg/data/TestGenericReaderDeletes.java: ## @@ -38,7 +38,8 @@ public class TestGenericReaderDeletes extends DeleteReadTests { @

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1549451494 ## data/src/test/java/org/apache/iceberg/data/TestDataFileIndexStatsFilters.java: ## @@ -85,33 +84,25 @@ public void createTableAndData() throws IOException { recor

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1549445167 ## data/src/test/java/org/apache/iceberg/data/DeleteReadTests.java: ## @@ -108,11 +112,8 @@ public void writeTestDataFile() throws IOException { records.add(record.

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-04-03 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1549444931 ## data/src/test/java/org/apache/iceberg/data/DeleteReadTests.java: ## @@ -18,6 +18,7 @@ */ package org.apache.iceberg.data; + Review Comment: unnecessary newli

Re: [I] Create JUnit5-version of FlinkCatalogTestBase [iceberg]

2024-04-03 Thread via GitHub
nastra commented on issue #9079: URL: https://github.com/apache/iceberg/issues/9079#issuecomment-2034192040 fixed by #9381 #9598 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] iceberg-core: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-04-03 Thread via GitHub
nastra closed issue #9085: iceberg-core: Switch tests to JUnit5 + AssertJ-style assertions URL: https://github.com/apache/iceberg/issues/9085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Create JUnit5-version of FlinkCatalogTestBase [iceberg]

2024-04-03 Thread via GitHub
nastra closed issue #9079: Create JUnit5-version of FlinkCatalogTestBase URL: https://github.com/apache/iceberg/issues/9079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Migrate TableTestBase related classes to JUnit5 [iceberg]

2024-04-03 Thread via GitHub
nastra merged PR #10080: URL: https://github.com/apache/iceberg/pull/10080 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Hive: Fix metadata file not found [iceberg]

2024-04-03 Thread via GitHub
lurnagao-dahua commented on code in PR #10069: URL: https://github.com/apache/iceberg/pull/10069#discussion_r1549380381 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCommits.java: ## @@ -397,6 +397,30 @@ public void testLockExceptionUnknownSuccessCommit() throw

Re: [PR] Migrate TableTestBase related classes to JUnit5 [iceberg]

2024-04-03 Thread via GitHub
tomtongue commented on PR #10080: URL: https://github.com/apache/iceberg/pull/10080#issuecomment-2034071501 > the changes you have so far LGTM, is this PR still WIP or ready? Thanks for the review. This PR is ready, and all the classes are already migrated. Remove "WIP". -- This is

  1   2   >