Re: [PR] Clarify which columns can be used for equality delete files. [iceberg]

2023-11-05 Thread via GitHub
liurenjie1024 commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1382867987 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify dele

Re: [I] Ability to the write Metadata JSON [iceberg-python]

2023-11-05 Thread via GitHub
HonahX commented on issue #22: URL: https://github.com/apache/iceberg-python/issues/22#issuecomment-1794191459 Hi @Fokko. Is there an update on this issue? I am interested in taking this if it's still open. In terms of implementation, I was thinking of something like this: ```pytho

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-05 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1382845319 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,351 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-05 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1382844788 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -132,74 +131,36 @@ protected void doRefresh() { @Override protected voi

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-05 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1382844582 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieUtil.java: ## @@ -165,4 +180,77 @@ public static TableMetadata updateTableMetadataWithNessieSpecificProper

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-05 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r138284 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -135,71 +135,26 @@ protected void doCommit(TableMetadata base, TableMetadata m

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-05 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1382844294 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [I] Flink write iceberg bug(org.apache.iceberg.exceptions.NotFoundException) [iceberg]

2023-11-05 Thread via GitHub
pvary commented on issue #5846: URL: https://github.com/apache/iceberg/issues/5846#issuecomment-1794170117 @lirui-apache: For the record: To restore the state of the Flink job, you need the previous snapshot (to identify the last committed snapshot), and the new data files and temporary

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-05 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1382711672 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,351 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-05 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1382710543 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,351 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-05 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1382704220 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -400,8 +400,15 @@ public void replaceTableViaTransactionThatAlreadyExistsAsView() {

Re: [PR] Spark 3.5: Don't throw exception when decoding dictionary of type INT96 [iceberg]

2023-11-05 Thread via GitHub
manuzhang commented on PR #8988: URL: https://github.com/apache/iceberg/pull/8988#issuecomment-1793942486 @yabola @nastra PTAL, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-05 Thread via GitHub
stevenzwu commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382661567 ## core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-05 Thread via GitHub
stevenzwu commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1382659616 ## api/src/main/java/org/apache/iceberg/Scan.java: ## @@ -77,6 +78,21 @@ public interface Scan> { */ ThisT includeColumnStats(); + /** + * Create a new sc

Re: [PR] added contributing.md file [iceberg-python]

2023-11-05 Thread via GitHub
onemriganka commented on PR #102: URL: https://github.com/apache/iceberg-python/pull/102#issuecomment-1793819378 OK sir, if you think the website is more helpful then ok... Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Bump version to 0.6.0 [iceberg-python]

2023-11-05 Thread via GitHub
Fokko commented on PR #72: URL: https://github.com/apache/iceberg-python/pull/72#issuecomment-1793816199 👍 Thanks for the review @rdblue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Bump version to 0.6.0 [iceberg-python]

2023-11-05 Thread via GitHub
Fokko merged PR #72: URL: https://github.com/apache/iceberg-python/pull/72 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] added contributing.md file [iceberg-python]

2023-11-05 Thread via GitHub
Fokko commented on PR #102: URL: https://github.com/apache/iceberg-python/pull/102#issuecomment-1793815821 But what are your thoughts on linking from the `CONTRIBUTING.md` to the website? Otherwise, it is abound to get out of sync. -- This is an automated message from the Apache Git Servi

Re: [PR] Consider moving to ParallelIterable in Deletes::toPositionIndex [iceberg]

2023-11-05 Thread via GitHub
rdblue closed pull request #6432: Consider moving to ParallelIterable in Deletes::toPositionIndex URL: https://github.com/apache/iceberg/pull/6432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Consider moving to ParallelIterable in Deletes::toPositionIndex [iceberg]

2023-11-05 Thread via GitHub
rdblue commented on PR #6432: URL: https://github.com/apache/iceberg/pull/6432#issuecomment-1793809960 #8805 was merged so I'll close this. I should also note that @aokolnychyi raised some concerns about this approach instead of a more comprehensive fix. This is probably a good start if we

Re: [PR] Support of before and after actions in preorderschema traversal [iceberg-python]

2023-11-05 Thread via GitHub
Fokko commented on PR #42: URL: https://github.com/apache/iceberg-python/pull/42#issuecomment-1793809220 This was suggested here: https://github.com/apache/iceberg/pull/7831/files#r1285259053 I'll leave it up to @rdblue to decide if he thinks this is valuable. -- This is an automated mes

Re: [PR] Bump version to 0.6.0 [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on PR #72: URL: https://github.com/apache/iceberg-python/pull/72#issuecomment-1793809215 Looks good to me. Merge when you're ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Replace black by Ruff Formatter [iceberg-python]

2023-11-05 Thread via GitHub
Fokko commented on code in PR #127: URL: https://github.com/apache/iceberg-python/pull/127#discussion_r1382622265 ## .pre-commit-config.yaml: ## @@ -29,15 +29,11 @@ repos: - id: check-ast - repo: https://github.com/astral-sh/ruff-pre-commit # Ruff version (Used fo

Re: [PR] Update pre-commit [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on PR #85: URL: https://github.com/apache/iceberg-python/pull/85#issuecomment-1793808567 Thanks, @Fokko! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Update pre-commit [iceberg-python]

2023-11-05 Thread via GitHub
rdblue merged PR #85: URL: https://github.com/apache/iceberg-python/pull/85 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Add flake8-pie to ruff [iceberg-python]

2023-11-05 Thread via GitHub
rdblue merged PR #86: URL: https://github.com/apache/iceberg-python/pull/86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382621381 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382621288 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382621051 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382620882 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382620705 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382620743 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382620579 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382620579 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382620295 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382619653 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382619528 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382619372 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382619167 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1382618622 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Support of before and after actions in preorderschema traversal [iceberg-python]

2023-11-05 Thread via GitHub
MehulBatra commented on PR #42: URL: https://github.com/apache/iceberg-python/pull/42#issuecomment-1793802279 > Maybe it's me, but I don't understand the value of adding before and after callbacks to this visitor. A node's children are traversed when the future is called and that allows you

Re: [PR] Support of before and after actions in preorderschema traversal [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on PR #42: URL: https://github.com/apache/iceberg-python/pull/42#issuecomment-1793797992 Maybe it's me, but I don't understand the value of adding before and after callbacks to this visitor. A node's children are traversed when the future is called and that allows you to do

Re: [PR] Replace black by Ruff Formatter [iceberg-python]

2023-11-05 Thread via GitHub
rdblue commented on PR #127: URL: https://github.com/apache/iceberg-python/pull/127#issuecomment-1793797091 Looks fine overall, but it seems like too many changes with string normalization. Why force string normalization? That's going to cause a ton of pull requests to fail formatting valid

Re: [I] Substitue in memory data struct's timestamp type for DataTime rather i64 to simplify usage. [iceberg-rust]

2023-11-05 Thread via GitHub
liurenjie1024 commented on issue #90: URL: https://github.com/apache/iceberg-rust/issues/90#issuecomment-1793783924 > I‘d like to have a try. Sure, welcome to contribute! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Substitue in memory data struct's timestamp type for DataTime rather i64 to simplify usage. [iceberg-rust]

2023-11-05 Thread via GitHub
my-vegetable-has-exploded commented on issue #90: URL: https://github.com/apache/iceberg-rust/issues/90#issuecomment-1793775352 I‘d like to have a try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.22.0 [iceberg-go]

2023-11-05 Thread via GitHub
dependabot[bot] opened a new pull request, #30: URL: https://github.com/apache/iceberg-go/pull/30 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.19.1 to 1.22.0. Commits https://github.com/aws/aws-sdk-go-v2/commit/61039fea9cc9e080c53382

[PR] build(deps): bump github.com/wolfeidau/s3iofs from 1.3.0 to 1.3.1 [iceberg-go]

2023-11-05 Thread via GitHub
dependabot[bot] opened a new pull request, #31: URL: https://github.com/apache/iceberg-go/pull/31 Bumps [github.com/wolfeidau/s3iofs](https://github.com/wolfeidau/s3iofs) from 1.3.0 to 1.3.1. Release notes Sourced from https://github.com/wolfeidau/s3iofs/releases";>github.com/wolfe

[PR] build(deps): bump github.com/google/uuid from 1.3.1 to 1.4.0 [iceberg-go]

2023-11-05 Thread via GitHub
dependabot[bot] opened a new pull request, #29: URL: https://github.com/apache/iceberg-go/pull/29 Bumps [github.com/google/uuid](https://github.com/google/uuid) from 1.3.1 to 1.4.0. Release notes Sourced from https://github.com/google/uuid/releases";>github.com/google/uuid's relea

[PR] build(deps): bump github.com/hamba/avro/v2 from 2.16.0 to 2.17.1 [iceberg-go]

2023-11-05 Thread via GitHub
dependabot[bot] opened a new pull request, #28: URL: https://github.com/apache/iceberg-go/pull/28 Bumps [github.com/hamba/avro/v2](https://github.com/hamba/avro) from 2.16.0 to 2.17.1. Release notes Sourced from https://github.com/hamba/avro/releases";>github.com/hamba/avro/v2's r

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.22.1 [iceberg-go]

2023-11-05 Thread via GitHub
dependabot[bot] opened a new pull request, #27: URL: https://github.com/apache/iceberg-go/pull/27 Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.21.2 to 1.22.1. Commits https://github.com/aws/aws-sdk-go-v2/commit/ee5e3f05637540596cc7aab1359742