Re: [PR] Add base headers in properties to signer_headers [iceberg-python]

2025-02-09 Thread via GitHub
Fokko commented on PR #1610: URL: https://github.com/apache/iceberg-python/pull/1610#issuecomment-2647193310 @tom-s-powell Could you run the linter using `make lint`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Update LICENSE/NOTICE files with last dependency updates [iceberg]

2025-02-09 Thread via GitHub
nastra merged PR #12214: URL: https://github.com/apache/iceberg/pull/12214 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Explore potential issue with `scan` returning the incorrect results [iceberg-python]

2025-02-09 Thread via GitHub
dongsupkim-onepredict commented on issue #1506: URL: https://github.com/apache/iceberg-python/issues/1506#issuecomment-2647162855 The result I found is that the number of data entries differs when the data count is not 100,000, compared to when the data entries count is 100,000. After logg

[PR] Update LICENSE/NOTICE files with last dependency updates [iceberg]

2025-02-09 Thread via GitHub
jbonofre opened a new pull request, #12214: URL: https://github.com/apache/iceberg/pull/12214 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.4.1 to 5.4.2 [iceberg]

2025-02-09 Thread via GitHub
jbonofre commented on PR #12209: URL: https://github.com/apache/iceberg/pull/12209#issuecomment-2647127584 > @jbonofre I'm going to merge this now Thanks. I think in the future, if a dependency update needs LICENSE/NOTICE change, it could be a commit on the dependabot PR. -

Re: [PR] Doc: update note about notebook in spark-quickstart [iceberg]

2025-02-09 Thread via GitHub
xxchan commented on PR #11996: URL: https://github.com/apache/iceberg/pull/11996#issuecomment-2647126070 Hi @Fokko, can you take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] fix: allow nullable field of equality delete writer [iceberg-rust]

2025-02-09 Thread via GitHub
Xuanwo merged PR #834: URL: https://github.com/apache/iceberg-rust/pull/834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Docs: Fix expire_snapshots output [iceberg]

2025-02-09 Thread via GitHub
nastra merged PR #12213: URL: https://github.com/apache/iceberg/pull/12213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.4.1 to 5.4.2 [iceberg]

2025-02-09 Thread via GitHub
nastra merged PR #12209: URL: https://github.com/apache/iceberg/pull/12209 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.4.1 to 5.4.2 [iceberg]

2025-02-09 Thread via GitHub
nastra commented on PR #12209: URL: https://github.com/apache/iceberg/pull/12209#issuecomment-2647075614 @jbonofre I'm going to merge this now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.48.0.0 to 3.49.0.0 [iceberg]

2025-02-09 Thread via GitHub
nastra merged PR #12206: URL: https://github.com/apache/iceberg/pull/12206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.30.11 to 2.30.16 [iceberg]

2025-02-09 Thread via GitHub
nastra merged PR #12208: URL: https://github.com/apache/iceberg/pull/12208 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.53.0 to 26.54.0 [iceberg]

2025-02-09 Thread via GitHub
nastra merged PR #12207: URL: https://github.com/apache/iceberg/pull/12207 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.30.11 to 2.30.16 [iceberg]

2025-02-09 Thread via GitHub
nastra commented on PR #12208: URL: https://github.com/apache/iceberg/pull/12208#issuecomment-2647074293 @jbonofre I'm going to merge this now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Spark: Support singular form of years, months, days, and hours functions [iceberg]

2025-02-09 Thread via GitHub
wypoon commented on PR #12117: URL: https://github.com/apache/iceberg/pull/12117#issuecomment-2647071222 @Fokko ping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-09 Thread via GitHub
jonathanc-n commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2647011594 Yes, i will put this out for review after I add some schema validation. should be put out tomorrow -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-09 Thread via GitHub
ajantha-bhat commented on PR #11216: URL: https://github.com/apache/iceberg/pull/11216#issuecomment-2646947773 I was travelling for past four days. Will be taking a look at the new comments tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Core: Fix move/update/makeRequire/makeOptional fail after rename schema (#10830) [iceberg]

2025-02-09 Thread via GitHub
imtzer commented on code in PR #12202: URL: https://github.com/apache/iceberg/pull/12202#discussion_r1948391338 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -240,7 +240,10 @@ public UpdateSchema makeColumnOptional(String name) { } private void intern

Re: [PR] fix: allow nullable field of equality delete writer [iceberg-rust]

2025-02-09 Thread via GitHub
ZENOTME commented on PR #834: URL: https://github.com/apache/iceberg-rust/pull/834#issuecomment-2646911048 Hi @liurenjie1024 I think this PR is ready to go -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-09 Thread via GitHub
ZENOTME commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2646905246 > > @liurenjie1024 I believe FastAppend is for directly appending `DataFiles` to the snapshot, whereas this pr takes existing parquet file paths, parses the Parquet metadata and conver

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-09 Thread via GitHub
liurenjie1024 commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2646853950 > @liurenjie1024 I believe FastAppend is for directly appending `DataFiles` to the snapshot, whereas this pr takes existing parquet file paths, parses the Parquet metadata and co

Re: [I] Enhance iceberg-go to Support Nessie API for All Catalog Operations [iceberg-go]

2025-02-09 Thread via GitHub
kevinjqliu commented on issue #291: URL: https://github.com/apache/iceberg-go/issues/291#issuecomment-2646849537 hey @zeroshade thats a great question. Based on my understanding, Nessie has 2 sets of APIs, its own catalog API and an Iceberg REST compliant API. Heres the nessie docs on ic

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-09 Thread via GitHub
jonathanc-n commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2646843193 @liurenjie1024 I believe FastAppend is for directly appending `DataFiles` to the snapshot, whereas this pr takes existing parquet file paths, parses the Parquet metadata and conver

Re: [PR] Data: Handle case where partition location is missing for `TableMigrationUtil` [iceberg]

2025-02-09 Thread via GitHub
manuzhang commented on PR #12212: URL: https://github.com/apache/iceberg/pull/12212#issuecomment-2646843754 @jshmchenxi Thanks for the fix. Can you add a test? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[PR] Docs: Fix expire_snapshots output [iceberg]

2025-02-09 Thread via GitHub
manuzhang opened a new pull request, #12213: URL: https://github.com/apache/iceberg/pull/12213 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Core: Fix move/update/makeRequire/makeOptional fail after rename schema (#10830) [iceberg]

2025-02-09 Thread via GitHub
manuzhang commented on code in PR #12202: URL: https://github.com/apache/iceberg/pull/12202#discussion_r1948330892 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -240,7 +240,10 @@ public UpdateSchema makeColumnOptional(String name) { } private void int

Re: [PR] chore(deps): Bump opendal from 0.51.1 to 0.51.2 [iceberg-rust]

2025-02-09 Thread via GitHub
liurenjie1024 merged PR #958: URL: https://github.com/apache/iceberg-rust/pull/958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] chore(deps): Bump bytes from 1.9.0 to 1.10.0 [iceberg-rust]

2025-02-09 Thread via GitHub
dependabot[bot] commented on PR #955: URL: https://github.com/apache/iceberg-rust/pull/955#issuecomment-2646801869 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] chore(deps): Bump bytes from 1.9.0 to 1.10.0 [iceberg-rust]

2025-02-09 Thread via GitHub
liurenjie1024 closed pull request #955: chore(deps): Bump bytes from 1.9.0 to 1.10.0 URL: https://github.com/apache/iceberg-rust/pull/955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Fix move/update/makeRequire/makeOptional fail after rename schema (#10830) [iceberg]

2025-02-09 Thread via GitHub
manuzhang commented on code in PR #12202: URL: https://github.com/apache/iceberg/pull/12202#discussion_r1948330892 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -240,7 +240,10 @@ public UpdateSchema makeColumnOptional(String name) { } private void int

Re: [PR] chore(deps): Bump uuid from 1.12.1 to 1.13.1 [iceberg-rust]

2025-02-09 Thread via GitHub
liurenjie1024 merged PR #957: URL: https://github.com/apache/iceberg-rust/pull/957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] chore(deps): Bump once_cell from 1.20.2 to 1.20.3 [iceberg-rust]

2025-02-09 Thread via GitHub
liurenjie1024 merged PR #954: URL: https://github.com/apache/iceberg-rust/pull/954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] chore(deps): Bump aws-sdk-s3tables from 1.8.0 to 1.9.0 [iceberg-rust]

2025-02-09 Thread via GitHub
liurenjie1024 merged PR #956: URL: https://github.com/apache/iceberg-rust/pull/956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Enhance iceberg-go to Support Nessie API for All Catalog Operations [iceberg-go]

2025-02-09 Thread via GitHub
zeroshade commented on issue #291: URL: https://github.com/apache/iceberg-go/issues/291#issuecomment-2646688804 @Fokko @kevinjqliu can you two chime in here? My personal opinion is that we were following the REST spec, so if Nessie isn't following the REST spec, we should do a separate cata

Re: [I] What should be Pyarrow's default buffer size? [iceberg-python]

2025-02-09 Thread via GitHub
github-actions[bot] commented on issue #1058: URL: https://github.com/apache/iceberg-python/issues/1058#issuecomment-2646676048 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Clean up TODOs in the codebase [iceberg-python]

2025-02-09 Thread via GitHub
github-actions[bot] commented on issue #1052: URL: https://github.com/apache/iceberg-python/issues/1052#issuecomment-2646676054 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Contribution Proposal: Kafka to Iceberg Tutorial Repo [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] commented on issue #10933: URL: https://github.com/apache/iceberg/issues/10933#issuecomment-2646674122 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Use griffe to find breaking changes [iceberg-python]

2025-02-09 Thread via GitHub
github-actions[bot] commented on issue #334: URL: https://github.com/apache/iceberg-python/issues/334#issuecomment-2646676094 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [PR] Doc:Hive 4.0 and later versions allow vectorized read and write opera… [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] commented on PR #11877: URL: https://github.com/apache/iceberg/pull/11877#issuecomment-2646674215 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Spark 3.5: Refactor scanning changelog table with timestamps [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] commented on PR #11612: URL: https://github.com/apache/iceberg/pull/11612#issuecomment-2646674179 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Introduce `MissingRequiredFilesToDeleteException` for Streaming Deletes [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] commented on PR #11887: URL: https://github.com/apache/iceberg/pull/11887#issuecomment-2646674229 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Spec: The spec about metadata key `schema-id` in manifest file do not match the lib implementation [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] commented on issue #10927: URL: https://github.com/apache/iceberg/issues/10927#issuecomment-2646674110 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Doc:Hive 4.0 and later versions allow vectorized read and write opera… [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] closed pull request #11877: Doc:Hive 4.0 and later versions allow vectorized read and write opera… URL: https://github.com/apache/iceberg/pull/11877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Introduce `MissingRequiredFilesToDeleteException` for Streaming Deletes [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] closed pull request #11887: Introduce `MissingRequiredFilesToDeleteException` for Streaming Deletes URL: https://github.com/apache/iceberg/pull/11887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Spark 3.5: Refactor scanning changelog table with timestamps [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] closed pull request #11612: Spark 3.5: Refactor scanning changelog table with timestamps URL: https://github.com/apache/iceberg/pull/11612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Core: Map methods should return immutable collections [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] commented on PR #11304: URL: https://github.com/apache/iceberg/pull/11304#issuecomment-2646674141 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Is dataFiles() Method Retryable? [iceberg]

2025-02-09 Thread via GitHub
github-actions[bot] commented on issue #10750: URL: https://github.com/apache/iceberg/issues/10750#issuecomment-2646674075 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

[PR] 61- Add commands for iceberg CLI. [iceberg-go]

2025-02-09 Thread via GitHub
subkanthi opened a new pull request, #298: URL: https://github.com/apache/iceberg-go/pull/298 closes: #61 Added based on this https://github.com/docopt/docopt#help-message-format ``` iceberg -h Usage: iceberg list [options] [PARENT] iceberg describe [options

Re: [I] [feat] add missing metadata tables [iceberg-python]

2025-02-09 Thread via GitHub
soumya-ghosh commented on issue #1053: URL: https://github.com/apache/iceberg-python/issues/1053#issuecomment-2646624859 I understand that `files` table by snapshot and `all_files` (and its derivatives) should respect schema evolution. The keys in column of `readable_metrics` is derived

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distributions [iceberg]

2025-02-09 Thread via GitHub
amogh-jahagirdar merged PR #12195: URL: https://github.com/apache/iceberg/pull/12195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distributions [iceberg]

2025-02-09 Thread via GitHub
amogh-jahagirdar commented on PR #12195: URL: https://github.com/apache/iceberg/pull/12195#issuecomment-2646608558 Thank you @jbonofre, I'll go ahead and merge, thanks @Fokko @bryanck for reviewing and providing context! -- This is an automated message from the Apache Git Service. To res

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distributions [iceberg]

2025-02-09 Thread via GitHub
amogh-jahagirdar commented on PR #12195: URL: https://github.com/apache/iceberg/pull/12195#issuecomment-2646608262 > > Big picture question I have though is if it makes sense for Iceberg to even be releasing a separate Kafka Connect distribution with HMS client. Since we're close to a relea

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on code in PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#discussion_r1948240540 ## pyiceberg/io/pyarrow.py: ## @@ -1359,14 +1359,11 @@ def _task_to_record_batches( # Apply the user filter if pyarrow_fil

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on code in PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#discussion_r1948240540 ## pyiceberg/io/pyarrow.py: ## @@ -1359,14 +1359,11 @@ def _task_to_record_batches( # Apply the user filter if pyarrow_fil

[PR] feat: Add existing parquet files [iceberg-rust]

2025-02-09 Thread via GitHub
jonathanc-n opened a new pull request, #960: URL: https://github.com/apache/iceberg-rust/pull/960 Completes #932 Allows for adding existing parquet files by using parquet metadata to create `DataFiles`. Then fast appending them using existing api -- This is an automated message fr

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-09 Thread via GitHub
gabeiglio commented on code in PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#discussion_r1948234324 ## pyiceberg/io/pyarrow.py: ## @@ -1359,14 +1359,11 @@ def _task_to_record_batches( # Apply the user filter if pyarrow_filt

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-09 Thread via GitHub
rdblue commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1948233800 ## core/src/main/java/org/apache/iceberg/schema/UnionByNameVisitor.java: ## @@ -157,7 +157,9 @@ private Type findFieldType(int fieldId) { private void addColumn(i

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-09 Thread via GitHub
rdblue commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1948233450 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -293,23 +282,15 @@ public UpdateSchema updateColumn(String name, Type.PrimitiveType newType) {

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-09 Thread via GitHub
rdblue commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1948233225 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -94,40 +95,24 @@ public SchemaUpdate allowIncompatibleChanges() { } @Override - public Updat

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-09 Thread via GitHub
rdblue commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1948233106 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -55,7 +56,7 @@ class SchemaUpdate implements UpdateSchema { private final Map idToParent; priva

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-09 Thread via GitHub
rdblue commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1948232917 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -559,15 +559,22 @@ public static Builder optional(String name) { return new Builder(true, name);

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on code in PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#discussion_r1948232906 ## pyiceberg/io/pyarrow.py: ## @@ -1359,14 +1359,11 @@ def _task_to_record_batches( # Apply the user filter if pyarrow_fil

[PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-09 Thread via GitHub
rdblue opened a new pull request, #12211: URL: https://github.com/apache/iceberg/pull/12211 This adds default value support to `UpdateSchema` and the implementation of it in core. The API has new methods that accept a default value: * `addColumn(String name, Type type, String doc,

Re: [PR] Remove old metadata [iceberg-python]

2025-02-09 Thread via GitHub
kaushiksrini commented on code in PR #1607: URL: https://github.com/apache/iceberg-python/pull/1607#discussion_r1948228781 ## pyiceberg/table/__init__.py: ## @@ -1179,6 +1182,11 @@ def refs(self) -> Dict[str, SnapshotRef]: def _do_commit(self, updates: Tuple[TableUpdate,

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-09 Thread via GitHub
gabeiglio commented on code in PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#discussion_r1948224048 ## pyiceberg/io/pyarrow.py: ## @@ -1359,14 +1359,11 @@ def _task_to_record_batches( # Apply the user filter if pyarrow_filt

Re: [I] [bug] Transaction new requirements handling [iceberg-python]

2025-02-09 Thread via GitHub
Fokko closed issue #1628: [bug] Transaction new requirements handling URL: https://github.com/apache/iceberg-python/issues/1628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] [bug] Transaction new requirements handling [iceberg-python]

2025-02-09 Thread via GitHub
Fokko closed issue #1628: [bug] Transaction new requirements handling URL: https://github.com/apache/iceberg-python/issues/1628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] fix new requirements handling [iceberg-python]

2025-02-09 Thread via GitHub
Fokko merged PR #1631: URL: https://github.com/apache/iceberg-python/pull/1631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] fix new requirements handling [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on code in PR #1631: URL: https://github.com/apache/iceberg-python/pull/1631#discussion_r1948165373 ## pyiceberg/table/__init__.py: ## @@ -269,7 +269,7 @@ def _apply(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequ existing_req

Re: [PR] Fix: `SqlCatalog` list_namespaces() should return only sub-namespaces [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on code in PR #1629: URL: https://github.com/apache/iceberg-python/pull/1629#discussion_r1948160074 ## pyiceberg/catalog/sql.py: ## @@ -610,15 +610,26 @@ def list_namespaces(self, namespace: Union[str, Identifier] = ()) -> List[Identi table_stmt =

Re: [PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on code in PR #1621: URL: https://github.com/apache/iceberg-python/pull/1621#discussion_r1948157569 ## pyiceberg/io/pyarrow.py: ## @@ -1359,14 +1359,11 @@ def _task_to_record_batches( # Apply the user filter if pyarrow_fil

Re: [PR] chore: fix Cargo.lock diff always present after `cargo build` [iceberg-rust]

2025-02-09 Thread via GitHub
Xuanwo merged PR #952: URL: https://github.com/apache/iceberg-rust/pull/952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Remove old metadata [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on code in PR #1607: URL: https://github.com/apache/iceberg-python/pull/1607#discussion_r1948155351 ## pyiceberg/table/__init__.py: ## @@ -1179,6 +1182,11 @@ def refs(self) -> Dict[str, SnapshotRef]: def _do_commit(self, updates: Tuple[TableUpdate, ..

Re: [PR] fix new requirements handling [iceberg-python]

2025-02-09 Thread via GitHub
ZENOTME commented on PR #1631: URL: https://github.com/apache/iceberg-python/pull/1631#issuecomment-2646424688 cc @kevinjqliu @Fokko @HonahX -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] fix new requirements handling [iceberg-python]

2025-02-09 Thread via GitHub
ZENOTME opened a new pull request, #1631: URL: https://github.com/apache/iceberg-python/pull/1631 Resolve #1628. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [I] [bug] Transaction new requirements handling [iceberg-python]

2025-02-09 Thread via GitHub
ZENOTME commented on issue #1628: URL: https://github.com/apache/iceberg-python/issues/1628#issuecomment-2646424250 > [@ZENOTME](https://github.com/ZENOTME) would you like to contribute this change? Sure, I will send the fix pr later -- This is an automated message from the Apache

Re: [PR] Remove old metadata [iceberg-python]

2025-02-09 Thread via GitHub
kaushiksrini commented on PR #1607: URL: https://github.com/apache/iceberg-python/pull/1607#issuecomment-2646423584 @kevinjqliu thanks for the review! I moved the logic after the table is committed -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] [bug] Transaction new requirements handling [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on issue #1628: URL: https://github.com/apache/iceberg-python/issues/1628#issuecomment-2646413186 @ZENOTME would you like to contribute this change? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[PR] Add PuffinWriter [iceberg-rust]

2025-02-09 Thread via GitHub
fqaiser94 opened a new pull request, #959: URL: https://github.com/apache/iceberg-rust/pull/959 Part of https://github.com/apache/iceberg-rust/issues/744 # Summary - Add PuffinWriter # Context - This is the fifth of a number of PRs to add support for Iceberg Puffin file

Re: [I] `py-io-impl` config propagation [iceberg-python]

2025-02-09 Thread via GitHub
kevinjqliu commented on issue #1589: URL: https://github.com/apache/iceberg-python/issues/1589#issuecomment-2646412685 > Obviously, when configuring the client, I'll need to use 3 different URLs depending on these 3 conditions, but the server will always return the same URL in the /config

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distributions [iceberg]

2025-02-09 Thread via GitHub
bryanck commented on PR #12195: URL: https://github.com/apache/iceberg/pull/12195#issuecomment-2646385638 > Big picture question I have though is if it makes sense for Iceberg to even be releasing a separate Kafka Connect distribution with HMS client. Since we're close to a release, and I a

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distributions [iceberg]

2025-02-09 Thread via GitHub
jbonofre commented on code in PR #12195: URL: https://github.com/apache/iceberg/pull/12195#discussion_r1948141074 ## kafka-connect/kafka-connect-runtime/main/LICENSE: ## @@ -0,0 +1,1442 @@ + + Apache License + Version 2.0

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distributions [iceberg]

2025-02-09 Thread via GitHub
jbonofre commented on code in PR #12195: URL: https://github.com/apache/iceberg/pull/12195#discussion_r1948140979 ## kafka-connect/kafka-connect-runtime/main/LICENSE: ## @@ -0,0 +1,1442 @@ + + Apache License + Version 2.0

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.30.11 to 2.30.16 [iceberg]

2025-02-09 Thread via GitHub
jbonofre commented on PR #12208: URL: https://github.com/apache/iceberg/pull/12208#issuecomment-2646360653 Please let me know when this is merged: I will update `LICENSE` and `NOTICE` accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.4.1 to 5.4.2 [iceberg]

2025-02-09 Thread via GitHub
jbonofre commented on PR #12209: URL: https://github.com/apache/iceberg/pull/12209#issuecomment-2646360341 Please ping me if this is merged before 1.8.0: I will update `LICENSE` and `NOTICE` accordingly. -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [I] Generation of Snapshot Summaries [iceberg-rust]

2025-02-09 Thread via GitHub
barronw commented on issue #724: URL: https://github.com/apache/iceberg-rust/issues/724#issuecomment-2646357548 @c-thiel Feel free to re-assign this to someone else (maybe @jonathanc-n). I won't be able to complete this soon, so I pushed what I have to my fork for reference ([link](https:/

[PR] chore(deps): Bump opendal from 0.51.1 to 0.51.2 [iceberg-rust]

2025-02-09 Thread via GitHub
dependabot[bot] opened a new pull request, #958: URL: https://github.com/apache/iceberg-rust/pull/958 Bumps [opendal](https://github.com/apache/opendal) from 0.51.1 to 0.51.2. Release notes Sourced from https://github.com/apache/opendal/releases";>opendal's releases. v0.51.2

[PR] chore(deps): Bump uuid from 1.12.1 to 1.13.1 [iceberg-rust]

2025-02-09 Thread via GitHub
dependabot[bot] opened a new pull request, #957: URL: https://github.com/apache/iceberg-rust/pull/957 Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.12.1 to 1.13.1. Release notes Sourced from https://github.com/uuid-rs/uuid/releases";>uuid's releases. 1.13.1 What's C

[PR] chore(deps): Bump aws-sdk-s3tables from 1.8.0 to 1.9.0 [iceberg-rust]

2025-02-09 Thread via GitHub
dependabot[bot] opened a new pull request, #956: URL: https://github.com/apache/iceberg-rust/pull/956 Bumps [aws-sdk-s3tables](https://github.com/awslabs/aws-sdk-rust) from 1.8.0 to 1.9.0. Commits See full diff in https://github.com/awslabs/aws-sdk-rust/commits";>compare view

[PR] chore(deps): Bump bytes from 1.9.0 to 1.10.0 [iceberg-rust]

2025-02-09 Thread via GitHub
dependabot[bot] opened a new pull request, #955: URL: https://github.com/apache/iceberg-rust/pull/955 Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.9.0 to 1.10.0. Release notes Sourced from https://github.com/tokio-rs/bytes/releases";>bytes's releases. Bytes v1.10.0

[PR] chore(deps): Bump once_cell from 1.20.2 to 1.20.3 [iceberg-rust]

2025-02-09 Thread via GitHub
dependabot[bot] opened a new pull request, #954: URL: https://github.com/apache/iceberg-rust/pull/954 Bumps [once_cell](https://github.com/matklad/once_cell) from 1.20.2 to 1.20.3. Changelog Sourced from https://github.com/matklad/once_cell/blob/master/CHANGELOG.md";>once_cell's c

Re: [I] Integrate CRT with Iceberg S3 client [iceberg]

2025-02-09 Thread via GitHub
guizmaii commented on issue #6739: URL: https://github.com/apache/iceberg/issues/6739#issuecomment-2646183453 Hi everyone, What's the status of this issue, please? Can we re-open the issue maybe? It seems to be quite an interesting improvement -- This is an automated message fro

Re: [I] Integrate CRT with Iceberg S3 client [iceberg]

2025-02-09 Thread via GitHub
jivanic-demystdata commented on issue #6739: URL: https://github.com/apache/iceberg/issues/6739#issuecomment-2646183137 Hi everyone, What's the status of this issue, please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] feat: View Metadata Builder [iceberg-rust]

2025-02-09 Thread via GitHub
c-thiel commented on code in PR #908: URL: https://github.com/apache/iceberg-rust/pull/908#discussion_r1948048445 ## crates/iceberg/src/spec/view_metadata_builder.rs: ## @@ -0,0 +1,1579 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [I] Enhance iceberg-go to Support Nessie API for All Catalog Operations [iceberg-go]

2025-02-09 Thread via GitHub
shubham-tomar commented on issue #291: URL: https://github.com/apache/iceberg-go/issues/291#issuecomment-2646137299 I have added a [PR](https://github.com/apache/iceberg-go/pull/297) to support Nessie via catalog/rest (namespace related functionalities) -- This is an automated message

[PR] Nessie REST API Support for Namespace Operations [iceberg-go]

2025-02-09 Thread via GitHub
shubham-tomar opened a new pull request, #297: URL: https://github.com/apache/iceberg-go/pull/297 ## Overview This PR adds support for using Apache Nessie as a REST catalog by implementing the Nessie API paths for namespace operations. The changes ensure compatibility with [Nessi

Re: [I] flink1.13.2+iceberg0.13.0+hive-metastore3.0.0+minio(S3) Forbidden (Service: Amazon S3; Status Code: 403 [iceberg]

2025-02-09 Thread via GitHub
ragin0ff commented on issue #8968: URL: https://github.com/apache/iceberg/issues/8968#issuecomment-2646133177 does it have solutions ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Fix LICENSE and NOTICE for the kafka-connect-runtime distributions [iceberg]

2025-02-09 Thread via GitHub
Fokko commented on code in PR #12195: URL: https://github.com/apache/iceberg/pull/12195#discussion_r1948032347 ## kafka-connect/kafka-connect-runtime/main/LICENSE: ## @@ -0,0 +1,1442 @@ + + Apache License + Version 2.0, J

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.55 to 1.17.59 [iceberg-go]

2025-02-09 Thread via GitHub
dependabot[bot] opened a new pull request, #296: URL: https://github.com/apache/iceberg-go/pull/296 Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.17.55 to 1.17.59. Commits https://github.com/aws/aws-sdk-go-v2/commit/78fa10aa9eaaa

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.74.1 to 1.76.0 [iceberg-go]

2025-02-09 Thread via GitHub
dependabot[bot] opened a new pull request, #295: URL: https://github.com/apache/iceberg-go/pull/295 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.74.1 to 1.76.0. Commits https://github.com/aws/aws-sdk-go-v2/commit/061fd1b6d9940c6b

  1   2   >