Re: [I] Extend the e2e tests with append a partitioned file [iceberg-rust]

2024-11-27 Thread via GitHub
Fokko commented on issue #720: URL: https://github.com/apache/iceberg-rust/issues/720#issuecomment-2505460545 @feniljain Thanks, I've merged the PR. I always want to give others the chance to chime in as well. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-11-27 Thread via GitHub
Guosmilesmile commented on PR #11662: URL: https://github.com/apache/iceberg/pull/11662#issuecomment-2505426989 @pvary Because a new flag has been added, the testSerializationSize test case needs to be modified to reflect the corresponding size value + 1byte for the flag . It has been adjus

Re: [PR] Docs: Add RisingWave [iceberg]

2024-11-27 Thread via GitHub
hengm3467 commented on PR #11642: URL: https://github.com/apache/iceberg/pull/11642#issuecomment-2505396460 @Fokko No worries. And thank you so much for the review! :) Have a nice day. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Docs: Add RisingWave [iceberg]

2024-11-27 Thread via GitHub
Fokko merged PR #11642: URL: https://github.com/apache/iceberg/pull/11642 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Docs: Add RisingWave [iceberg]

2024-11-27 Thread via GitHub
hengm3467 commented on code in PR #11642: URL: https://github.com/apache/iceberg/pull/11642#discussion_r1861611438 ## docs/docs/risingwave.md: ## @@ -0,0 +1,92 @@ +--- +title: "RisingWave" +--- + + +# RisingWave + +[RisingWave](https://risingwave.com/) is a Postgres-compatible S

Re: [PR] Docs: Add guidelines for contributors to become committers [iceberg]

2024-11-27 Thread via GitHub
jbonofre commented on code in PR #11670: URL: https://github.com/apache/iceberg/pull/11670#discussion_r1861603745 ## site/docs/community.md: ## @@ -100,3 +102,56 @@ Recruitment of community members should not be conducted through direct messages related to contributing to or u

Re: [PR] Docs: Add guidelines for contributors to become committers [iceberg]

2024-11-27 Thread via GitHub
jbonofre commented on code in PR #11670: URL: https://github.com/apache/iceberg/pull/11670#discussion_r1861603745 ## site/docs/community.md: ## @@ -100,3 +102,56 @@ Recruitment of community members should not be conducted through direct messages related to contributing to or u

Re: [PR] Data: Add partition stats writer and reader [iceberg]

2024-11-27 Thread via GitHub
jbonofre commented on PR #11216: URL: https://github.com/apache/iceberg/pull/11216#issuecomment-2505384190 @RussellSpitzer @aokolnychyi I'm reviewing the stale PRs, and this one is open for month. Do we have a way to move forward ? I can do a new review, but at the end of the day, it won't

Re: [PR] Docs: Add RisingWave [iceberg]

2024-11-27 Thread via GitHub
Fokko commented on code in PR #11642: URL: https://github.com/apache/iceberg/pull/11642#discussion_r1861604454 ## docs/docs/risingwave.md: ## @@ -0,0 +1,92 @@ +--- +title: "RisingWave" +--- + + +# RisingWave + +[RisingWave](https://risingwave.com/) is a Postgres-compatible SQL d

Re: [PR] Docs: Add guidelines for contributors to become committers [iceberg]

2024-11-27 Thread via GitHub
jbonofre commented on code in PR #11670: URL: https://github.com/apache/iceberg/pull/11670#discussion_r1861603745 ## site/docs/community.md: ## @@ -100,3 +102,56 @@ Recruitment of community members should not be conducted through direct messages related to contributing to or u

Re: [PR] chore: Bump upload-artifact@v3 to v4 [iceberg-rust]

2024-11-27 Thread via GitHub
Xuanwo merged PR #725: URL: https://github.com/apache/iceberg-rust/pull/725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] Logic to determine the partitions [iceberg-rust]

2024-11-27 Thread via GitHub
Fokko commented on issue #728: URL: https://github.com/apache/iceberg-rust/issues/728#issuecomment-2505367587 > I think you mean arrow record batch? Yes :) This could be a broader discussion on where the responsibilities lie between iceberg-rust and the query engine. On

Re: [I] Support setting Table Properties [iceberg-rust]

2024-11-27 Thread via GitHub
Fokko closed issue #730: Support setting Table Properties URL: https://github.com/apache/iceberg-rust/issues/730 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Docs: Add guidelines for contributors to become committers [iceberg]

2024-11-27 Thread via GitHub
nastra commented on code in PR #11670: URL: https://github.com/apache/iceberg/pull/11670#discussion_r1861566021 ## site/docs/community.md: ## @@ -76,7 +76,9 @@ Iceberg has four mailing lists: The Apache Iceberg community is built on the principles described in the [Apache Wa

Re: [PR] Docs: Add guidelines for contributors to become committers [iceberg]

2024-11-27 Thread via GitHub
nastra commented on PR #11670: URL: https://github.com/apache/iceberg/pull/11670#issuecomment-2505348525 thanks @rdblue for writing this up. This will be very helpful for people in the Iceberg community -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-11-27 Thread via GitHub
pvary commented on code in PR #11662: URL: https://github.com/apache/iceberg/pull/11662#discussion_r1861570865 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySerializer.java: ## @@ -124,6 +124,14 @@ public void serialize(SortKey record, DataOutpu

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
jeesou commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861546270 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + * Returns th

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
jeesou commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861546270 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + * Returns th

Re: [PR] feat: Add equality delete writer [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 merged PR #703: URL: https://github.com/apache/iceberg-rust/pull/703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: support position delete writer [iceberg-rust]

2024-11-27 Thread via GitHub
ZENOTME commented on PR #704: URL: https://github.com/apache/iceberg-rust/pull/704#issuecomment-2505206247 > > > There are two kinds of writers in iceberg: > > > > > > 1. Plain position delete writer: https://github.com/apache/iceberg/blob/da2ad389fd9ba8222f6fb3f57922209c239a7045/core

Re: [PR] Docs: Add RisingWave [iceberg]

2024-11-27 Thread via GitHub
hengm3467 commented on PR #11642: URL: https://github.com/apache/iceberg/pull/11642#issuecomment-2505176722 @Fokko Could you please take a look? Thanks! And let me know if you have any questions. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] [KafkaConnect] Fix RecordConverter for UUID and Fixed Types [iceberg]

2024-11-27 Thread via GitHub
Gezi-lzq commented on PR #11346: URL: https://github.com/apache/iceberg/pull/11346#issuecomment-2505163794 When writing UUIDs, should we handle the conversion directly within BaseParquetWriter, by modifying BaseParquetWriter#primitive to check if the LogicalTypeAnnotation is a UUID and then

Re: [I] Extend the `DataFileWriterBuilder` tests [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 commented on issue #726: URL: https://github.com/apache/iceberg-rust/issues/726#issuecomment-2505163082 > @Fokko I would like to try working on this, may I be assigned this? @jonathanc-n Welcome to contribute! -- This is an automated message from the Apache Git Servic

Re: [PR] fix: expand arrow to iceberg schema to handle nanosecond timestamp [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 merged PR #710: URL: https://github.com/apache/iceberg-rust/pull/710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Arrow schema to iceberg conversion fails with nanosecond timestamps [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 closed issue #709: Arrow schema to iceberg conversion fails with nanosecond timestamps URL: https://github.com/apache/iceberg-rust/issues/709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Logic to determine the partitions [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 commented on issue #728: URL: https://github.com/apache/iceberg-rust/issues/728#issuecomment-2505151151 I think you mean arrow record batch? Could you elaborate the use case of this method? -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [I] Arrow Writer [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 commented on issue #729: URL: https://github.com/apache/iceberg-rust/issues/729#issuecomment-2505151941 Currently file writers all accept arrow record batch as input since it's easy to convert arrow to parquet file. -- This is an automated message from the Apache Git Service

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-11-27 Thread via GitHub
Guosmilesmile commented on code in PR #11662: URL: https://github.com/apache/iceberg/pull/11662#discussion_r1861359801 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySerializer.java: ## @@ -124,6 +124,14 @@ public void serialize(SortKey record, D

Re: [PR] Derive Clone for IcebergTableProvider [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 merged PR #722: URL: https://github.com/apache/iceberg-rust/pull/722 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-11-27 Thread via GitHub
Guosmilesmile commented on code in PR #11662: URL: https://github.com/apache/iceberg/pull/11662#discussion_r1861436171 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySerializer.java: ## @@ -124,6 +124,14 @@ public void serialize(SortKey record, D

Re: [PR] chore: Typo in test :) [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 merged PR #727: URL: https://github.com/apache/iceberg-rust/pull/727 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Error running data fusion queries - Physical input schema should be the same as the one converted from logical input schema [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 closed issue #655: Error running data fusion queries - Physical input schema should be the same as the one converted from logical input schema URL: https://github.com/apache/iceberg-rust/issues/655 -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] Tracking: Reading iceberg tables. [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 closed issue #123: Tracking: Reading iceberg tables. URL: https://github.com/apache/iceberg-rust/issues/123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Fix error running data fusion queries - Physical input schema should be the same as the one converted from logical input schema [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 merged PR #664: URL: https://github.com/apache/iceberg-rust/pull/664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: support position delete writer [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 commented on PR #704: URL: https://github.com/apache/iceberg-rust/pull/704#issuecomment-2505139679 > > There are two kinds of writers in iceberg: > > > > 1. Plain position delete writer: https://github.com/apache/iceberg/blob/da2ad389fd9ba8222f6fb3f57922209c239a7045/core

Re: [PR] feat: Add equality delete writer [iceberg-rust]

2024-11-27 Thread via GitHub
liurenjie1024 commented on code in PR #703: URL: https://github.com/apache/iceberg-rust/pull/703#discussion_r1861419228 ## crates/iceberg/src/writer/base_writer/equality_delete_writer.rs: ## @@ -0,0 +1,502 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [I] Extend the `DataFileWriterBuilder` tests [iceberg-rust]

2024-11-27 Thread via GitHub
jonathanc-n commented on issue #726: URL: https://github.com/apache/iceberg-rust/issues/726#issuecomment-2505119512 @Fokko I would like to try working on this, may I be assigned this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [I] Tracking issue: Writing iceberg tables [iceberg-rust]

2024-11-27 Thread via GitHub
kevinzwang commented on issue #346: URL: https://github.com/apache/iceberg-rust/issues/346#issuecomment-2505089607 Thank you for compiling this @c-thiel @ZENOTME! I'm part of the Daft team and we're eagerly looking forward to write support. The tasks enumerated by @ZENOTME match our priorit

Re: [PR] Core: Add Variant implementation to read serialized objects [iceberg]

2024-11-27 Thread via GitHub
rdblue commented on code in PR #11415: URL: https://github.com/apache/iceberg/pull/11415#discussion_r1861390948 ## core/src/main/java/org/apache/iceberg/VariantPrimitive.java: ## @@ -0,0 +1,126 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one + * * o

Re: [PR] Core: Add Variant implementation to read serialized objects [iceberg]

2024-11-27 Thread via GitHub
rdblue commented on code in PR #11415: URL: https://github.com/apache/iceberg/pull/11415#discussion_r1861391300 ## core/src/main/java/org/apache/iceberg/Variants.java: ## @@ -0,0 +1,207 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one + * * or more c

[PR] "how to release", add `gh` command to trigger workflow [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu opened a new pull request, #1387: URL: https://github.com/apache/iceberg-python/pull/1387 Add gh commands instead of trigger manually on github -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] [infra] Prevent running `python-release` action without a specified rc version [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu commented on code in PR #1386: URL: https://github.com/apache/iceberg-python/pull/1386#discussion_r1861373849 ## .github/workflows/python-release.yml: ## @@ -29,12 +29,30 @@ on: jobs: + validate_version: +name: Validate version input +runs-on: ubuntu-lat

Re: [PR] Docs: Section referring to the ASF roles, guidelines and process [iceberg]

2024-11-27 Thread via GitHub
github-actions[bot] commented on PR #10804: URL: https://github.com/apache/iceberg/pull/10804#issuecomment-2505032982 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] wip: add loadMetricsContext method [iceberg]

2024-11-27 Thread via GitHub
sullis closed pull request #11384: wip: add loadMetricsContext method URL: https://github.com/apache/iceberg/pull/11384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] wip: add loadMetricsContext method [iceberg]

2024-11-27 Thread via GitHub
sullis commented on PR #11384: URL: https://github.com/apache/iceberg/pull/11384#issuecomment-2505039518 Closing this PR. I do not have any plans to continue this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

2024-11-27 Thread via GitHub
github-actions[bot] commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2505035047 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [PR] wip: add loadMetricsContext method [iceberg]

2024-11-27 Thread via GitHub
github-actions[bot] commented on PR #11384: URL: https://github.com/apache/iceberg/pull/11384#issuecomment-2505033068 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-11-27 Thread via GitHub
Guosmilesmile commented on code in PR #11662: URL: https://github.com/apache/iceberg/pull/11662#discussion_r1861359801 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySerializer.java: ## @@ -124,6 +124,14 @@ public void serialize(SortKey record, D

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861360140 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + *

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861360140 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + *

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2024-11-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #11040: URL: https://github.com/apache/iceberg/pull/11040#discussion_r1861360067 ## api/src/main/java/org/apache/iceberg/Table.java: ## @@ -373,4 +374,14 @@ default Snapshot snapshot(String name) { return null; } + + /** + *

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-11-27 Thread via GitHub
Guosmilesmile commented on code in PR #11662: URL: https://github.com/apache/iceberg/pull/11662#discussion_r1861359801 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySerializer.java: ## @@ -124,6 +124,14 @@ public void serialize(SortKey record, D

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861354496 ## core/src/test/java/org/apache/iceberg/rest/requests/TestPlanTableScanRequestParser.java: ## @@ -0,0 +1,151 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861350352 ## core/src/test/java/org/apache/iceberg/rest/responses/TestPlanTableScanResponseParser.java: ## @@ -0,0 +1,244 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861347016 ## core/src/test/java/org/apache/iceberg/rest/responses/TestFetchScanTasksResponseParser.java: ## @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861347257 ## core/src/test/java/org/apache/iceberg/rest/responses/TestFetchScanTasksResponseParser.java: ## @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861334737 ## core/src/test/java/org/apache/iceberg/rest/responses/TestFetchScanTasksResponseParser.java: ## @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-11-27 Thread via GitHub
rdblue commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1861332491 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -924,6 +934,7 @@ public List apply(TableMetadata base, Snapshot snapshot) {

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-11-27 Thread via GitHub
stevenzwu commented on code in PR #11662: URL: https://github.com/apache/iceberg/pull/11662#discussion_r1861329534 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySerializer.java: ## @@ -124,6 +124,14 @@ public void serialize(SortKey record, DataO

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-11-27 Thread via GitHub
rdblue commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1861325992 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) { fil

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861310725 ## api/src/main/java/org/apache/iceberg/exceptions/EntityNotFoundException.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861314221 ## core/src/test/java/org/apache/iceberg/rest/responses/TestFetchScanTasksResponseParser.java: ## @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861313142 ## core/src/test/java/org/apache/iceberg/rest/responses/TestFetchScanTasksResponseParser.java: ## @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] chore: Bump upload-artifact@v3 to v4 [iceberg-rust]

2024-11-27 Thread via GitHub
sungwy commented on PR #725: URL: https://github.com/apache/iceberg-rust/pull/725#issuecomment-2504805615 > thanks for adding this! @kevinjqliu Thanks for the review! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861311628 ## core/src/main/java/org/apache/iceberg/rest/requests/PlanTableScanRequestParser.java: ## @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Spark-3.5: make `where` sql case sensitive setting alterable in rewrite data files procedure [iceberg]

2024-11-27 Thread via GitHub
huaxingao commented on code in PR #11439: URL: https://github.com/apache/iceberg/pull/11439#discussion_r1861308936 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -102,11 +102,13 @@ public class RewriteDataFilesSparkActi

Re: [PR] Materialized View Spec [iceberg]

2024-11-27 Thread via GitHub
stevenzwu commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1861267203 ## format/view-spec.md: ## @@ -82,9 +98,13 @@ Each version in `versions` is a struct with the following fields: | _required_ | `representations` | A list of [

Re: [PR] Materialized View Spec [iceberg]

2024-11-27 Thread via GitHub
stevenzwu commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1861263489 ## format/view-spec.md: ## @@ -158,6 +174,57 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when

Re: [PR] Flink: Fix range distribution npe when value is null [iceberg]

2024-11-27 Thread via GitHub
Guosmilesmile commented on PR #11662: URL: https://github.com/apache/iceberg/pull/11662#issuecomment-2504787892 @pvary I'm sorry, I just re-uploaded the missing parts. Please help me trigger the test again. I apologize for the inconvenience. -- This is an automated message from the Apache

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861239915 ## core/src/test/java/org/apache/iceberg/rest/responses/TestFetchScanTasksResponseParser.java: ## @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Snapshot: Make manifest-list required [iceberg-python]

2024-11-27 Thread via GitHub
Fokko commented on PR #1385: URL: https://github.com/apache/iceberg-python/pull/1385#issuecomment-2504747642 Then we should read the `manifests` field, but that has never been implemented on PyIceberg. No engine is really using that, and I think we should deprecate it. By making the manifes

Re: [PR] [release] Pyiceberg 0.8.1 [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu merged PR #1384: URL: https://github.com/apache/iceberg-python/pull/1384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Core: Support aggregated basic stats in partition summary [iceberg]

2024-11-27 Thread via GitHub
pvary commented on PR #11669: URL: https://github.com/apache/iceberg/pull/11669#issuecomment-2504732771 This discussion could be relevant here too: https://lists.apache.org/thread/0q1csnkfg8jc11zo1dlssjkr4v8s8zz0 -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Core: Support aggregated basic stats in partition summary [iceberg]

2024-11-27 Thread via GitHub
pvary commented on PR #11669: URL: https://github.com/apache/iceberg/pull/11669#issuecomment-2504724953 @deniskuzZ: Could you please provide a short description what data is stored in the summary and in what format? I think it is important to understand the cost for keeping this stat

Re: [PR] Backport Avro planned reader (and corresponding tests) on Flink v1.18 and v1.19 [iceberg]

2024-11-27 Thread via GitHub
pvary merged PR #11668: URL: https://github.com/apache/iceberg/pull/11668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861183796 ## core/src/test/java/org/apache/iceberg/rest/responses/TestFetchScanTasksResponseParser.java: ## @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861183629 ## core/src/main/java/org/apache/iceberg/RESTFileScanTaskParser.java: ## @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

Re: [PR] chore: Bump upload-artifact@v3 to v4 [iceberg-rust]

2024-11-27 Thread via GitHub
kevinjqliu commented on PR #725: URL: https://github.com/apache/iceberg-rust/pull/725#issuecomment-2504672024 nice! I see it at https://test.pypi.org/project/pyiceberg-core/ and I was able to install it locally ``` poetry run pip install -i https://test.pypi.org/simple/ pyiceberg

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861181045 ## core/src/main/java/org/apache/iceberg/rest/requests/PlanTableScanRequestParser.java: ## @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] chore: Bump upload-artifact@v3 to v4 [iceberg-rust]

2024-11-27 Thread via GitHub
sungwy commented on PR #725: URL: https://github.com/apache/iceberg-rust/pull/725#issuecomment-2504663428 @kevinjqliu - please refer to: https://test.pypi.org/manage/project/pyiceberg-core/release/0.1.0/ -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861165401 ## core/src/main/java/org/apache/iceberg/rest/requests/FetchScanTasksRequest.java: ## @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861173069 ## core/src/main/java/org/apache/iceberg/rest/requests/PlanTableScanRequestParser.java: ## @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Snapshot: Make manifest-list required [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu commented on PR #1385: URL: https://github.com/apache/iceberg-python/pull/1385#issuecomment-2504651007 > we can assume that the manifest-list is there all the time. what about for V1 tables where the `manifest-list` field is optional? -- This is an automated message from

Re: [PR] Core,API: Set `503: added_snapshot_id` as required [iceberg]

2024-11-27 Thread via GitHub
rdblue commented on code in PR #11626: URL: https://github.com/apache/iceberg/pull/11626#discussion_r1861156965 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -166,8 +166,14 @@ private String createManifestListWithManifestFiles(long snapshotId, Long paren

Re: [PR] [release] Pyiceberg 0.8.1 [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu commented on PR #1369: URL: https://github.com/apache/iceberg-python/pull/1369#issuecomment-2504632380 accidentally squashed the commits... redoing in #1384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[PR] Snapshot: Make manifest-list required [iceberg-python]

2024-11-27 Thread via GitHub
Fokko opened a new pull request, #1385: URL: https://github.com/apache/iceberg-python/pull/1385 Since we don't support `manifests`, we can assume that the manifest-list is there all the time. https://github.com/user-attachments/assets/3c438a98-318b-4dfd-8624-78996ce17c19";> --

[PR] [release] Pyiceberg 0.8.1 [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu opened a new pull request, #1384: URL: https://github.com/apache/iceberg-python/pull/1384 [pyiceberg 0.8.1 proposal on devlist](https://lists.apache.org/thread/j4l5xk92j6z3pb31wyqtxk7fbf1621kv) ## 0.8.1 Release Note The behavior of `Table.name` is changed to return the t

Re: [PR] Bump pyarrow from 18.0.0 to 18.1.0 [iceberg-python]

2024-11-27 Thread via GitHub
Fokko commented on PR #1381: URL: https://github.com/apache/iceberg-python/pull/1381#issuecomment-2504630235 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Bump pydantic from 2.10.1 to 2.10.2 [iceberg-python]

2024-11-27 Thread via GitHub
Fokko merged PR #1382: URL: https://github.com/apache/iceberg-python/pull/1382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861149852 ## core/src/main/java/org/apache/iceberg/rest/requests/FetchScanTasksRequest.java: ## @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861148874 ## core/src/main/java/org/apache/iceberg/ContentFileParser.java: ## @@ -48,6 +48,97 @@ class ContentFileParser { private ContentFileParser() {} + public stati

Re: [PR] Add scan planning api request and response models, parsers [iceberg]

2024-11-27 Thread via GitHub
rahil-c commented on code in PR #11369: URL: https://github.com/apache/iceberg/pull/11369#discussion_r1861144615 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -511,7 +511,6 @@ public Table loadTable(SessionContext context, TableIdentifier identifi

Re: [PR] [release] Pyiceberg 0.8.1 [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu merged PR #1369: URL: https://github.com/apache/iceberg-python/pull/1369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu commented on issue #1363: URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2504544750 cc @djouallah from https://github.com/apache/iceberg-python/issues/1018#issuecomment-2471827257 -- This is an automated message from the Apache Git Service. To respond t

[PR] Core: Support aggregated basic stats in partition summary [iceberg]

2024-11-27 Thread via GitHub
deniskuzZ opened a new pull request, #11669: URL: https://github.com/apache/iceberg/pull/11669 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] chore: Bump upload-artifact@v3 to v4 [iceberg-rust]

2024-11-27 Thread via GitHub
sungwy commented on PR #725: URL: https://github.com/apache/iceberg-rust/pull/725#issuecomment-2504570448 > is it possible to test this on your action on your fork? I think the easier way would be for me to test it against this repository using the `pull_request` trigger by temporaril

Re: [PR] chore: Bump upload-artifact@v3 to v4 [iceberg-rust]

2024-11-27 Thread via GitHub
sungwy commented on code in PR #725: URL: https://github.com/apache/iceberg-rust/pull/725#discussion_r1861113350 ## .github/workflows/release_python.yml: ## @@ -132,7 +133,7 @@ jobs: steps: - name: Download all the dists - uses: actions/download-artifact@v3 +

Re: [I] CommitReport not reported through custom MetricsReporter [iceberg]

2024-11-27 Thread via GitHub
singhpk234 commented on issue #11664: URL: https://github.com/apache/iceberg/issues/11664#issuecomment-2504554697 Looks like we just made this property honoured in RestCatalog : https://github.com/apache/iceberg/pull/6404/files#diff-d3f2fbf7b78c8015a3f0c2f9b83b2c49119490a9659f6a1270efbf5

Re: [PR] [release] Pyiceberg 0.8.1 [iceberg-python]

2024-11-27 Thread via GitHub
kevinjqliu commented on PR #1369: URL: https://github.com/apache/iceberg-python/pull/1369#issuecomment-2504540699 Added #1383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]

2024-11-27 Thread via GitHub
JasperHG90 closed issue #1363: `catalog.table_exists()` returns 'False' when table exists in Polaris catalog URL: https://github.com/apache/iceberg-python/issues/1363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

  1   2   >