[PR] build(deps): bump github.com/hamba/avro/v2 from 2.26.0 to 2.27.0 [iceberg-go]

2024-10-20 Thread via GitHub
dependabot[bot] opened a new pull request, #181: URL: https://github.com/apache/iceberg-go/pull/181 Bumps [github.com/hamba/avro/v2](https://github.com/hamba/avro) from 2.26.0 to 2.27.0. Release notes Sourced from https://github.com/hamba/avro/releases";>github.com/hamba/avro/v2's

[PR] build(deps): bump github.com/apache/arrow-go/v18 from 18.0.0-20240924011512-14844aea3205 to 18.0.0-rc0 [iceberg-go]

2024-10-20 Thread via GitHub
dependabot[bot] opened a new pull request, #180: URL: https://github.com/apache/iceberg-go/pull/180 Bumps [github.com/apache/arrow-go/v18](https://github.com/apache/arrow-go) from 18.0.0-20240924011512-14844aea3205 to 18.0.0-rc0. Commits See full diff in https://github.com/apac

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.65.3 to 1.66.0 [iceberg-go]

2024-10-20 Thread via GitHub
dependabot[bot] opened a new pull request, #182: URL: https://github.com/apache/iceberg-go/pull/182 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.65.3 to 1.66.0. Commits https://github.com/aws/aws-sdk-go-v2/commit/ebfeafa763ea779f

[I] flink write to iceberg raises warn: Unclosed S3FileIO instance in GlueTableOperations [iceberg]

2024-10-20 Thread via GitHub
a8356555 opened a new issue, #11364: URL: https://github.com/apache/iceberg/issues/11364 ### Apache Iceberg version 1.5.2 ### Query engine Flink ### Please describe the bug 🐞 use this flink sql ```sql CREATE CATALOG glue_catalog WITH ( 'type

[PR] [Views] Update view spec with table identifier requirements [iceberg]

2024-10-20 Thread via GitHub
wmoustafa opened a new pull request, #11365: URL: https://github.com/apache/iceberg/pull/11365 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Spark:read iceberg table data error [iceberg]

2024-10-20 Thread via GitHub
nastra commented on issue #11336: URL: https://github.com/apache/iceberg/issues/11336#issuecomment-2425633608 @beyond-up in that case you might want to use a more recent Spark version that includes a potential fix for this -- This is an automated message from the Apache Git Service. To re

Re: [PR] Build: Bump parquet from 1.13.1 to 1.14.3 [iceberg]

2024-10-20 Thread via GitHub
nastra merged PR #11264: URL: https://github.com/apache/iceberg/pull/11264 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Proxy support unavailable for iceberg spark client [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9174: URL: https://github.com/apache/iceberg/issues/9174#issuecomment-2425313651 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Flink: Add IcebergSinkBuilder interface allowed unification of most of operations on FlinkSink and IcebergSink Builders [iceberg]

2024-10-20 Thread via GitHub
pvary merged PR #11305: URL: https://github.com/apache/iceberg/pull/11305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Flink: Add RowConverter for Iceberg Source [iceberg]

2024-10-20 Thread via GitHub
stevenzwu commented on code in PR #11301: URL: https://github.com/apache/iceberg/pull/11301#discussion_r1808069850 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/reader/RowConverter.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-10-20 Thread via GitHub
wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2425520426 > For instance, if two position-based delete operations remove the same set of records, both of them will succeed. I was not aware of this. Is this specifically the case of two con

Re: [PR] Flink: Add IcebergSinkBuilder interface allowed unification of most of operations on FlinkSink and IcebergSink Builders [iceberg]

2024-10-20 Thread via GitHub
pvary commented on PR #11305: URL: https://github.com/apache/iceberg/pull/11305#issuecomment-2425565481 Thanks for the patience @arkadius. Merged the PR. Thanks for the review @rodmeneses -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Flink: Tests alignment for the Flink Sink v2-based implemenation (IcebergSink) [iceberg]

2024-10-20 Thread via GitHub
pvary commented on PR #11219: URL: https://github.com/apache/iceberg/pull/11219#issuecomment-2425564578 Merged #11305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] flink:FlinkSink support dynamically changed schema [iceberg]

2024-10-20 Thread via GitHub
pvary commented on issue #4190: URL: https://github.com/apache/iceberg/issues/4190#issuecomment-2425567002 The Iceberg table could be used as a schema registry. I would be reluctant to add any new requirements if possible -- This is an automated message from the Apache Git Service. To re

Re: [PR] (AWS) Docs: List all AWS S3 properties from all language impl. [iceberg]

2024-10-20 Thread via GitHub
TiansuYu commented on code in PR #11321: URL: https://github.com/apache/iceberg/pull/11321#discussion_r1807900307 ## docs/docs/aws-s3-fileio-properties.md: ## @@ -0,0 +1,341 @@ +--- +title: "AWS S3 Properties" +--- + + +### Language Implementations + +| S3 Property

Re: [I] Spark:read iceberg table data error [iceberg]

2024-10-20 Thread via GitHub
beyond-up commented on issue #11336: URL: https://github.com/apache/iceberg/issues/11336#issuecomment-2425438404 > @beyond-up so far the NPE seems to be coming from Spark itself, not from Iceberg. Do you have a small reproducible example? This problem will be reproduced when a string

Re: [PR] Add `view_exists` method to REST Catalog [iceberg-python]

2024-10-20 Thread via GitHub
shiv-io commented on PR #1242: URL: https://github.com/apache/iceberg-python/pull/1242#issuecomment-2425262513 When I tested `catalog.view_exists('default.bar')` with a local REST catalog, I got the following exception. This also occurs with the existing `catalog.table_exists()` method. Is

Re: [PR] Update flink docs with alter column support [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on PR #9756: URL: https://github.com/apache/iceberg/pull/9756#issuecomment-2425313949 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] REST Catalog Spec: Snapshot Summary Class [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9837: URL: https://github.com/apache/iceberg/issues/9837#issuecomment-2425314268 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Support metadata tables for views [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9844: URL: https://github.com/apache/iceberg/issues/9844#issuecomment-2425314301 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spark job application finished with failed status when trying to read iceberg hive tables from remote jupyter notebook pod [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9824: URL: https://github.com/apache/iceberg/issues/9824#issuecomment-2425314184 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] The Ambiguity in the Docs "Hive type to Iceberg type." [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9863: URL: https://github.com/apache/iceberg/issues/9863#issuecomment-2425314337 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Parquet, Arrow: Refactor vectorized reader [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on PR #9772: URL: https://github.com/apache/iceberg/pull/9772#issuecomment-2425314026 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Can we load iceberg table using external volume instead of external stage ? [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9828: URL: https://github.com/apache/iceberg/issues/9828#issuecomment-2425314215 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Under Concepts -> Iceberg Catalog Page Hyperlink seems not to be routing to correct page [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9847: URL: https://github.com/apache/iceberg/issues/9847#issuecomment-2425314315 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spec inconsistency: partition_spec_id column in ManifestList vs. partition_specs in metadata.json [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9739: URL: https://github.com/apache/iceberg/issues/9739#issuecomment-2425313892 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Bug: Flink data loss after failed to refresh table [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9753: URL: https://github.com/apache/iceberg/issues/9753#issuecomment-2425313919 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] branch schema affected by main table schema [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9737: URL: https://github.com/apache/iceberg/issues/9737#issuecomment-2425313868 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] [Docs, Flink] Iceberg Flink docs do not include support for enhanced DDL support added in #7628 [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9755: URL: https://github.com/apache/iceberg/issues/9755#issuecomment-2425313931 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] truncate partitioning underflows, leads to wrong results [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9767: URL: https://github.com/apache/iceberg/issues/9767#issuecomment-2425313998 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spark read failed when migrate hive orc table with `timestamp` column [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9784: URL: https://github.com/apache/iceberg/issues/9784#issuecomment-2425314041 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark 3.5: Add deleted_snapshots_count to result of expire_snapshots procedure [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on PR #9605: URL: https://github.com/apache/iceberg/pull/9605#issuecomment-2425313678 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Support rewrite file with z-order for nested Struct type [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on PR #9818: URL: https://github.com/apache/iceberg/pull/9818#issuecomment-2425314163 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Is it possible to add a set of existing partitioned parquet files to the Iceberg table via the Java Standalone API [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9763: URL: https://github.com/apache/iceberg/issues/9763#issuecomment-2425313974 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Confusion about latest_schema_id in metadata_log_entries [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9758: URL: https://github.com/apache/iceberg/issues/9758#issuecomment-2425313966 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spec is ambiguous w.r.t. optional fields in field_summary [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9740: URL: https://github.com/apache/iceberg/issues/9740#issuecomment-2425313909 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Writing Equality Deletes using Iceberg Java API [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9808: URL: https://github.com/apache/iceberg/issues/9808#issuecomment-2425314144 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Calling `rewrite_position_delete_files` rewrites into same amount of files [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9833: URL: https://github.com/apache/iceberg/issues/9833#issuecomment-2425314254 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] start-timestamp not utilized in create_changelog_view [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9791: URL: https://github.com/apache/iceberg/issues/9791#issuecomment-2425314072 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Cannot find constructor for interface org.apache.parquet.column.page.PageWriteStore? [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9802: URL: https://github.com/apache/iceberg/issues/9802#issuecomment-2425314124 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Views, Spark: Add support for Materialized Views; Integrate with Spark SQL [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on PR #9830: URL: https://github.com/apache/iceberg/pull/9830#issuecomment-2425314230 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] S3FileIO does not support Iceberg Cross-Region API Calls to Amazon S3 buckets [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9785: URL: https://github.com/apache/iceberg/issues/9785#issuecomment-2425314056 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Supports limiting the count of manifests to merge when committing [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] closed pull request #9609: Core: Supports limiting the count of manifests to merge when committing URL: https://github.com/apache/iceberg/pull/9609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Spark 3.5: Add deleted_snapshots_count to result of expire_snapshots procedure [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] closed pull request #9605: Spark 3.5: Add deleted_snapshots_count to result of expire_snapshots procedure URL: https://github.com/apache/iceberg/pull/9605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [I] [Docs] Add an open-api build that is added with the other spec implementations. [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9850: URL: https://github.com/apache/iceberg/issues/9850#issuecomment-2425314325 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] [DOCS] Update format specs to use the mkdocs material markdown [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9842: URL: https://github.com/apache/iceberg/issues/9842#issuecomment-2425314286 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Proxy support unavailable for iceberg spark client [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] closed issue #9174: Proxy support unavailable for iceberg spark client URL: https://github.com/apache/iceberg/issues/9174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Core: Supports limiting the count of manifests to merge when committing [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on PR #9609: URL: https://github.com/apache/iceberg/pull/9609#issuecomment-2425313706 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [I] Inconsistency in deleting manifest and data files [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9792: URL: https://github.com/apache/iceberg/issues/9792#issuecomment-2425314103 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Parquet, Arrow: Refactor vectorized reader [iceberg]

2024-10-20 Thread via GitHub
PaulLiang1 commented on PR #9772: URL: https://github.com/apache/iceberg/pull/9772#issuecomment-2425321714 Bump comment to keep the bot from closing this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] rewrite_data_files procedure fails with Premature end of Content-Length when using S3 client [iceberg]

2024-10-20 Thread via GitHub
github-actions[bot] commented on issue #9679: URL: https://github.com/apache/iceberg/issues/9679#issuecomment-2425313755 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Unable to use GlueCatalog in flink environments without hadoop [iceberg]

2024-10-20 Thread via GitHub
j3-signalroom commented on issue #3044: URL: https://github.com/apache/iceberg/issues/3044#issuecomment-2425331532 > I found this thread while trying to make Flink, Iceberg, and Glue work together without the Hadoop dependencies, so I'm hoping this is useful for someone else in the future.

Re: [PR] Flink: Add RowConverter for Iceberg Source [iceberg]

2024-10-20 Thread via GitHub
abharath9 commented on code in PR #11301: URL: https://github.com/apache/iceberg/pull/11301#discussion_r1807179938 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/reader/RowConverter.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation

[PR] Add `all_manifests` metadata table with tests [iceberg-python]

2024-10-20 Thread via GitHub
soumya-ghosh opened a new pull request, #1241: URL: https://github.com/apache/iceberg-python/pull/1241 Implements `all_manifests` metadata table - https://github.com/apache/iceberg-python/issues/1053 Have refactored the code tor re-use logic of `manifests` metadata table. -- This i

Re: [I] [feat] add missing metadata tables [iceberg-python]

2024-10-20 Thread via GitHub
soumya-ghosh commented on issue #1053: URL: https://github.com/apache/iceberg-python/issues/1053#issuecomment-2425165343 @kevinjqliu added PR - https://github.com/apache/iceberg-python/pull/1241 for `all_manifests`. Will get on with `all_files`, `all_data_files` and `all_delete_files

Re: [I] bug: ArrowSchemaConverter can't handle unsigned datatypes from arrow [iceberg-rust]

2024-10-20 Thread via GitHub
Xuanwo commented on issue #675: URL: https://github.com/apache/iceberg-rust/issues/675#issuecomment-2425581938 Thank you for the report. I will take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[I] Implement list_views Method and __is_view Utility Function [iceberg-python]

2024-10-20 Thread via GitHub
omkenge opened a new issue, #1240: URL: https://github.com/apache/iceberg-python/issues/1240 ### Feature Request / Improvement You can refer this PR #1239 ( @sungwy @kevinjqliu ) -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Kafka Connect: Add table to topics mapping property [iceberg]

2024-10-20 Thread via GitHub
igorvoltaic commented on PR #10422: URL: https://github.com/apache/iceberg/pull/10422#issuecomment-2425109486 > I believe when we add https://github.com/apache/iceberg/pull/11313 you should be able to accomplish mapping topics to tables. Also I think this PR isn't complete, the new config i

Re: [PR] API, Core: Add scan planning apis to REST Catalog [iceberg]

2024-10-20 Thread via GitHub
rahil-c commented on code in PR #11180: URL: https://github.com/apache/iceberg/pull/11180#discussion_r1807950377 ## core/src/test/java/org/apache/iceberg/TestBase.java: ## @@ -63,7 +63,7 @@ public class TestBase { public static final PartitionSpec SPEC = PartitionSpec.

Re: [I] Add view support to the Rest Catalog [iceberg-python]

2024-10-20 Thread via GitHub
shiv-io commented on issue #818: URL: https://github.com/apache/iceberg-python/issues/818#issuecomment-2425171379 Would love to take a first stab at this @kevinjqliu, could you assign? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] Add `view_exists` method to REST Catalog [iceberg-python]

2024-10-20 Thread via GitHub
shiv-io opened a new pull request, #1242: URL: https://github.com/apache/iceberg-python/pull/1242 Part of the adding view support to the REST catalog: #818 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above