Re: [PR] View Spec implementation [iceberg-rust]

2024-07-24 Thread via GitHub
c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1690895876 ## crates/iceberg/src/spec/view_version.rs: ## @@ -71,8 +74,18 @@ impl ViewVersion { /// Get the timestamp of when the view version was created #[inline]

Re: [PR] Kafka Connect: Runtime distribution with integration tests [iceberg]

2024-07-24 Thread via GitHub
ajantha-bhat commented on code in PR #10739: URL: https://github.com/apache/iceberg/pull/10739#discussion_r1690897385 ## kafka-connect/kafka-connect-runtime/docker/docker-compose.yml: ## @@ -0,0 +1,107 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more c

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-24 Thread via GitHub
c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1690895876 ## crates/iceberg/src/spec/view_version.rs: ## @@ -71,8 +74,18 @@ impl ViewVersion { /// Get the timestamp of when the view version was created #[inline]

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-24 Thread via GitHub
c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1690900328 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -144,8 +144,24 @@ impl TableMetadata { /// Returns last updated time. #[inline] -pub fn last_upda

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-24 Thread via GitHub
c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1690898177 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -144,8 +144,24 @@ impl TableMetadata { /// Returns last updated time. #[inline] -pub fn last_upda

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-24 Thread via GitHub
c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1690897640 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -917,8 +933,24 @@ pub struct SnapshotLog { impl SnapshotLog { /// Returns the last updated timestamp as a

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-24 Thread via GitHub
c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1690895876 ## crates/iceberg/src/spec/view_version.rs: ## @@ -71,8 +74,18 @@ impl ViewVersion { /// Get the timestamp of when the view version was created #[inline]

[PR] DRAFT: Strawman proposal for PR merging [iceberg]

2024-07-24 Thread via GitHub
emkornfield opened a new pull request, #10780: URL: https://github.com/apache/iceberg/pull/10780 Based on mailing list discussion add focused language to contributing.md on committing PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Spark Action to Analyze table [iceberg]

2024-07-24 Thread via GitHub
jeesou commented on PR #10288: URL: https://github.com/apache/iceberg/pull/10288#issuecomment-2249540721 Hi @karuppayya is there any more changes that are supposed to be added to this PR? If no, when are we planning to get it merged? Apart from that is there any plan to back port t

Re: [I] Parquet column array> with null value is read in as empty list [iceberg-python]

2024-07-24 Thread via GitHub
HonahX commented on issue #251: URL: https://github.com/apache/iceberg-python/issues/251#issuecomment-2249538702 Reply to re-activate the issue : ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Incorrect Avro schema generated for Tables with a `Transform::Day` partition causes manifest file parsing to fail [iceberg-rust]

2024-07-24 Thread via GitHub
Fokko commented on issue #478: URL: https://github.com/apache/iceberg-rust/issues/478#issuecomment-2249537321 @sdd Thanks for digging into this, and I agree that it should be a `date`: https://github.com/apache/iceberg/issues/10616#issuecomment-2200191427 -- This is an automated message f

Re: [I] Incorrect Avro schema generated for Tables with a `Transform::Day` partition causes manifest file parsing to fail [iceberg-rust]

2024-07-24 Thread via GitHub
Fokko closed issue #478: Incorrect Avro schema generated for Tables with a `Transform::Day` partition causes manifest file parsing to fail URL: https://github.com/apache/iceberg-rust/issues/478 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Alter `Transform::Day` to map partition types to `Date` rather than `Int` for consistency with reference implementation [iceberg-rust]

2024-07-24 Thread via GitHub
Fokko merged PR #479: URL: https://github.com/apache/iceberg-rust/pull/479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
Xuanwo commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1690861564 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1496 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-24 Thread via GitHub
c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1690858992 ## crates/iceberg/src/spec/view_version.rs: ## @@ -155,6 +155,11 @@ impl ViewRepresentations { pub fn is_empty(&self) -> bool { self.0.is_empty()

Re: [I] Is dataFiles() Method Retryable? [iceberg]

2024-07-24 Thread via GitHub
osy497 commented on issue #10750: URL: https://github.com/apache/iceberg/issues/10750#issuecomment-2249482744 @nk1506 Most of cases seems timeout problem, but i'm not sure about that. I will ask for this in Slack channel later. Thanks for your explanation :) -- This is an automat

Re: [PR] Flink: refactor sink tests to reduce the number of combinations with parameterized tests [iceberg]

2024-07-24 Thread via GitHub
pvary commented on PR #10777: URL: https://github.com/apache/iceberg/pull/10777#issuecomment-2249435360 @rodmeneses: Could you please take a look. I think it will greatly affect your testing for Sink V2 -- This is an automated message from the Apache Git Service. To respond to the message

[I] On droping table with shared location the data got deleted which should not be case after version 0.14.0 of Apache iceberg [iceberg]

2024-07-24 Thread via GitHub
Muskan-m opened a new issue, #10779: URL: https://github.com/apache/iceberg/issues/10779 ### Apache Iceberg version 1.2.0 ### Query engine Spark ### Please describe the bug 🐞 I have created a two tables which share same location and i dropped one of the tab

[PR] API: Fix typo in RewriteManifestFiles java doc [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar opened a new pull request, #10778: URL: https://github.com/apache/iceberg/pull/10778 Fixes a small typo I noticed when reading the rewrite manifests API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
Xuanwo commented on PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#issuecomment-2249346483 Request review from @Fokko and @liurenjie1024 for another eyes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Alter `Transform::Day` to map partition types to `Date` rather than `Int` for consistency with reference implementation [iceberg-rust]

2024-07-24 Thread via GitHub
Xuanwo commented on PR #479: URL: https://github.com/apache/iceberg-rust/pull/479#issuecomment-2249344205 cc @liurenjie1024 and @Fokko for another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] feat(catalog!): Deprecate rest.authorization-url in favor of oauth2-server-uri [iceberg-rust]

2024-07-24 Thread via GitHub
Xuanwo merged PR #480: URL: https://github.com/apache/iceberg-rust/pull/480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1690620132 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1511 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-07-24 Thread via GitHub
stevenzwu commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1690550344 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergSink.java: ## @@ -0,0 +1,395 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Concurrent table scans [iceberg-rust]

2024-07-24 Thread via GitHub
liurenjie1024 commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1690704340 ## crates/iceberg/src/scan.rs: ## @@ -197,134 +193,103 @@ impl<'a> TableScanBuilder<'a> { field_ids.push(field_id); } -Ok(Table

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-24 Thread via GitHub
advancedxy commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1690700326 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1102,6 +1121,22 @@ public Builder setDefaultPartitionSpec(int specId) { return this;

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-24 Thread via GitHub
advancedxy commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1690698977 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1102,6 +1121,22 @@ public Builder setDefaultPartitionSpec(int specId) { return this;

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-24 Thread via GitHub
advancedxy commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1690697748 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1102,6 +1121,22 @@ public Builder setDefaultPartitionSpec(int specId) { return this;

Re: [I] table.delete()/overwrite() with null values in table and with non-null filter will delete null rows [iceberg-python]

2024-07-24 Thread via GitHub
sungwy closed issue #954: table.delete()/overwrite() with null values in table and with non-null filter will delete null rows URL: https://github.com/apache/iceberg-python/issues/954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Remove support for catalog_name in table identifier string [iceberg-python]

2024-07-24 Thread via GitHub
sungwy commented on PR #963: URL: https://github.com/apache/iceberg-python/pull/963#issuecomment-2249217253 There is some more work to be done on this PR, including: - deprecating `identifier_to_tuple_without_catalog` public method on the catalog - removing its usage across the board,

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690686528 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs"

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
sungwy merged PR #955: URL: https://github.com/apache/iceberg-python/pull/955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Unable to create catalog with Hive catalog [iceberg]

2024-07-24 Thread via GitHub
ideal commented on issue #10772: URL: https://github.com/apache/iceberg/issues/10772#issuecomment-2249208114 > These are very old versions of Iceberg and Flink? Would it be possible to retry it with one of the current releases? > > Thanks, Peter OK, I will have a try. --

Re: [I] Unable to create catalog with Hive catalog [iceberg]

2024-07-24 Thread via GitHub
ideal closed issue #10772: Unable to create catalog with Hive catalog URL: https://github.com/apache/iceberg/issues/10772 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[PR] Remove support for catalog_name in table identifier string [iceberg-python]

2024-07-24 Thread via GitHub
sungwy opened a new pull request, #963: URL: https://github.com/apache/iceberg-python/pull/963 Currently, we optionally support catalog names being specified in the identifier string. In other words, this means that we currently support identifier names that look like `catalog_name.table_na

Re: [PR] Add `IcebergAnalysisException` in iceberg-spark module [iceberg]

2024-07-24 Thread via GitHub
huaxingao commented on PR #10766: URL: https://github.com/apache/iceberg/pull/10766#issuecomment-2249186701 > Curious what's the plan for Spark 4.0 then? I can't find a formally written down plan for this. > Just brainstorming out loud the value of IcebergAnalysisException, anot

Re: [PR] Add `IcebergAnalysisException` in iceberg-spark module [iceberg]

2024-07-24 Thread via GitHub
huaxingao commented on code in PR #10766: URL: https://github.com/apache/iceberg/pull/10766#discussion_r1690665089 ## spark/v3.5/spark/src/main/java/org/apache/spark/sql/catalyst/analysis/IcebergAnalysisException.java: ## @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: refactor sink tests to reduce the number of combinations with parameterized tests [iceberg]

2024-07-24 Thread via GitHub
stevenzwu commented on code in PR #10777: URL: https://github.com/apache/iceberg/pull/10777#discussion_r1690630724 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/CatalogTestBase.java: ## @@ -118,26 +118,4 @@ protected String getFullQualifiedTableName(String tableNa

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
jeremysong commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690631304 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs" +

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
fqaiser94 commented on PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#issuecomment-2249126369 @Xuanwo just FYI, all comments besides the [package-naming](https://github.com/apache/iceberg-rust/pull/475#discussion_r1690026333) discussion should be addressed now -- This is

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1690627063 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1511 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

[PR] Flink: refactor sink tests to reduce the number of combinations with parameterized tests [iceberg]

2024-07-24 Thread via GitHub
stevenzwu opened a new pull request, #10777: URL: https://github.com/apache/iceberg/pull/10777 E.g. it is not really necessary to test different file format for distribution mode (hash, range). I found this necessary when trying to add new tests for the new range distribution feature. W

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690624433 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs" +Pre

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690623146 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs" +Pre

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690623689 ## open-api/rest-catalog-open-api.yaml: ## @@ -2774,6 +2920,30 @@ components: additionalProperties: type: string +PreplanTableResult: +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690622005 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs" +Pre

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1690621507 ## crates/catalog/inmemory/src/lib.rs: ## @@ -0,0 +1,26 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreement

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690622344 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs" +Pre

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1690620132 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1511 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690617398 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs" +Pre

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1690616688 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1511 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [PR] Add in-memory catalog implementation [iceberg-rust]

2024-07-24 Thread via GitHub
fqaiser94 commented on code in PR #475: URL: https://github.com/apache/iceberg-rust/pull/475#discussion_r1690614275 ## crates/catalog/inmemory/src/catalog.rs: ## @@ -0,0 +1,1511 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agr

Re: [I] Parquet column array> with null value is read in as empty list [iceberg-python]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #251: URL: https://github.com/apache/iceberg-python/issues/251#issuecomment-2249099707 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [PR] Spark: vectorized/non-vectorized read compound constant type exception (#3139) [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on PR #3186: URL: https://github.com/apache/iceberg/pull/3186#issuecomment-2249098314 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] the document error of sort order evolution [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3150: URL: https://github.com/apache/iceberg/issues/3150#issuecomment-2249098212 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] beeline query iceberg table error [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3146: URL: https://github.com/apache/iceberg/issues/3146#issuecomment-2249098192 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink reads iceberg in real time and reports errors [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3156: URL: https://github.com/apache/iceberg/issues/3156#issuecomment-2249098235 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Unable to create iceberg database / table using spark sql with AWS S3 + Glue integration [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3131: URL: https://github.com/apache/iceberg/issues/3131#issuecomment-2249098165 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] iceberg HiveCatalog insert exception of GSS initiate failed [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3127: URL: https://github.com/apache/iceberg/issues/3127#issuecomment-2249098107 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] ORC: support metadata column structs with vectorization [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3192: URL: https://github.com/apache/iceberg/issues/3192#issuecomment-2249098350 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Data: Add equality delete file cache [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on PR #3174: URL: https://github.com/apache/iceberg/pull/3174#issuecomment-2249098283 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Does anyone know the maven dependency of flink1.12+iceberg0.12.0? [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3166: URL: https://github.com/apache/iceberg/issues/3166#issuecomment-2249098251 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Read ORC table with nested partition column can lead to ArrayIndexOutOfBoundsException [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3139: URL: https://github.com/apache/iceberg/issues/3139#issuecomment-2249098176 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] flink +iceberg +s3 Data is written in real time [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3129: URL: https://github.com/apache/iceberg/issues/3129#issuecomment-2249098151 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] how create iceberg hive catalog kerberos [iceberg]

2024-07-24 Thread via GitHub
github-actions[bot] commented on issue #3128: URL: https://github.com/apache/iceberg/issues/3128#issuecomment-2249098128 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1689261963 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs" +Pr

Re: [PR] Add `IcebergAnalysisException` in iceberg-spark module [iceberg]

2024-07-24 Thread via GitHub
szehon-ho commented on PR #10766: URL: https://github.com/apache/iceberg/pull/10766#issuecomment-2249088638 > Spark 4.0 requires AnalysisException to have error class Curious what's the plan for Spark 4.0 then? Just brainstorming out loud the value of IcebergAnalysisException,

Re: [PR] Add `IcebergAnalysisException` in iceberg-spark module [iceberg]

2024-07-24 Thread via GitHub
szehon-ho commented on code in PR #10766: URL: https://github.com/apache/iceberg/pull/10766#discussion_r1690602060 ## spark/v3.5/spark/src/main/java/org/apache/spark/sql/catalyst/analysis/IcebergAnalysisException.java: ## @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-07-24 Thread via GitHub
stevenzwu commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1690550344 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergSink.java: ## @@ -0,0 +1,395 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690550621 ## pyiceberg/io/pyarrow.py: ## @@ -576,11 +572,11 @@ def _convert_scalar(value: Any, iceberg_type: IcebergType) -> pa.scalar: class _ConvertToArrowExpression(

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-07-24 Thread via GitHub
stevenzwu commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1690515381 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSinkWriter.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] UpdatePartitionSpec: Added ability to not set the new partition spec as default [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #10736: URL: https://github.com/apache/iceberg/pull/10736#discussion_r1690525854 ## api/src/main/java/org/apache/iceberg/UpdatePartitionSpec.java: ## @@ -122,4 +122,13 @@ public interface UpdatePartitionSpec extends PendingUpdate { *

Re: [PR] Update last-updated-ms for DDL operations [iceberg-python]

2024-07-24 Thread via GitHub
HonahX commented on code in PR #956: URL: https://github.com/apache/iceberg-python/pull/956#discussion_r1690519566 ## tests/table/test_init.py: ## @@ -689,7 +689,7 @@ def test_update_metadata_add_snapshot(table_v2: Table) -> None: snapshot_id=25, parent_snapsh

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
HonahX commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690505567 ## pyiceberg/io/pyarrow.py: ## @@ -576,11 +572,11 @@ def _convert_scalar(value: Any, iceberg_type: IcebergType) -> pa.scalar: class _ConvertToArrowExpression(

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
HonahX commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690505567 ## pyiceberg/io/pyarrow.py: ## @@ -576,11 +572,11 @@ def _convert_scalar(value: Any, iceberg_type: IcebergType) -> pa.scalar: class _ConvertToArrowExpression(

[I] Incorrect Avro schema generated for Tables with a `Transform::Day` partition causes manifest file parsing to fail [iceberg-rust]

2024-07-24 Thread via GitHub
sdd opened a new issue, #478: URL: https://github.com/apache/iceberg-rust/issues/478 I've been working on creating a performance testing suite to measure the performance impact of the concurrent table scan work that I've been doing. I created a docker-compose file that uses the Tabular spar

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-07-24 Thread via GitHub
sl255051 commented on PR #10678: URL: https://github.com/apache/iceberg/pull/10678#issuecomment-2248955661 @RussellSpitzer @amogh-jahagirdar I understand your position on case sensitivity. Given that position it seems to me that PartitionSpec.java does not enable case-insensitivity because

Re: [PR] Support for Flink's SpeculativeExecution in batch execution mode - Backport of PR #10548 [iceberg]

2024-07-24 Thread via GitHub
rodmeneses commented on PR #10776: URL: https://github.com/apache/iceberg/pull/10776#issuecomment-2248921732 > The 1st point is fine. @rodmeneses: How did we handle the drop database stuff in previous Flink releases? > > Thanks, Peter We didn't have `dropDatabase` on 1.17 and 1

Re: [PR] UpdatePartitionSpec: Added ability to not set the new partition spec as default [iceberg]

2024-07-24 Thread via GitHub
RussellSpitzer commented on code in PR #10736: URL: https://github.com/apache/iceberg/pull/10736#discussion_r1690468371 ## api/src/main/java/org/apache/iceberg/UpdatePartitionSpec.java: ## @@ -122,4 +122,13 @@ public interface UpdatePartitionSpec extends PendingUpdate { *

Re: [PR] UpdatePartitionSpec: Added ability to not set the new partition spec as default [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #10736: URL: https://github.com/apache/iceberg/pull/10736#discussion_r1690457935 ## core/src/main/java/org/apache/iceberg/BaseUpdatePartitionSpec.java: ## @@ -327,7 +336,12 @@ public PartitionSpec apply() { @Override public void c

Re: [PR] UpdatePartitionSpec: Added ability to not set the new partition spec as default [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #10736: URL: https://github.com/apache/iceberg/pull/10736#discussion_r1690448386 ## api/src/main/java/org/apache/iceberg/UpdatePartitionSpec.java: ## @@ -122,4 +122,13 @@ public interface UpdatePartitionSpec extends PendingUpdate { *

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-24 Thread via GitHub
RussellSpitzer commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1690422091 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1102,6 +1121,22 @@ public Builder setDefaultPartitionSpec(int specId) { return this;

Re: [PR] API: Add RemoveUnusedSpecs in Table [iceberg]

2024-07-24 Thread via GitHub
RussellSpitzer commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1690420193 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1102,6 +1121,22 @@ public Builder setDefaultPartitionSpec(int specId) { return this;

Re: [PR] Update checkstyle definition [iceberg]

2024-07-24 Thread via GitHub
RussellSpitzer commented on PR #10681: URL: https://github.com/apache/iceberg/pull/10681#issuecomment-2248879112 Thanks @attilakreiner for the commit and @findepi for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Update checkstyle definition [iceberg]

2024-07-24 Thread via GitHub
RussellSpitzer merged PR #10681: URL: https://github.com/apache/iceberg/pull/10681 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690408866 ## tests/integration/test_deletes.py: ## @@ -417,3 +452,107 @@ def test_delete_truncate(session_catalog: RestCatalog) -> None: assert len(entries) == 1

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690328141 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,152 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-24 Thread via GitHub
hussein-awala commented on code in PR #10771: URL: https://github.com/apache/iceberg/pull/10771#discussion_r1690391598 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -512,7 +531,10 @@ private boolean isValidateNamespace(Namespace namespace) {

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #10771: URL: https://github.com/apache/iceberg/pull/10771#discussion_r1690386787 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -512,7 +531,10 @@ private boolean isValidateNamespace(Namespace namespace) {

Re: [I] Flaky Test `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on issue #10356: URL: https://github.com/apache/iceberg/issues/10356#issuecomment-2248835965 Actually maybe it's not backported to Flink 1.17/1.18 I'll double check that -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Hive: close the fileIO client when closing the hive catalog [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on PR #10771: URL: https://github.com/apache/iceberg/pull/10771#issuecomment-2248834321 Looks like the flink test failure is unrelated, I reopened https://github.com/apache/iceberg/issues/10356 since it looks like that test is still flaky. I'm retriggering CI.

Re: [I] Flaky Test `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on issue #10356: URL: https://github.com/apache/iceberg/issues/10356#issuecomment-2248831540 Reopening because we recently hit this on another PR https://github.com/apache/iceberg/actions/runs/10081030094/job/27872164965?pr=10771 -- This is an automated message

[I] Flaky Test `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]

2024-07-24 Thread via GitHub
manuzhang opened a new issue, #10356: URL: https://github.com/apache/iceberg/issues/10356 ### Apache Iceberg version main (development) ### Query engine Flink ### Please describe the bug 🐞 ``` TestIcebergSourceFailover > testBoundedWithSavepoint FAILED

Re: [PR] Core: Add estimateRowCount for Files and Entries Metadata Tables [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar merged PR #10759: URL: https://github.com/apache/iceberg/pull/10759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Support for Flink's SpeculativeExecution in batch execution mode - Backport of PR #10548 [iceberg]

2024-07-24 Thread via GitHub
pvary commented on PR #10776: URL: https://github.com/apache/iceberg/pull/10776#issuecomment-224880 The 1st point is fine. @rodmeneses: How did we handle the drop database stuff in previous Flink releases? Thanks, Peter -- This is an automated message from the Apache Git Se

Re: [PR] Support for Flink's SpeculativeExecution in batch execution mode - Backport of PR #10548 [iceberg]

2024-07-24 Thread via GitHub
venkata91 commented on PR #10776: URL: https://github.com/apache/iceberg/pull/10776#issuecomment-2248775314 cc @pvary for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Support for Flink's SpeculativeExecution in batch execution mode [iceberg]

2024-07-24 Thread via GitHub
pvary commented on PR #10548: URL: https://github.com/apache/iceberg/pull/10548#issuecomment-2248770983 > > Could you please create the backport PR to the other Flink versions? > > @pvary Should we have 2 backport PRs one for 1.17 and 1.18 or is it fine to do it in a single PR?

Re: [PR] fix table.delete()/overwrite() with null values [iceberg-python]

2024-07-24 Thread via GitHub
jqin61 commented on code in PR #955: URL: https://github.com/apache/iceberg-python/pull/955#discussion_r1690328141 ## pyiceberg/io/pyarrow.py: ## @@ -638,10 +634,152 @@ def visit_or(self, left_result: pc.Expression, right_result: pc.Expression) -> p return left_result

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690318781 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{p

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-07-24 Thread via GitHub
jeremysong commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1690319854 ## open-api/rest-catalog-open-api.yaml: ## @@ -3647,6 +3818,176 @@ components: type: integer description: "List of equality field IDs" +

  1   2   3   >