Re: [PR] Spark: Remove deprecated SparkAppenderFactory [iceberg]

2024-12-08 Thread via GitHub
findepi commented on code in PR #11727: URL: https://github.com/apache/iceberg/pull/11727#discussion_r1875507755 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMergingMetrics.java: ## @@ -1,70 +0,0 @@ -/* - * Licensed to the Apache Software Foundation

Re: [I] flink restore failed with filenotfound [iceberg]

2024-12-08 Thread via GitHub
rahulkr25 commented on issue #6066: URL: https://github.com/apache/iceberg/issues/6066#issuecomment-2527161916 Thanks for the confirmation @jad-grepr However, In the last thread mentioned, you were facing issue with Iceberg 1.5.2 and Flink 1.18.1 but now as in current setup of yours i

Re: [I] Tracking issues of iceberg rust v0.4.0 Release [iceberg-rust]

2024-12-08 Thread via GitHub
c-thiel commented on issue #739: URL: https://github.com/apache/iceberg-rust/issues/739#issuecomment-2527134335 > > @sungwy I think we should add #694 to the blockers as well. If we decide to not introduce a `SchemalessPartitionSpec`, we shouldn't release it as part of 0.4.0 as it would be

Re: [I] why spark ddl rename iceberg table name not change location? does it matter? [iceberg]

2024-12-08 Thread via GitHub
madeirak commented on issue #10436: URL: https://github.com/apache/iceberg/issues/10436#issuecomment-2527139517 > Two tables should _not_ share the same location as `remove_orphan_files` procedure can clean the files not related to current table as orphan files. > > Table names can ch

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

2024-12-08 Thread via GitHub
Ppei-Wang commented on PR #11729: URL: https://github.com/apache/iceberg/pull/11729#issuecomment-2527135445 @rdblue hello,Could you please help approve workflow runs and assign a Reviewer? -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Spark3.4,3.5,Api,Hive: Fix using NullType in View. [iceberg]

2024-12-08 Thread via GitHub
Ppei-Wang commented on PR #11728: URL: https://github.com/apache/iceberg/pull/11728#issuecomment-2527135079 @rdblue hello,Could you please help approve workflow runs and assign a Reviewer? -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Implement `initial-default` [iceberg-rust]

2024-12-08 Thread via GitHub
c-thiel commented on issue #737: URL: https://github.com/apache/iceberg-rust/issues/737#issuecomment-2527132014 @jonathanc-n sure, I'll assign the issue to you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Core: Unimplement Map from CharSequenceMap to obey contract [iceberg]

2024-12-08 Thread via GitHub
nastra commented on code in PR #11704: URL: https://github.com/apache/iceberg/pull/11704#discussion_r1875488258 ## .palantir/revapi.yml: ## @@ -1145,6 +1145,22 @@ acceptedBreaks: new: "method org.apache.iceberg.BaseMetastoreOperations.CommitStatus org.apache.iceberg.Base

[I] Support Type `unknown` for v3 Schema [iceberg]

2024-12-08 Thread via GitHub
HonahX opened a new issue, #11732: URL: https://github.com/apache/iceberg/issues/11732 ### Feature Request / Improvement Implement `unknown` type which is added by version 3: https://iceberg.apache.org/spec/#primitive-types ### Query engine None ### Willingness to

Re: [I] Sort Order Replacement API [iceberg-rust]

2024-12-08 Thread via GitHub
c-thiel closed issue #734: Sort Order Replacement API URL: https://github.com/apache/iceberg-rust/issues/734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [I] Sort Order Replacement API [iceberg-rust]

2024-12-08 Thread via GitHub
c-thiel commented on issue #734: URL: https://github.com/apache/iceberg-rust/issues/734#issuecomment-2527130294 Ah, you are right - sorry -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

2024-12-08 Thread via GitHub
HonahX commented on code in PR #11730: URL: https://github.com/apache/iceberg/pull/11730#discussion_r1875481834 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -307,6 +311,13 @@ public String toString() { Preconditions.checkArgument( metadataFile

Re: [PR] Core, Flink, Spark: Drop deprecated APIs scheduled for removal in 1.8.0 [iceberg]

2024-12-08 Thread via GitHub
ajantha-bhat commented on code in PR #11721: URL: https://github.com/apache/iceberg/pull/11721#discussion_r1875424451 ## .palantir/revapi.yml: ## @@ -1145,6 +1145,32 @@ acceptedBreaks: new: "method org.apache.iceberg.BaseMetastoreOperations.CommitStatus org.apache.iceber

[PR] Bump Spark 3.5.4 RC1 [iceberg]

2024-12-08 Thread via GitHub
pan3793 opened a new pull request, #11731: URL: https://github.com/apache/iceberg/pull/11731 https://lists.apache.org/thread/btdsos7b92962lh10hzswkfy3oj1v0fb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

2024-12-08 Thread via GitHub
HonahX opened a new pull request, #11730: URL: https://github.com/apache/iceberg/pull/11730 Fixes #10764 - Add new fields: `row-lineage` and `next-row-id` to TableMetadata - Update the parser to parse v3 new fields - Add new unit tests for V3 metadata - Refactor TableMetadata

[PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

2024-12-08 Thread via GitHub
Ppei-Wang opened a new pull request, #11729: URL: https://github.com/apache/iceberg/pull/11729 This PR Fixed the BUG of iceberg views when resolved "group/order by ordinals". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[PR] Spark3.4,3.5,Api,Hive: Fix using NullType in View. [iceberg]

2024-12-08 Thread via GitHub
Ppei-Wang opened a new pull request, #11728: URL: https://github.com/apache/iceberg/pull/11728 This PR fixed using nulltype in views returns an exception. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[PR] Spark: Remove deprecated SparkAppenderFactory [iceberg]

2024-12-08 Thread via GitHub
ajantha-bhat opened a new pull request, #11727: URL: https://github.com/apache/iceberg/pull/11727 Follow up from my previous PR: https://github.com/apache/iceberg/pull/11076#issuecomment-2378384268 `SparkAppenderFactory` is not used in production code. -- This is an automated messa

Re: [PR] Core, Flink, Spark: Drop deprecated APIs scheduled for removal in 1.8.0 [iceberg]

2024-12-08 Thread via GitHub
ajantha-bhat commented on PR #11721: URL: https://github.com/apache/iceberg/pull/11721#issuecomment-2527025673 I created https://github.com/apache/iceberg/pull/11727 for `SparkAppenderFactor` now -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Kafka Connect: Add mechanisms for routing records by topic name [iceberg]

2024-12-08 Thread via GitHub
igorvoltaic commented on PR #11623: URL: https://github.com/apache/iceberg/pull/11623#issuecomment-2526891816 > I think that "basic routing" like consuming from multiple topics and "map" topic -> table could make sense. That's something I've suggested here https://github.com/apache/i

Re: [PR] chore(deps): Bump crate-ci/typos from 1.27.3 to 1.28.2 [iceberg-rust]

2024-12-08 Thread via GitHub
liurenjie1024 merged PR #767: URL: https://github.com/apache/iceberg-rust/pull/767 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Core: Try create Iceberg metadata table for Jdbc catalog in initialization [iceberg]

2024-12-08 Thread via GitHub
FANNG1 commented on PR #11427: URL: https://github.com/apache/iceberg/pull/11427#issuecomment-2526709541 will work on it recently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] chore(deps): bump crate-ci/typos from 1.27.3 to 1.28.1 [iceberg-rust]

2024-12-08 Thread via GitHub
liurenjie1024 merged PR #769: URL: https://github.com/apache/iceberg-rust/pull/769 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] chore(deps): Bump crate-ci/typos from 1.27.3 to 1.28.1 [iceberg-rust]

2024-12-08 Thread via GitHub
liurenjie1024 closed issue #749: chore(deps): Bump crate-ci/typos from 1.27.3 to 1.28.1 URL: https://github.com/apache/iceberg-rust/issues/749 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] chore(deps): bump crate-ci/typos from 1.27.3 to 1.28.1 [iceberg-rust]

2024-12-08 Thread via GitHub
liurenjie1024 commented on PR #769: URL: https://github.com/apache/iceberg-rust/pull/769#issuecomment-252573 Thanks @jonathanc-n for fixing this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] closed issue #10390: Iceberg Roadmap is 404 URL: https://github.com/apache/iceberg/issues/10390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] (AWS) Docs: List all AWS S3 properties from all language impl. [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] commented on PR #11383: URL: https://github.com/apache/iceberg/pull/11383#issuecomment-2526534293 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] (AWS) Docs: List all AWS S3 properties from all language impl. [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] closed pull request #11383: (AWS) Docs: List all AWS S3 properties from all language impl. URL: https://github.com/apache/iceberg/pull/11383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Flink-1.19: Fix the file offset mismatch when Flink reader first seek… [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] commented on PR #10567: URL: https://github.com/apache/iceberg/pull/10567#issuecomment-2526534152 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Copy iceberg table from hdfs to GCS and register table to BLMS [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] commented on issue #10389: URL: https://github.com/apache/iceberg/issues/10389#issuecomment-2526533971 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Add view support for Hadoop catalog [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] closed issue #10387: Add view support for Hadoop catalog URL: https://github.com/apache/iceberg/issues/10387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Spark 3.5: Display write metrics on SQL UI [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] commented on PR #11340: URL: https://github.com/apache/iceberg/pull/11340#issuecomment-2526534243 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Getting schema of the all tables instead of actual data [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] commented on issue #10483: URL: https://github.com/apache/iceberg/issues/10483#issuecomment-2526534120 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Relax client.credentials-provider requirements [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] commented on PR #10604: URL: https://github.com/apache/iceberg/pull/10604#issuecomment-2526534180 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] commented on issue #10390: URL: https://github.com/apache/iceberg/issues/10390#issuecomment-2526534001 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Copy iceberg table from hdfs to GCS and register table to BLMS [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] closed issue #10389: Copy iceberg table from hdfs to GCS and register table to BLMS URL: https://github.com/apache/iceberg/issues/10389 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Add view support for Hadoop catalog [iceberg]

2024-12-08 Thread via GitHub
github-actions[bot] commented on issue #10387: URL: https://github.com/apache/iceberg/issues/10387#issuecomment-2526533958 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Core: Assign fresh IDs to view schema [iceberg]

2024-12-08 Thread via GitHub
amogh-jahagirdar commented on code in PR #10253: URL: https://github.com/apache/iceberg/pull/10253#discussion_r1875107084 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1798,8 +1798,9 @@ public void createOrReplaceViewKeep

Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2024-12-08 Thread via GitHub
lidavidm commented on PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#issuecomment-2526460512 e.g. it seems we could integrate find_package and FetchContent more naturally and not rely on the pile of code Arrow uses: https://stackoverflow.com/questions/73621403/cmake-integrating-fe

Re: [I] Support dynamic overwrite [iceberg-python]

2024-12-08 Thread via GitHub
sundaresanr commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2526399105 It would be nice to support multiple partition overwrites in a single transaction => produce single snapshot -- This is an automated message from the Apache Git Service

Re: [I] Questions of using flink to update iceberg table partial columns and delete [iceberg]

2024-12-08 Thread via GitHub
pvary commented on issue #11720: URL: https://github.com/apache/iceberg/issues/11720#issuecomment-2526388257 Currently there is no native Iceberg way to partially update rows. Iceberg provides way to delete and insert rows. There is two types of delete: 1. Delete a given row from a file

Re: [I] Implement `initial-default` [iceberg-rust]

2024-12-08 Thread via GitHub
jonathanc-n commented on issue #737: URL: https://github.com/apache/iceberg-rust/issues/737#issuecomment-2526373263 May i work on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Iceberg-rust Delete support [iceberg-rust]

2024-12-08 Thread via GitHub
jonathanc-n commented on issue #735: URL: https://github.com/apache/iceberg-rust/issues/735#issuecomment-2526364557 I would also like to look into this, I will probably be working on the Strict Metrics Evaluator. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Spec: add variant type [iceberg]

2024-12-08 Thread via GitHub
aihuaxu commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1874623947 ## format/spec.md: ## @@ -178,6 +178,21 @@ A **`list`** is a collection of values with some element type. The element field A **`map`** is a collection of key-val

[PR] feat: add `DataFileWriter` test for schema and partition [iceberg-rust]

2024-12-08 Thread via GitHub
jonathanc-n opened a new pull request, #768: URL: https://github.com/apache/iceberg-rust/pull/768 Closes #726 Added `DataFileWriter` tests for schema and partition -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] table_exists error raisng 400 error code [iceberg-python]

2024-12-08 Thread via GitHub
Fokko commented on issue #1378: URL: https://github.com/apache/iceberg-python/issues/1378#issuecomment-2526175930 Hey @dongsupkim-onepredict unfortunally the `tabulario/iceberg-rest` is deprecated in favor of `apache/iceberg-rest-fixture`. You should be able to use Postgres as a backend, b

Re: [PR] Build: Delete branch automatically on PR merge [iceberg-python]

2024-12-08 Thread via GitHub
Fokko merged PR #1408: URL: https://github.com/apache/iceberg-python/pull/1408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Build: Delete branch automatically on PR merge [iceberg-python]

2024-12-08 Thread via GitHub
Fokko commented on PR #1408: URL: https://github.com/apache/iceberg-python/pull/1408#issuecomment-2526173738 I like this! @manuzhang thanks for adding this, and thanks @kevinjqliu for the review 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, ple

[PR] chore(deps): Bump crate-ci/typos from 1.27.3 to 1.28.2 [iceberg-rust]

2024-12-08 Thread via GitHub
dependabot[bot] opened a new pull request, #767: URL: https://github.com/apache/iceberg-rust/pull/767 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.27.3 to 1.28.2. Release notes Sourced from https://github.com/crate-ci/typos/releases";>crate-ci/typos's releases.

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-12-08 Thread via GitHub
jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1874771178 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean canContai

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-12-08 Thread via GitHub
jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1874771178 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean canContai

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-12-08 Thread via GitHub
jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1874771178 ## core/src/main/java/org/apache/iceberg/ManifestFilterManager.java: ## @@ -363,6 +363,10 @@ private ManifestFile filterManifest( } private boolean canContai

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.69.0 [iceberg-go]

2024-12-08 Thread via GitHub
dependabot[bot] closed pull request #219: build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.69.0 URL: https://github.com/apache/iceberg-go/pull/219 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.69.0 [iceberg-go]

2024-12-08 Thread via GitHub
dependabot[bot] commented on PR #219: URL: https://github.com/apache/iceberg-go/pull/219#issuecomment-2525543979 Superseded by #225. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.71.0 [iceberg-go]

2024-12-08 Thread via GitHub
dependabot[bot] opened a new pull request, #225: URL: https://github.com/apache/iceberg-go/pull/225 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.67.1 to 1.71.0. Commits https://github.com/aws/aws-sdk-go-v2/commit/9a211f9a8eb9923a

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/glue from 1.102.0 to 1.103.0 [iceberg-go]

2024-12-08 Thread via GitHub
dependabot[bot] opened a new pull request, #224: URL: https://github.com/apache/iceberg-go/pull/224 Bumps [github.com/aws/aws-sdk-go-v2/service/glue](https://github.com/aws/aws-sdk-go-v2) from 1.102.0 to 1.103.0. Changelog Sourced from https://github.com/aws/aws-sdk-go-v2/blob/ser

[PR] build(deps): bump golang.org/x/sync from 0.9.0 to 0.10.0 [iceberg-go]

2024-12-08 Thread via GitHub
dependabot[bot] opened a new pull request, #223: URL: https://github.com/apache/iceberg-go/pull/223 Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.9.0 to 0.10.0. Commits https://github.com/golang/sync/commit/913fb63af28f446cd10c684ee847b5606cf328f7";>913fb63