Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on PR #787: URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2146758073 Hmm, it's my first time to see this error. I've merged a PR that bumps `duckdb` to 1.0.0: #793. Hope that can fix the issue -- This is an automated message from the Apache Git Servi

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
chinmay-bhat commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625482510 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

Re: [PR] Cache Manifest files [iceberg-python]

2024-06-04 Thread via GitHub
HonahX commented on code in PR #787: URL: https://github.com/apache/iceberg-python/pull/787#discussion_r1625475590 ## pyiceberg/table/snapshots.py: ## @@ -247,9 +248,12 @@ def __str__(self) -> str: result_str = f"{operation}id={self.snapshot_id}{parent_id}{schema_id}"

[PR] Core: Use TestTemplate instead of Test annotation in TestPartitionSpecParser [iceberg]

2024-06-04 Thread via GitHub
nastra opened a new pull request, #10435: URL: https://github.com/apache/iceberg/pull/10435 I've noticed that the `formatVersion` parameter wasn't properly initialized for this test class because the tests weren't annotated with `@TestTemplate` -- This is an automated message from the Apa

Re: [PR] Open-API: TableRequirements subclasses should inherit 'type' property [iceberg]

2024-06-04 Thread via GitHub
Fokko commented on code in PR #10434: URL: https://github.com/apache/iceberg/pull/10434#discussion_r1625486376 ## open-api/rest-catalog-open-api.py: ## @@ -361,23 +361,29 @@ class RemovePartitionStatisticsUpdate(BaseUpdate): class TableRequirement(BaseModel): -type: str

Re: [PR] Core: Fix create v1 table on REST Catalog [iceberg]

2024-06-04 Thread via GitHub
nastra commented on code in PR #10369: URL: https://github.com/apache/iceberg/pull/10369#discussion_r1625500488 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -991,7 +991,7 @@ public Builder assignUUID(String newUuid) { // it is only safe to set the fo

Re: [I] New Types: JSONB / JSON [iceberg]

2024-06-04 Thread via GitHub
Fokko closed issue #9066: New Types: JSONB / JSON URL: https://github.com/apache/iceberg/issues/9066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: iss

Re: [I] Questions on some requirement in view spec [iceberg]

2024-06-04 Thread via GitHub
ajantha-bhat commented on issue #10410: URL: https://github.com/apache/iceberg/issues/10410#issuecomment-2146929374 @nastra: 1. Thanks for the info on the `summary`, I didn't know we had required `operation` before. Agree that making it optional now is not a good idea. 2. Trin

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-06-04 Thread via GitHub
ajantha-bhat commented on PR #10176: URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2147007461 @RussellSpitzer, @szehon-ho: It seems Anton is on holiday for 2-3 weeks. Is it possible for any of you to support for review? cc: @jbonofre -- This is an automated mes

[I] why spark ddl rename iceberg table name not change location? does it matter? [iceberg]

2024-06-04 Thread via GitHub
madeirak opened a new issue, #10436: URL: https://github.com/apache/iceberg/issues/10436 ### Query engine _No response_ ### Question ![image](https://github.com/apache/iceberg/assets/23215922/1f8e3989-90de-4c50-86bc-5e3edb841f19) If we create a table with the same nam

Re: [PR] Core: Use TestTemplate instead of Test annotation in TestPartitionSpecParser/Info [iceberg]

2024-06-04 Thread via GitHub
nastra merged PR #10435: URL: https://github.com/apache/iceberg/pull/10435 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] REST: disallow overriding "credential" in table sessions [iceberg]

2024-06-04 Thread via GitHub
adutra commented on PR #10345: URL: https://github.com/apache/iceberg/pull/10345#issuecomment-2147204756 @nastra could you have a look at this one as well please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-06-04 Thread via GitHub
adutra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2147206036 @nastra or @amogh-jahagirdar is it possible for you to have a another look here please? Thanks πŸ™ -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Docs: Point links in metrics-reporting.md to GitHub Java source [iceberg]

2024-06-04 Thread via GitHub
nastra merged PR #10397: URL: https://github.com/apache/iceberg/pull/10397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[I] Iceberg could support Hive1.1.0? [iceberg]

2024-06-04 Thread via GitHub
davidyuan1223 opened a new issue, #10437: URL: https://github.com/apache/iceberg/issues/10437 ### Query engine spark-3.1.2 kyuubi-1.7.0 iceberg-spark-3.1.2_1.3.1 ### Question config ``` spark.authenticate=false spark.sql.hive.metastore.version=1.1.0 spar

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625817607 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MonitorSource.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625822325 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/SingleThreadedIteratorSource.java: ## @@ -0,0 +1,195 @@ +/* + * Licensed to the Apache

Re: [PR] kafka-connect: correct partition transform support [iceberg]

2024-06-04 Thread via GitHub
ajantha-bhat closed pull request #10185: kafka-connect: correct partition transform support URL: https://github.com/apache/iceberg/pull/10185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] kafka-connect: correct partition transform support [iceberg]

2024-06-04 Thread via GitHub
ajantha-bhat commented on PR #10185: URL: https://github.com/apache/iceberg/pull/10185#issuecomment-2147282348 closing this as many people don't see it as a problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625828013 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestMonitorSource.java: ## @@ -0,0 +1,344 @@ +/* + * Licensed to the Apache Software Fo

Re: [I] Iceberg could support Hive1.1.0? [iceberg]

2024-06-04 Thread via GitHub
ajantha-bhat commented on issue #10437: URL: https://github.com/apache/iceberg/issues/10437#issuecomment-2147295539 Isn't Hive 1.1 is pretty ancient? > 11 April 2024: EOL for release 1.x line The Apache Hive Community has voted to declare the 1.x release line as End of Life (EOL).

Re: [I] Iceberg could support Hive1.1.0? [iceberg]

2024-06-04 Thread via GitHub
davidyuan1223 commented on issue #10437: URL: https://github.com/apache/iceberg/issues/10437#issuecomment-2147310089 > Isn't Hive 1.1 is pretty ancient? > > > 11 April 2024: EOL for release 1.x line > > The Apache Hive Community has voted to declare the 1.x release line as End of L

Re: [I] Iceberg could support Hive1.1.0? [iceberg]

2024-06-04 Thread via GitHub
davidyuan1223 closed issue #10437: Iceberg could support Hive1.1.0? URL: https://github.com/apache/iceberg/issues/10437 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Spark: Add CopyTable spark action [iceberg]

2024-06-04 Thread via GitHub
ajantha-bhat commented on PR #10024: URL: https://github.com/apache/iceberg/pull/10024#issuecomment-2147313645 @laithalzyoud : Are you planning to address the comments on this? This feature is definitely useful. If not, I would like to take it up. -- This is an automated message from

Re: [I] Make ManifestEntry and ManifestReader.liveEntries() as public [iceberg]

2024-06-04 Thread via GitHub
ajantha-bhat commented on issue #10425: URL: https://github.com/apache/iceberg/issues/10425#issuecomment-2147315645 Another PR that requires it to be public: https://github.com/apache/iceberg/pull/10024/files#diff-2cb5b09259db131b4d85059ab8b42a9f3bdfd233bd565539cd78c7f485b44b5dR27 -- This

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625855246 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MonitorSource.java: ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625856014 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MonitorSource.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625860108 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MonitorSource.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625861166 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/SingleThreadedIteratorSource.java: ## @@ -0,0 +1,195 @@ +/* + * Licensed to the Apache

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625862231 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/SingleThreadedIteratorSource.java: ## @@ -0,0 +1,195 @@ +/* + * Licensed to the Apache

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625864926 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/SingleThreadedIteratorSource.java: ## @@ -0,0 +1,195 @@ +/* + * Licensed to the Apache

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625866835 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/CollectingSink.java: ## @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625867192 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/FlinkSqlExtension.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625868760 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestMonitorSource.java: ## @@ -0,0 +1,344 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1625869218 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestMonitorSource.java: ## @@ -0,0 +1,344 @@ +/* + * Licensed to the Apache Software Fo

Re: [I] Flink sink writes duplicate data in upsert mode [iceberg]

2024-06-04 Thread via GitHub
pvary commented on issue #10431: URL: https://github.com/apache/iceberg/issues/10431#issuecomment-2147373677 @zhongqishang: Do you see anything more in the logs? Exceptions/retries, or something like this? Also, I don't fully understand your statement here: > I think it is because

Re: [PR] REST: disallow overriding "credential" in table sessions [iceberg]

2024-06-04 Thread via GitHub
nastra commented on code in PR #10345: URL: https://github.com/apache/iceberg/pull/10345#discussion_r1625998903 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -915,7 +923,13 @@ private FileIO tableFileIO(SessionContext context, Map config) { }

Re: [PR] docs: Broken link from catalog concepts to configuration page [iceberg]

2024-06-04 Thread via GitHub
Fokko commented on code in PR #10438: URL: https://github.com/apache/iceberg/pull/10438#discussion_r1626100232 ## site/docs/concepts/catalog.md: ## @@ -26,7 +26,7 @@ You may think of Iceberg as a format for managing data in a single table, but th The first step when using an

[PR] docs: Broken link from catalog concepts to configuration page [iceberg]

2024-06-04 Thread via GitHub
Fokko opened a new pull request, #10438: URL: https://github.com/apache/iceberg/pull/10438 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [PR] Open-API: TableRequirements subclasses should inherit 'type' property [iceberg]

2024-06-04 Thread via GitHub
flyrain commented on code in PR #10434: URL: https://github.com/apache/iceberg/pull/10434#discussion_r1626133536 ## open-api/rest-catalog-open-api.py: ## @@ -361,23 +361,29 @@ class RemovePartitionStatisticsUpdate(BaseUpdate): class TableRequirement(BaseModel): -type: s

Re: [PR] Open-API: TableRequirements subclasses should inherit 'type' property [iceberg]

2024-06-04 Thread via GitHub
flyrain commented on code in PR #10434: URL: https://github.com/apache/iceberg/pull/10434#discussion_r1626139854 ## open-api/rest-catalog-open-api.py: ## @@ -361,23 +361,29 @@ class RemovePartitionStatisticsUpdate(BaseUpdate): class TableRequirement(BaseModel): -type: s

[PR] docs: Backport fixes for broken links [iceberg]

2024-06-04 Thread via GitHub
Fokko opened a new pull request, #10439: URL: https://github.com/apache/iceberg/pull/10439 My IDEA autoformatted the tables, but there is no git history in the `docs/` branch anyway. Found this using the link checker: ``` cat link_warnings.csv LinkChecker 10.4.0 Start checkin

Re: [PR] Core: Use correct headers for Multi-Table commits [iceberg]

2024-06-04 Thread via GitHub
nastra closed pull request #9523: Core: Use correct headers for Multi-Table commits URL: https://github.com/apache/iceberg/pull/9523 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Flink: refactor sink shuffling statistics collection [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10331: URL: https://github.com/apache/iceberg/pull/10331#discussion_r1626211093 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SketchUtil.java: ## @@ -0,0 +1,148 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[PR] Core, Parquet: Don't write column sizes when metrics mode is None [iceberg]

2024-06-04 Thread via GitHub
amogh-jahagirdar opened a new pull request, #10440: URL: https://github.com/apache/iceberg/pull/10440 Currently, the Iceberg ParquetWriter will write out column sizes even when metrics are disabled. This should not be the case since column sizes are optional in the spec and we should respec

Re: [PR] Flink: refactor sink shuffling statistics collection [iceberg]

2024-06-04 Thread via GitHub
stevenzwu commented on code in PR #10331: URL: https://github.com/apache/iceberg/pull/10331#discussion_r1626238602 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SketchUtil.java: ## @@ -0,0 +1,148 @@ +/* + * Licensed to the Apache Software Foundation (A

[PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
huaxingao opened a new pull request, #10442: URL: https://github.com/apache/iceberg/pull/10442 ``` sql("CREATE VIEW %s AS SELECT 1 AS id", "test"); sql("SHOW VIEWS") assertion failed java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:208)

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
stevenzwu commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1626255534 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MonitorSource.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-06-04 Thread via GitHub
epgif commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2147874958 Friendly ping @nastra @rdblue :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
stevenzwu commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1626263978 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/FlinkSqlExtension.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] Core, Parquet: Don't write column sizes when metrics mode is None [iceberg]

2024-06-04 Thread via GitHub
amogh-jahagirdar commented on PR #10440: URL: https://github.com/apache/iceberg/pull/10440#issuecomment-2147891419 Seems like ORC and possibly Avro writers also need to be updated, the ORC tests are failing -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626273063 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/WriteResultSerializer.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
stevenzwu commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1626274743 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestMonitorSource.java: ## @@ -0,0 +1,344 @@ +/* + * Licensed to the Apache Softwar

Re: [I] Broken links in Spark Writes documentation [iceberg]

2024-06-04 Thread via GitHub
manuzhang commented on issue #10409: URL: https://github.com/apache/iceberg/issues/10409#issuecomment-2147920793 @gphilipp please check again whether these links have been fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
stevenzwu commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1626255534 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MonitorSource.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626290543 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergCommitter.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
huaxingao commented on PR #10442: URL: https://github.com/apache/iceberg/pull/10442#issuecomment-2147946347 cc @nastra @szehon-ho @flyrain -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] docs: Backport fixes for broken links [iceberg]

2024-06-04 Thread via GitHub
manuzhang commented on PR #10439: URL: https://github.com/apache/iceberg/pull/10439#issuecomment-2147979503 @Fokko which command do you run? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
szehon-ho commented on code in PR #10442: URL: https://github.com/apache/iceberg/pull/10442#discussion_r1626346354 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -66,10 +66,13 @@ case class RewriteViewCommand

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
huaxingao commented on code in PR #10442: URL: https://github.com/apache/iceberg/pull/10442#discussion_r1626362527 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -66,10 +66,13 @@ case class RewriteViewCommand

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626390005 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergCommitter.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626390005 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergCommitter.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626393891 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergCommitter.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Flink: refactor sink shuffling statistics collection [iceberg]

2024-06-04 Thread via GitHub
stevenzwu commented on code in PR #10331: URL: https://github.com/apache/iceberg/pull/10331#discussion_r1626398331 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/AggregatedStatisticsTracker.java: ## @@ -30,71 +42,99 @@ * {@link AggregatedStatistics} r

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626400067 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergCommitter.java: ## @@ -0,0 +1,434 @@ +/* + * Licensed to the Apache Software Foun

Re: [I] Broken links in Spark Writes documentation [iceberg]

2024-06-04 Thread via GitHub
gphilipp commented on issue #10409: URL: https://github.com/apache/iceberg/issues/10409#issuecomment-2148099388 It’s fixed thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[I] Rest Catalog - Bug in list namespaces. Namespaces not underneath the parent namespace are returned [iceberg]

2024-06-04 Thread via GitHub
jurossiar opened a new issue, #10443: URL: https://github.com/apache/iceberg/issues/10443 ### Apache Iceberg version 1.5.0 ### Query engine None ### Please describe the bug 🐞 We have the rest catalog configured with a Postgres DB. In some cases, where nam

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626416286 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/WriteResultSerializer.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-06-04 Thread via GitHub
syun64 commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1626436716 ## pyiceberg/table/__init__.py: ## @@ -2751,10 +2824,12 @@ def _parquet_files_to_data_files(table_metadata: TableMetadata, file_paths: List class _MergingSnaps

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1626475908 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/FlinkSqlExtension.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-06-04 Thread via GitHub
syun64 commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1626456000 ## pyiceberg/table/__init__.py: ## @@ -3735,3 +3894,92 @@ def _determine_partitions(spec: PartitionSpec, schema: Schema, arrow_table: pa.T table_partitions: L

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
viirya commented on code in PR #10442: URL: https://github.com/apache/iceberg/pull/10442#discussion_r1626506489 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -66,10 +66,13 @@ case class RewriteViewCommands(s

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
viirya commented on code in PR #10442: URL: https://github.com/apache/iceberg/pull/10442#discussion_r1626512474 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -66,10 +66,13 @@ case class RewriteViewCommands(s

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1626516787 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestMonitorSource.java: ## @@ -0,0 +1,344 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
viirya commented on code in PR #10442: URL: https://github.com/apache/iceberg/pull/10442#discussion_r1626517298 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -66,10 +66,13 @@ case class RewriteViewCommands(s

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1626517533 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MonitorSource.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: refactor sink shuffling statistics collection [iceberg]

2024-06-04 Thread via GitHub
pvary commented on code in PR #10331: URL: https://github.com/apache/iceberg/pull/10331#discussion_r1626530485 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/AggregatedStatisticsTracker.java: ## @@ -30,104 +42,225 @@ * {@link AggregatedStatistics} rec

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
szehon-ho commented on PR #10442: URL: https://github.com/apache/iceberg/pull/10442#issuecomment-2148340152 @nastra fyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626559065 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergWriteAggregator.java: ## @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626559065 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergWriteAggregator.java: ## @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] docs: Backport fixes for broken links [iceberg]

2024-06-04 Thread via GitHub
Fokko commented on PR #10439: URL: https://github.com/apache/iceberg/pull/10439#issuecomment-2148367000 @manuzhang Slightly modified version of the [links checker](https://github.com/apache/iceberg/blob/main/site/README.md#validate-links) in the docs: ```sh linkchecker http://loca

Re: [PR] Open-API: TableRequirements subclasses should inherit 'type' property [iceberg]

2024-06-04 Thread via GitHub
Fokko commented on code in PR #10434: URL: https://github.com/apache/iceberg/pull/10434#discussion_r1626572332 ## open-api/rest-catalog-open-api.py: ## @@ -361,23 +361,29 @@ class RemovePartitionStatisticsUpdate(BaseUpdate): class TableRequirement(BaseModel): -type: str

Re: [PR] Open-API: TableRequirements subclasses should inherit 'type' property [iceberg]

2024-06-04 Thread via GitHub
Fokko commented on code in PR #10434: URL: https://github.com/apache/iceberg/pull/10434#discussion_r1626572332 ## open-api/rest-catalog-open-api.py: ## @@ -361,23 +361,29 @@ class RemovePartitionStatisticsUpdate(BaseUpdate): class TableRequirement(BaseModel): -type: str

[PR] [WIP]Spark: Asynchronous Spark Micro Batch Planner [iceberg]

2024-06-04 Thread via GitHub
hiloboy0119 opened a new pull request, #10444: URL: https://github.com/apache/iceberg/pull/10444 This PR adds a Spark micro-batch planner that reads table snapshots asynchronously in the background and fills a queue of files which can be consumed to compute latestOffset. This allows the me

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
viirya commented on code in PR #10442: URL: https://github.com/apache/iceberg/pull/10442#discussion_r1626612224 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -66,10 +66,13 @@ case class RewriteViewCommands(s

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
stevenzwu commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626642809 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergWriteAggregator.java: ## @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-06-04 Thread via GitHub
stevenzwu commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1626655524 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/FlinkSqlExtension.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-06-04 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1626656514 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergWriteAggregator.java: ## @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] Core, Parquet, ORC: Don't write column sizes when metrics mode is None [iceberg]

2024-06-04 Thread via GitHub
amogh-jahagirdar commented on PR #10440: URL: https://github.com/apache/iceberg/pull/10440#issuecomment-2148497567 Avro didn't need any changes which makes sense it's row oriented anyways. Fixed ORC. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] support python 3.12 [iceberg-python]

2024-06-04 Thread via GitHub
pdpark commented on PR #254: URL: https://github.com/apache/iceberg-python/pull/254#issuecomment-2148516083 FYI: [Ray issue 45477](https://github.com/ray-project/ray/issues/45477) was recently completed. -- This is an automated message from the Apache Git Service. To respond to the messag

[PR] Bump getdaft from 0.2.25 to 0.2.26 [iceberg-python]

2024-06-04 Thread via GitHub
dependabot[bot] opened a new pull request, #796: URL: https://github.com/apache/iceberg-python/pull/796 Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.2.25 to 0.2.26. Release notes Sourced from https://github.com/Eventual-Inc/Daft/releases";>getdaft's releases.

[PR] Repair manifest action [iceberg]

2024-06-04 Thread via GitHub
tabmatfournier opened a new pull request, #10445: URL: https://github.com/apache/iceberg/pull/10445 introduces a spark action to tackle two corrupt manifest issues: - duplicate files existing within the same manifest, or between manifests - missing files referenced by a manifest (co

Re: [PR] Repair manifest action [iceberg]

2024-06-04 Thread via GitHub
szehon-ho commented on PR #10445: URL: https://github.com/apache/iceberg/pull/10445#issuecomment-2148582001 It looks similar to my attempt in https://github.com/apache/iceberg/pull/2608 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] SHOW VIEWS failed with AssertionError [iceberg]

2024-06-04 Thread via GitHub
szehon-ho commented on PR #10442: URL: https://github.com/apache/iceberg/pull/10442#issuecomment-2148582710 Ill commit this tomorrow if no further comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Core, Parquet, ORC: Don't write column sizes when metrics mode is None [iceberg]

2024-06-04 Thread via GitHub
amogh-jahagirdar commented on PR #10440: URL: https://github.com/apache/iceberg/pull/10440#issuecomment-2148603104 @szehon-ho Yeah at least from my analysis nothing is relying on this internally in the library (which makes sense it is optional as per the spec). -- This is an automated mes

Re: [PR] Repair manifest action [iceberg]

2024-06-04 Thread via GitHub
tabmatfournier commented on PR #10445: URL: https://github.com/apache/iceberg/pull/10445#issuecomment-2148609656 > It looks similar to my attempt in #2608 Similar. Didn't realize that was up. We tackle different corrupted issues! -- This is an automated message from the Apache Gi

  1   2   >