[I] Docs: Fix MkDocs ASF nav links [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev opened a new issue, #8965: URL: https://github.com/apache/iceberg/issues/8965 There's an issue with how mkdocs renders. https://github.com/squidfunk/mkdocs-material/issues/868 The solution is to provide an [override](https://squidfunk.github.io/mkdocs-material/cus

[I] Why are updateSchema and UpdatePartitionSpec commit not retried? [iceberg]

2023-10-31 Thread via GitHub
waltczhang opened a new issue, #8964: URL: https://github.com/apache/iceberg/issues/8964 ### Query engine _No response_ ### Question Theoretically, it is possible for updateSchema and UpdatePartitionSpec to fail during the commit process. I want to know why retries have

[PR] Spark 3.4: Remove usage of AssertHelpers [iceberg]

2023-10-31 Thread via GitHub
coded9 opened a new pull request, #8963: URL: https://github.com/apache/iceberg/pull/8963 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

2023-10-31 Thread via GitHub
wypoon commented on code in PR #8805: URL: https://github.com/apache/iceberg/pull/8805#discussion_r1378364590 ## core/src/main/java/org/apache/iceberg/deletes/Deletes.java: ## @@ -127,14 +130,25 @@ public static StructLikeSet toEqualitySet( public static PositionDeleteInde

Re: [PR] feat: Implement load table api. [iceberg-rust]

2023-10-31 Thread via GitHub
liurenjie1024 commented on code in PR #89: URL: https://github.com/apache/iceberg-rust/pull/89#discussion_r1378337379 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -109,50 +110,139 @@ pub struct TableMetadata { } impl TableMetadata { +/// Returns format version of

Re: [PR] feat: Implement load table api. [iceberg-rust]

2023-10-31 Thread via GitHub
liurenjie1024 commented on code in PR #89: URL: https://github.com/apache/iceberg-rust/pull/89#discussion_r1378335610 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -109,50 +110,139 @@ pub struct TableMetadata { } impl TableMetadata { +/// Returns format version of

Re: [PR] feat: Implement load table api. [iceberg-rust]

2023-10-31 Thread via GitHub
liurenjie1024 commented on code in PR #89: URL: https://github.com/apache/iceberg-rust/pull/89#discussion_r1378335162 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -109,50 +110,139 @@ pub struct TableMetadata { } impl TableMetadata { +/// Returns format version of

Re: [PR] feat: Implement load table api. [iceberg-rust]

2023-10-31 Thread via GitHub
liurenjie1024 commented on code in PR #89: URL: https://github.com/apache/iceberg-rust/pull/89#discussion_r1378334579 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -109,50 +110,139 @@ pub struct TableMetadata { } impl TableMetadata { +/// Returns format version of

Re: [PR] feat: Implement load table api. [iceberg-rust]

2023-10-31 Thread via GitHub
liurenjie1024 commented on code in PR #89: URL: https://github.com/apache/iceberg-rust/pull/89#discussion_r1378325205 ## crates/catalog/rest/src/catalog.rs: ## @@ -312,11 +316,43 @@ impl Catalog for RestCatalog { } /// Load table from the catalog. -async fn load_

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313906 ## site/README.md: ## @@ -125,17 +124,16 @@ mkdocs serve ## Release process Deploying a version of the docs is a two step process: - 1. ~~Cut a new release fr

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313906 ## site/README.md: ## @@ -125,17 +124,16 @@ mkdocs serve ## Release process Deploying a version of the docs is a two step process: - 1. ~~Cut a new release fr

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313906 ## site/README.md: ## @@ -125,17 +124,16 @@ mkdocs serve ## Release process Deploying a version of the docs is a two step process: - 1. ~~Cut a new release fr

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313906 ## site/README.md: ## @@ -125,17 +124,16 @@ mkdocs serve ## Release process Deploying a version of the docs is a two step process: - 1. ~~Cut a new release fr

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378313362 ## site/README.md: ## @@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar > This build process is currently missi

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378304225 ## site/README.md: ## @@ -125,17 +124,16 @@ mkdocs serve ## Release process Deploying a version of the docs is a two step process: - 1. ~~Cut a new release fr

Re: [PR] Spark 3.5: Honor Spark conf spark.sql.files.maxPartitionBytes in read split [iceberg]

2023-10-31 Thread via GitHub
aokolnychyi commented on PR #8922: URL: https://github.com/apache/iceberg/pull/8922#issuecomment-1788252194 I am not sure it is a good idea too, we always avoided respecting Spark configs for the built-in file sources. Iceberg split planning is different. -- This is an automated message

[PR] Core, Spark: Avoid manifest copies when importing data to V2 tables [iceberg]

2023-10-31 Thread via GitHub
aokolnychyi opened a new pull request, #8962: URL: https://github.com/apache/iceberg/pull/8962 This PR extends the idea from #8928 to `FastAppend` and `MergeAppend`, which are used in data imports and migration. -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Partitions metadata table returns inconsistent results size between 1.3.1 and 1.4.1 [iceberg]

2023-10-31 Thread via GitHub
dramaticlly commented on issue #8951: URL: https://github.com/apache/iceberg/issues/8951#issuecomment-1788241135 Thank you @zhangbutao , looks like this was indeed introduced by #7581, where we changed to scan manifest entry instead of files. I am able to reproduce the problem using provide

[PR] Remove outdated python autoformating command from doc [iceberg]

2023-10-31 Thread via GitHub
hussein-awala opened a new pull request, #8961: URL: https://github.com/apache/iceberg/pull/8961 This PR removes outdated Python autoformatting command from the site doc, where tox was replaced by pre-commit by #4811. I'm preparing a new contribution guideline for iceberg-python, so n

Re: [I] API: StructProjection returns a null Projection object when the nested struct is null [iceberg]

2023-10-31 Thread via GitHub
github-actions[bot] commented on issue #7507: URL: https://github.com/apache/iceberg/issues/7507#issuecomment-1788203155 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] delete files with spark sql failed [iceberg]

2023-10-31 Thread via GitHub
github-actions[bot] commented on issue #7522: URL: https://github.com/apache/iceberg/issues/7522#issuecomment-1788203108 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark 3.3: drop_namespace with CASCADE support [iceberg]

2023-10-31 Thread via GitHub
vinitamaloo-asu commented on PR #7275: URL: https://github.com/apache/iceberg/pull/7275#issuecomment-1788177314 @abmo-x any when do you plan to merge this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-31 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1378254490 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieUtil.java: ## @@ -165,4 +180,77 @@ public static TableMetadata updateTableMetadataWithNessieSpecificProperties(

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-31 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1378253133 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-31 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1378251837 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -400,8 +400,15 @@ public void replaceTableViaTransactionThatAlreadyExistsAsView() {

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-31 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1378251837 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -400,8 +400,15 @@ public void replaceTableViaTransactionThatAlreadyExistsAsView() {

[PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-10-31 Thread via GitHub
jacobmarble opened a new pull request, #8960: URL: https://github.com/apache/iceberg/pull/8960 Helps #8657 This change adds field `ChronoUnit unit` to `TimestampType`, such that `TimestampType` now represents four specified types: - `timestamp` (existing) - `timestamptz` (existi

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1378017052 ## site/README.md: ## @@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar > This build process is currently missi

Re: [PR] Spark: Fix usage of staging location when optimizing metadata [iceberg]

2023-10-31 Thread via GitHub
aokolnychyi commented on code in PR #8959: URL: https://github.com/apache/iceberg/pull/8959#discussion_r1377997350 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -133,15 +143,17 @@ public RewriteManifestsSparkAction re

Re: [PR] Spark: Avoid extra copies of manifests while optimizing V2 tables [iceberg]

2023-10-31 Thread via GitHub
aokolnychyi commented on code in PR #8928: URL: https://github.com/apache/iceberg/pull/8928#discussion_r1377994540 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -337,7 +337,7 @@ private void replaceManifests( ad

[PR] Spark: Fix usage of staging location when optimizing metadata [iceberg]

2023-10-31 Thread via GitHub
aokolnychyi opened a new pull request, #8959: URL: https://github.com/apache/iceberg/pull/8959 This PR fixes our actions for rewriting manifests to use the provided staging location only if the manifests are staged. If snapshot ID inheritance is enabled (true for all V2 tables), the manifes

Re: [PR] Spark: Use SerializableTableWithSize when optimizing metadata [iceberg]

2023-10-31 Thread via GitHub
aokolnychyi merged PR #8957: URL: https://github.com/apache/iceberg/pull/8957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark: Use SerializableTableWithSize when optimizing metadata [iceberg]

2023-10-31 Thread via GitHub
aokolnychyi commented on PR #8957: URL: https://github.com/apache/iceberg/pull/8957#issuecomment-1787717661 Thank you, @singhpk234 @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Spark: Avoid extra copies of manifests while optimizing V2 tables [iceberg]

2023-10-31 Thread via GitHub
aokolnychyi merged PR #8928: URL: https://github.com/apache/iceberg/pull/8928 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.18.44 to 1.19.1 [iceberg-go]

2023-10-31 Thread via GitHub
nastra merged PR #26: URL: https://github.com/apache/iceberg-go/pull/26 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] API: add nanosecond variant of Timestamp type [iceberg]

2023-10-31 Thread via GitHub
jacobmarble closed pull request #8658: API: add nanosecond variant of Timestamp type URL: https://github.com/apache/iceberg/pull/8658 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.40.1 to 1.40.2 [iceberg-go]

2023-10-31 Thread via GitHub
nastra merged PR #22: URL: https://github.com/apache/iceberg-go/pull/22 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] chore(deps): Update typed-builder requirement from ^0.17 to ^0.18 [iceberg-rust]

2023-10-31 Thread via GitHub
Fokko merged PR #87: URL: https://github.com/apache/iceberg-rust/pull/87 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Partitions metadata table returns inconsistent results size between 1.3.1 and 1.4.1 [iceberg]

2023-10-31 Thread via GitHub
dramaticlly commented on issue #8951: URL: https://github.com/apache/iceberg/issues/8951#issuecomment-1787562193 I'll take a look and try to reproduce the problem today -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Docs: Document UNORDERED for spark write [iceberg]

2023-10-31 Thread via GitHub
nastra merged PR #8958: URL: https://github.com/apache/iceberg/pull/8958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.13.42 to 1.13.43 [iceberg-go]

2023-10-31 Thread via GitHub
nastra merged PR #21: URL: https://github.com/apache/iceberg-go/pull/21 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Spec: add nanosecond timestamp types [iceberg]

2023-10-31 Thread via GitHub
rdblue commented on PR #8683: URL: https://github.com/apache/iceberg/pull/8683#issuecomment-1787544377 @jacobmarble, I updated the table to be more clear about when types can be used and merge this. Thanks! -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Spec: add nanosecond timestamp types [iceberg]

2023-10-31 Thread via GitHub
rdblue merged PR #8683: URL: https://github.com/apache/iceberg/pull/8683 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.1 to 1.21.2 [iceberg-go]

2023-10-31 Thread via GitHub
nastra merged PR #20: URL: https://github.com/apache/iceberg-go/pull/20 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] build(deps): bump golang.org/x/net from 0.15.0 to 0.17.0 [iceberg-go]

2023-10-31 Thread via GitHub
nastra merged PR #18: URL: https://github.com/apache/iceberg-go/pull/18 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Disable merge commit [iceberg-go]

2023-10-31 Thread via GitHub
nastra merged PR #17: URL: https://github.com/apache/iceberg-go/pull/17 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] Partitions metadata table returns inconsistent results size between 1.3.1 and 1.4.1 [iceberg]

2023-10-31 Thread via GitHub
zhangbutao commented on issue #8951: URL: https://github.com/apache/iceberg/issues/8951#issuecomment-1787442557 I want to know if this is Iceberg bug or not? Or this issue should be handled by engine/user side? Hope to get a feedback. Thanks in advance. -- This is an automated message

Re: [I] Duplicate file name in Iceberg's metadata [iceberg]

2023-10-31 Thread via GitHub
Fokko commented on issue #8953: URL: https://github.com/apache/iceberg/issues/8953#issuecomment-1787411953 Slack conversation for reference that provides some more interesting details: https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1698676018510089 -- This is an automated message

Re: [PR] FlinkV1.16 : Remove usage of AssertHelpers [iceberg]

2023-10-31 Thread via GitHub
nastra merged PR #8946: URL: https://github.com/apache/iceberg/pull/8946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-31 Thread via GitHub
nastra merged PR #8945: URL: https://github.com/apache/iceberg/pull/8945 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-10-31 Thread via GitHub
cccs-jc commented on issue #8902: URL: https://github.com/apache/iceberg/issues/8902#issuecomment-1787207952 @singhpk234 I have a fix. Unit tested and ran it on our live data. I will soon create a PR for you to review. -- This is an automated message from the Apache Git Service. To

Re: [PR] Docs: Document UNORDERED for spark write [iceberg]

2023-10-31 Thread via GitHub
bitsondatadev commented on PR #8958: URL: https://github.com/apache/iceberg/pull/8958#issuecomment-1787193669 > @bitsondatadev: Should I also modify the same doc in `site` folder or later you will sync from docs to site folder? Continue with the same docs folder I will sync any chang

Re: [PR] Docs: Document UNORDERED for spark write [iceberg]

2023-10-31 Thread via GitHub
ajantha-bhat commented on PR #8958: URL: https://github.com/apache/iceberg/pull/8958#issuecomment-1787119657 @bitsondatadev: Should I also modify the same doc in `site` folder or later you will sync from docs to site folder? -- This is an automated message from the Apache Git Service. To

Re: [PR] Docs: Document UNORDERED for spark write [iceberg]

2023-10-31 Thread via GitHub
ajantha-bhat commented on PR #8958: URL: https://github.com/apache/iceberg/pull/8958#issuecomment-1787117950 cc: @RussellSpitzer, @puchengy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[PR] Docs: Document UNORDERED for spark write [iceberg]

2023-10-31 Thread via GitHub
ajantha-bhat opened a new pull request, #8958: URL: https://github.com/apache/iceberg/pull/8958 We have testcase in https://github.com/apache/iceberg/blob/b9a4478b0f8f5eeae553eb3900b08a7a08dbdb40/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSetWriteDistri

Re: [I] Support UNSET of sortOrder from the SQL [iceberg]

2023-10-31 Thread via GitHub
ajantha-bhat commented on issue #5071: URL: https://github.com/apache/iceberg/issues/5071#issuecomment-1787065904 There is a test case also https://github.com/apache/iceberg/blob/b9a4478b0f8f5eeae553eb3900b08a7a08dbdb40/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/ex

Re: [I] Support UNSET of sortOrder from the SQL [iceberg]

2023-10-31 Thread via GitHub
ajantha-bhat closed issue #5071: Support UNSET of sortOrder from the SQL URL: https://github.com/apache/iceberg/issues/5071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Support UNSET of sortOrder from the SQL [iceberg]

2023-10-31 Thread via GitHub
ajantha-bhat commented on issue #5071: URL: https://github.com/apache/iceberg/issues/5071#issuecomment-1787059232 > Isn't this supported via UNORDERED or did that not get into OSS yet? I saw the code, looks like syntax is there but no where documented or have testcase. I will add

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-31 Thread via GitHub
nastra commented on PR #8725: URL: https://github.com/apache/iceberg/pull/8725#issuecomment-1787057228 Closing this one as it has been superseded by https://github.com/apache/iceberg/pull/8804 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-31 Thread via GitHub
nastra closed pull request #8725: Thread.sleep() method is replaced with Awaitility URL: https://github.com/apache/iceberg/pull/8725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] feat: Implement load table api. [iceberg-rust]

2023-10-31 Thread via GitHub
Xuanwo commented on code in PR #89: URL: https://github.com/apache/iceberg-rust/pull/89#discussion_r1377416089 ## crates/catalog/rest/src/catalog.rs: ## @@ -312,11 +316,43 @@ impl Catalog for RestCatalog { } /// Load table from the catalog. -async fn load_table(&

[I] Support UNSET of sortOrder from the SQL [iceberg]

2023-10-31 Thread via GitHub
ajantha-bhat opened a new issue, #5071: URL: https://github.com/apache/iceberg/issues/5071 Currently, users can set sort_order using `ALTER TABLE ... WRITE ORDER BY `in SQL. But if the users feel data loading is slow because of sorting and want to reset it. There are no options from SQL.

Re: [I] Support UNSET of sortOrder from the SQL [iceberg]

2023-10-31 Thread via GitHub
RussellSpitzer commented on issue #5071: URL: https://github.com/apache/iceberg/issues/5071#issuecomment-1786975161 Isn't this supported via UNORDERED or did that not get into OSS yet? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377381629 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -166,8 +217,8 @@ public void testInitialize() { assertThatNoException()

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-31 Thread via GitHub
coded9 commented on code in PR #8945: URL: https://github.com/apache/iceberg/pull/8945#discussion_r1377358392 ## flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/TestStreamScanSql.java: ## @@ -219,15 +219,13 @@ public void testConsumeFilesWithBranch() throws Excep

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-31 Thread via GitHub
coded9 commented on code in PR #8945: URL: https://github.com/apache/iceberg/pull/8945#discussion_r1377358822 ## flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/TestStreamScanSql.java: ## @@ -306,15 +304,13 @@ public void testConsumeFromStartTag() throws Exceptio

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-31 Thread via GitHub
nastra merged PR #8948: URL: https://github.com/apache/iceberg/pull/8948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spark 3.4: Support pushing down system functions by V2 filters [iceberg]

2023-10-31 Thread via GitHub
BsoBird commented on PR #7886: URL: https://github.com/apache/iceberg/pull/7886#issuecomment-1786887107 Although I am only an iceberg user, I think this issue should be discussed further. This solution solves the problem at hand, but it inherently hurts the fairness of ICEBERG's support for

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377283966 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -82,12 +83,20 @@ import org.apache.iceberg.types.Types; import org.apache.iceberg.u

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377281522 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreExtension.java: ## @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377281522 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreExtension.java: ## @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [I] Long overflow when Iceberg reading INT96 timestamp column from Spark parquet table [iceberg]

2023-10-31 Thread via GitHub
manuzhang commented on issue #8949: URL: https://github.com/apache/iceberg/issues/8949#issuecomment-1786757877 If you check [Spark's VectorizedColumnReader](https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnR

Re: [I] Iceberg Materialized View Spec [iceberg]

2023-10-31 Thread via GitHub
JanKaul commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1786737489 You are right, I was confused because the REST catalog stores the metadata internally and I thought this would include the storage table. But it would be possible to store the metada

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1377217101 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,346 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1377215123 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,346 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1377205765 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -75,14 +74,9 @@ public class HiveTableOperations extends BaseMetastoreTableOpera

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1377204878 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -51,7 +51,7 @@ public abstract class ViewCatalogTests

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377190081 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,15 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377186180 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,15 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377186180 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,15 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377181258 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -217,7 +217,7 @@ protected void doCommit(TableMetadata base, TableMetadata metad

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1377179041 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreExtension.java: ## @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-31 Thread via GitHub
coded9 commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377177076 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRequiredDistributionAndOrdering.java: ## @@ -186,21 +186,22 @@ public void testDisab

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377154810 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -2390,32 +2377,29 @@ public void testMergeWithInvalidUpdates() {

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377152579 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestUpdate.java: ## @@ -1425,59 +1420,57 @@ public void testUpdateWithInvalidAssignments

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377150647 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestUpdate.java: ## @@ -1333,45 +1332,42 @@ public void testUpdateWithInvalidUpdates() {

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377149920 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestUpdate.java: ## @@ -1333,45 +1332,42 @@ public void testUpdateWithInvalidUpdates() {

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-31 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377146775 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRequiredDistributionAndOrdering.java: ## @@ -186,21 +186,22 @@ public void testDisab

Re: [I] Long overflow when Iceberg reading INT96 timestamp column from Spark parquet table [iceberg]

2023-10-31 Thread via GitHub
manuzhang commented on issue #8949: URL: https://github.com/apache/iceberg/issues/8949#issuecomment-1786591885 @yabola timestamp96 column is using dictionary with encodings `PLAIN_DICTIONARY` and `PLAIN`, which is not covered in your case. -- This is an automated message from the Apache

Re: [I] Does the Java API support primary keys for creating tables [iceberg]

2023-10-31 Thread via GitHub
nastra commented on issue #8950: URL: https://github.com/apache/iceberg/issues/8950#issuecomment-1786574081 @FWLamb you might want to take a look at https://tabular.io/blog/java-api-part-1/ to achieve that -- This is an automated message from the Apache Git Service. To respond to the mess