Re: [PR] Hive: Refactor HiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-26 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1372801587 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Refactor HiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-26 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1372806656 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreSetup.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-26 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1372831403 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -504,6 +508,27 @@ private static Map toReadableByteBufferMap(Map Map filterColumnsStats( + Map map, Se

[I] Vulnerabilities found on latest version - jackson, avro, openssl [iceberg]

2023-10-26 Thread via GitHub
mrmadira opened a new issue, #8923: URL: https://github.com/apache/iceberg/issues/8923 ### Apache Iceberg version 1.4.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Package | Version | Path | Type | CVE | CVSS | Severity | Status

[PR] Api docs refactor [iceberg-python]

2023-10-26 Thread via GitHub
mobley-trent opened a new pull request, #106: URL: https://github.com/apache/iceberg-python/pull/106 Fixes #78 - Adds the `location` parameter to the `catalog.create_table(...)` example for more context. - Adds the code to initialize the catalog in a database if this was not done

Re: [PR] Api docs refactor [iceberg-python]

2023-10-26 Thread via GitHub
mobley-trent commented on PR #106: URL: https://github.com/apache/iceberg-python/pull/106#issuecomment-1781175804 @Fokko do you mind reviewing this PR ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-10-26 Thread via GitHub
dimas-b commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1373340680 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +612,59 @@ public void close() { api.close(); } } + + private v

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-26 Thread via GitHub
stevenzwu commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1373440573 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -504,6 +508,27 @@ private static Map toReadableByteBufferMap(Map Map filterColumnsStats( + Map map

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-26 Thread via GitHub
stevenzwu commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1373440573 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -504,6 +508,27 @@ private static Map toReadableByteBufferMap(Map Map filterColumnsStats( + Map map

Re: [PR] Core: Use avro compression properties from table properties when writing manifests and manifest lists [iceberg]

2023-10-26 Thread via GitHub
wypoon commented on code in PR #6799: URL: https://github.com/apache/iceberg/pull/6799#discussion_r1373486156 ## core/src/main/java/org/apache/iceberg/ManifestFiles.java: ## @@ -157,11 +157,34 @@ public static ManifestWriter write(PartitionSpec spec, OutputFile outp */ p

Re: [PR] GCP: Add Iceberg Catalog for GCP BigLake Metastore [iceberg]

2023-10-26 Thread via GitHub
dchristle commented on PR #7412: URL: https://github.com/apache/iceberg/pull/7412#issuecomment-1781549984 > We released these code here (https://cloud.google.com/bigquery/docs/manage-open-source-metadata#connect-dataproc-vm): > > Iceberg 1.2.0: gs://spark-lib/biglake/biglake-catalog-

[I] java.lang.IllegalArgumentException: info must be ExtendedLogicalWriteInfo [iceberg]

2023-10-26 Thread via GitHub
rafoid opened a new issue, #8926: URL: https://github.com/apache/iceberg/issues/8926 ### Apache Iceberg version 1.4.0 ### Query engine Spark ### Please describe the bug 🐞 Starting `spark-sql` with the following config params: ``` spark-sql \ --packa

[I] Spark write abort result in table miss metadata location file [iceberg]

2023-10-26 Thread via GitHub
dyno opened a new issue, #8927: URL: https://github.com/apache/iceberg/issues/8927 https://github.com/apache/iceberg/blob/333227fbd13821365cec1bdbfcb9314a239bea0f/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L244-L255 recently we encounter a few cases

Re: [PR] feat: manifest list writer [iceberg-rust]

2023-10-26 Thread via GitHub
Fokko merged PR #76: URL: https://github.com/apache/iceberg-rust/pull/76 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] feat: manifest list writer [iceberg-rust]

2023-10-26 Thread via GitHub
Fokko commented on PR #76: URL: https://github.com/apache/iceberg-rust/pull/76#issuecomment-1781749468 @barronw No, I think we're good 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] feat: manifest list writer [iceberg-rust]

2023-10-26 Thread via GitHub
Fokko commented on PR #76: URL: https://github.com/apache/iceberg-rust/pull/76#issuecomment-1781750232 Thanks for working on this @barronw and @liurenjie1024, @Xuanwo and @ZENOTME for the review 👍 -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Api docs refactor [iceberg-python]

2023-10-26 Thread via GitHub
Fokko commented on code in PR #106: URL: https://github.com/apache/iceberg-python/pull/106#discussion_r1373699406 ## mkdocs/docs/api.md: ## @@ -53,6 +53,11 @@ catalog = load_catalog( } ) ``` +If the catalog has not been initialized before, you need to run: Review Comment

Re: [PR] Update roadmap.md [iceberg-docs]

2023-10-26 Thread via GitHub
Fokko commented on code in PR #272: URL: https://github.com/apache/iceberg-docs/pull/272#discussion_r1373721010 ## landing-page/content/common/roadmap.md: ## @@ -22,28 +22,36 @@ disableSidebar: true # Roadmap Overview -This roadmap outlines projects that the Iceberg communi

Re: [PR] Update roadmap.md [iceberg-docs]

2023-10-26 Thread via GitHub
Fokko commented on code in PR #272: URL: https://github.com/apache/iceberg-docs/pull/272#discussion_r1373722476 ## landing-page/content/common/roadmap.md: ## @@ -22,28 +22,36 @@ disableSidebar: true # Roadmap Overview -This roadmap outlines projects that the Iceberg communi

Re: [PR] Update roadmap.md [iceberg-docs]

2023-10-26 Thread via GitHub
Fokko merged PR #272: URL: https://github.com/apache/iceberg-docs/pull/272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Update roadmap.md [iceberg-docs]

2023-10-26 Thread via GitHub
Fokko commented on PR #272: URL: https://github.com/apache/iceberg-docs/pull/272#issuecomment-1781779396 I'll merge this since this is such an improvement over the current state. If there are any follow-up questions, we can address them in a new PR. Thanks @bitsondatadev for taking the lead

Re: [PR] Api docs refactor [iceberg-python]

2023-10-26 Thread via GitHub
Fokko merged PR #106: URL: https://github.com/apache/iceberg-python/pull/106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Docs: PostgreSql integration [iceberg-python]

2023-10-26 Thread via GitHub
Fokko closed issue #78: Docs: PostgreSql integration URL: https://github.com/apache/iceberg-python/issues/78 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-10-26 Thread via GitHub
Fokko commented on code in PR #104: URL: https://github.com/apache/iceberg-python/pull/104#discussion_r1373726454 ## mkdocs/docs/configuration.md: ## @@ -195,6 +195,19 @@ catalog: table-name: iceberg ``` +If you prefer to pass the credentials explicitly to the client ins

Re: [PR] Add docs dir branch [iceberg]

2023-10-26 Thread via GitHub
rdblue merged PR #8917: URL: https://github.com/apache/iceberg/pull/8917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Ignore split offsets array when split offset is past file length [iceberg]

2023-10-26 Thread via GitHub
rdblue commented on PR #8925: URL: https://github.com/apache/iceberg/pull/8925#issuecomment-1781867237 @amogh-jahagirdar, maybe this time we should create a test to validate `FileScanTask.split` when there are bad split offsets? I think that would have caught this in the last PR. -- This

Re: [I] Manifest List Writer Design [iceberg-rust]

2023-10-26 Thread via GitHub
barronw closed issue #72: Manifest List Writer Design URL: https://github.com/apache/iceberg-rust/issues/72 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [PR] Spec: add nanosecond timestamp types [iceberg]

2023-10-26 Thread via GitHub
rdblue commented on code in PR #8683: URL: https://github.com/apache/iceberg/pull/8683#discussion_r1373910153 ## format/spec.md: ## @@ -187,10 +189,11 @@ A **`map`** is a collection of key-value pairs with a key type and a value type. Notes: 1. Decimal scale is fixed and ca

Re: [PR] Spec: add nanosecond timestamp types [iceberg]

2023-10-26 Thread via GitHub
rdblue commented on PR #8683: URL: https://github.com/apache/iceberg/pull/8683#issuecomment-1782035943 @jacobmarble sorry for the delay, I was travelling. I think we just need to call out more clearly that the new types are v3 only and can't be used in v2 without breaking forward compatibil

Re: [PR] Spec: Clarify spec_id field in Data File [iceberg]

2023-10-26 Thread via GitHub
rdblue commented on code in PR #8730: URL: https://github.com/apache/iceberg/pull/8730#discussion_r1373911335 ## format/spec.md: ## @@ -443,13 +443,13 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _optional_ | _optional_ | **`132 split_

Re: [PR] Spec: Clarify spec_id field in Data File [iceberg]

2023-10-26 Thread via GitHub
rdblue commented on code in PR #8730: URL: https://github.com/apache/iceberg/pull/8730#discussion_r1373912012 ## format/spec.md: ## @@ -443,13 +443,13 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _optional_ | _optional_ | **`132 split_

Re: [I] Move JUnit4 tests to JUnit5 [iceberg]

2023-10-26 Thread via GitHub
github-actions[bot] commented on issue #7160: URL: https://github.com/apache/iceberg/issues/7160#issuecomment-1782099308 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark: Clean up FileIO instances on executors for metadata tables [iceberg]

2023-10-26 Thread via GitHub
aokolnychyi commented on PR #8924: URL: https://github.com/apache/iceberg/pull/8924#issuecomment-1782123678 Thank you @singhpk234 @RussellSpitzer! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Spark: Clean up FileIO instances on executors for metadata tables [iceberg]

2023-10-26 Thread via GitHub
aokolnychyi merged PR #8924: URL: https://github.com/apache/iceberg/pull/8924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark: Avoid extra copies of manifests while optimizing V2 tables [iceberg]

2023-10-26 Thread via GitHub
aokolnychyi commented on code in PR #8928: URL: https://github.com/apache/iceberg/pull/8928#discussion_r1373976469 ## core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java: ## @@ -160,11 +163,10 @@ public RewriteManifests addManifest(ManifestFile manifest) { }

[I] Missing serialVersionUID in Serializable implementation [iceberg]

2023-10-26 Thread via GitHub
v-qiunan opened a new issue, #8929: URL: https://github.com/apache/iceberg/issues/8929 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 # Question To implement the serialization interface, do you need to implement a ser

Re: [PR] Test: Add a test utility method to programmatically create expected partition specs [iceberg]

2023-10-26 Thread via GitHub
jerqi commented on PR #8467: URL: https://github.com/apache/iceberg/pull/8467#issuecomment-1782198625 @aokolnychyi @RussellSpitzer Gently ping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[I] Flink: Add support for Flink 1.18 [iceberg]

2023-10-26 Thread via GitHub
YesOrNo828 opened a new issue, #8930: URL: https://github.com/apache/iceberg/issues/8930 ### Feature Request / Improvement ### Feature Request: Recently Flink 1.18 was released. https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.18/ ### What

Re: [PR] feat: support ser/deser of value [iceberg-rust]

2023-10-26 Thread via GitHub
Xuanwo commented on code in PR #82: URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1374016647 ## crates/iceberg/src/spec/values.rs: ## @@ -1309,4 +1981,277 @@ mod tests { &Type::Primitive(PrimitiveType::String), ); } + +#[test] +

Re: [I] Flink: Add support for Flink 1.18 [iceberg]

2023-10-26 Thread via GitHub
pvary commented on issue #8930: URL: https://github.com/apache/iceberg/issues/8930#issuecomment-1782313012 I am concerned about the dependency tree of the flink/v1.17 module. If the flink-common depends on Flink 1.18, and the flink/v1.17 depends on flink-common, then we will have a transiti

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-26 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1374098273 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -400,8 +400,15 @@ public void replaceTableViaTransactionThatAlreadyExistsAsView() {

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-26 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1374101581 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +630,72 @@ public void close() { api.close(); } } + + publ

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-26 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1374101581 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +630,72 @@ public void close() { api.close(); } } + + publ

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-26 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1374109329 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -504,6 +508,27 @@ private static Map toReadableByteBufferMap(Map Map filterColumnsStats( + Map map, Se

Re: [I] Spark write abort result in table miss metadata location file [iceberg]

2023-10-26 Thread via GitHub
nk1506 commented on issue #8927: URL: https://github.com/apache/iceberg/issues/8927#issuecomment-1782354754 @dyno , as per log message > W1024 22:07:51.303 pool-38-thread-273 o.a.i.s.s.SparkWrite:226] Skipping cleanup of written files It means files should not be deleted.

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-26 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1374114025 ## nessie/src/test/java/org/apache/iceberg/nessie/TestBranchVisibilityForView.java: ## @@ -0,0 +1,291 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-26 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1374114239 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNessieView.java: ## @@ -0,0 +1,349 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-26 Thread via GitHub
ajantha-bhat commented on PR #8909: URL: https://github.com/apache/iceberg/pull/8909#issuecomment-1782356812 PR is ready for review. cc: @nastra, @dimas-b, @snazy, @adutra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu