[GitHub] [iceberg-rust] JanKaul commented on issue #52: No builder for TableMetadata and no public field

2023-09-08 Thread via GitHub
JanKaul commented on issue #52: URL: https://github.com/apache/iceberg-rust/issues/52#issuecomment-1711180269 I would also be in favor of using the builder pattern for the pub structs. If I'm correct all pub structs except for TableMetadata already have a builder. With the `derive_bui

[GitHub] [iceberg] Fokko commented on pull request #8521: Python: Non-Cython fallback Avro parser

2023-09-08 Thread via GitHub
Fokko commented on PR #8521: URL: https://github.com/apache/iceberg/pull/8521#issuecomment-1711190176 @rustyconover Yes I agree. It looks like it is pulling the wheel correctly but it is missing the `decoder_fast` module. Maybe still good to just add this fallback anyway. -- This is an a

[GitHub] [iceberg] GoGoWen opened a new issue, #8527: Why Iceberg do not support column with default value?

2023-09-08 Thread via GitHub
GoGoWen opened a new issue, #8527: URL: https://github.com/apache/iceberg/issues/8527 ### Query engine why Iceberg do not support column with default value? like mysql "k1 INT DEFAULT '1'"? ### Question why Iceberg do not support column with default value? like mys

[GitHub] [iceberg] Fokko commented on issue #8527: Why Iceberg do not support column with default value?

2023-09-08 Thread via GitHub
Fokko commented on issue #8527: URL: https://github.com/apache/iceberg/issues/8527#issuecomment-1711211295 This is actually in the works: https://iceberg.apache.org/spec/#default-values This will be part of Spec version 3 that's being finalized. -- This is an automated message from the A

[GitHub] [iceberg] getAlexRibeiro closed issue #7537: Error reading version hint file

2023-09-08 Thread via GitHub
getAlexRibeiro closed issue #7537: Error reading version hint file URL: https://github.com/apache/iceberg/issues/7537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

[GitHub] [iceberg-rust] JanKaul opened a new pull request, #57: Metadata integration tests

2023-09-08 Thread via GitHub
JanKaul opened a new pull request, #57: URL: https://github.com/apache/iceberg-rust/pull/57 This PR adds integration tests for reading the table metadata from files. Some of the tests are designed to fail. With the current design of the serialization/deserialization the error doesn't specif

[GitHub] [iceberg-rust] JanKaul commented on pull request #57: Metadata integration tests

2023-09-08 Thread via GitHub
JanKaul commented on PR #57: URL: https://github.com/apache/iceberg-rust/pull/57#issuecomment-1711222438 @liurenjie1024, @Xuanwo , @Fokko it would be great if you could take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #56: feat: support read Manifest List

2023-09-08 Thread via GitHub
ZENOTME commented on code in PR #56: URL: https://github.com/apache/iceberg-rust/pull/56#discussion_r1319653288 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -0,0 +1,881 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

[GitHub] [iceberg] andreacfm opened a new pull request, #8528: Schema Merge docs

2023-09-08 Thread via GitHub
andreacfm opened a new pull request, #8528: URL: https://github.com/apache/iceberg/pull/8528 Documentation about schemaMerge See #8005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [iceberg] zeddit commented on issue #8515: Python: Support vectorization read which improve read performance

2023-09-08 Thread via GitHub
zeddit commented on issue #8515: URL: https://github.com/apache/iceberg/issues/8515#issuecomment-1711486546 great thanks for your help. I have tried a poc about `minio + hive metastore + iceberg`, and I am using `pyiceberg` to conduct some performance test. I have a poor performance a

[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #57: Metadata integration tests

2023-09-08 Thread via GitHub
liurenjie1024 commented on code in PR #57: URL: https://github.com/apache/iceberg-rust/pull/57#discussion_r1319737798 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -346,21 +349,29 @@ pub(super) mod _serde { } else { value.current_snapshot_id

[GitHub] [iceberg-rust] liurenjie1024 commented on pull request #56: feat: support read Manifest List

2023-09-08 Thread via GitHub
liurenjie1024 commented on PR #56: URL: https://github.com/apache/iceberg-rust/pull/56#issuecomment-1711520015 > And I find some place is inconsistent with spec. > > > https://iceberg.apache.org/spec/#manifests:~:text=504-,added_files_count,-int In partice, this field in avro is

[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #56: feat: support read Manifest List

2023-09-08 Thread via GitHub
liurenjie1024 commented on code in PR #56: URL: https://github.com/apache/iceberg-rust/pull/56#discussion_r1319745001 ## crates/iceberg/src/avro/mod.rs: ## @@ -18,3 +18,4 @@ //! Avro related codes. #[allow(dead_code)] mod schema; +pub use schema::*; Review Comment: ```sug

[GitHub] [iceberg] xuqi1633 commented on issue #3028: i can't import class which start with org.apache.iceberg.relocated

2023-09-08 Thread via GitHub
xuqi1633 commented on issue #3028: URL: https://github.com/apache/iceberg/issues/3028#issuecomment-1711568897 After compiling the project, a relocated guava jar file will be generated under the bundled-guava module ``` ./gradlew clean build -x test -x javadoc -x integrationTest ```

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #7105: Spec: Add partition stats spec

2023-09-08 Thread via GitHub
ajantha-bhat commented on code in PR #7105: URL: https://github.com/apache/iceberg/pull/7105#discussion_r1319799186 ## format/spec.md: ## @@ -702,6 +703,41 @@ Blob metadata is a struct with the following fields: | _optional_ | _optional_ | **`properties`** | `map` | Additional

[GitHub] [iceberg] ajantha-bhat commented on pull request #7105: Spec: Add partition stats spec

2023-09-08 Thread via GitHub
ajantha-bhat commented on PR #7105: URL: https://github.com/apache/iceberg/pull/7105#issuecomment-1711585128 @RussellSpitzer, @flyrain, @szehon-ho, @rdblue: I have addressed the new suggestions. Please approve the PR if it is ok or comment more if we need further changes. Thanks. --

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #7105: Spec: Add partition stats spec

2023-09-08 Thread via GitHub
ajantha-bhat commented on code in PR #7105: URL: https://github.com/apache/iceberg/pull/7105#discussion_r1319801698 ## format/spec.md: ## @@ -702,6 +703,41 @@ Blob metadata is a struct with the following fields: | _optional_ | _optional_ | **`properties`** | `map` | Additional

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #7105: Spec: Add partition stats spec

2023-09-08 Thread via GitHub
ajantha-bhat commented on code in PR #7105: URL: https://github.com/apache/iceberg/pull/7105#discussion_r1319802967 ## format/spec.md: ## @@ -702,6 +703,41 @@ Blob metadata is a struct with the following fields: | _optional_ | _optional_ | **`properties`** | `map` | Additional

[GitHub] [iceberg] juanrondineau commented on issue #8333: Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.la

2023-09-08 Thread via GitHub
juanrondineau commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1711594714 @chandu-1101 , thanks for your welcome i share 2 printscreens the first simulate on a dbeaver session connected to spark the operations that dbt internaly executes, in th

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #8528: Schema Merge docs

2023-09-08 Thread via GitHub
RussellSpitzer commented on code in PR #8528: URL: https://github.com/apache/iceberg/pull/8528#discussion_r1319920277 ## docs/spark-writes.md: ## @@ -313,6 +313,22 @@ data.writeTo("prod.db.table") .createOrReplace() ``` +### Schema Merge + +Iceberg support dynamic `schem

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #7105: Spec: Add partition stats spec

2023-09-08 Thread via GitHub
szehon-ho commented on code in PR #7105: URL: https://github.com/apache/iceberg/pull/7105#discussion_r1319945486 ## format/spec.md: ## @@ -702,6 +703,49 @@ Blob metadata is a struct with the following fields: | _optional_ | _optional_ | **`properties`** | `map` | Additional pr

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #7105: Spec: Add partition stats spec

2023-09-08 Thread via GitHub
szehon-ho commented on code in PR #7105: URL: https://github.com/apache/iceberg/pull/7105#discussion_r1319942204 ## format/spec.md: ## @@ -702,6 +703,49 @@ Blob metadata is a struct with the following fields: | _optional_ | _optional_ | **`properties`** | `map` | Additional pr

[GitHub] [iceberg-docs] amogh-jahagirdar merged pull request #274: Update vendors.md

2023-09-08 Thread via GitHub
amogh-jahagirdar merged PR #274: URL: https://github.com/apache/iceberg-docs/pull/274 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #7105: Spec: Add partition stats spec

2023-09-08 Thread via GitHub
ajantha-bhat commented on code in PR #7105: URL: https://github.com/apache/iceberg/pull/7105#discussion_r1320008836 ## format/spec.md: ## @@ -702,6 +703,49 @@ Blob metadata is a struct with the following fields: | _optional_ | _optional_ | **`properties`** | `map` | Additional

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #7105: Spec: Add partition stats spec

2023-09-08 Thread via GitHub
ajantha-bhat commented on code in PR #7105: URL: https://github.com/apache/iceberg/pull/7105#discussion_r1320009356 ## format/spec.md: ## @@ -702,6 +703,49 @@ Blob metadata is a struct with the following fields: | _optional_ | _optional_ | **`properties`** | `map` | Additional

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #8491: Python: Improved Readability and Alignment of Regex Patterns

2023-09-08 Thread via GitHub
amogh-jahagirdar commented on PR #8491: URL: https://github.com/apache/iceberg/pull/8491#issuecomment-1711851893 @hiteshbedre Since this is more of a cleanup, I'll merge after the checks pass. -- This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [iceberg-go] delaneyj opened a new issue, #4: Implementations?

2023-09-08 Thread via GitHub
delaneyj opened a new issue, #4: URL: https://github.com/apache/iceberg-go/issues/4 ### Question Iceberg has subprojects targetting arrow/orc/parquet/etc. Is there plans to have adapters be part of this repo? Are there plans to have interfaces for `SchemaToDatastore`? -- Th

[GitHub] [iceberg] amogh-jahagirdar merged pull request #8491: Python: Improved Readability and Alignment of Regex Patterns

2023-09-08 Thread via GitHub
amogh-jahagirdar merged PR #8491: URL: https://github.com/apache/iceberg/pull/8491 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

[GitHub] [iceberg] amogh-jahagirdar commented on pull request #8491: Python: Improved Readability and Alignment of Regex Patterns

2023-09-08 Thread via GitHub
amogh-jahagirdar commented on PR #8491: URL: https://github.com/apache/iceberg/pull/8491#issuecomment-1711878925 Thanks for the contribution @hiteshbedre ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [iceberg-docs] melvynator opened a new pull request, #275: Update vendors.md

2023-09-08 Thread via GitHub
melvynator opened a new pull request, #275: URL: https://github.com/apache/iceberg-docs/pull/275 Fixed a typo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[GitHub] [iceberg-go] zeroshade commented on issue #4: Implementations?

2023-09-08 Thread via GitHub
zeroshade commented on issue #4: URL: https://github.com/apache/iceberg-go/issues/4#issuecomment-1711921361 I plan on supporting Arrow, Parquet, Avro and Orc in this repo as much as I can. That said, I'm not familiar with `SchemaToDatastore`, but I want to support as much as possibl

[GitHub] [iceberg-go] delaneyj commented on issue #4: Implementations?

2023-09-08 Thread via GitHub
delaneyj commented on issue #4: URL: https://github.com/apache/iceberg-go/issues/4#issuecomment-1712023219 Oh its not a library, I meant include an interface to be able to plugin any of these options or others. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [iceberg] kunal-nandwana opened a new issue, #5556: Feature Request: Support mergeSchema option when using Spark MERGE INTO

2023-09-08 Thread via GitHub
kunal-nandwana opened a new issue, #5556: URL: https://github.com/apache/iceberg/issues/5556 ### Feature Request / Improvement Hi Team, I am using Iceberg in my project and I found a big thing which is missing from Iceberg which is easily available in Apache Hudi and Deltalake that

[GitHub] [iceberg] vinitamaloo-asu commented on issue #2442: cannot insert value in hive command shell

2023-09-08 Thread via GitHub
vinitamaloo-asu commented on issue #2442: URL: https://github.com/apache/iceberg/issues/2442#issuecomment-1712329399 I created a new catalog "iceberg_catalog" using spark config like below: `.set("spark.sql.catalog.iceberg_catalog", "org.apache.iceberg.spark.SparkCatalog")

[GitHub] [iceberg] vinitamaloo-asu opened a new issue, #8529: CASCADE WITH Drop Namespace Gives exception

2023-09-08 Thread via GitHub
vinitamaloo-asu opened a new issue, #8529: URL: https://github.com/apache/iceberg/issues/8529 ### Apache Iceberg version 1.3.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Running this command with: `spark.sql(DROP DATABASE IF EX

[GitHub] [iceberg] github-actions[bot] closed issue #6914: change partition led to query bug

2023-09-08 Thread via GitHub
github-actions[bot] closed issue #6914: change partition led to query bug URL: https://github.com/apache/iceberg/issues/6914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [iceberg] github-actions[bot] commented on issue #6914: change partition led to query bug

2023-09-08 Thread via GitHub
github-actions[bot] commented on issue #6914: URL: https://github.com/apache/iceberg/issues/6914#issuecomment-1712351531 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi