Re: [I] [Feature Request] Implement `equals` for `RESTMessage` [iceberg]

2023-11-07 Thread via GitHub
liurenjie1024 commented on issue #9003: URL: https://github.com/apache/iceberg/issues/9003#issuecomment-1801251776 > `RESTMessage` is an interface that does not carry any contents. Are there any particular messages that you would like to compare? Oh, sorry, I mean all the concrete imp

Re: [I] [Feature Request] Implement `equals` for `RESTMessage` [iceberg]

2023-11-07 Thread via GitHub
Fokko commented on issue #9003: URL: https://github.com/apache/iceberg/issues/9003#issuecomment-1801250346 `RESTMessage` is an interface that does not carry any contents. Are there any particular messages that you would like to compare? -- This is an automated message from the Apache Git

Re: [I] [Feature Request] Implement `equals` for `RESTMessage` [iceberg]

2023-11-07 Thread via GitHub
liurenjie1024 commented on issue #9003: URL: https://github.com/apache/iceberg/issues/9003#issuecomment-1801248035 I would be happy to take this if there is no objection. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Core: Add a constructor to StaticTableOperations [iceberg]

2023-11-07 Thread via GitHub
leewjae commented on PR #8996: URL: https://github.com/apache/iceberg/pull/8996#issuecomment-1801232845 @nastra done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Core: Add a constructor to StaticTableOperations [iceberg]

2023-11-07 Thread via GitHub
nastra commented on PR #8996: URL: https://github.com/apache/iceberg/pull/8996#issuecomment-1801219420 @leewjae can you please adjust `BaseSparkAction` to use the new constructor? After that the PR should be good to go -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Shift site build to use monorepo and gh-pages [iceberg]

2023-11-07 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1386108273 ## site/docs/docs/nightly/docs/configuration.md: ## Review Comment: All of the changes under `site/docs/docs/**` are all in a seperate commit. This was just

Re: [I] Iceberg Materialized View Spec [iceberg]

2023-11-07 Thread via GitHub
JanKaul commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1801212017 Thanks for your input. The discussion has moved to the Google doc (https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?usp=sharing). It would be grea

Re: [I] java.lang.IllegalArgumentException: requirement failed while read migrated parquet table [iceberg]

2023-11-07 Thread via GitHub
camper42 commented on issue #8863: URL: https://github.com/apache/iceberg/issues/8863#issuecomment-1801209401 after 1.4.2, no IllegalArgumentException -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Infra: Add 1.4.2 as latest release to issue template [iceberg]

2023-11-07 Thread via GitHub
nastra merged PR #9001: URL: https://github.com/apache/iceberg/pull/9001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Docs: Add note that snapshot expiration and cleanup orphan files could corrupt Flink job state [iceberg]

2023-11-07 Thread via GitHub
lirui-apache commented on PR #9002: URL: https://github.com/apache/iceberg/pull/9002#issuecomment-1801201336 @pvary please have a look, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-07 Thread via GitHub
jbonofre commented on PR #8971: URL: https://github.com/apache/iceberg/pull/8971#issuecomment-1801192678 I agree with @rdblue For partition stat we had a PR dedicated for spec. As it's not necessary the same reviewers, I think it's good to have one PR for spec (even if it's very

Re: [I] Iceberg Materialized View Spec [iceberg]

2023-11-07 Thread via GitHub
wmoustafa commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1801155675 > This leaves us again with the question of what to use for the storage table pointer. I think table UUID or location is fine. Same should apply when referencing the base ta

Re: [PR] Core: Add a constructor to StaticTableOperations [iceberg]

2023-11-07 Thread via GitHub
ajantha-bhat commented on PR #8996: URL: https://github.com/apache/iceberg/pull/8996#issuecomment-1801121344 > @leewjae do you have a specific use case for using this? Most of the time StaticTableOperations will be used in cases where only the location is serialized and available and not th

[PR] Docs: Add note that snapshot expiration and cleanup orphan files could corrupt Flink job state [iceberg]

2023-11-07 Thread via GitHub
lirui-apache opened a new pull request, #9002: URL: https://github.com/apache/iceberg/pull/9002 Update the doc per the discussions in #5846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-11-07 Thread via GitHub
singhpk234 commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1385993967 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java: ## @@ -309,6 +309,19 @@ private static StreamingOffset determineStarti

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-11-07 Thread via GitHub
singhpk234 commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1385993310 ## core/src/main/java/org/apache/iceberg/MicroBatches.java: ## @@ -92,7 +92,7 @@ private static List> indexManifests( for (ManifestFile manifest : manifestFi

Re: [PR] Spec: Clarify time travel implementation in Iceberg [iceberg]

2023-11-07 Thread via GitHub
emkornfield commented on code in PR #8982: URL: https://github.com/apache/iceberg/pull/8982#discussion_r1385985134 ## format/spec.md: ## @@ -1370,3 +1370,16 @@ Writing v2 metadata: * `sort_columns` was removed Note that these requirements apply when writing data to a v2

Re: [I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-11-07 Thread via GitHub
zeddit commented on issue #132: URL: https://github.com/apache/iceberg-python/issues/132#issuecomment-1800969358 https://stackoverflow.com/questions/74638658/delta-lake-table-storage-sorting here is an expected behavior for me when pulling data with `pyiceberg` to pyarrow or pandas, espec

Re: [PR] API: Add default value API [iceberg]

2023-11-07 Thread via GitHub
lirui-apache commented on PR #4732: URL: https://github.com/apache/iceberg/pull/4732#issuecomment-1800965092 Hey guys, I have some questions about the API change here. 1. Do we need to differentiate the case when default value is not specified, and the case when default value is expli

Re: [I] Flink: Add support for Flink 1.18 [iceberg]

2023-11-07 Thread via GitHub
kevnzhao commented on issue #8930: URL: https://github.com/apache/iceberg/issues/8930#issuecomment-1800919921 Quick check on any progress or plan/timeline for this request? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-11-07 Thread via GitHub
zeddit commented on issue #132: URL: https://github.com/apache/iceberg-python/issues/132#issuecomment-1800900225 @rdblue on the other hand, I wanna know if pyiceberg read data in a consistent way. that is to say, if the data read out has the same order between runs. e.g. I load the

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-07 Thread via GitHub
aokolnychyi commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1385865215 ## api/src/main/java/org/apache/iceberg/Scan.java: ## @@ -77,6 +77,21 @@ public interface Scan> { */ ThisT includeColumnStats(); + /** + * Create a new

Re: [I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-11-07 Thread via GitHub
zeddit commented on issue #132: URL: https://github.com/apache/iceberg-python/issues/132#issuecomment-1800892211 @rdblue thanks for your reply. I wanna know if this feature is challenge by design, why we don't guarantee it on read. I think sorting or reordering in python is quite

Re: [PR] Test: Add a test utility method to programmatically create expected partition specs [iceberg]

2023-11-07 Thread via GitHub
jerqi commented on PR #8467: URL: https://github.com/apache/iceberg/pull/8467#issuecomment-1800866840 @RussellSpitzer @boushphong Thanks for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Does the Java API support primary keys for creating tables [iceberg]

2023-11-07 Thread via GitHub
FWLamb commented on issue #8950: URL: https://github.com/apache/iceberg/issues/8950#issuecomment-1800866028 > primary key in flink is identifier fields in iceberg @Zhangg7723 Can you show me a simple example of creating a primary key table? Thank you. -- This is an automated messag

Re: [I] hive integration iceberg related problems [iceberg]

2023-11-07 Thread via GitHub
dragon-feng commented on issue #8993: URL: https://github.com/apache/iceberg/issues/8993#issuecomment-1800865688 @pvary However, writing data does have problems. The data directory is generated normally, and the data file is also generated normally, but the metadata directory only has the m

Re: [PR] Spec: Clarify which columns can be used for equality delete files. [iceberg]

2023-11-07 Thread via GitHub
liurenjie1024 commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1384272688 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify dele

Re: [PR] Spec: Clarify which columns can be used for equality delete files. [iceberg]

2023-11-07 Thread via GitHub
liurenjie1024 commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1385849323 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify dele

Re: [I] hive integration iceberg related problems [iceberg]

2023-11-07 Thread via GitHub
dragon-feng commented on issue #8993: URL: https://github.com/apache/iceberg/issues/8993#issuecomment-1800861761 @pvary :In contrast, the tez engine I use supports writing, but not reading. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Spec: Clarify time travel implementation in Iceberg [iceberg]

2023-11-07 Thread via GitHub
aokolnychyi commented on code in PR #8982: URL: https://github.com/apache/iceberg/pull/8982#discussion_r1385798919 ## format/spec.md: ## @@ -1370,3 +1370,16 @@ Writing v2 metadata: * `sort_columns` was removed Note that these requirements apply when writing data to a v2

Re: [PR] Spec: Clarify time travel implementation in Iceberg [iceberg]

2023-11-07 Thread via GitHub
aokolnychyi commented on code in PR #8982: URL: https://github.com/apache/iceberg/pull/8982#discussion_r1385797897 ## format/spec.md: ## @@ -1370,3 +1370,16 @@ Writing v2 metadata: * `sort_columns` was removed Note that these requirements apply when writing data to a v2

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-07 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1385795066 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +630,72 @@ public void close() { api.close(); } } + + publ

Re: [I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-11-07 Thread via GitHub
rdblue commented on issue #132: URL: https://github.com/apache/iceberg-python/issues/132#issuecomment-1800679283 The sort order for a table is a suggested write order. It is not guaranteed at read time. To get a sort order in your dataframe, you'll need to sort it. -- This is an automated

Re: [I] Rewrite with zORDER results in 'Cannot find field 'ICEZVALUE' in struct' Error [iceberg]

2023-11-07 Thread via GitHub
github-actions[bot] commented on issue #7586: URL: https://github.com/apache/iceberg/issues/7586#issuecomment-1800646075 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Support replacing delete manifests [iceberg]

2023-11-07 Thread via GitHub
aokolnychyi commented on code in PR #9000: URL: https://github.com/apache/iceberg/pull/9000#discussion_r1385766686 ## core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java: ## @@ -242,7 +241,7 @@ private void performRewrite(List currentManifests) { .execute

Re: [PR] Core: Support replacing delete manifests [iceberg]

2023-11-07 Thread via GitHub
aokolnychyi commented on code in PR #9000: URL: https://github.com/apache/iceberg/pull/9000#discussion_r1385766686 ## core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java: ## @@ -242,7 +241,7 @@ private void performRewrite(List currentManifests) { .execute

[PR] Core: Support replacing delete manifests [iceberg]

2023-11-07 Thread via GitHub
aokolnychyi opened a new pull request, #9000: URL: https://github.com/apache/iceberg/pull/9000 This PR enables delete manifest replacement, basis for delete manifest compaction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-07 Thread via GitHub
dimas-b commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1385587751 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -540,4 +630,72 @@ public void close() { api.close(); } } + + public vo

Re: [PR] Docs: Update docs to reflect the use of hive, not jdbc [iceberg]

2023-11-07 Thread via GitHub
zachlefevre closed pull request #8999: Docs: Update docs to reflect the use of hive, not jdbc URL: https://github.com/apache/iceberg/pull/8999 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] Docs: Update docs to reflect the use of hive, not jdbc [iceberg]

2023-11-07 Thread via GitHub
zachlefevre opened a new pull request, #8999: URL: https://github.com/apache/iceberg/pull/8999 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Test: Add a test utility method to programmatically create expected partition specs [iceberg]

2023-11-07 Thread via GitHub
RussellSpitzer commented on PR #8467: URL: https://github.com/apache/iceberg/pull/8467#issuecomment-1800102549 Thanks @jerqi and @boushphong for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Have a test utility method to programmatically create expected specs [iceberg]

2023-11-07 Thread via GitHub
RussellSpitzer closed issue #8434: Have a test utility method to programmatically create expected specs URL: https://github.com/apache/iceberg/issues/8434 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Test: Add a test utility method to programmatically create expected partition specs [iceberg]

2023-11-07 Thread via GitHub
RussellSpitzer merged PR #8467: URL: https://github.com/apache/iceberg/pull/8467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceb

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-07 Thread via GitHub
rdblue commented on PR #8971: URL: https://github.com/apache/iceberg/pull/8971#issuecomment-1799846173 @jacobmarble can you break this into smaller commits? There are a ton of files changed here and I'm concerned about catching problems with such a large PR. -- This is an automated messa

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-07 Thread via GitHub
jbonofre commented on PR #8971: URL: https://github.com/apache/iceberg/pull/8971#issuecomment-1799781805 I'm starting to review. Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-07 Thread via GitHub
jacobmarble commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1385434928 ## format/spec.md: ## Review Comment: > Don't the Avro and Parquet details need to be updated? Yes. (I'm new to the project.) My original intent

Re: [PR] Fix Github Pages path [iceberg-python]

2023-11-07 Thread via GitHub
Fokko merged PR #133: URL: https://github.com/apache/iceberg-python/pull/133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-07 Thread via GitHub
emkornfield commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1385387452 ## format/spec.md: ## Review Comment: A few high level questions. Don't the Avro and Parquet details need to be updated? Also, are these changes inten

Re: [PR] Clarify which columns can be used for equality delete files. [iceberg]

2023-11-07 Thread via GitHub
emkornfield commented on code in PR #8981: URL: https://github.com/apache/iceberg/pull/8981#discussion_r1385383435 ## format/spec.md: ## @@ -842,7 +842,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify delete

Re: [PR] Core: Add a constructor to StaticTableOperations [iceberg]

2023-11-07 Thread via GitHub
nastra commented on PR #8996: URL: https://github.com/apache/iceberg/pull/8996#issuecomment-1799291906 @leewjae do you have a specific use case for using this? Most of the time `StaticTableOperations` will be used in cases where only the location is serialized and available and not the full

[PR] Fix Github Pages path [iceberg-python]

2023-11-07 Thread via GitHub
Fokko opened a new pull request, #133: URL: https://github.com/apache/iceberg-python/pull/133 `~` is not a valid path, updating this to `/`. From https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features: ![image](https://github.com/apache/iceberg-python/asset

Re: [PR] Core: Add a constructor to StaticTableOperations [iceberg]

2023-11-07 Thread via GitHub
leewjae commented on code in PR #8996: URL: https://github.com/apache/iceberg/pull/8996#discussion_r1385261354 ## core/src/main/java/org/apache/iceberg/StaticTableOperations.java: ## @@ -34,11 +34,24 @@ public class StaticTableOperations implements TableOperations { /** Cr

[PR] Add Description on Using a Separate Authorization Server [iceberg]

2023-11-07 Thread via GitHub
syun64 opened a new pull request, #8998: URL: https://github.com/apache/iceberg/pull/8998 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2023-11-07 Thread via GitHub
nastra commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1385087168 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -186,8 +186,21 @@ private static void throwFailure( throw new RESTException("Unhandled error: %

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2023-11-07 Thread via GitHub
nastra commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1385056741 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Properties.java: ## @@ -27,6 +27,9 @@ private OAuth2Properties() {} /** A credential to exchange for a token

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-11-07 Thread via GitHub
nastra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1385038063 ## nessie/src/test/java/org/apache/iceberg/nessie/TestMultipleClients.java: ## @@ -78,10 +81,19 @@ public void testListNamespaces() { Assertions.assertThat(catalo

Re: [PR] Parquet: Support reading INT96 column in row group filter [iceberg]

2023-11-07 Thread via GitHub
manuzhang commented on PR #8988: URL: https://github.com/apache/iceberg/pull/8988#issuecomment-1798726896 @nastra #8949 is a different RC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Parquet: Support reading INT96 column in row group filter [iceberg]

2023-11-07 Thread via GitHub
nastra commented on PR #8988: URL: https://github.com/apache/iceberg/pull/8988#issuecomment-1798689331 @manuzhang does that also fix https://github.com/apache/iceberg/issues/8949 or is that a different root cause? -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Parquet: Support reading INT96 column in row group filter [iceberg]

2023-11-07 Thread via GitHub
nastra commented on code in PR #8988: URL: https://github.com/apache/iceberg/pull/8988#discussion_r1384934043 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetConversions.java: ## @@ -112,6 +113,10 @@ static Function converterFromParquet(PrimitiveType type) { c

Re: [I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-11-07 Thread via GitHub
zeddit commented on issue #132: URL: https://github.com/apache/iceberg-python/issues/132#issuecomment-1798490930 I will check the concept of `local sort` and `global sort`, there are many documents, thanks. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-11-07 Thread via GitHub
cccs-jc commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1384895852 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java: ## @@ -309,6 +309,19 @@ private static StreamingOffset determineStartingO

Re: [PR] Core: Add a constructor to StaticTableOperations [iceberg]

2023-11-07 Thread via GitHub
nastra commented on code in PR #8996: URL: https://github.com/apache/iceberg/pull/8996#discussion_r1384893965 ## core/src/main/java/org/apache/iceberg/StaticTableOperations.java: ## @@ -34,11 +34,24 @@ public class StaticTableOperations implements TableOperations { /** Cre

Re: [I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-11-07 Thread via GitHub
zeddit commented on issue #132: URL: https://github.com/apache/iceberg-python/issues/132#issuecomment-1798481255 great thanks for your help. I am sorry for not knowing too much about how to conduct a global sort, could you please give me some of documents about how to doing so by usin

Re: [PR] Core: Add a constructor to StaticTableOperations [iceberg]

2023-11-07 Thread via GitHub
nastra commented on code in PR #8996: URL: https://github.com/apache/iceberg/pull/8996#discussion_r1384886730 ## core/src/main/java/org/apache/iceberg/StaticTableOperations.java: ## @@ -34,11 +34,24 @@ public class StaticTableOperations implements TableOperations { /** Cre

Re: [I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-11-07 Thread via GitHub
Fokko commented on issue #132: URL: https://github.com/apache/iceberg-python/issues/132#issuecomment-1798424236 @zeddit Thanks for raising this. I think the `sorted_by` is a local sort. Could you try a global sort by explicitly sorting on the two columns? The manifests in Iceberg are read s

[PR] Core: Add a constructor to StaticTableOperations [iceberg]

2023-11-07 Thread via GitHub
leewjae opened a new pull request, #8996: URL: https://github.com/apache/iceberg/pull/8996 If the table is already having table metadata object, no need to do an extra IO to read from the table metadata object. So, we need a constructor that accepts table metadata object directly. -- Th

Re: [PR] Build: Bump mkdocs-material from 9.4.7 to 9.4.8 [iceberg-python]

2023-11-07 Thread via GitHub
Fokko merged PR #131: URL: https://github.com/apache/iceberg-python/pull/131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] hive integration iceberg related problems [iceberg]

2023-11-07 Thread via GitHub
dragon-feng commented on issue #8993: URL: https://github.com/apache/iceberg/issues/8993#issuecomment-1798338476 Commands containing the catalog configuration: Use iceberg_db; list jars; add jar hdfs://mycluster/tmp/ftp/checkpoint/iceberg-hive-runtime-1.4.2.jar; add jar hd

Re: [I] hive integration iceberg related problems [iceberg]

2023-11-07 Thread via GitHub
dragon-feng commented on issue #8993: URL: https://github.com/apache/iceberg/issues/8993#issuecomment-1798333993 Both version clusters use the same command, The following commands are used: Use iceberg_db; list jars; add jar hdfs://mycluster/tmp/ftp/checkpoint/iceberg-hive-runtim

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1384763093 ## core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [I] manifest exception [iceberg]

2023-11-07 Thread via GitHub
pvary commented on issue #8994: URL: https://github.com/apache/iceberg/issues/8994#issuecomment-1798296469 @innocent123: I do not really understand your question, but I think your problem might be similar to #5846. -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384749968 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogUtil.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384749968 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogUtil.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384743347 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +276,158 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on PR #8907: URL: https://github.com/apache/iceberg/pull/8907#issuecomment-1798279159 @nk1506: I have a few concerns here: 1. `listAllTables` - This configuration has been explicitly created to prevent reading all of the actual table data for the given database (names

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384740902 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -115,42 +129,13 @@ public void initialize(String inputName, Map properties) { @Overri

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384733798 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +276,158 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384737093 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -220,6 +210,28 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {

Re: [I] Adding new columns (mergeSchema) [iceberg]

2023-11-07 Thread via GitHub
josgarmar14 commented on issue #8908: URL: https://github.com/apache/iceberg/issues/8908#issuecomment-1798268560 Thank you @FabricioZGalvani !! It worked!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Adding new columns (mergeSchema) [iceberg]

2023-11-07 Thread via GitHub
josgarmar14 closed issue #8908: Adding new columns (mergeSchema) URL: https://github.com/apache/iceberg/issues/8908 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384733798 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +276,158 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384734112 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -168,6 +153,11 @@ public String name() { return name; } + @Override + public b

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
pvary commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384694865 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +251,146 @@ public void renameTable(TableIdentifier from, TableIdentifier original

[I] I can't find any detailed explanation about column metric options on the official docs for Iceberg configuration [iceberg]

2023-11-07 Thread via GitHub
okayhooni opened a new issue, #8995: URL: https://github.com/apache/iceberg/issues/8995 ### Query engine Spark/Trino ### Question As I asked in https://github.com/tabular-io/iceberg-kafka-connect/issues/149#issuecomment-1797636602 , there is no detailed explanation abou

[I] manifest exception [iceberg]

2023-11-07 Thread via GitHub
innocent123 opened a new issue, #8994: URL: https://github.com/apache/iceberg/issues/8994 ### Query engine When I write tables with flink, I periodically manipulate metadata with spark, rewriteDataFiles, rewriteManifests, expireSnapshots, deleteOrphanFiles are created in sequence, an

[I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-11-07 Thread via GitHub
zeddit opened a new issue, #132: URL: https://github.com/apache/iceberg-python/issues/132 ### Apache Iceberg version 0.5.0 (latest release) ### Please describe the bug 🐞 the code snippets look like below: ``` table = catalog.load_table("default.sample_table") sca

Re: [I] Does the Java API support primary keys for creating tables [iceberg]

2023-11-07 Thread via GitHub
Zhangg7723 commented on issue #8950: URL: https://github.com/apache/iceberg/issues/8950#issuecomment-1798091532 primary key in flink is identifier fields in iceberg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384563404 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +251,146 @@ public void renameTable(TableIdentifier from, TableIdentifier origina

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384561963 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -115,42 +126,13 @@ public void initialize(String inputName, Map properties) { @Overr

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384563404 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +251,146 @@ public void renameTable(TableIdentifier from, TableIdentifier origina

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384560320 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogUtil.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384559141 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogUtil.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-07 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1384554218 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,389 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[I] hive integration iceberg related problems [iceberg]

2023-11-07 Thread via GitHub
dragon-feng opened a new issue, #8993: URL: https://github.com/apache/iceberg/issues/8993 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 hadoop3.1.2 hive3.1.2 tez0.9.2 iceberg1.4.2 Table creation and data insertion a

Re: [I] Ability to the write Metadata JSON [iceberg-python]

2023-11-07 Thread via GitHub
Fokko commented on issue #22: URL: https://github.com/apache/iceberg-python/issues/22#issuecomment-1798012290 > @Fokko, please correct me if I've missed anything about the RestCatalog part No I think that's about right. > Could you please elaborate on this a bit? Are there addi