Re: [PR] Build: Replace Thread.Sleep() usage with org.Awaitility from Tests. [iceberg]

2023-10-25 Thread via GitHub
nk1506 commented on code in PR #8804: URL: https://github.com/apache/iceberg/pull/8804#discussion_r1372617190 ## aws/src/integration/java/org/apache/iceberg/aws/TestAssumeRoleAwsClientFactory.java: ## @@ -189,7 +192,18 @@ public void testAssumeRoleS3FileIO() throws Exception {

Re: [PR] Build: Replace Thread.Sleep() usage with org.Awaitility from Tests. [iceberg]

2023-10-25 Thread via GitHub
nk1506 commented on code in PR #8804: URL: https://github.com/apache/iceberg/pull/8804#discussion_r1372614898 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -435,13 +437,11 @@ public void testConcurrentFastAppends(@TempDir File dir) throws Excepti

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-25 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1372578783 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCatalog.java: ## @@ -0,0 +1,1358 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-25 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1372577053 ## api/src/main/java/org/apache/iceberg/exceptions/NoSuchIcebergViewException.java: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-25 Thread via GitHub
stevenzwu commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1372568261 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -504,6 +508,27 @@ private static Map toReadableByteBufferMap(Map Map filterColumnsStats( + Map map

Re: [I] java.lang.IllegalArgumentException: requirement failed while read migrated parquet table [iceberg]

2023-10-25 Thread via GitHub
camper42 commented on issue #8863: URL: https://github.com/apache/iceberg/issues/8863#issuecomment-1780377369 Can this caused by critical bug described in #8834 ? @bryanck If it is, this issue could be closed -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] Update roadmap.md [iceberg-docs]

2023-10-25 Thread via GitHub
bitsondatadev commented on PR #272: URL: https://github.com/apache/iceberg-docs/pull/272#issuecomment-1780366373 > Hey all, I'm looping back to this today. There are lies, damned lies, and random promises I make on GitHub. Pardon my 🧠. Just one open question but we can update t

[PR] [WIP] Spark 3.5: Honor spark.sql.files.maxPartitionBytes in read split [iceberg]

2023-10-25 Thread via GitHub
jzhuge opened a new pull request, #8922: URL: https://github.com/apache/iceberg/pull/8922 Honor Spark conf `spark.sql.files.maxPartitionBytes` in read split size, right before the default value. Here is the proposed order: 1. DataFrame read option `split-size` 2. Table prope

Re: [PR] Update roadmap.md [iceberg-docs]

2023-10-25 Thread via GitHub
bitsondatadev commented on code in PR #272: URL: https://github.com/apache/iceberg-docs/pull/272#discussion_r1372537159 ## landing-page/content/common/roadmap.md: ## @@ -22,28 +22,36 @@ disableSidebar: true # Roadmap Overview -This roadmap outlines projects that the Iceberg

Re: [PR] Spark 3.3: Change the default dest catalog name for TableSnapshot [iceberg]

2023-10-25 Thread via GitHub
tomtongue closed pull request #6806: Spark 3.3: Change the default dest catalog name for TableSnapshot URL: https://github.com/apache/iceberg/pull/6806 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-25 Thread via GitHub
aokolnychyi commented on PR #8803: URL: https://github.com/apache/iceberg/pull/8803#issuecomment-1780243850 I took a quick look. My original guess was to do this in `ManifestReader` but it seems Peter is right: it should be sufficient to do this in `ManifestGroup` as we are using either `en

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-10-25 Thread via GitHub
HonahX commented on code in PR #104: URL: https://github.com/apache/iceberg-python/pull/104#discussion_r1372446210 ## pyiceberg/catalog/dynamodb.py: ## @@ -80,7 +80,15 @@ class DynamoDbCatalog(Catalog): def __init__(self, name: str, **properties: str): super().__i

Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

2023-10-25 Thread via GitHub
wypoon commented on code in PR #8805: URL: https://github.com/apache/iceberg/pull/8805#discussion_r1372439543 ## core/src/test/java/org/apache/iceberg/deletes/TestPositionFilter.java: ## @@ -282,6 +286,16 @@ public void testPositionSetRowFilter() { @Test public void test

Re: [I] UnsupportedOperationException when using `IcebergGenerics.read` to read metadata tables [iceberg]

2023-10-25 Thread via GitHub
github-actions[bot] closed issue #7351: UnsupportedOperationException when using `IcebergGenerics.read` to read metadata tables URL: https://github.com/apache/iceberg/issues/7351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [I] UnsupportedOperationException when using `IcebergGenerics.read` to read metadata tables [iceberg]

2023-10-25 Thread via GitHub
github-actions[bot] commented on issue #7351: URL: https://github.com/apache/iceberg/issues/7351#issuecomment-1780223574 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] spark insert partition table Data skew [iceberg]

2023-10-25 Thread via GitHub
github-actions[bot] commented on issue #7455: URL: https://github.com/apache/iceberg/issues/7455#issuecomment-1780223493 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Predicate Pushdown Not Working as Expected [iceberg]

2023-10-25 Thread via GitHub
github-actions[bot] commented on issue #7461: URL: https://github.com/apache/iceberg/issues/7461#issuecomment-1780223461 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Use avro compression properties from table properties when writing manifests and manifest lists [iceberg]

2023-10-25 Thread via GitHub
aokolnychyi commented on code in PR #6799: URL: https://github.com/apache/iceberg/pull/6799#discussion_r1372337825 ## core/src/main/java/org/apache/iceberg/ManifestFiles.java: ## @@ -157,11 +157,34 @@ public static ManifestWriter write(PartitionSpec spec, OutputFile outp */

Re: [I] Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-10-25 Thread via GitHub
singhpk234 commented on issue #8902: URL: https://github.com/apache/iceberg/issues/8902#issuecomment-1780043095 Ya sure take your time you are pretty close :) let me know if you need some help here, apologies I am short on bandwidth too due to upcoming re:invent. -- This is an automated m

Re: [I] Iceberg streaming using checkpoint does not ignore the stream-from-timestamp option [iceberg]

2023-10-25 Thread via GitHub
singhpk234 commented on issue #8921: URL: https://github.com/apache/iceberg/issues/8921#issuecomment-1780040874 I am not entirely sure still how would this give us or being us to snapshot id not found as the offset returned here would be pointing to -1 snapshot id but as you suggested earli

Re: [I] Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-10-25 Thread via GitHub
cccs-jc commented on issue #8902: URL: https://github.com/apache/iceberg/issues/8902#issuecomment-1780032370 I did manage to reproduce the issue. So we'll see how it goes. ```java @Test public void testReadStreamWithSnapshotTypeReplaceIgnoresReplace() throws Exception {

Re: [I] Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-10-25 Thread via GitHub
cccs-jc commented on issue #8902: URL: https://github.com/apache/iceberg/issues/8902#issuecomment-1780029801 a CL, that's a pull request? right now I'm trying to add unit tests but it's going slowly. I don't have too many free cycles.. -- This is an automated message from the Apach

Re: [I] Iceberg streaming using checkpoint does not ignore the stream-from-timestamp option [iceberg]

2023-10-25 Thread via GitHub
cccs-jc commented on issue #8921: URL: https://github.com/apache/iceberg/issues/8921#issuecomment-1780023262 ya seems pretty clear that it should be picking up the offset from the checkpoint when it exists. could this come into the picture. There are two checks about the timestamp he

Re: [I] Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-10-25 Thread via GitHub
singhpk234 commented on issue #8902: URL: https://github.com/apache/iceberg/issues/8902#issuecomment-1780015926 yes @cccs-jc i think the right way to skip over snapshots (for the overwrite ones) should be latestOffset Api so that we coud just say the new offset is the one ignoring un-necess

Re: [I] Iceberg streaming using checkpoint does not ignore the stream-from-timestamp option [iceberg]

2023-10-25 Thread via GitHub
singhpk234 commented on issue #8921: URL: https://github.com/apache/iceberg/issues/8921#issuecomment-1780012237 I am not entirely sure why this would happen, i was under the impression that initialOffset() is used from checkpoint file and the way it's calculated is same as you stated above

Re: [PR] Build: Bump werkzeug from 2.3.7 to 3.0.1 [iceberg-python]

2023-10-25 Thread via GitHub
Fokko merged PR #105: URL: https://github.com/apache/iceberg-python/pull/105 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[I] Iceberg streaming using checkpoint does not ignore the stream-from-timestamp option [iceberg]

2023-10-25 Thread via GitHub
cccs-jc opened a new issue, #8921: URL: https://github.com/apache/iceberg/issues/8921 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 @singhpk234 I suspect there is a bug in the Spark readStream. I'm not 100% but we had s

[PR] Build: Bump werkzeug from 2.3.7 to 3.0.1 [iceberg-python]

2023-10-25 Thread via GitHub
dependabot[bot] opened a new pull request, #105: URL: https://github.com/apache/iceberg-python/pull/105 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.3.7 to 3.0.1. Release notes Sourced from https://github.com/pallets/werkzeug/releases";>werkzeug's releases. 3

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-10-25 Thread via GitHub
waifairer commented on PR #104: URL: https://github.com/apache/iceberg-python/pull/104#issuecomment-1779931470 Done, thanks @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on PR #104: URL: https://github.com/apache/iceberg-python/pull/104#issuecomment-1779912483 Thanks @waifairer for raising this! Could you also update the docs under `mkdocs/docs/configuration.md`? -- This is an automated message from the Apache Git Service. To respond to th

[PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-10-25 Thread via GitHub
waifairer opened a new pull request, #104: URL: https://github.com/apache/iceberg-python/pull/104 Looks like Dynamo is lacking some properties which are documented for other AWS-centric catalog types. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1372141271 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1372141271 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1372144754 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1372143737 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1372141271 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1372139436 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1372139436 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1372134598 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add javadoc dir branch [iceberg]

2023-10-25 Thread via GitHub
rdblue merged PR #8912: URL: https://github.com/apache/iceberg/pull/8912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[PR] Core: Add table session cache [iceberg]

2023-10-25 Thread via GitHub
nastra opened a new pull request, #8920: URL: https://github.com/apache/iceberg/pull/8920 The purpose of caching auth session for tables is mainly so that we can stop refreshing the session when it isn't used anymore -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Fix the nullability of `snapshot-id` on `AssertRefSnapshotId` [iceberg-python]

2023-10-25 Thread via GitHub
Fokko commented on PR #103: URL: https://github.com/apache/iceberg-python/pull/103#issuecomment-1779742734 Thanks for the prompt review @nastra & @amogh-jahagirdar 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Fix the nullability of `snapshot-id` on `AssertRefSnapshotId` [iceberg-python]

2023-10-25 Thread via GitHub
Fokko merged PR #103: URL: https://github.com/apache/iceberg-python/pull/103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Built: Update revapi to 1.4.1 [iceberg]

2023-10-25 Thread via GitHub
Fokko commented on PR #8906: URL: https://github.com/apache/iceberg/pull/8906#issuecomment-1779741555 @ajantha-bhat thanks for the context, closing this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Built: Update revapi to 1.4.1 [iceberg]

2023-10-25 Thread via GitHub
Fokko closed pull request #8906: Built: Update revapi to 1.4.1 URL: https://github.com/apache/iceberg/pull/8906 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Core: Reduce unnecessary add operations in deletedPaths set [iceberg]

2023-10-25 Thread via GitHub
amogh-jahagirdar merged PR #8868: URL: https://github.com/apache/iceberg/pull/8868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Open-API: Error response in the spec don’t align with the expected model. [iceberg]

2023-10-25 Thread via GitHub
jackye1995 merged PR #8914: URL: https://github.com/apache/iceberg/pull/8914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Open-API: Error response in the spec don’t align with the expected model. [iceberg]

2023-10-25 Thread via GitHub
jackye1995 commented on PR #8914: URL: https://github.com/apache/iceberg/pull/8914#issuecomment-1779669087 thanks for the fix, and thanks for the reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-25 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371956676 ## api/src/main/java/org/apache/iceberg/ContentFile.java: ## @@ -165,6 +166,20 @@ default Long fileSequenceNumber() { */ F copyWithoutStats(); + /** + * Copie

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-25 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371955308 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -174,8 +177,10 @@ public PartitionData copy() { * * @param toCopy a generic data file to copy. *

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-25 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371954933 ## api/src/main/java/org/apache/iceberg/ContentFile.java: ## @@ -165,6 +166,20 @@ default Long fileSequenceNumber() { */ F copyWithoutStats(); + /** + * Copie

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-25 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371953259 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -504,6 +508,27 @@ private static Map toReadableByteBufferMap(Map Map filterColumnsStats( + Map map, Se

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-25 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371948395 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -504,6 +508,27 @@ private static Map toReadableByteBufferMap(Map Map filterColumnsStats( + Map map, Se

Re: [PR] Core: Enable column statistics filtering after planning [iceberg]

2023-10-25 Thread via GitHub
pvary commented on code in PR #8803: URL: https://github.com/apache/iceberg/pull/8803#discussion_r1371944613 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -504,6 +508,27 @@ private static Map toReadableByteBufferMap(Map Map filterColumnsStats( + Map map, Se

Re: [PR] Nessie: reimplement namespace operations [iceberg]

2023-10-25 Thread via GitHub
adutra commented on code in PR #8857: URL: https://github.com/apache/iceberg/pull/8857#discussion_r1371727433 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -181,133 +185,223 @@ public IcebergTable table(TableIdentifier tableIdentifier) { }

Re: [PR] INFRA: Disable merging explicitly [iceberg]

2023-10-25 Thread via GitHub
Fokko commented on PR #8878: URL: https://github.com/apache/iceberg/pull/8878#issuecomment-1779220520 Thanks everyone for the review and comment 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] INFRA: Disable merging explicitly [iceberg]

2023-10-25 Thread via GitHub
Fokko merged PR #8878: URL: https://github.com/apache/iceberg/pull/8878 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] INFRA: Disable merging explicitly [iceberg]

2023-10-25 Thread via GitHub
Fokko commented on PR #8878: URL: https://github.com/apache/iceberg/pull/8878#issuecomment-1779218865 > I didn't find the rules to decide when to rebase and when to squash the PR in the docs. Are they both used/needed, or only the squash? The squash and merge is the normal way of merg

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1371717288 ## .github/workflows/spark-ci.yml: ## @@ -141,4 +142,4 @@ jobs: with: name: test logs path: | -**/build/testlogs \ No newline a

Re: [PR] Build: Replace Thread.Sleep() usage with org.Awaitility from Tests. [iceberg]

2023-10-25 Thread via GitHub
amogh-jahagirdar commented on code in PR #8804: URL: https://github.com/apache/iceberg/pull/8804#discussion_r1371705732 ## aws/src/integration/java/org/apache/iceberg/aws/lakeformation/LakeFormationTestBase.java: ## @@ -417,7 +431,16 @@ private static void registerResource(Strin

[PR] Add latest version to menu [iceberg-docs]

2023-10-25 Thread via GitHub
nastra opened a new pull request, #291: URL: https://github.com/apache/iceberg-docs/pull/291 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Build: Replace Thread.Sleep() usage with org.Awaitility from Tests. [iceberg]

2023-10-25 Thread via GitHub
nastra commented on code in PR #8804: URL: https://github.com/apache/iceberg/pull/8804#discussion_r1371698377 ## aws/src/integration/java/org/apache/iceberg/aws/TestAssumeRoleAwsClientFactory.java: ## @@ -189,7 +192,18 @@ public void testAssumeRoleS3FileIO() throws Exception {

Re: [PR] Build: Replace Thread.Sleep() usage with org.Awaitility from Tests. [iceberg]

2023-10-25 Thread via GitHub
nastra commented on code in PR #8804: URL: https://github.com/apache/iceberg/pull/8804#discussion_r1371693544 ## aws/src/integration/java/org/apache/iceberg/aws/TestAssumeRoleAwsClientFactory.java: ## @@ -189,7 +192,18 @@ public void testAssumeRoleS3FileIO() throws Exception {

Re: [PR] Open-API: Error response in the spec don’t align with the expected model. [iceberg]

2023-10-25 Thread via GitHub
Fokko commented on code in PR #8914: URL: https://github.com/apache/iceberg/pull/8914#discussion_r1371690577 ## open-api/rest-catalog-open-api.yaml: ## @@ -2322,13 +2322,22 @@ components: description: Authorization scope for client credentials or token exchange

Re: [PR] Add latest version to menu [iceberg-docs]

2023-10-25 Thread via GitHub
Fokko merged PR #290: URL: https://github.com/apache/iceberg-docs/pull/290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Release: Versioned 1.4.1 docs [iceberg-docs]

2023-10-25 Thread via GitHub
Fokko merged PR #288: URL: https://github.com/apache/iceberg-docs/pull/288 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[PR] Add latest version to menu [iceberg-docs]

2023-10-25 Thread via GitHub
nastra opened a new pull request, #290: URL: https://github.com/apache/iceberg-docs/pull/290 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-25 Thread via GitHub
bitsondatadev commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1371640541 ## .github/workflows/spark-ci.yml: ## @@ -141,4 +142,4 @@ jobs: with: name: test logs path: | -**/build/testlogs \ No n

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.21.0 to 2.21.5 [iceberg]

2023-10-25 Thread via GitHub
nastra merged PR #8896: URL: https://github.com/apache/iceberg/pull/8896 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.43.2.0 to 3.43.2.1 [iceberg]

2023-10-25 Thread via GitHub
nastra merged PR #8893: URL: https://github.com/apache/iceberg/pull/8893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.22.0 to 2.23.0 [iceberg]

2023-10-25 Thread via GitHub
nastra merged PR #8897: URL: https://github.com/apache/iceberg/pull/8897 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Update release template [iceberg]

2023-10-25 Thread via GitHub
nastra merged PR #8879: URL: https://github.com/apache/iceberg/pull/8879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

2023-10-25 Thread via GitHub
cccs-jc commented on issue #8902: URL: https://github.com/apache/iceberg/issues/8902#issuecomment-1779110745 @singhpk234 just a few quick questions. see above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Build: Bump nessie from 0.72.0 to 0.72.1 [iceberg]

2023-10-25 Thread via GitHub
nastra merged PR #8900: URL: https://github.com/apache/iceberg/pull/8900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-25 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1371621034 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCatalog.java: ## @@ -0,0 +1,1358 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-25 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1371613818 ## api/src/main/java/org/apache/iceberg/exceptions/NoSuchIcebergViewException.java: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-25 Thread via GitHub
nastra commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1371601349 ## .github/workflows/spark-ci.yml: ## @@ -141,4 +142,4 @@ jobs: with: name: test logs path: | -**/build/testlogs \ No newline

[PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-25 Thread via GitHub
bitsondatadev opened a new pull request, #8919: URL: https://github.com/apache/iceberg/pull/8919 Revert back to using the `site` directory which contains the monorepo of the site and versioned docs. Also move back to using the asf infra standards including the [asf-site](https://github.com/

Re: [PR] Hive: Refactor HiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-25 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1371583467 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalogNew.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Hive: Refactor HiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-25 Thread via GitHub
ajantha-bhat commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1371580522 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreSetup.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Hive: Refactor HiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-25 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1371579935 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalogNew.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Hive: Refactor HiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-25 Thread via GitHub
ajantha-bhat commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1371550325 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,12 @@ public void renameTable(TableIdentifier from, TableIdentifier or

Re: [PR] Hive: Refactor HiveCatalog tests to use the core CatalogTests [iceberg]

2023-10-25 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1371576645 ## hive-metastore/src/test/java/org/apache/iceberg/hive/HiveMetastoreSetup.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-25 Thread via GitHub
ismailsimsek commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1371571888 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -601,15 +601,81 @@ public void testDropNamespace() { public void testCreateNamespace()

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-25 Thread via GitHub
ismailsimsek commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1371566535 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -604,6 +604,27 @@ public void testCreateNamespace() { // Test with no metadata

Re: [I] Adding new columns (mergeSchema) [iceberg]

2023-10-25 Thread via GitHub
josgarmar14 commented on issue #8908: URL: https://github.com/apache/iceberg/issues/8908#issuecomment-1778946237 I think everything is correct. Once the table is created I execute: ``` query = f"""ALTER TABLE job_catalog.{args['glue_db']}.{args['glue_iceberg_tablename']} SET TBLPR

Re: [PR] AWS: Glue catalog strip trailing slash on DB URI [iceberg]

2023-10-25 Thread via GitHub
okayhooni commented on PR #8870: URL: https://github.com/apache/iceberg/pull/8870#issuecomment-1778670668 @amogh-jahagirdar Sorry for the late response and thanks to handle this issue..! Yes, there is a trailing slash on the database location, as you said..! so, your commit wi