Re: [I] S3 compression Issue with Iceberg [iceberg]

2023-10-05 Thread via GitHub
swat1234 commented on issue #8713: URL: https://github.com/apache/iceberg/issues/8713#issuecomment-1748235072 I have tried with huge data. Below are the outcomes. 1. File with UNCOMPRESSED codec. - 1.8GB 2. File with gzip codec - 1.8GB 3. File with code snappy codec - 2.8GB

Re: [PR] Spec: Add partition stats spec [iceberg]

2023-10-05 Thread via GitHub
ajantha-bhat commented on PR #7105: URL: https://github.com/apache/iceberg/pull/7105#issuecomment-1748239349 @rdblue: Can you also please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Core: Include summary without 'operation' when comparing view versions [iceberg]

2023-10-05 Thread via GitHub
nastra commented on code in PR #8678: URL: https://github.com/apache/iceberg/pull/8678#discussion_r1346975335 ## core/src/main/java/org/apache/iceberg/view/ViewMetadata.java: ## @@ -321,14 +321,20 @@ private int reuseOrCreateNewViewVersionId(ViewVersion viewVersion) { /*

Re: [I] Need to support AWS storage class [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on issue #8153: URL: https://github.com/apache/iceberg/issues/8153#issuecomment-1748352603 Was going through issues, looks like this is already fixed by #8154 . Closing this issue -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [I] Need to support AWS storage class [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar closed issue #8153: Need to support AWS storage class URL: https://github.com/apache/iceberg/issues/8153 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Add logic to generate a new snapshot-id [iceberg-python]

2023-10-05 Thread via GitHub
Fokko commented on PR #37: URL: https://github.com/apache/iceberg-python/pull/37#issuecomment-1748353368 In Python, you cannot xor on ints/longs (probably because they are also unbound in terms of size :), so I had to deviate a bit there. -- This is an automated message from the Apache G

Re: [PR] Add logic to generate a new snapshot-id [iceberg-python]

2023-10-05 Thread via GitHub
Fokko commented on code in PR #37: URL: https://github.com/apache/iceberg-python/pull/37#discussion_r1346999502 ## pyiceberg/table/__init__.py: ## @@ -1566,3 +1575,15 @@ def _add_and_move_fields( elif len(moves) > 0: return _move_fields(fields, moves) return N

Re: [PR] Add logic to generate a new snapshot-id [iceberg-python]

2023-10-05 Thread via GitHub
Fokko merged PR #37: URL: https://github.com/apache/iceberg-python/pull/37 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
jbonofre opened a new pull request, #8722: URL: https://github.com/apache/iceberg/pull/8722 This PR rename resources using `master` to `main`. This PR should be merged only when the renaming has been done by Apache INRA. -- This is an automated message from the Apache Git Service. To resp

Re: [I] Upsert support for keyless Apache Flink tables [iceberg]

2023-10-05 Thread via GitHub
pvary commented on issue #8719: URL: https://github.com/apache/iceberg/issues/8719#issuecomment-1748445456 Have you tried turning off the upsert mode? https://iceberg.apache.org/docs/latest/flink-configuration/#write-options ``` upsert-enabled | Table write.upsert.enabled | Overrides

Re: [I] Null support in Apache Flink [iceberg]

2023-10-05 Thread via GitHub
pvary commented on issue #8720: URL: https://github.com/apache/iceberg/issues/8720#issuecomment-1748453178 I would guess the `NULL` handling becomes non-trivial, if you start executing queries filtering for the fields containing `NULL` values. What is the results / expected result for

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
nastra commented on code in PR #8722: URL: https://github.com/apache/iceberg/pull/8722#discussion_r1347081505 ## README.md: ## @@ -36,9 +36,9 @@ The core Java library that tracks table snapshots and metadata is complete, but The [Iceberg format specification][iceberg-spec] i

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
jbonofre commented on code in PR #8722: URL: https://github.com/apache/iceberg/pull/8722#discussion_r1347085171 ## README.md: ## @@ -36,9 +36,9 @@ The core Java library that tracks table snapshots and metadata is complete, but The [Iceberg format specification][iceberg-spec]

Re: [PR] OpenAPI: Add AssignUUID update to metadata updates [iceberg]

2023-10-05 Thread via GitHub
nastra merged PR #8716: URL: https://github.com/apache/iceberg/pull/8716 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] OpenAPI: Add AssignUUID update to metadata updates [iceberg]

2023-10-05 Thread via GitHub
nastra commented on PR #8716: URL: https://github.com/apache/iceberg/pull/8716#issuecomment-1748467044 thanks for reviewing @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
ajantha-bhat commented on code in PR #8722: URL: https://github.com/apache/iceberg/pull/8722#discussion_r1347108071 ## docs/metrics-reporting.md: ## @@ -171,4 +171,4 @@ TableScan tableScan = try (CloseableIterable fileScanTasks = tableScan.planFiles()) { // ... } -``` \ No

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
jbonofre commented on code in PR #8722: URL: https://github.com/apache/iceberg/pull/8722#discussion_r1347144486 ## docs/flink-writes.md: ## @@ -270,4 +270,4 @@ INSERT INTO tableName /*+ OPTIONS('upsert-enabled'='true') */ ... ``` -Check out all the options here: [write-opt

[PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
nastra opened a new pull request, #8723: URL: https://github.com/apache/iceberg/pull/8723 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1347232147 ## docs-new/site/releases.md: ## @@ -0,0 +1,777 @@ +--- +title: "Releases" +--- + + +## Downloads + +The latest version of Iceberg is [{{ icebergVersion }}](http

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1347234000 ## docs-new/site/releases.md: ## @@ -0,0 +1,777 @@ +--- +title: "Releases" +--- + + +## Downloads + +The latest version of Iceberg is [{{ icebergVersion }}](http

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
ajantha-bhat commented on code in PR #8722: URL: https://github.com/apache/iceberg/pull/8722#discussion_r1347234482 ## docs/flink-writes.md: ## @@ -270,4 +270,4 @@ INSERT INTO tableName /*+ OPTIONS('upsert-enabled'='true') */ ... ``` -Check out all the options here: [write

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1347234000 ## docs-new/site/releases.md: ## @@ -0,0 +1,777 @@ +--- +title: "Releases" +--- + + +## Downloads + +The latest version of Iceberg is [{{ icebergVersion }}](http

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
ajantha-bhat commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347242330 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throws IO

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
jbonofre commented on code in PR #8722: URL: https://github.com/apache/iceberg/pull/8722#discussion_r1347247075 ## docs/flink-writes.md: ## @@ -270,4 +270,4 @@ INSERT INTO tableName /*+ OPTIONS('upsert-enabled'='true') */ ... ``` -Check out all the options here: [write-opt

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
shreyanshR7 commented on PR #8715: URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1748692640 I've made the changes :)@nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
nastra commented on code in PR #8715: URL: https://github.com/apache/iceberg/pull/8715#discussion_r1347350422 ## api/src/test/java/org/apache/iceberg/metrics/TestDefaultTimer.java: ## @@ -104,7 +104,7 @@ public void measureRunnable() { Runnable runnable = () -> {

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
nastra commented on code in PR #8715: URL: https://github.com/apache/iceberg/pull/8715#discussion_r1347351094 ## core/src/test/java/org/apache/iceberg/util/TestInMemoryLockManager.java: ## @@ -112,7 +114,7 @@ public void testAcquireSingleProcess() throws Exception { Complet

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
nastra commented on code in PR #8715: URL: https://github.com/apache/iceberg/pull/8715#discussion_r1347351835 ## flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -208,7 +210,7 @@ private void testContinuousIcebergSource(Failov

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
nastra commented on code in PR #8715: URL: https://github.com/apache/iceberg/pull/8715#discussion_r1347352377 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java: ## @@ -1955,7 +1958,6 @@ public void testRemoveOrphanFilesActionSuppo

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347356619 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
shreyanshR7 commented on PR #8715: URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1748822301 wait let me check -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
ajantha-bhat commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347371858 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throws IO

Re: [I] Adopt `Catalog` API to include references to the `TableMetadata` and the `metadata_location` in the `TableCommit` payload for the `update_table` method [iceberg-rust]

2023-10-05 Thread via GitHub
Xuanwo commented on issue #75: URL: https://github.com/apache/iceberg-rust/issues/75#issuecomment-1748854108 > It would therefore be helpful to include references to the `metadata_location` and the `TableMetadata` in the [`TableCommit`](https://github.com/apache/iceberg-rust/blob/main/crate

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
nastra commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347392925 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throws IOExcept

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347395043 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347395043 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347395043 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347395043 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347395043 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347411761 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347411761 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [I] Upgrade to gradle 8.4 [iceberg]

2023-10-05 Thread via GitHub
jbonofre commented on issue #8485: URL: https://github.com/apache/iceberg/issues/8485#issuecomment-1748905073 As gradle 8.4 is now available, I'm updating the PR (I'm also re-testing `revapi` but I'm pretty sure it will be the same as in Gradle 8.4 :) ). -- This is an automated message fr

Re: [PR] Core: Make view metadata properties optional in JSON parser [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8723: URL: https://github.com/apache/iceberg/pull/8723#discussion_r1347436052 ## core/src/main/java/org/apache/iceberg/view/ViewMetadataParser.java: ## @@ -66,7 +67,9 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throw

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-05 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1345991299 ## pyiceberg/manifest.py: ## @@ -262,15 +346,13 @@ class DataFile(Record): "split_offsets", "equality_ids", "sort_order_id", -"spec_

Re: [PR] Build: Upgrade to gradle 8.4 [iceberg]

2023-10-05 Thread via GitHub
jbonofre commented on PR #8486: URL: https://github.com/apache/iceberg/pull/8486#issuecomment-1748930989 Re-testing `revapi` on Gradle 8.4 (it should be the same as with Gradle 8.3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1347419881 ## core/src/main/java/org/apache/iceberg/GenericManifestFile.java: ## @@ -92,7 +92,7 @@ public GenericManifestFile(Schema avroSchema) { this.specId = spec

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-05 Thread via GitHub
Fokko commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1347459324 ## core/src/main/java/org/apache/iceberg/GenericManifestFile.java: ## @@ -92,7 +92,7 @@ public GenericManifestFile(Schema avroSchema) { this.specId = specId; th

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-05 Thread via GitHub
Fokko commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1347466602 ## core/src/main/java/org/apache/iceberg/GenericManifestFile.java: ## @@ -92,7 +92,7 @@ public GenericManifestFile(Schema avroSchema) { this.specId = specId; th

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-05 Thread via GitHub
Fokko commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1347466602 ## core/src/main/java/org/apache/iceberg/GenericManifestFile.java: ## @@ -92,7 +92,7 @@ public GenericManifestFile(Schema avroSchema) { this.specId = specId; th

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-10-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1347539389 ## core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java: ## @@ -374,4 +385,107 @@ static TableMetadata commit(TableOperations ops, UpdateTableReque

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
shreyanshR7 commented on PR #8715: URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1749145628 I tried to fix the issues :) @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
nastra commented on PR #8715: URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1749152592 @shreyanshR7 there are still many unrelated changes to files. Can you please fix all of those? -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] feat: add builder to TableMetadata interface [iceberg-rust]

2023-10-05 Thread via GitHub
y0psolo commented on PR #62: URL: https://github.com/apache/iceberg-rust/pull/62#issuecomment-1749197704 Ok i added some validation steps. Tell me if its ok i don't know if we need to have all fields visible on the builder. Tell me. We can choose to open and restrict later or do the rever

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
shreyanshR7 commented on PR #8715: URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1749205309 Let me check, I will try my best to undo these unrelated changes, these changes may be occurred due to adding awaitility method. sorry for inconvenience @nastra -- This is an auto

Re: [PR] Thread.sleep() method is replaced with Awaitility [iceberg]

2023-10-05 Thread via GitHub
shreyanshR7 commented on PR #8715: URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1749406399 @nastra I made another pull request addressing the issue with some more clean changes, please consider that. -- This is an automated message from the Apache Git Service. To respond

Re: [I] Support get partition table with filter [iceberg-python]

2023-10-05 Thread via GitHub
puchengy commented on issue #24: URL: https://github.com/apache/iceberg-python/issues/24#issuecomment-1749412122 @Fokko gentle ping, thanks ^ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] Write data to iceberg with using java api multi thread, cause data lost [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on issue #8610: URL: https://github.com/apache/iceberg/issues/8610#issuecomment-1749415259 Yes, I agree. The safety of the HadoopCatalog depends on the file system, which is one reason why we don't recommend using that catalog. -- This is an automated message from the Apa

Re: [I] Write data to iceberg with using java api multi thread, cause data lost [iceberg]

2023-10-05 Thread via GitHub
rdblue closed issue #8610: Write data to iceberg with using java api multi thread, cause data lost URL: https://github.com/apache/iceberg/issues/8610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-05 Thread via GitHub
Fokko commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1347814559 ## core/src/main/java/org/apache/iceberg/GenericManifestFile.java: ## @@ -92,7 +92,7 @@ public GenericManifestFile(Schema avroSchema) { this.specId = specId; th

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8722: URL: https://github.com/apache/iceberg/pull/8722#discussion_r1347814726 ## .github/workflows/python-release.yml: ## @@ -51,7 +51,7 @@ jobs: - name: Set version run: python -m poetry version "${{ inputs.version }}" wo

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on PR #8722: URL: https://github.com/apache/iceberg/pull/8722#issuecomment-1749425255 @jbonofre, how will this affect existing PRs? Do all of them need to be reopened against main instead of master? -- This is an automated message from the Apache Git Service. To respond t

Re: [I] Support get partition table with filter [iceberg-python]

2023-10-05 Thread via GitHub
Fokko commented on issue #24: URL: https://github.com/apache/iceberg-python/issues/24#issuecomment-1749431518 @puchengy Yes, certainly. Would this be something that you're interested in working on? From the snapshot, we can load the manifest list, and from there the manifests themselves, wh

Re: [PR] Hive: Push filtering for Iceberg table type to Hive MetaStore when listing tables [iceberg]

2023-10-05 Thread via GitHub
hankfanchiu commented on PR #2722: URL: https://github.com/apache/iceberg/pull/2722#issuecomment-1749443748 @mderoy, thanks for the interest! I don't plan on following through with apache/hive#2484, #2751, and this PR. Please feel free to take over. -- This is an automated message from th

Re: [PR] Rename master branch to main [iceberg]

2023-10-05 Thread via GitHub
jbonofre commented on PR #8722: URL: https://github.com/apache/iceberg/pull/8722#issuecomment-1749444235 @rdblue no impact on the existing PRs: they will be "attached" to main instead of master. The only impact is for the remotes for our users. -- This is an automated message from the Ap

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1347851058 ## format/spec.md: ## @@ -128,13 +128,13 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1347851474 ## format/spec.md: ## @@ -128,13 +128,13 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1347854144 ## format/spec.md: ## @@ -128,13 +128,13 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables

[PR] Revert "Spec: Mark added_snapshot_id as optional (#8600)" [iceberg]

2023-10-05 Thread via GitHub
Fokko opened a new pull request, #8726: URL: https://github.com/apache/iceberg/pull/8726 This reverts commit f7a7eb2c10cb4a9b6b3ea5bfdfc5d085be8b9c31. This should stay required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Revert "Spec: Mark added_snapshot_id as optional (#8600)" [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on PR #8726: URL: https://github.com/apache/iceberg/pull/8726#issuecomment-1749494924 Thank you! Good to get this fixed quickly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Spec: Inconsistency around files_count [iceberg]

2023-10-05 Thread via GitHub
Fokko commented on PR #5338: URL: https://github.com/apache/iceberg/pull/5338#issuecomment-1749550427 Thanks for the review @rdblue 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Spec disagrees with implementation on field names [iceberg]

2023-10-05 Thread via GitHub
Fokko closed issue #8684: Spec disagrees with implementation on field names URL: https://github.com/apache/iceberg/issues/8684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Spec: Inconsistency around files_count [iceberg]

2023-10-05 Thread via GitHub
Fokko merged PR #5338: URL: https://github.com/apache/iceberg/pull/5338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Revert "Spec: Mark added_snapshot_id as optional (#8600)" [iceberg]

2023-10-05 Thread via GitHub
rdblue merged PR #8726: URL: https://github.com/apache/iceberg/pull/8726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-05 Thread via GitHub
Fokko commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1347949961 ## api/src/main/java/org/apache/iceberg/ManifestFile.java: ## @@ -49,7 +49,7 @@ public interface ManifestFile { Types.LongType.get(), "Lowest sequen

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348043125 ## docs-new/mkdocs.yml: ## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTI

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
bitsondatadev commented on PR #8659: URL: https://github.com/apache/iceberg/pull/8659#issuecomment-1749722304 I've addressed the comments I intend to hit this review round, but have some open questions: - https://github.com/apache/iceberg/pull/8659#discussion_r1343890396 - https://gith

[PR] add support of before and after in preorderschema visitor for traversal [iceberg-python]

2023-10-05 Thread via GitHub
MehulBatra opened a new pull request, #42: URL: https://github.com/apache/iceberg-python/pull/42 Add support to perform an action before and after field traversal in preorder fashion for schema for object: structtype, maptype & listtype -- This is an automated message from the Apache

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348075495 ## docs-new/mkdocs.yml: ## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348076686 ## docs-new/mkdocs.yml: ## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348077823 ## docs-new/mkdocs.yml: ## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348077823 ## docs-new/mkdocs.yml: ## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348087050 ## docs-new/.github/bin/deploy_docs.sh: ## @@ -0,0 +1,39 @@ +#!/bin/bash + +while [[ "$#" -gt 0 ]]; do +case $1 in +-v|--version) ICEBERG_VERSION="$2"; shift

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348087991 ## docs-new/README.md: ## @@ -0,0 +1,123 @@ + + +# making-iceberg-docs + +Testbed for the Iceberg docs. + +## Requirements + +* Python >=3.9 +* pip + + +## Build proces

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348089146 ## docs-new/README.md: ## @@ -0,0 +1,175 @@ + + +# Iceberg site and documentation + +This holds the static files that define and build the documentation site for Apache

Re: [PR] add support of before and after actions in preorderschema traversal [iceberg-python]

2023-10-05 Thread via GitHub
rdblue commented on PR #42: URL: https://github.com/apache/iceberg-python/pull/42#issuecomment-1749783634 I don't think that this is necessary and I think it also breaks future uses of the visitor. Despite the name, the visitor this updates doesn't actually perform pre-order traversa

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1348099841 ## core/src/main/java/org/apache/iceberg/V2Metadata.java: ## @@ -39,7 +39,7 @@ private V2Metadata() {} ManifestFile.MANIFEST_CONTENT.asRequired(),

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1348100841 ## core/src/main/java/org/apache/iceberg/GenericManifestFile.java: ## @@ -84,15 +84,24 @@ public GenericManifestFile(Schema avroSchema) { } } + /** + * @dep

Re: [PR] Spec: add types timstamp_ns and timestamptz_ns [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on PR #8683: URL: https://github.com/apache/iceberg/pull/8683#issuecomment-1749792874 @jacobmarble, I was talking with @danielcweeks a couple days ago about v3 additions and we came up with a good reason to add `timestamp_ms` and `timestamptz_ms`. Currently, type promotion

Re: [PR] Spec: add types timstamp_ns and timestamptz_ns [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8683: URL: https://github.com/apache/iceberg/pull/8683#discussion_r1348103716 ## format/spec.md: ## @@ -167,30 +167,34 @@ A **`map`** is a collection of key-value pairs with a key type and a value type. Primitive Types -| Primitive type

Re: [PR] Spec: add types timstamp_ns and timestamptz_ns [iceberg]

2023-10-05 Thread via GitHub
jacobmarble commented on PR #8683: URL: https://github.com/apache/iceberg/pull/8683#issuecomment-1749799494 > What do you think? If you agree, then we should probably add `ms` types at the same time. Sounds good to me. I'll add milliseconds to this PR. -- This is an automated messa

Re: [PR] Kafka Connect: Initial project setup and event data structures [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8701: URL: https://github.com/apache/iceberg/pull/8701#discussion_r1348114157 ## kafka-connect/kafka-connect-events/src/main/java/org/apache/iceberg/connect/events/CommitCompletePayload.java: ## @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Kafka Connect: Initial project setup and event data structures [iceberg]

2023-10-05 Thread via GitHub
rdblue commented on code in PR #8701: URL: https://github.com/apache/iceberg/pull/8701#discussion_r1348115184 ## kafka-connect/kafka-connect-events/src/main/java/org/apache/iceberg/connect/events/Element.java: ## @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348126895 ## docs-new/.github/bin/deploy_docs.sh: ## @@ -0,0 +1,39 @@ +#!/bin/bash + +while [[ "$#" -gt 0 ]]; do +case $1 in +-v|--version) ICEBERG_VERSION="$2"

Re: [PR] Python: Remove python directory and references [iceberg]

2023-10-05 Thread via GitHub
ajantha-bhat commented on PR #8695: URL: https://github.com/apache/iceberg/pull/8695#issuecomment-1749882115 Shall we merge this if no comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Support get partition table with filter [iceberg-python]

2023-10-05 Thread via GitHub
puchengy commented on issue #24: URL: https://github.com/apache/iceberg-python/issues/24#issuecomment-1749897984 @Fokko Yes, I can help. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Phase 1 - New Docs Deployment [iceberg]

2023-10-05 Thread via GitHub
bitsondatadev commented on code in PR #8659: URL: https://github.com/apache/iceberg/pull/8659#discussion_r1348198680 ## docs-new/mkdocs.yml: ## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTI

[PR] Build: Add 1.4.0 to issue template [iceberg]

2023-10-05 Thread via GitHub
aokolnychyi opened a new pull request, #8728: URL: https://github.com/apache/iceberg/pull/8728 This PR adds 1.4.0 to our issue template. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Build: Add 1.4.0 to issue template [iceberg]

2023-10-05 Thread via GitHub
aokolnychyi commented on PR #8728: URL: https://github.com/apache/iceberg/pull/8728#issuecomment-1749928447 @Fokko @nastra @rdblue @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Build: Let revapi compare against 1.4.0 [iceberg]

2023-10-05 Thread via GitHub
aokolnychyi commented on PR #8727: URL: https://github.com/apache/iceberg/pull/8727#issuecomment-1749928311 @Fokko @nastra @rdblue @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] Common docs for 1.4.0 [iceberg-docs]

2023-10-05 Thread via GitHub
aokolnychyi opened a new pull request, #276: URL: https://github.com/apache/iceberg-docs/pull/276 This PR is the first step to update our docs as described [here](https://iceberg.apache.org/how-to-release/#common-documentation-update). -- This is an automated message from the Apache Git S

  1   2   >