Re: [I] table_exists error raisng 400 error code [iceberg-python]

2024-12-06 Thread via GitHub
sungwy commented on issue #1378: URL: https://github.com/apache/iceberg-python/issues/1378#issuecomment-2524880844 Hi @dongsupkim-onepredict ! Please try using this docker image from the dockerhub instead: `apache/iceberg-rest-fixture` I'm closing this issue as there is now a table_e

Re: [I] table_exists error raisng 400 error code [iceberg-python]

2024-12-06 Thread via GitHub
sungwy closed issue #1378: table_exists error raisng 400 error code URL: https://github.com/apache/iceberg-python/issues/1378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] fix warnings from newer versions of dependencies [iceberg-python]

2024-12-06 Thread via GitHub
kevinjqliu opened a new pull request, #1414: URL: https://github.com/apache/iceberg-python/pull/1414 Closes #1409 This PR updates the following packages, (using `poetry update `): - pandas `2.0.3` -> `2.2.3` - getdaft `0.3.14` -> `0.3.15` Fixed warning for daft, allow list

Re: [PR] Remove support for catalog_name in table identifier string [iceberg-python]

2024-12-06 Thread via GitHub
sungwy commented on PR #963: URL: https://github.com/apache/iceberg-python/pull/963#issuecomment-2524829316 Now that the deprecation is out for 0.8.0 - I'll get working on this large PR again -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] Spark 3.3/3.4: Align RewritePositionDeleteFilesSparkAction filter case sensitivity with Spark case sensitivity [iceberg]

2024-12-06 Thread via GitHub
huaxingao commented on PR #11710: URL: https://github.com/apache/iceberg/pull/11710#issuecomment-2524826084 Thanks @szehon-ho @singhpk234 @dramaticlly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Add `view_exists` method to REST Catalog [iceberg-python]

2024-12-06 Thread via GitHub
sungwy commented on PR #1242: URL: https://github.com/apache/iceberg-python/pull/1242#issuecomment-2524819444 Hi @shiv-io ! Sorry this fell off my radar. Could we actually add an integration test here as well, now that we have a new REST Catalog image we are testing against that supp

Re: [PR] TEST: adopt new rest catalog image and enable tableExists tests [iceberg-python]

2024-12-06 Thread via GitHub
sungwy commented on PR #1389: URL: https://github.com/apache/iceberg-python/pull/1389#issuecomment-2524818861 Thanks for the reviews @Fokko and @kevinjqliu ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] TEST: adopt new rest catalog image and enable tableExists tests [iceberg-python]

2024-12-06 Thread via GitHub
sungwy merged PR #1389: URL: https://github.com/apache/iceberg-python/pull/1389 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Spark 3.3/3.4: Align RewritePositionDeleteFilesSparkAction filter case sensitivity with Spark case sensitivity [iceberg]

2024-12-06 Thread via GitHub
szehon-ho merged PR #11710: URL: https://github.com/apache/iceberg/pull/11710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark 3.3/3.4: Align RewritePositionDeleteFilesSparkAction filter case sensitivity with Spark case sensitivity [iceberg]

2024-12-06 Thread via GitHub
szehon-ho commented on PR #11710: URL: https://github.com/apache/iceberg/pull/11710#issuecomment-2524810180 Merged, thanks @huaxingao , and also @singhpk234 and @dramaticlly for additional review -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2024-12-06 Thread via GitHub
szehon-ho commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1874218545 ## api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java: ## @@ -91,9 +92,10 @@ interface Result { String stagingLocation(); /** - * Pa

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2024-12-06 Thread via GitHub
szehon-ho commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1874217873 ## core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java: ## @@ -60,28 +69,63 @@ public static CharSequence referencedDataFile(DeleteFile deleteFile) {

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2024-12-06 Thread via GitHub
szehon-ho commented on PR #11555: URL: https://github.com/apache/iceberg/pull/11555#issuecomment-2524803591 @flyrain thanks for review! I spent some time cleaning it up. Now more things are moved to RewriteTablePathsUtil, removing the need to make ManifestLists public. Also , I moved the

Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

2024-12-06 Thread via GitHub
fqaiser94 commented on code in PR #765: URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1874212744 ## crates/puffin/src/lib.rs: ## @@ -23,3 +23,8 @@ mod compression; pub use compression::CompressionCodec; + +mod metadata; Review Comment: Note that I have

Re: [I] [DISCUSSION] Project Goal [iceberg-cpp]

2024-12-06 Thread via GitHub
lidavidm commented on issue #2: URL: https://github.com/apache/iceberg-cpp/issues/2#issuecomment-2524781385 > Another possibility perhaps would be an IO-less abstraction (the Iceberg library tells you what it is waiting for, and you give it what it asks for). Probably more complex to design

[PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

2024-12-06 Thread via GitHub
fqaiser94 opened a new pull request, #765: URL: https://github.com/apache/iceberg-rust/pull/765 Part of https://github.com/apache/iceberg-rust/issues/744 # Summary - Add support for parsing Puffin FileMetadata # Context This is the second of a number of PRs to add support

[PR] Docs: Add Javadocs for 1.7.1 release [iceberg]

2024-12-06 Thread via GitHub
bryanck opened a new pull request, #11718: URL: https://github.com/apache/iceberg/pull/11718 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-06 Thread via GitHub
aokolnychyi commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1874176834 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DVIterator.java: ## @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-06 Thread via GitHub
aokolnychyi commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1874176544 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DVIterator.java: ## @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] [WIP] API, Core, Spark: Proof of concept implementation of the geo support proposal [iceberg]

2024-12-06 Thread via GitHub
github-actions[bot] commented on PR #11293: URL: https://github.com/apache/iceberg/pull/11293#issuecomment-2524686864 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Build: Bump com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.18.1 [iceberg]

2024-12-06 Thread via GitHub
github-actions[bot] closed pull request #10799: Build: Bump com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.18.1 URL: https://github.com/apache/iceberg/pull/10799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Storage Partitioned Join (SPJ) fails when >2 tables are joined [iceberg]

2024-12-06 Thread via GitHub
github-actions[bot] commented on issue #10450: URL: https://github.com/apache/iceberg/issues/10450#issuecomment-2524686685 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] [WIP] API, Core, Spark: Proof of concept implementation of the geo support proposal [iceberg]

2024-12-06 Thread via GitHub
github-actions[bot] closed pull request #11293: [WIP] API, Core, Spark: Proof of concept implementation of the geo support proposal URL: https://github.com/apache/iceberg/pull/11293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Build: Bump com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.18.1 [iceberg]

2024-12-06 Thread via GitHub
dependabot[bot] commented on PR #10799: URL: https://github.com/apache/iceberg/pull/10799#issuecomment-2524686853 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, l

Re: [PR] Build: Bump com.aliyun.oss:aliyun-sdk-oss from 3.10.2 to 3.18.1 [iceberg]

2024-12-06 Thread via GitHub
github-actions[bot] commented on PR #10799: URL: https://github.com/apache/iceberg/pull/10799#issuecomment-2524686794 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Spark 3.5: Spark property for avoiding write failures due to missing fields in complex data types [iceberg]

2024-12-06 Thread via GitHub
github-actions[bot] commented on PR #10361: URL: https://github.com/apache/iceberg/pull/10361#issuecomment-2524686560 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Try create Iceberg metadata table for Jdbc catalog in initialization [iceberg]

2024-12-06 Thread via GitHub
github-actions[bot] commented on PR #11427: URL: https://github.com/apache/iceberg/pull/11427#issuecomment-2524686913 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

[PR] Docs: Add release notes for 1.7.1 [iceberg]

2024-12-06 Thread via GitHub
bryanck opened a new pull request, #11717: URL: https://github.com/apache/iceberg/pull/11717 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

[PR] Relocate parquet to core [iceberg]

2024-12-06 Thread via GitHub
danielcweeks opened a new pull request, #11716: URL: https://github.com/apache/iceberg/pull/11716 This PR relocates the parquet implementation from the `iceberg-parquet` project to `iceberg-core` so that parquet can be used as part of metadata operations. The project and artifacts ar

[PR] Add version 1.7.1 release docs [iceberg]

2024-12-06 Thread via GitHub
bryanck opened a new pull request, #11715: URL: https://github.com/apache/iceberg/pull/11715 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Flink: Maintenance - RewriteDataFiles [iceberg]

2024-12-06 Thread via GitHub
pvary commented on code in PR #11497: URL: https://github.com/apache/iceberg/pull/11497#discussion_r1874112948 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LogUtil.java: ## @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Flink: Maintenance - RewriteDataFiles [iceberg]

2024-12-06 Thread via GitHub
pvary commented on code in PR #11497: URL: https://github.com/apache/iceberg/pull/11497#discussion_r1874111940 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestRewriteDataFiles.java: ## @@ -0,0 +1,447 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Docs: Add guidelines for contributors to become committers [iceberg]

2024-12-06 Thread via GitHub
dramaticlly commented on code in PR #11670: URL: https://github.com/apache/iceberg/pull/11670#discussion_r1874095478 ## site/docs/community.md: ## @@ -100,3 +102,56 @@ Recruitment of community members should not be conducted through direct messages related to contributing to o

[PR] Bump coverage from 7.6.8 to 7.6.9 [iceberg-python]

2024-12-06 Thread via GitHub
dependabot[bot] opened a new pull request, #1413: URL: https://github.com/apache/iceberg-python/pull/1413 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.6.8 to 7.6.9. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's cha

Re: [PR] Update ASF doap.rdf to release 1.7.1 [iceberg]

2024-12-06 Thread via GitHub
RussellSpitzer merged PR #11712: URL: https://github.com/apache/iceberg/pull/11712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Update ASF doap.rdf to release 1.7.1 [iceberg]

2024-12-06 Thread via GitHub
RussellSpitzer commented on PR #11712: URL: https://github.com/apache/iceberg/pull/11712#issuecomment-2524237586 Thanks @bryanck and all the reviewers @singhpk234 and @Fokko ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [I] Allow `file_format` to be lower-case [iceberg-python]

2024-12-06 Thread via GitHub
jiakai-li commented on issue #1340: URL: https://github.com/apache/iceberg-python/issues/1340#issuecomment-2524188435 Not sure if I'm still able to work on this or not. But I submitted another PR, rather than modifying `__new__`, I think `_missing_` is a better candidate given its semantic

Re: [PR] [INFRA] make integration test more resilient to intermittent failures [iceberg-python]

2024-12-06 Thread via GitHub
kevinjqliu merged PR #1412: URL: https://github.com/apache/iceberg-python/pull/1412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

[PR] Add C++ to the list of languages in `doap.rdf` [iceberg]

2024-12-06 Thread via GitHub
Fokko opened a new pull request, #11714: URL: https://github.com/apache/iceberg/pull/11714 From the list of accepted values: https://projects.apache.org/validation.json -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Update `doap.rdf` to Iceberg 1.7.1 [iceberg]

2024-12-06 Thread via GitHub
Fokko commented on PR #11713: URL: https://github.com/apache/iceberg/pull/11713#issuecomment-2524160320 Wrong handle @sfc-gh-rspitzer, let's approve https://github.com/apache/iceberg/pull/11712 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Update `doap.rdf` to Iceberg 1.7.1 [iceberg]

2024-12-06 Thread via GitHub
RussellSpitzer commented on PR #11713: URL: https://github.com/apache/iceberg/pull/11713#issuecomment-2524160874 Woops wrong account -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Update `doap.rdf` to Iceberg 1.7.1 [iceberg]

2024-12-06 Thread via GitHub
Fokko closed pull request #11713: Update `doap.rdf` to Iceberg 1.7.1 URL: https://github.com/apache/iceberg/pull/11713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

[PR] Update `doap.rdf` to Iceberg 1.7.1 [iceberg]

2024-12-06 Thread via GitHub
Fokko opened a new pull request, #11713: URL: https://github.com/apache/iceberg/pull/11713 And also add C++ along the way. From the list of accepted values: https://projects.apache.org/validation.json -- This is an automated message from the Apache Git Service. To respond to the message,

[PR] integration test mor resilient to intermittent failures [iceberg-python]

2024-12-06 Thread via GitHub
kevinjqliu opened a new pull request, #1412: URL: https://github.com/apache/iceberg-python/pull/1412 CI constantly fails because `curl` errors intermittently, up curl retry to `5`. By default `--retry` uses exponential backoff. From the docs, https://curl.se/docs/manpage.html ```

Re: [PR] Add `_missing_` to make `FileFormat` case insensitive [iceberg-python]

2024-12-06 Thread via GitHub
jiakai-li commented on PR #1411: URL: https://github.com/apache/iceberg-python/pull/1411#issuecomment-2524090648 Hey @kevinjqliu @Fokko, this PR is ready for review. Rather than modifying the `__new__` method, I think it's probably a better option to modify `_missing_` based on its semantic

Re: [PR] Infra: Add 1.7.1 to issue template [iceberg]

2024-12-06 Thread via GitHub
RussellSpitzer merged PR #11711: URL: https://github.com/apache/iceberg/pull/11711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

[PR] Add `_missing_` to make `FileFormat` case insensitive [iceberg-python]

2024-12-06 Thread via GitHub
jiakai-li opened a new pull request, #1411: URL: https://github.com/apache/iceberg-python/pull/1411 **This pull request makes the following changes:** * Fixes issue #1362 * The change is to make `FileFormat` case insensitve. The idea is to add `_missing_` funtion to handle lower ca

Re: [PR] Infra: Add 1.7.1 to issue template [iceberg]

2024-12-06 Thread via GitHub
RussellSpitzer commented on PR #11711: URL: https://github.com/apache/iceberg/pull/11711#issuecomment-2524072372 Thanks @bryanck for the PR and @Fokko for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] 0.8.1 post release [iceberg-python]

2024-12-06 Thread via GitHub
kevinjqliu opened a new pull request, #1410: URL: https://github.com/apache/iceberg-python/pull/1410 Post-release steps for 0.8.1 release https://py.iceberg.apache.org/how-to-release/#post-release Similar to #1334 -- This is an automated message from the Apache Git Service. To resp

Re: [PR] Added curl to the iceberg rest fixture Docker image #11688 [iceberg]

2024-12-06 Thread via GitHub
Fokko commented on code in PR #11705: URL: https://github.com/apache/iceberg/pull/11705#discussion_r1873897394 ## docker/iceberg-rest-fixture/Dockerfile: ## @@ -36,6 +39,10 @@ ENV CATALOG_JDBC_USER=user ENV CATALOG_JDBC_PASSWORD=password ENV REST_PORT=8181 +# Healthcheck for

Re: [I] Support for S3 catalog to work with S3 Tables [iceberg-python]

2024-12-06 Thread via GitHub
nlm4145 commented on issue #1404: URL: https://github.com/apache/iceberg-python/issues/1404#issuecomment-2524033719 I also would be interested in this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] API table.scan does not conform to Iceberg spec for identity partition columns [iceberg-python]

2024-12-06 Thread via GitHub
kevinjqliu commented on issue #1401: URL: https://github.com/apache/iceberg-python/issues/1401#issuecomment-2524020723 @gabeiglio assigned to you for now since I realized OP checked "I cannot contribute a fix for this bug at this time" -- This is an automated message from the Apache Git

Re: [I] API table.scan does not conform to Iceberg spec for identity partition columns [iceberg-python]

2024-12-06 Thread via GitHub
gabeiglio commented on issue #1401: URL: https://github.com/apache/iceberg-python/issues/1401#issuecomment-2524000604 If its available I would want to give it a go! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-06 Thread via GitHub
nastra closed pull request #11657: Spark: Read DVs when reading from .position_deletes table URL: https://github.com/apache/iceberg/pull/11657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] API table.scan does not conform to Iceberg spec for identity partition columns [iceberg-python]

2024-12-06 Thread via GitHub
kevinjqliu commented on issue #1401: URL: https://github.com/apache/iceberg-python/issues/1401#issuecomment-2523936829 @rkuhlercadent thanks a bunch for reporting this issue! Would you like to help contribute this fix? -- This is an automated message from the Apache Git Service. To resp

Re: [PR] Spark 3.3/3.4: Align RewritePositionDeleteFilesSparkAction filter case sensitivity with Spark case sensitivity [iceberg]

2024-12-06 Thread via GitHub
huaxingao commented on PR #11710: URL: https://github.com/apache/iceberg/pull/11710#issuecomment-2523933071 @szehon-ho I changed to the correct title. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] API table.scan does not conform to Iceberg spec for identity partition columns [iceberg-python]

2024-12-06 Thread via GitHub
kevinjqliu commented on issue #1401: URL: https://github.com/apache/iceberg-python/issues/1401#issuecomment-2523936027 Heres the code path for arrow table scan: `Table.to_arrow`: https://github.com/apache/iceberg-python/blob/bfc0d9a62176803094da0867ee793808f105d352/pyiceberg/table

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-12-06 Thread via GitHub
huaxingao commented on PR #9841: URL: https://github.com/apache/iceberg/pull/9841#issuecomment-2523905361 @aokolnychyi Could you please take a look again? I have changed the default to `Comet` to make sure all the tests run successfully with `Comet`. I will switch back to the regular iceber

Re: [I] API table.scan does not conform to Iceberg spec for identity partition columns [iceberg-python]

2024-12-06 Thread via GitHub
kevinjqliu commented on issue #1401: URL: https://github.com/apache/iceberg-python/issues/1401#issuecomment-2523929896 Thanks for providing the test! I added a few print statements ``` import os import datetime from pyiceberg.catalog.sql import SqlCatalog from pyiceberg.io

[PR] Spark 3.4/3.5: Align RewritePositionDeleteFilesSparkAction filter case sensitivity with Spark case sensitivity [iceberg]

2024-12-06 Thread via GitHub
huaxingao opened a new pull request, #11710: URL: https://github.com/apache/iceberg/pull/11710 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-06 Thread via GitHub
singhpk234 commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1873786968 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DVIterable.java: ## @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [I] expire-snapshots isn't removing data files after DELETE + expire [iceberg]

2024-12-06 Thread via GitHub
paul-bormans-pcgw commented on issue #11695: URL: https://github.com/apache/iceberg/issues/11695#issuecomment-2523846982 I'm debugging this issue by adding LOG statements to https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java#L261 it

Re: [I] API table.scan does not conform to Iceberg spec for identity partition columns [iceberg-python]

2024-12-06 Thread via GitHub
rkuhlercadent commented on issue #1401: URL: https://github.com/apache/iceberg-python/issues/1401#issuecomment-2523813801 Here is a python script that will demonstrate the issue. ``` import os import datetime from pyiceberg.catalog.sql import SqlCatalog from pyiceberg.

Re: [PR] Spark: Write DVs for V3 MoR tables [iceberg]

2024-12-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #11561: URL: https://github.com/apache/iceberg/pull/11561#discussion_r1873686457 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java: ## @@ -162,13 +162,14 @@ public void filter(Predicate[] predicate

Re: [PR] Spark: Write DVs for V3 MoR tables [iceberg]

2024-12-06 Thread via GitHub
amogh-jahagirdar merged PR #11561: URL: https://github.com/apache/iceberg/pull/11561 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spark: Write DVs for V3 MoR tables [iceberg]

2024-12-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #11561: URL: https://github.com/apache/iceberg/pull/11561#discussion_r1873680851 ## core/src/main/java/org/apache/iceberg/io/PartitioningDVWriter.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark: Write DVs for V3 MoR tables [iceberg]

2024-12-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #11561: URL: https://github.com/apache/iceberg/pull/11561#discussion_r1873681281 ## core/src/main/java/org/apache/iceberg/io/PartitioningDVWriter.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
pitrou commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873677036 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# dist

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-06 Thread via GitHub
nastra commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1873645147 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPositionDeletesReader.java: ## @@ -0,0 +1,247 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-06 Thread via GitHub
nastra commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1873645147 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPositionDeletesReader.java: ## @@ -0,0 +1,247 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
gaborkaszab commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873639268 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +#

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-06 Thread via GitHub
nastra commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1873619891 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/PositionDeletesRowReader.java: ## @@ -90,6 +91,10 @@ protected CloseableIterator open(PositionDelete

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-06 Thread via GitHub
gaborkaszab commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1873615824 ## cmake-format.py: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file

Re: [PR] Added curl to the iceberg rest fixture Docker image #11688 [iceberg]

2024-12-06 Thread via GitHub
dominikhei commented on PR #11705: URL: https://github.com/apache/iceberg/pull/11705#issuecomment-2523583449 @danielcweeks let me know if it is fine if I just probe the `$REST_PORT `. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] WIP: Use localhost instead of container hostname [iceberg-rust]

2024-12-06 Thread via GitHub
nastra commented on PR #748: URL: https://github.com/apache/iceberg-rust/pull/748#issuecomment-2523569678 @Fokko you're not alone. Those tests fail for me too using native Docker on OSX due to networking: ``` test tests::gcs_exists has been running for over 60 seconds test tests

Re: [PR] Build: Delete branch automatically on PR merge [iceberg-rust]

2024-12-06 Thread via GitHub
Xuanwo merged PR #764: URL: https://github.com/apache/iceberg-rust/pull/764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] test: append partition data file [iceberg-rust]

2024-12-06 Thread via GitHub
ZENOTME commented on PR #742: URL: https://github.com/apache/iceberg-rust/pull/742#issuecomment-2523503652 > Hey @Fokko 👋🏻 > > Thanks a lot for checking up in detail! Can I take up both of the issues as both are related to this test itself? 😅 > > Also, slightly tangential, but

Re: [PR] Flink: Maintenance - RewriteDataFiles [iceberg]

2024-12-06 Thread via GitHub
mxm commented on code in PR #11497: URL: https://github.com/apache/iceberg/pull/11497#discussion_r1873507376 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DataFileRewriteExecutor.java: ## @@ -0,0 +1,257 @@ +/* + * Licensed to the Apache Softwar

[PR] Build: Delete branch automatically on PR merge [iceberg-rust]

2024-12-06 Thread via GitHub
manuzhang opened a new pull request, #764: URL: https://github.com/apache/iceberg-rust/pull/764 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[PR] Build: Delete branch automatically on PR merge [iceberg-python]

2024-12-06 Thread via GitHub
manuzhang opened a new pull request, #1408: URL: https://github.com/apache/iceberg-python/pull/1408 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Flink: Maintenance - RewriteDataFiles [iceberg]

2024-12-06 Thread via GitHub
mxm commented on code in PR #11497: URL: https://github.com/apache/iceberg/pull/11497#discussion_r1873496955 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LogUtil.java: ## @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] test: append partition data file [iceberg-rust]

2024-12-06 Thread via GitHub
ZENOTME commented on code in PR #742: URL: https://github.com/apache/iceberg-rust/pull/742#discussion_r1873487448 ## crates/integration_tests/tests/append_partition_data_file_test.rs: ## @@ -0,0 +1,214 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
raulcd commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873469974 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: I am not familiar with the `cpp-linter-action` or t

Re: [I] Kafka connect iceberg connector seems to stop after having processed all messages in Kafka topic [iceberg]

2024-12-06 Thread via GitHub
thjaeckle commented on issue #11703: URL: https://github.com/apache/iceberg/issues/11703#issuecomment-2523430074 Maybe this has something to do with me setting `"iceberg.connect.group-id"` explicitly? This is not documented at https://iceberg.apache.org/docs/nightly/kafka-connect/#config

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-06 Thread via GitHub
zhjwpku commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1873455730 ## cmake-format.py: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# d

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-06 Thread via GitHub
zhjwpku commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1873451868 ## src/puffin/CMakeLists.txt: ## @@ -18,11 +18,10 @@ set(ICEBERG_PUFFIN_SOURCES demo_puffin.cc) set(ICEBERG_PUFFIN_INCLUDES "${ICEBERG_API_DIR}") -ADD_ICEBERG_LIB( -

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
wgtmac commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873449223 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: I don't think they are blockers at this moment. Eve

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
zhjwpku commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873444269 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: I agree we should have a pre-commit file and Githu

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
zhjwpku commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873436117 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# dis

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-06 Thread via GitHub
wgtmac commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1873421386 ## src/puffin/CMakeLists.txt: ## @@ -18,11 +18,10 @@ set(ICEBERG_PUFFIN_SOURCES demo_puffin.cc) set(ICEBERG_PUFFIN_INCLUDES "${ICEBERG_API_DIR}") -ADD_ICEBERG_LIB( -

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
wgtmac commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873409074 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: Can we integrate a Github action like https://gith

[PR] Simplify partition structures [iceberg-rust]

2024-12-06 Thread via GitHub
Fokko opened a new pull request, #763: URL: https://github.com/apache/iceberg-rust/pull/763 This PR removes `SchemalessPartitionSpec` and `UnboundPartitionSpecField`. We could also combine `BoundPartitionSpec` and `UnboundPartitionSpec` if we like, but this is already quite a big change.

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
wgtmac commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873409074 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: Can we integrate a Github action like https://gith

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
wgtmac commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873402979 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# dist

Re: [I] [DISCUSSION] Project Goal [iceberg-cpp]

2024-12-06 Thread via GitHub
wgtmac commented on issue #2: URL: https://github.com/apache/iceberg-cpp/issues/2#issuecomment-2523303832 > The FileIO is designed to avoid certain operations (move/list/etc), and it only does a few things (read, create, and delete). If wrap this into an abstraction, that would work just as

Re: [I] Kafka connect iceberg connector seems to stop after having processed all messages in Kafka topic [iceberg]

2024-12-06 Thread via GitHub
thjaeckle commented on issue #11703: URL: https://github.com/apache/iceberg/issues/11703#issuecomment-2523319487 I now figured out (by adding some more logs to the Iceberg connector) that at this point: https://github.com/apache/iceberg/blob/38c8daa4eae8a75ab46571f1efce1609100f53dd/kafka

Re: [PR] refine: refine interface of ManifestWriter [iceberg-rust]

2024-12-06 Thread via GitHub
ZENOTME commented on code in PR #738: URL: https://github.com/apache/iceberg-rust/pull/738#discussion_r1873384095 ## crates/iceberg/src/spec/manifest.rs: ## @@ -203,12 +206,80 @@ impl ManifestWriter { partition_summary } -/// Write a manifest. -pub async

Re: [PR] refine: refine writer interface [iceberg-rust]

2024-12-06 Thread via GitHub
ZENOTME commented on code in PR #741: URL: https://github.com/apache/iceberg-rust/pull/741#discussion_r1873379608 ## crates/iceberg/src/writer/file_writer/mod.rs: ## @@ -37,11 +37,11 @@ pub trait FileWriterBuilder: Send + Clone + 'static { /// The associated file writer ty

Re: [I] [Request] Area of Improvements for Documentation [iceberg-python]

2024-12-06 Thread via GitHub
jeppe-dos commented on issue #1407: URL: https://github.com/apache/iceberg-python/issues/1407#issuecomment-2523199638 The schema evolution on the struct is incorrect in the pyiceberg api documention [Schema evolution](https://py.iceberg.apache.org/api/#add-column). It says that to use dot

Re: [PR] Add clang format [iceberg-cpp]

2024-12-06 Thread via GitHub
pitrou commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1873276234 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: How about adding a pre-commit file? -- This is

  1   2   >