Re: [PR] New iceberg blogs nov 2024 [iceberg]

2024-10-29 Thread via GitHub
nastra merged PR #11424: URL: https://github.com/apache/iceberg/pull/11424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Count rows as a metadata-only operation [iceberg-python]

2024-10-29 Thread via GitHub
Fokko commented on issue #1223: URL: https://github.com/apache/iceberg-python/issues/1223#issuecomment-2445969277 Thanks @Visorgood for reaching out here, and that's an excellent idea. We actually already do this in a project like Datahub, see: https://github.com/datahub-project/datahub/bl

Re: [I] [bug] Cannot perform table scan on V1 table [iceberg-python]

2024-10-29 Thread via GitHub
Fokko commented on issue #1194: URL: https://github.com/apache/iceberg-python/issues/1194#issuecomment-2445957746 @kevinjqliu Thanks for raising this. Can you elaborate on what you encountered when reading a V1 table? The Iceberg metadata is forward compatible, meaning we can turn any V1 t

Re: [I] NumPy Hardpin 1.26 issue [iceberg-python]

2024-10-29 Thread via GitHub
Fokko commented on issue #1198: URL: https://github.com/apache/iceberg-python/issues/1198#issuecomment-2445953768 I think we can remove the numpy dependency completely once we get https://github.com/apache/iceberg-python/issues/1259 fixed. This will give flexibility to the user on which ve

Re: [PR] GCS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
nastra commented on PR #11282: URL: https://github.com/apache/iceberg/pull/11282#issuecomment-2445954106 thanks for the review @danielcweeks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] AWS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
nastra commented on code in PR #11389: URL: https://github.com/apache/iceberg/pull/11389#discussion_r1821941647 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -136,6 +156,12 @@ public void applyClientCredentialConfigurations(T b @SuppressWarning

Re: [PR] GCS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
nastra merged PR #11282: URL: https://github.com/apache/iceberg/pull/11282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] AWS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
nastra commented on code in PR #11389: URL: https://github.com/apache/iceberg/pull/11389#discussion_r1821939030 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -66,21 +67,39 @@ public class AwsClientProperties implements Serializable { */ publi

Re: [I] Hive metastore 4.0.1 remove deprecated thrift APIs [iceberg-python]

2024-10-29 Thread via GitHub
Fokko commented on issue #1222: URL: https://github.com/apache/iceberg-python/issues/1222#issuecomment-2445948742 Thanks for reporting this @mattheusv To get to the bottom of this: - I would first suggest bumping the Hive container to 4.0.1 - Maybe regenerate the Hive Thrif

Re: [PR] Core: delete temp metadata file when version already exists [iceberg]

2024-10-29 Thread via GitHub
Fokko commented on PR #11350: URL: https://github.com/apache/iceberg/pull/11350#issuecomment-2445940060 @leesf Thanks! Can you run `./gradlew :iceberg-core:spotlessApply` to fix the formatting? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Core: delete temp metadata file when version already exists [iceberg]

2024-10-29 Thread via GitHub
leesf commented on PR #11350: URL: https://github.com/apache/iceberg/pull/11350#issuecomment-2445898551 > This makes sense. @leesf Could you add a test so we don't break this in the future? @Fokko Added check to the existing UT with dist file already exists failure. -- This is an a

Re: [I] API to optimize table metadata [iceberg]

2024-10-29 Thread via GitHub
ajantha-bhat commented on issue #11425: URL: https://github.com/apache/iceberg/issues/11425#issuecomment-2445925564 This one seems to be related: https://github.com/apache/iceberg/pull/10755, but it just handles the spec clean up. We need to handle the same for schema and sort orders. --

[I] API to optimize table metadata [iceberg]

2024-10-29 Thread via GitHub
ajantha-bhat opened a new issue, #11425: URL: https://github.com/apache/iceberg/issues/11425 ### Feature Request / Improvement Table metadata has history of partition spec, sort orders, schemas. There is no way to shorten the history from the table metadata. It ends up bloatin

Re: [PR] Remove `initial_change` when CreateTableTransaction apply table updates on an empty metadata [iceberg-python]

2024-10-29 Thread via GitHub
HonahX commented on PR #1219: URL: https://github.com/apache/iceberg-python/pull/1219#issuecomment-2445891924 @kevinjqliu Thanks for reviewing this! I've updated the PR. It will be great to include this in the next release. -- This is an automated message from the Apache Git Service. To

Re: [PR] Remove `initial_change` when CreateTableTransaction apply table updates on an empty metadata [iceberg-python]

2024-10-29 Thread via GitHub
HonahX commented on code in PR #1219: URL: https://github.com/apache/iceberg-python/pull/1219#discussion_r1821898893 ## dev/Dockerfile: ## @@ -36,9 +36,9 @@ ENV PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip:$ RUN mkdir -p ${HADOOP_HOME} && mkdir -p

Re: [PR] Core: Add Variant implementation to read serialized objects [iceberg]

2024-10-29 Thread via GitHub
aihuaxu commented on code in PR #11415: URL: https://github.com/apache/iceberg/pull/11415#discussion_r1821861473 ## core/src/main/java/org/apache/iceberg/Variants.java: ## @@ -0,0 +1,207 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one + * * or more

Re: [I] pyarrow 18 regression: ValueError: type(schema)= [iceberg-python]

2024-10-29 Thread via GitHub
kevinjqliu commented on issue #1265: URL: https://github.com/apache/iceberg-python/issues/1265#issuecomment-2445614118 i was not able to reproduce on my local setup. On `main` branch, ``` poetry add pyarrow==18.0.0 poetry run python run.py ``` I noticed that in your e

Re: [PR] Core: Suppress exceptions in case of dropTableData [iceberg]

2024-10-29 Thread via GitHub
caohoangha126 commented on PR #9184: URL: https://github.com/apache/iceberg/pull/9184#issuecomment-2445697418 @nk1506 some checks failed and block merging, do you mind taking a look? Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] core: Filter on live entries when reading the manifest [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9996: URL: https://github.com/apache/iceberg/pull/9996#issuecomment-2445551686 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] [DRAFT]: Adjust site links to absolute from site_url [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9887: URL: https://github.com/apache/iceberg/pull/9887#issuecomment-2445551330 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core: use ManifestFiles.open when possible [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer merged PR #11414: URL: https://github.com/apache/iceberg/pull/11414 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] AWS: Enable RetryMode for AWS KMS client [iceberg]

2024-10-29 Thread via GitHub
hsiang-c commented on code in PR #11420: URL: https://github.com/apache/iceberg/pull/11420#discussion_r1821695163 ## aws/src/main/java/org/apache/iceberg/aws/kms/KmsClientProperties.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] fix: do not sort indices for `ProjectionMask::leaves` [iceberg-rust]

2024-10-29 Thread via GitHub
wcy-fdu closed pull request #682: fix: do not sort indices for `ProjectionMask::leaves` URL: https://github.com/apache/iceberg-rust/pull/682 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] fix: do not sort indices for `ProjectionMask::leaves` [iceberg-rust]

2024-10-29 Thread via GitHub
wcy-fdu commented on PR #682: URL: https://github.com/apache/iceberg-rust/pull/682#issuecomment-2445726838 After investigation, I found that although [ProjectionMask::leaves()](https://github.com/apache/iceberg-rust/blob/2f3554b30a05028aef1d46d37209e24e1ed96582/crates/iceberg/src/arrow/reade

Re: [PR] AWS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #11389: URL: https://github.com/apache/iceberg/pull/11389#discussion_r1821787031 ## aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java: ## @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] AWS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #11389: URL: https://github.com/apache/iceberg/pull/11389#discussion_r1821787031 ## aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java: ## @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [I] when using SQL catalog, table location is not optional [iceberg-python]

2024-10-29 Thread via GitHub
djouallah closed issue #1254: when using SQL catalog, table location is not optional URL: https://github.com/apache/iceberg-python/issues/1254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9924: Hive: Common namespace validation for HiveCatalog URL: https://github.com/apache/iceberg/pull/9924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

2024-10-29 Thread via GitHub
manuzhang commented on PR #11421: URL: https://github.com/apache/iceberg/pull/11421#issuecomment-2445695391 We might need to update [this part of table spec](https://iceberg.apache.org/spec/#version-2) -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] core,api: Refactor code with `hasLiveEntries` [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9993: URL: https://github.com/apache/iceberg/pull/9993#issuecomment-2445551655 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #10002: AWS: Glue table operations hang when aws authentication parameters are illegal URL: https://github.com/apache/iceberg/pull/10002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Core:Add test for removing partition filed [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9903: Core:Add test for removing partition filed URL: https://github.com/apache/iceberg/pull/9903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] hdfs file delete check trash configration [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9950: URL: https://github.com/apache/iceberg/pull/9950#issuecomment-2445551532 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Bump PyArrow to 18.0.0 [iceberg-python]

2024-10-29 Thread via GitHub
kevinjqliu commented on PR #1256: URL: https://github.com/apache/iceberg-python/pull/1256#issuecomment-2445610968 #1265 potential issue with arrow 18 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] core,api: Refactor code with `hasLiveEntries` [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9993: core,api: Refactor code with `hasLiveEntries` URL: https://github.com/apache/iceberg/pull/9993 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Fix hive mapreduce insert error in kerberos environment [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9944: Fix hive mapreduce insert error in kerberos environment URL: https://github.com/apache/iceberg/pull/9944 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Docs: Fix inconsistency in branching and tagging scenario [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9968: Docs: Fix inconsistency in branching and tagging scenario URL: https://github.com/apache/iceberg/pull/9968 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [DRAFT]: Adjust site links to absolute from site_url [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9887: [DRAFT]: Adjust site links to absolute from site_url URL: https://github.com/apache/iceberg/pull/9887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] hdfs file delete check trash configration [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9950: hdfs file delete check trash configration URL: https://github.com/apache/iceberg/pull/9950 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [DRAFT]: Create iceberg_docs_improvement issues template. [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9896: [DRAFT]: Create iceberg_docs_improvement issues template. URL: https://github.com/apache/iceberg/pull/9896 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9902: Spark: Add a test to check if the bloom filters are added to the parquet files URL: https://github.com/apache/iceberg/pull/9902 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] Fix hive mapreduce insert error in kerberos environment [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9944: URL: https://github.com/apache/iceberg/pull/9944#issuecomment-2445551495 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9902: URL: https://github.com/apache/iceberg/pull/9902#issuecomment-2445551417 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9924: URL: https://github.com/apache/iceberg/pull/9924#issuecomment-2445551471 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core:Add test for removing partition filed [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9903: URL: https://github.com/apache/iceberg/pull/9903#issuecomment-2445551444 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Minor: Parse rowGroupSize as a long [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9900: URL: https://github.com/apache/iceberg/pull/9900#issuecomment-2445551389 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Minor: Parse rowGroupSize as a long [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9900: Minor: Parse rowGroupSize as a long URL: https://github.com/apache/iceberg/pull/9900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] [DRAFT]: Create iceberg_docs_improvement issues template. [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9896: URL: https://github.com/apache/iceberg/pull/9896#issuecomment-2445551351 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark: Reconcile derived partitioning from source table with target table specs in AddFilesProcedure [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #10133: URL: https://github.com/apache/iceberg/pull/10133#issuecomment-2445551962 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Docs: Fix inconsistency in branching and tagging scenario [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9968: URL: https://github.com/apache/iceberg/pull/9968#issuecomment-2445551610 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] AWS: Glue table operations hang when aws authentication parameters are illegal [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #10002: URL: https://github.com/apache/iceberg/pull/10002#issuecomment-2445551716 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] core: Filter on live entries when reading the manifest [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9996: core: Filter on live entries when reading the manifest URL: https://github.com/apache/iceberg/pull/9996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [Draft] Fixing #9923 updating partitioned table with more than 1k columns fails [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] commented on PR #9957: URL: https://github.com/apache/iceberg/pull/9957#issuecomment-2445551564 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] [Draft] Fixing #9923 updating partitioned table with more than 1k columns fails [iceberg]

2024-10-29 Thread via GitHub
github-actions[bot] closed pull request #9957: [Draft] Fixing #9923 updating partitioned table with more than 1k columns fails URL: https://github.com/apache/iceberg/pull/9957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Another way to remove `initial_change` when CreateTableTransaction apply table updates on an empty metadata [iceberg-python]

2024-10-29 Thread via GitHub
kevinjqliu commented on code in PR #1219: URL: https://github.com/apache/iceberg-python/pull/1219#discussion_r1821594252 ## dev/Dockerfile: ## @@ -36,9 +36,9 @@ ENV PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip:$ RUN mkdir -p ${HADOOP_HOME} && mkdi

Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]

2024-10-29 Thread via GitHub
jad-grepr commented on issue #10147: URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2445506849 Hi just wanted to check if you figured this one out @sannaroby -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Spec v3: Add deletion vectors to the table spec [iceberg]

2024-10-29 Thread via GitHub
aokolnychyi commented on code in PR #11240: URL: https://github.com/apache/iceberg/pull/11240#discussion_r1821553001 ## format/spec.md: ## @@ -841,19 +855,45 @@ Notes: ## Delete Formats -This section details how to encode row-level deletes in Iceberg delete files. Row-leve

[PR] Bump sqlalchemy from 2.0.35 to 2.0.36 [iceberg-python]

2024-10-29 Thread via GitHub
dependabot[bot] opened a new pull request, #1268: URL: https://github.com/apache/iceberg-python/pull/1268 Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.35 to 2.0.36. Release notes Sourced from https://github.com/sqlalchemy/sqlalchemy/releases";>sqlalchemy's

Re: [PR] AWS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
danielcweeks commented on code in PR #11389: URL: https://github.com/apache/iceberg/pull/11389#discussion_r1821633197 ## aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java: ## @@ -0,0 +1,141 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Another way to remove `initial_change` when CreateTableTransaction apply table updates on an empty metadata [iceberg-python]

2024-10-29 Thread via GitHub
kevinjqliu commented on PR #1219: URL: https://github.com/apache/iceberg-python/pull/1219#issuecomment-2445438246 I want to get this PR in before the next release, please let me know if there's anything I can do to move it along! -- This is an automated message from the Apache Git Service

[PR] Bump pre-commit from 3.5.0 to 4.0.1 [iceberg-python]

2024-10-29 Thread via GitHub
dependabot[bot] opened a new pull request, #1267: URL: https://github.com/apache/iceberg-python/pull/1267 Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 3.5.0 to 4.0.1. Release notes Sourced from https://github.com/pre-commit/pre-commit/releases";>pre-commit's r

[PR] Deprecate for 0.8.0 release [iceberg-python]

2024-10-29 Thread via GitHub
kevinjqliu opened a new pull request, #1269: URL: https://github.com/apache/iceberg-python/pull/1269 Searching for `removed_in="0.8.0": https://github.com/search?q=repo%3Aapache%2Ficeberg-python+removed_in%3D%220.8&type=code Removed items: * `DEPRECATED_PROPERTY_NAMES` * `to_

Re: [PR] Deprecate for 0.8.0 release [iceberg-python]

2024-10-29 Thread via GitHub
kevinjqliu commented on PR #1269: URL: https://github.com/apache/iceberg-python/pull/1269#issuecomment-2445456953 I think this might be the first time we're removing deprecated functions/parameters from the library. I'm a bit concerned about the `DEPRECATED_PROPERTY_NAMES` and whether pe

[PR] Bump mkdocs-material from 9.5.36 to 9.5.42 [iceberg-python]

2024-10-29 Thread via GitHub
dependabot[bot] opened a new pull request, #1266: URL: https://github.com/apache/iceberg-python/pull/1266 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.36 to 9.5.42. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

Re: [PR] Remove `initial_change` when dealing with table updates [iceberg-python]

2024-10-29 Thread via GitHub
kevinjqliu commented on PR #950: URL: https://github.com/apache/iceberg-python/pull/950#issuecomment-2445432447 Closing this in favor of #1219 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Remove `initial_change` when dealing with table updates [iceberg-python]

2024-10-29 Thread via GitHub
kevinjqliu closed pull request #950: Remove `initial_change` when dealing with table updates URL: https://github.com/apache/iceberg-python/pull/950 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Parquet: Correctly prune nested columns [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11373: URL: https://github.com/apache/iceberg/pull/11373#discussion_r1821560025 ## parquet/src/test/java/org/apache/iceberg/parquet/TestParquetSchemaUtil.java: ## @@ -464,4 +588,103 @@ private Type map(Integer id, String name, Repetition

Re: [PR] Add REST Catalog tests to Spark 3.5 integration test [iceberg]

2024-10-29 Thread via GitHub
haizhou-zhao commented on code in PR #11093: URL: https://github.com/apache/iceberg/pull/11093#discussion_r1821584310 ## open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java: ## @@ -64,7 +65,9 @@ public Map configuration() { private CatalogContext i

Re: [PR] PyArrow: Pass in null-mask [iceberg-python]

2024-10-29 Thread via GitHub
Fokko merged PR #1264: URL: https://github.com/apache/iceberg-python/pull/1264 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Add REST Catalog tests to Spark 3.5 integration test [iceberg]

2024-10-29 Thread via GitHub
haizhou-zhao commented on code in PR #11093: URL: https://github.com/apache/iceberg/pull/11093#discussion_r1821577479 ## open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java: ## @@ -64,7 +72,8 @@ public Map configuration() { private CatalogContext i

Re: [PR] Add REST Catalog tests to Spark 3.5 integration test [iceberg]

2024-10-29 Thread via GitHub
haizhou-zhao commented on code in PR #11093: URL: https://github.com/apache/iceberg/pull/11093#discussion_r1821577268 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java: ## @@ -59,18 +70,45 @@ protected static Object[][] parameters() { }

Re: [PR] Core: Add Variant implementation to read serialized objects [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on PR #11415: URL: https://github.com/apache/iceberg/pull/11415#issuecomment-2445394163 My main overall question on this is whether or not this implementation belongs in the Iceberg project or in the Parquet project? I'm a little worried about a proliferation of imp

Re: [PR] Parquet: Correctly prune nested columns [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11373: URL: https://github.com/apache/iceberg/pull/11373#discussion_r1821571529 ## parquet/src/test/java/org/apache/iceberg/parquet/TestParquetSchemaUtil.java: ## @@ -464,4 +588,103 @@ private Type map(Integer id, String name, Repetition

Re: [PR] Parquet: Correctly prune nested columns [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11373: URL: https://github.com/apache/iceberg/pull/11373#discussion_r1821559558 ## parquet/src/test/java/org/apache/iceberg/parquet/TestParquetSchemaUtil.java: ## @@ -61,6 +63,128 @@ public class TestParquetSchemaUtil { required

Re: [I] when using SQL catalog, table location is not optional [iceberg-python]

2024-10-29 Thread via GitHub
Fokko commented on issue #1254: URL: https://github.com/apache/iceberg-python/issues/1254#issuecomment-2445382202 @djouallah I don't think it is obvious, but you can add the location to the catalog itself: ```python catalog = SqlCatalog( "default", **{

Re: [PR] PyArrow: Pass in null-mask [iceberg-python]

2024-10-29 Thread via GitHub
Fokko commented on PR #1264: URL: https://github.com/apache/iceberg-python/pull/1264#issuecomment-2445351586 Thanks for the prompt review @kevinjqliu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] AWS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11389: URL: https://github.com/apache/iceberg/pull/11389#discussion_r1821537570 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -136,6 +156,12 @@ public void applyClientCredentialConfigurations(T b @Suppres

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

2024-10-29 Thread via GitHub
Fokko commented on code in PR #11421: URL: https://github.com/apache/iceberg/pull/11421#discussion_r1821514825 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -176,4 +176,41 @@ private String createManifestListWithManifestFiles(long snapshotId, Long parent

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

2024-10-29 Thread via GitHub
Fokko commented on code in PR #11421: URL: https://github.com/apache/iceberg/pull/11421#discussion_r1821522106 ## core/src/main/java/org/apache/iceberg/SnapshotParser.java: ## @@ -142,6 +145,14 @@ static Snapshot fromJson(JsonNode node) { summary = builder.build(); }

Re: [PR] API: Add Variant data type [iceberg]

2024-10-29 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1821493402 ## api/src/main/java/org/apache/iceberg/VariantLike.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contribut

Re: [PR] API: Add Variant data type [iceberg]

2024-10-29 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1821493642 ## api/src/main/java/org/apache/iceberg/VariantLike.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contribut

[PR] Core: Suppress exceptions in case of dropTableData [iceberg]

2024-10-29 Thread via GitHub
nk1506 opened a new pull request, #9184: URL: https://github.com/apache/iceberg/pull/9184 With [dropTableData](https://github.com/apache/iceberg/blob/d247b20f166ccb0b92443d4b05330b1e0d9c5d49/core/src/main/java/org/apache/iceberg/CatalogUtil.java#L86) we plan to delete orphan files as many a

Re: [PR] API: Add Variant data type [iceberg]

2024-10-29 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1821492918 ## api/src/main/java/org/apache/iceberg/VariantLike.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contribut

Re: [PR] Core: use ManifestFiles.open when possible [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on PR #11414: URL: https://github.com/apache/iceberg/pull/11414#issuecomment-2445220619 Thanks @dramaticlly for the fix - Thanks @szehon-ho, @singhpk234 and @hsiang-c for review -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] AWS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11389: URL: https://github.com/apache/iceberg/pull/11389#discussion_r1821469987 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -66,21 +67,39 @@ public class AwsClientProperties implements Serializable { */

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11421: URL: https://github.com/apache/iceberg/pull/11421#discussion_r1821458413 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -176,4 +176,41 @@ private String createManifestListWithManifestFiles(long snapshotId, Lo

Re: [I] Support for timestamp downcasting when loading data to iceberg tables [iceberg-python]

2024-10-29 Thread via GitHub
tusharchou commented on issue #1045: URL: https://github.com/apache/iceberg-python/issues/1045#issuecomment-2445205707 `import os os.environ['PYICEBERG_DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE'] = 'true'` I think we can add this to the documentation -- This is an automated message f

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11421: URL: https://github.com/apache/iceberg/pull/11421#discussion_r1821455896 ## core/src/main/java/org/apache/iceberg/SnapshotParser.java: ## @@ -142,6 +145,14 @@ static Snapshot fromJson(JsonNode node) { summary = builder.build(

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11421: URL: https://github.com/apache/iceberg/pull/11421#discussion_r1821455896 ## core/src/main/java/org/apache/iceberg/SnapshotParser.java: ## @@ -142,6 +145,14 @@ static Snapshot fromJson(JsonNode node) { summary = builder.build(

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

2024-10-29 Thread via GitHub
RussellSpitzer commented on code in PR #11421: URL: https://github.com/apache/iceberg/pull/11421#discussion_r1821455896 ## core/src/main/java/org/apache/iceberg/SnapshotParser.java: ## @@ -142,6 +145,14 @@ static Snapshot fromJson(JsonNode node) { summary = builder.build(

Re: [I] Support for timestamp downcasting when loading data to iceberg tables [iceberg-python]

2024-10-29 Thread via GitHub
tusharchou commented on issue #1045: URL: https://github.com/apache/iceberg-python/issues/1045#issuecomment-2445177842 @kevinjqliu @sungwy can I help here? I faced the same issue while trying to write a csv into iceberg using pyiceberg/catalog/sql.py -- This is an automated message from

Re: [I] Delete orphan files [iceberg-python]

2024-10-29 Thread via GitHub
sungwy commented on issue #1200: URL: https://github.com/apache/iceberg-python/issues/1200#issuecomment-2445095723 Hey sure thing! I'll assign it to you @omkenge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] API, Core: Add scan planning apis to REST Catalog [iceberg]

2024-10-29 Thread via GitHub
rahil-c commented on code in PR #11180: URL: https://github.com/apache/iceberg/pull/11180#discussion_r1821364081 ## core/src/main/java/org/apache/iceberg/ScanTasksIterable.java: ## @@ -120,7 +146,21 @@ private void fetchScanTasks(String planTask) { headers,

Re: [PR] AWS: Refresh vended credentials [iceberg]

2024-10-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #11389: URL: https://github.com/apache/iceberg/pull/11389#discussion_r1821351642 ## aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java: ## @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Core: Suppress exceptions in case of dropTableData [iceberg]

2024-10-29 Thread via GitHub
caohoangha126 commented on PR #9184: URL: https://github.com/apache/iceberg/pull/9184#issuecomment-2445040832 @nk1506 @Fokko Hi, this bug is impacting our work and we are hoping that we can get the fix in a future upgrade. Is it possible to get it merged? Thanks! -- This is an automated m

Re: [PR] open-api: Fix testFixtures dependencies [iceberg]

2024-10-29 Thread via GitHub
Fokko merged PR #11422: URL: https://github.com/apache/iceberg/pull/11422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] open-api: Fix testFixtures dependencies [iceberg]

2024-10-29 Thread via GitHub
Fokko commented on PR #11422: URL: https://github.com/apache/iceberg/pull/11422#issuecomment-2445022155 @ajantha-bhat Great catch, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Bump moto from 5.0.17 to 5.0.18 [iceberg-python]

2024-10-29 Thread via GitHub
Fokko merged PR #1261: URL: https://github.com/apache/iceberg-python/pull/1261 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Core: use ManifestFiles.open when possible [iceberg]

2024-10-29 Thread via GitHub
dramaticlly commented on code in PR #11414: URL: https://github.com/apache/iceberg/pull/11414#discussion_r1821313502 ## core/src/main/java/org/apache/iceberg/BaseFilesTable.java: ## @@ -166,7 +166,6 @@ public CloseableIterable rows() { if (readableMetricsField == null) {

Re: [PR] Core: use ManifestFiles.open when possible [iceberg]

2024-10-29 Thread via GitHub
szehon-ho commented on code in PR #11414: URL: https://github.com/apache/iceberg/pull/11414#discussion_r1821309914 ## core/src/main/java/org/apache/iceberg/BaseFilesTable.java: ## @@ -166,7 +166,6 @@ public CloseableIterable rows() { if (readableMetricsField == null) {

  1   2   >