Re: [PR] Build: Bump Comet from 0.5.0 to 0.8.1 [iceberg]

2025-05-06 Thread via GitHub
nastra merged PR #12974: URL: https://github.com/apache/iceberg/pull/12974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Fix KryoException with S3FileIO and SerializableMap after upgrading from 1.7.1 to 1.7.2 [iceberg]

2025-05-06 Thread via GitHub
nastra commented on code in PR #12984: URL: https://github.com/apache/iceberg/pull/12984#discussion_r2076845190 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -98,7 +99,7 @@ public AwsClientProperties() { } public AwsClientProperties(Map prop

Re: [PR] Fix KryoException with S3FileIO and SerializableMap after upgrading from 1.7.1 to 1.7.2 [iceberg]

2025-05-06 Thread via GitHub
nastra commented on code in PR #12984: URL: https://github.com/apache/iceberg/pull/12984#discussion_r2076845190 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -98,7 +99,7 @@ public AwsClientProperties() { } public AwsClientProperties(Map prop

Re: [PR] Flink: Backport Maintenance - RewriteDataFiles to Flink 1.19, 1.20 [iceberg]

2025-05-06 Thread via GitHub
stevenzwu merged PR #12991: URL: https://github.com/apache/iceberg/pull/12991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Fix KryoException with S3FileIO and SerializableMap after upgrading from 1.7.1 to 1.7.2 [iceberg]

2025-05-06 Thread via GitHub
nastra commented on code in PR #12984: URL: https://github.com/apache/iceberg/pull/12984#discussion_r2076845190 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -98,7 +99,7 @@ public AwsClientProperties() { } public AwsClientProperties(Map prop

Re: [PR] Docs: Remove links to iceberg-spark-runtime-3.3 jars in 1.9 downloads [iceberg]

2025-05-06 Thread via GitHub
ajantha-bhat commented on code in PR #12989: URL: https://github.com/apache/iceberg/pull/12989#discussion_r2076855993 ## site/docs/releases.md: ## @@ -29,8 +29,6 @@ The latest version of Iceberg is [{{ icebergVersion }}](https://github.com/apach * [{{ icebergVersion }} Spark 3

Re: [PR] Remove `row-lineage` field in V3 metadata [iceberg-python]

2025-05-06 Thread via GitHub
Fokko merged PR #1975: URL: https://github.com/apache/iceberg-python/pull/1975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Docs: Remove links to iceberg-spark-runtime-3.3 jars in 1.9 downloads [iceberg]

2025-05-06 Thread via GitHub
ajantha-bhat commented on code in PR #12989: URL: https://github.com/apache/iceberg/pull/12989#discussion_r2076854408 ## site/docs/releases.md: ## @@ -29,8 +29,6 @@ The latest version of Iceberg is [{{ icebergVersion }}](https://github.com/apach * [{{ icebergVersion }} Spark 3

Re: [I] Remove `row_lineage` field in V3 metadata [iceberg-python]

2025-05-06 Thread via GitHub
Fokko closed issue #1974: Remove `row_lineage` field in V3 metadata URL: https://github.com/apache/iceberg-python/issues/1974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Fix KryoException with S3FileIO and SerializableMap after upgrading from 1.7.1 to 1.7.2 [iceberg]

2025-05-06 Thread via GitHub
nastra commented on code in PR #12984: URL: https://github.com/apache/iceberg/pull/12984#discussion_r2076845190 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -98,7 +99,7 @@ public AwsClientProperties() { } public AwsClientProperties(Map prop

Re: [PR] Flink: Backport Maintenance - RewriteDataFiles to Flink 1.19, 1.20 [iceberg]

2025-05-06 Thread via GitHub
stevenzwu commented on PR #12991: URL: https://github.com/apache/iceberg/pull/12991#issuecomment-2857145351 thanks @gyfora for the backport -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Update Comet version to 0.8.1 [iceberg]

2025-05-06 Thread via GitHub
huaxingao commented on code in PR #12974: URL: https://github.com/apache/iceberg/pull/12974#discussion_r2076834256 ## gradle/libs.versions.toml: ## @@ -84,6 +84,7 @@ spark35 = "3.5.5" sqlite-jdbc = "3.49.1.0" testcontainers = "1.21.0" tez08 = { strictly = "0.8.4"} # see rich

Re: [PR] Core: Add commit timestamp to CommitReport [iceberg]

2025-05-06 Thread via GitHub
nastra commented on code in PR #12990: URL: https://github.com/apache/iceberg/pull/12990#discussion_r2076833232 ## .palantir/revapi.yml: ## @@ -1178,6 +1178,9 @@ acceptedBreaks: new: "class org.apache.iceberg.Metrics" justification: "Java serialization across versi

Re: [PR] Core: Add commit timestamp to CommitReport [iceberg]

2025-05-06 Thread via GitHub
nastra commented on code in PR #12990: URL: https://github.com/apache/iceberg/pull/12990#discussion_r2076832653 ## core/src/main/java/org/apache/iceberg/metrics/CommitReport.java: ## @@ -27,6 +27,8 @@ public interface CommitReport extends MetricsReport { String tableName();

Re: [I] RewriteDataFiles with merging equality deletes [iceberg]

2025-05-06 Thread via GitHub
shanielh commented on issue #12838: URL: https://github.com/apache/iceberg/issues/12838#issuecomment-2857098062 It's a known issue and, unfortunately, by design. Many underlying design decisions fail to take streaming data into account. We solved it in Upsolver by reading the manifes

Re: [PR] Core: Refactor and use InternalData for partition stats [iceberg]

2025-05-06 Thread via GitHub
ajantha-bhat commented on PR #12946: URL: https://github.com/apache/iceberg/pull/12946#issuecomment-2857105987 Rebased the PR due to spark flaky test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Update Comet version to 0.8.1 [iceberg]

2025-05-06 Thread via GitHub
nastra commented on code in PR #12974: URL: https://github.com/apache/iceberg/pull/12974#discussion_r2076813658 ## gradle/libs.versions.toml: ## @@ -84,6 +84,7 @@ spark35 = "3.5.5" sqlite-jdbc = "3.49.1.0" testcontainers = "1.21.0" tez08 = { strictly = "0.8.4"} # see rich ve

Re: [PR] introduce a new config class to manage configurable options following design ideas from Velox [iceberg-cpp]

2025-05-06 Thread via GitHub
lidavidm commented on code in PR #92: URL: https://github.com/apache/iceberg-cpp/pull/92#discussion_r2076809784 ## src/iceberg/util/config.h: ## @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

[PR] Flink: Backport Maintenance - RewriteDataFiles to Flink 1.19, 1.20 [iceberg]

2025-05-06 Thread via GitHub
gyfora opened a new pull request, #12991: URL: https://github.com/apache/iceberg/pull/12991 This PR backports https://github.com/apache/iceberg/pull/11497 to Flink 1.19 and 2.0 Flink 1.19 could be cleanly applied without any code changes required to the logic. Flink 2.0 port requi

[PR] Core: Add commit timestamp to CommitReport [iceberg]

2025-05-06 Thread via GitHub
manuzhang opened a new pull request, #12990: URL: https://github.com/apache/iceberg/pull/12990 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] feat: Support tag and branch operations [iceberg-rust]

2025-05-06 Thread via GitHub
Xuanwo commented on PR #1177: URL: https://github.com/apache/iceberg-rust/pull/1177#issuecomment-2857006163 Hi @jonathanc-n sorry for the delayed review. Are you still interested in continuing with this? -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [I] [EPIC] Iceberg-rust Write support [iceberg-rust]

2025-05-06 Thread via GitHub
Xuanwo commented on issue #700: URL: https://github.com/apache/iceberg-rust/issues/700#issuecomment-2857002337 Hi @liurenjie1024, does this epic issue reflect the most recent status? Could you update it and add any PRs that are awaiting review? I’m happy to see this move forward. -- This

[PR] Docs: Remove links to iceberg-spark-runtime-3.3 jars in 1.9 downloads [iceberg]

2025-05-06 Thread via GitHub
manuzhang opened a new pull request, #12989: URL: https://github.com/apache/iceberg/pull/12989 Spark 3.3 support has been removed in 1.9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] [SPARK] Fix add_files type conversion exception and incorrect partition value when handling null partitions [iceberg]

2025-05-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #12886: URL: https://github.com/apache/iceberg/pull/12886#discussion_r2076728051 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java: ## @@ -946,7 +946,7 @@ public static List getPartitions(

Re: [PR] REST spec: Add encryption keys [iceberg]

2025-05-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #12987: URL: https://github.com/apache/iceberg/pull/12987#discussion_r2076676484 ## open-api/rest-catalog-open-api.py: ## @@ -399,6 +406,16 @@ class EnableRowLineageUpdate(BaseUpdate): action: str = Field('enable-row-lineage', const=

Re: [PR] REST spec: Add encryption keys [iceberg]

2025-05-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #12987: URL: https://github.com/apache/iceberg/pull/12987#discussion_r2076676377 ## open-api/rest-catalog-open-api.py: ## @@ -399,6 +406,16 @@ class EnableRowLineageUpdate(BaseUpdate): action: str = Field('enable-row-lineage', const=

Re: [PR] Spark: Make maxRecordPerMicrobatch a soft limit [iceberg]

2025-05-06 Thread via GitHub
singhpk234 commented on PR #12988: URL: https://github.com/apache/iceberg/pull/12988#issuecomment-2856852701 seems un-related failure ``` TestRewriteDataFilesAction > testParallelPartialProgressWithMaxFailedCommitsLargerThanTotalFileGroup() > formatVersion = 2 FAILED java.lang

Re: [PR] Core: Fix numeric overflow of timestamp nano literal [iceberg]

2025-05-06 Thread via GitHub
ebyhr commented on code in PR #11775: URL: https://github.com/apache/iceberg/pull/11775#discussion_r2074715304 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -300,8 +300,7 @@ public Literal to(Type type) { case TIMESTAMP: return (Li

Re: [PR] Core: Fix numeric overflow of timestamp nano literal [iceberg]

2025-05-06 Thread via GitHub
ebyhr commented on code in PR #11775: URL: https://github.com/apache/iceberg/pull/11775#discussion_r2074715304 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -300,8 +300,7 @@ public Literal to(Type type) { case TIMESTAMP: return (Li

Re: [PR] Spark: Make maxRecordPerMicrobatch a soft limit [iceberg]

2025-05-06 Thread via GitHub
singhpk234 closed pull request #12988: Spark: Make maxRecordPerMicrobatch a soft limit URL: https://github.com/apache/iceberg/pull/12988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Support retrieving the latest Iceberg table on table scan [iceberg-rust]

2025-05-06 Thread via GitHub
phillipleblanc opened a new pull request, #1297: URL: https://github.com/apache/iceberg-rust/pull/1297 ## Which issue does this PR close? - Closes #1296 ## What changes are included in this PR? Makes the IcebergTableProvider::try_new method public that takes an Arc and a

[I] The iceberg-datafusion Table Provider has a way to scan the latest snapshots dynamically [iceberg-rust]

2025-05-06 Thread via GitHub
phillipleblanc opened a new issue, #1296: URL: https://github.com/apache/iceberg-rust/issues/1296 ### Is your feature request related to a problem or challenge? Currently the DataFusion IcebergTableProvider takes a fixed table metadata on creation, and data that is added to the Iceber

[PR] Spark: Make maxRecordPerMicrobatch a soft limit [iceberg]

2025-05-06 Thread via GitHub
singhpk234 opened a new pull request, #12988: URL: https://github.com/apache/iceberg/pull/12988 ### About the change Make _maxRecordsPerMicrobatch_ a soft limit, as the cases like for ex number of records is less than the maxRecords of a file would expect us to read the file partiall

Re: [PR] Spec: Add details on GZIP compressed metadata files [iceberg]

2025-05-06 Thread via GitHub
emkornfield commented on PR #12598: URL: https://github.com/apache/iceberg/pull/12598#issuecomment-2856732662 > Since this is just documenting how implementations work, do we need to vote on it? It doesn't change requirements. I think this covers that GZIP is supported which is probab

Re: [PR] Support changelog scan for table with delete files [iceberg]

2025-05-06 Thread via GitHub
guykhazma commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r2076587215 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -170,4 +240,10 @@ private static Stream> addedRowsScanTaskFiles(Ad

Re: [I] PyIceberg Cookbook [iceberg-python]

2025-05-06 Thread via GitHub
github-actions[bot] commented on issue #1201: URL: https://github.com/apache/iceberg-python/issues/1201#issuecomment-2856648135 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] kafka connect iceberg connect: option to fail connector on N number of failed commit cycles [iceberg]

2025-05-06 Thread via GitHub
github-actions[bot] commented on issue #11468: URL: https://github.com/apache/iceberg/issues/11468#issuecomment-2856643855 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Spark-hive catalog drop table XX purge not use purge code in HiveCatalog.dropTable() [iceberg]

2025-05-06 Thread via GitHub
github-actions[bot] commented on issue #11484: URL: https://github.com/apache/iceberg/issues/11484#issuecomment-2856644027 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] What's the use of old metadata file, why not delete by default? [iceberg]

2025-05-06 Thread via GitHub
github-actions[bot] commented on issue #11206: URL: https://github.com/apache/iceberg/issues/11206#issuecomment-2856643163 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] List all AWS S3 properties in the docs [iceberg]

2025-05-06 Thread via GitHub
github-actions[bot] commented on issue #10674: URL: https://github.com/apache/iceberg/issues/10674#issuecomment-2856642933 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] How does client use hadoopcatlog to read the iceberg table writen by hivecatalog? [iceberg]

2025-05-06 Thread via GitHub
github-actions[bot] commented on issue #11375: URL: https://github.com/apache/iceberg/issues/11375#issuecomment-2856643399 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] [Spark] Identity partition on required column generates nullable partition tuple in manifest file [iceberg]

2025-05-06 Thread via GitHub
github-actions[bot] commented on issue #11300: URL: https://github.com/apache/iceberg/issues/11300#issuecomment-2856643302 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] List all AWS S3 properties in the docs [iceberg]

2025-05-06 Thread via GitHub
github-actions[bot] closed issue #10674: List all AWS S3 properties in the docs URL: https://github.com/apache/iceberg/issues/10674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

2025-05-06 Thread via GitHub
coderfender commented on PR #12824: URL: https://github.com/apache/iceberg/pull/12824#issuecomment-2856622960 @RussellSpitzer , @pvary Rebased the branch and fixed tests. Please take a look whenever you get a chance . Thank you very much for the guidance -- This is an automated messag

Re: [I] Hive metastore 4.0.1 remove deprecated thrift APIs [iceberg-python]

2025-05-06 Thread via GitHub
rkarthik29 commented on issue #1222: URL: https://github.com/apache/iceberg-python/issues/1222#issuecomment-2856609006 qq is there a way to workaround this issue? We are using hive4 metastore and would like to use pyiceberg to manage data, but blocked by this issue. -- This is an automat

Re: [PR] Support changelog scan for table with delete files [iceberg]

2025-05-06 Thread via GitHub
wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r2076563323 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -170,4 +240,10 @@ private static Stream> addedRowsScanTaskFiles(Added

Re: [PR] Status: Split read/write and add deletion vectors [iceberg]

2025-05-06 Thread via GitHub
singhpk234 commented on code in PR #12958: URL: https://github.com/apache/iceberg/pull/12958#discussion_r2076552320 ## site/docs/status.md: ## @@ -163,9 +163,18 @@ This section lists the libraries that implement the Apache Iceberg specification | Operation | Java

Re: [PR] Fix KryoException with S3FileIO and SerializableMap after upgrading from 1.7.1 to 1.7.2 [iceberg]

2025-05-06 Thread via GitHub
singhpk234 commented on code in PR #12984: URL: https://github.com/apache/iceberg/pull/12984#discussion_r2076548312 ## aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java: ## @@ -98,7 +99,7 @@ public AwsClientProperties() { } public AwsClientProperties(Map

Re: [PR] Update Comet version to 0.8.1 [iceberg]

2025-05-06 Thread via GitHub
huaxingao commented on code in PR #12974: URL: https://github.com/apache/iceberg/pull/12974#discussion_r2076496240 ## spark/v3.4/build.gradle: ## @@ -75,7 +75,7 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") { exclude group: 'org.roaringb

Re: [PR] Spec: Add details on GZIP compressed metadata files [iceberg]

2025-05-06 Thread via GitHub
rdblue commented on PR #12598: URL: https://github.com/apache/iceberg/pull/12598#issuecomment-2856399957 Since this is just documenting how implementations work, do we need to vote on it? It doesn't change requirements. -- This is an automated message from the Apache Git Service. To respo

Re: [I] Format-versioned `Snapshot`s in light of V3 additions [iceberg-python]

2025-05-06 Thread via GitHub
smaheshwar-pltr commented on issue #1973: URL: https://github.com/apache/iceberg-python/issues/1973#issuecomment-2856395938 Ooh thanks a lot for pointing that out @Fokko, I think the upgrade procedure would indeed make versioning complicated. Siding with you now -- This is an automated m

Re: [PR] Enable HTTP proxy support for the client used by REST Catalog [iceberg]

2025-05-06 Thread via GitHub
flyrain merged PR #12406: URL: https://github.com/apache/iceberg/pull/12406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] Deleting a column from an iceberg table breaks schema in AWS Glue catalog [iceberg]

2025-05-06 Thread via GitHub
d4r3topk commented on issue #6340: URL: https://github.com/apache/iceberg/issues/6340#issuecomment-2856376562 Still facing this in 1.4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Spec: Add details on GZIP compressed metadata files [iceberg]

2025-05-06 Thread via GitHub
rdblue commented on code in PR #12598: URL: https://github.com/apache/iceberg/pull/12598#discussion_r2076448013 ## format/spec.md: ## @@ -1761,6 +1763,10 @@ The reference Java implementation uses a type 4 uuid and XORs the 4 most signifi Java writes `-1` for "no current snap

Re: [PR] Spec: Add details on GZIP compressed metadata files [iceberg]

2025-05-06 Thread via GitHub
rdblue commented on code in PR #12598: URL: https://github.com/apache/iceberg/pull/12598#discussion_r2076448013 ## format/spec.md: ## @@ -1761,6 +1763,10 @@ The reference Java implementation uses a type 4 uuid and XORs the 4 most signifi Java writes `-1` for "no current snap

Re: [PR] logging error when deleting old metadata [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade commented on code in PR #422: URL: https://github.com/apache/iceberg-go/pull/422#discussion_r2076439116 ## table/table.go: ## @@ -230,12 +228,12 @@ func deleteOldMetadata(fs io.IO, baseMeta, newMeta Metadata) error { for _, file := range toRemove {

Re: [PR] Build and test hive-metastore with Hive 2, 3 and 4 with a single source set [iceberg]

2025-05-06 Thread via GitHub
wypoon commented on PR #12721: URL: https://github.com/apache/iceberg/pull/12721#issuecomment-2856320947 @danielcweeks thanks for the update. In this case, it appears that `org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo.getAllStructFieldNames()` has changed its return type from `j

Re: [PR] Allow HTTP proxy to be used to talk to REST catalog server [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade merged PR #420: URL: https://github.com/apache/iceberg-go/pull/420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] dont fail when last-column-id is 0 [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade commented on PR #421: URL: https://github.com/apache/iceberg-go/pull/421#issuecomment-2856286543 Looks good once the linting is taken care of :smile: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] dont fail when last-column-id is 0 [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade commented on code in PR #421: URL: https://github.com/apache/iceberg-go/pull/421#discussion_r2076436259 ## table/metadata.go: ## @@ -1190,6 +1194,9 @@ func (m *metadataV2) UnmarshalJSON(b []byte) error { type Alias metadataV2 aux := (*Alias)(m) +

Re: [PR] dont fail when last-column-id is 0 [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade commented on code in PR #421: URL: https://github.com/apache/iceberg-go/pull/421#discussion_r2076436060 ## table/metadata.go: ## @@ -1145,6 +1146,9 @@ func (m *metadataV1) UnmarshalJSON(b []byte) error { type Alias metadataV1 aux := (*Alias)(m) +

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
rdblue commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076418940 ## format/spec.md: ## @@ -1680,6 +1680,25 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one is creat

Re: [I] Format-versioned `Snapshot`s in light of V3 additions [iceberg-python]

2025-05-06 Thread via GitHub
Fokko commented on issue #1973: URL: https://github.com/apache/iceberg-python/issues/1973#issuecomment-2856216333 Hey @smaheshwar-pltr Thanks for bringing this up. > I'm therefore wondering about "versioning" `Snapshot` similar to `TableMetadata`, so that V3 `TableMetadata` would con

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
rdblue commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076417558 ## format/spec.md: ## @@ -1680,6 +1680,25 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one is creat

Re: [I] Clarify Error Logs for GCS Directory/Object Conflicts [iceberg-python]

2025-05-06 Thread via GitHub
HyunWooZZ commented on issue #1952: URL: https://github.com/apache/iceberg-python/issues/1952#issuecomment-2856146318 @kevinjqliu location = 'gs://seoul_data_v1/warehouse/test/metadata/0-6f9631fe-18db-4743-915b-36d7562d4073.metadata.json' -- This is an automated message from t

[PR] Add Avro compression [iceberg-python]

2025-05-06 Thread via GitHub
Fokko opened a new pull request, #1976: URL: https://github.com/apache/iceberg-python/pull/1976 # Rationale for this change PyIceberg did not compress the Avro. This will make gzip/deflate the same as in Java. # Are these changes tested? Existing round-trip tests with Fa

Re: [PR] Remove `row-lineage` field in V3 metadata [iceberg-python]

2025-05-06 Thread via GitHub
smaheshwar-pltr commented on PR #1975: URL: https://github.com/apache/iceberg-python/pull/1975#issuecomment-2856169671 @Fokko, would you mind taking a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Add all filles metadata tables [iceberg-python]

2025-05-06 Thread via GitHub
soumya-ghosh commented on code in PR #1626: URL: https://github.com/apache/iceberg-python/pull/1626#discussion_r2076356376 ## pyiceberg/table/inspect.py: ## @@ -657,3 +671,30 @@ def all_manifests(self) -> "pa.Table": lambda args: self._generate_manifests_table(*args

[PR] Remove `row-lineage` field in V3 metadata [iceberg-python]

2025-05-06 Thread via GitHub
smaheshwar-pltr opened a new pull request, #1975: URL: https://github.com/apache/iceberg-python/pull/1975 Closes #1974. # Rationale for this change # Are these changes tested? # Are there any user-facing changes? -- This is an automated

[I] Format-versioned `Snapshot`s in light of V3 additions [iceberg-python]

2025-05-06 Thread via GitHub
smaheshwar-pltr opened a new issue, #1973: URL: https://github.com/apache/iceberg-python/issues/1973 ### Feature Request / Improvement While thinking about https://github.com/apache/iceberg-python/issues/1971 and https://github.com/apache/iceberg-python/issues/1972, I realised that V3

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
RussellSpitzer commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076321787 ## format/spec.md: ## @@ -1680,6 +1680,25 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
RussellSpitzer commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076321787 ## format/spec.md: ## @@ -1680,6 +1680,25 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
RussellSpitzer commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076321787 ## format/spec.md: ## @@ -1680,6 +1680,25 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
RussellSpitzer commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076321787 ## format/spec.md: ## @@ -1680,6 +1680,25 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one

Re: [PR] feat: Add basic support for `PartitionAndMetricsEvaluator` [iceberg-rust]

2025-05-06 Thread via GitHub
jonathanc-n commented on PR #1189: URL: https://github.com/apache/iceberg-rust/pull/1189#issuecomment-2856007698 cc @Xuanwo @liurenjie1024 This should be part of the push towards getting row delta for write support. -- This is an automated message from the Apache Git Service. To respond

[I] Extend V3 metadata read support with encryption changes [iceberg-python]

2025-05-06 Thread via GitHub
smaheshwar-pltr opened a new issue, #1972: URL: https://github.com/apache/iceberg-python/issues/1972 ### Feature Request / Improvement The V3 spec changes for encryption have merged in https://github.com/apache/iceberg/pull/12162. https://github.com/apache/iceberg/pull/12927 has also

[I] V3 row lineage metadata parsing of `first-row-id` in `snapshots` [iceberg-python]

2025-05-06 Thread via GitHub
smaheshwar-pltr opened a new issue, #1971: URL: https://github.com/apache/iceberg-python/issues/1971 ### Feature Request / Improvement https://github.com/apache/iceberg-python/pull/1554 that introduced V3 read support reads `next_row_id` on table metadata. We also need (see [spec](ht

Re: [PR] Spec: Clarify behavior of special geo objects for lower/upper bounds [iceberg]

2025-05-06 Thread via GitHub
szehon-ho commented on code in PR #12956: URL: https://github.com/apache/iceberg/pull/12956#discussion_r2076277911 ## format/spec.md: ## @@ -649,6 +649,8 @@ Notes: For `geometry` and `geography` types, `lower_bounds` and `upper_bounds` are both points of the following coordi

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
rdblue commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076264206 ## format/spec.md: ## @@ -1680,6 +1680,23 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one is creat

Re: [PR] Add all filles metadata tables [iceberg-python]

2025-05-06 Thread via GitHub
soumya-ghosh commented on code in PR #1626: URL: https://github.com/apache/iceberg-python/pull/1626#discussion_r2076246023 ## tests/integration/test_inspect_table.py: ## @@ -938,3 +947,37 @@ def test_inspect_all_manifests(spark: SparkSession, session_catalog: Catalog, fo l

Re: [PR] Catalogs: Add support for unique table locations via catalog property [iceberg]

2025-05-06 Thread via GitHub
RussellSpitzer commented on PR #12892: URL: https://github.com/apache/iceberg/pull/12892#issuecomment-2855905738 I think this makes a lot of sense but I'm not sure if this should be a client side decision. I'd like us to explore the idea of "owned locations" for tables and talk more about c

[PR] dont fail when last-column-id is 0 or schema len is 0 [iceberg-go]

2025-05-06 Thread via GitHub
Shreyas220 opened a new pull request, #421: URL: https://github.com/apache/iceberg-go/pull/421 solves #405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Catalogs: Add support for unique table locations via catalog property [iceberg]

2025-05-06 Thread via GitHub
mrcnc commented on PR #12892: URL: https://github.com/apache/iceberg/pull/12892#issuecomment-2855833233 +1 to having a catalog property for unique table locations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] feat(manifest): Expose the Avro Metadata for manifests [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade closed pull request #415: feat(manifest): Expose the Avro Metadata for manifests URL: https://github.com/apache/iceberg-go/pull/415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Build and test hive-metastore with Hive 2, 3 and 4 with a single source set [iceberg]

2025-05-06 Thread via GitHub
danielcweeks commented on PR #12721: URL: https://github.com/apache/iceberg/pull/12721#issuecomment-2855780956 Hey @wypoon, I spent some time working on getting the test only approach working, but ran into some other issues that I'm worried are even bigger problems with our Hive compatibili

Re: [PR] Basic version of expireSnapshots operation. [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade commented on PR #401: URL: https://github.com/apache/iceberg-go/pull/401#issuecomment-2855771394 > Regarding file deletion, maybe we can add a new method to the Update interface where we can put some post-commit code. It's a good question. If we have the table, then we have

Re: [PR] Basic version of expireSnapshots operation. [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade commented on code in PR #401: URL: https://github.com/apache/iceberg-go/pull/401#discussion_r2076149034 ## table/transaction.go: ## @@ -142,6 +143,120 @@ func (t *Transaction) SetProperties(props iceberg.Properties) error { return nil } +type expireSnapshot

Re: [PR] Add ManifestReader, which provides access to manifest metadata [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade merged PR #417: URL: https://github.com/apache/iceberg-go/pull/417 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] feat(manifest): Expose the Avro Metadata for manifests [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade commented on PR #415: URL: https://github.com/apache/iceberg-go/pull/415#issuecomment-2855711636 closing in favor of #417 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Provide access to a manifest's schema and partition spec [iceberg-go]

2025-05-06 Thread via GitHub
zeroshade closed issue #386: Provide access to a manifest's schema and partition spec URL: https://github.com/apache/iceberg-go/issues/386 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] feat(catalog/sql): Add views related ops [iceberg-go]

2025-05-06 Thread via GitHub
dttung2905 commented on code in PR #414: URL: https://github.com/apache/iceberg-go/pull/414#discussion_r2076094123 ## catalog/sql/sql.go: ## @@ -844,3 +871,235 @@ func (c *Catalog) UpdateNamespaceProperties(ctx context.Context, namespace table func (c *Catalog) CheckNamespaceE

Re: [PR] Spark 4.0 integration [iceberg]

2025-05-06 Thread via GitHub
aihuaxu commented on PR #12494: URL: https://github.com/apache/iceberg/pull/12494#issuecomment-2855641850 > I think we should get this in even before Spark 4.0 official release because I see we are having some trouble with keeping things in sync now. I think this is very close and most othe

[PR] Allow HTTP proxy to be used to talk to REST catalog server [iceberg-go]

2025-05-06 Thread via GitHub
jhump opened a new pull request, #420: URL: https://github.com/apache/iceberg-go/pull/420 This one-liner uses the "net/http" package's built-in support for HTTP proxies, activated via standard environment variables like `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`. -- This is an automated

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
RussellSpitzer commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076074302 ## format/spec.md: ## @@ -1680,6 +1680,25 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
RussellSpitzer commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076067192 ## format/spec.md: ## @@ -1680,6 +1680,23 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one

Re: [PR] Spec: Update v3 summary, add row lineage [iceberg]

2025-05-06 Thread via GitHub
RussellSpitzer commented on code in PR #12982: URL: https://github.com/apache/iceberg/pull/12982#discussion_r2076067610 ## format/spec.md: ## @@ -1680,6 +1680,23 @@ Row-level delete changes: * These position delete files must be merged into the DV for a data file when one

Re: [PR] Merge python-integration.yml into python-ci.yml [iceberg-python]

2025-05-06 Thread via GitHub
kevinjqliu commented on PR #1963: URL: https://github.com/apache/iceberg-python/pull/1963#issuecomment-2855483829 Thanks @petern48 for the contribution and @Fokko for the review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Core: Deep copy Record values for equality deletes [iceberg]

2025-05-06 Thread via GitHub
hsingh574 commented on PR #12855: URL: https://github.com/apache/iceberg/pull/12855#issuecomment-2855502963 @RussellSpitzer Appreciate the reviews! Addressed the nits in the latest revision. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] API, Core: Add table metadata keys for encryption [iceberg]

2025-05-06 Thread via GitHub
rdblue merged PR #12927: URL: https://github.com/apache/iceberg/pull/12927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] API, Core: Add table metadata keys for encryption [iceberg]

2025-05-06 Thread via GitHub
rdblue commented on PR #12927: URL: https://github.com/apache/iceberg/pull/12927#issuecomment-2855500450 Thanks for reviewing, @nastra and @RussellSpitzer! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

  1   2   3   >