Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
pvary commented on PR #10996: URL: https://github.com/apache/iceberg/pull/10996#issuecomment-2480424096 > It's been back-ported to hive 3.1, but there's no new release yet. @pvary @nastra thoughts? Basically the missing release means that there is no Hive 3 with Java 11 (contrary to

Re: [I] Documentation page returning 404 [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] closed issue #10249: Documentation page returning 404 URL: https://github.com/apache/iceberg/issues/10249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Core: Reimplement CharSequenceMap to obey Map contract [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on PR #11308: URL: https://github.com/apache/iceberg/pull/11308#issuecomment-2480194387 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Hive: Cache deletes on executors [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] closed pull request #10666: Hive: Cache deletes on executors URL: https://github.com/apache/iceberg/pull/10666 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Remove usage of deprecated functions from the codebase [iceberg-python]

2024-11-15 Thread via GitHub
ndrluis commented on issue #1327: URL: https://github.com/apache/iceberg-python/issues/1327#issuecomment-2480372639 @kevinjqliu Could you explain what you expect? I didn't understand the purpose of this issue. I believe we are supposed to warn the user when they use a deprecated API. --

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
pvary commented on PR #10996: URL: https://github.com/apache/iceberg/pull/10996#issuecomment-2479686648 > > I expect that there will be users trying to use HiveCatalog with old Hive versions. > > @pvary We don't expect them to use iceberg 1.8+ with old Hive versions, right? I

Re: [I] ERROR AppendDataExec: Data source write support IcebergBatchWrite(xxx) aborting [iceberg]

2024-11-15 Thread via GitHub
unni-pure commented on issue #11562: URL: https://github.com/apache/iceberg/issues/11562#issuecomment-2479516825 [iceberg-bug.md](https://github.com/user-attachments/files/17778996/iceberg-bug.md) -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Add interfaces for Action CheckSnapshotIntegrity [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] closed pull request #10642: Add interfaces for Action CheckSnapshotIntegrity URL: https://github.com/apache/iceberg/pull/10642 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Iceberg Hidden Partitioning and Spark SQL Wide Transformation Optimization [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on issue #10187: URL: https://github.com/apache/iceberg/issues/10187#issuecomment-2480193694 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Nested namespace support is broken in 1.7.0 [iceberg]

2024-11-15 Thread via GitHub
flyrain commented on issue #11539: URL: https://github.com/apache/iceberg/issues/11539#issuecomment-2480263584 The fix seems a good candidate for 1.7.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Add interfaces for Action CheckSnapshotIntegrity [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on PR #10642: URL: https://github.com/apache/iceberg/pull/10642#issuecomment-2480193846 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Documentation page returning 404 [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on issue #10249: URL: https://github.com/apache/iceberg/issues/10249#issuecomment-2480193721 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] API: Add Variant data type [iceberg]

2024-11-15 Thread via GitHub
rdblue commented on PR #11324: URL: https://github.com/apache/iceberg/pull/11324#issuecomment-2480238364 Late +1 from me! Thanks @aihuaxu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Hive: Cache deletes on executors [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on PR #10666: URL: https://github.com/apache/iceberg/pull/10666#issuecomment-2480193926 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Speed up Spark + Flink unit test execution [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] closed pull request #10581: Speed up Spark + Flink unit test execution URL: https://github.com/apache/iceberg/pull/10581 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Add interfaces for Action RemoveExpiredFiles [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on PR #10643: URL: https://github.com/apache/iceberg/pull/10643#issuecomment-2480193886 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Spark-3.5: Refactor BaseProcedure to support views [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on PR #11326: URL: https://github.com/apache/iceberg/pull/11326#issuecomment-2480194419 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on PR #10953: URL: https://github.com/apache/iceberg/pull/10953#issuecomment-2480194235 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Add interfaces for Action RemoveExpiredFiles [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] closed pull request #10643: Add interfaces for Action RemoveExpiredFiles URL: https://github.com/apache/iceberg/pull/10643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Let `./gradlew clean` clean everything [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] closed pull request #10601: Let `./gradlew clean` clean everything URL: https://github.com/apache/iceberg/pull/10601 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] API: Align CharSequenceSet impl with Data/DeleteFileSet [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on PR #11322: URL: https://github.com/apache/iceberg/pull/11322#issuecomment-2480194403 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Speed up Spark + Flink unit test execution [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] commented on PR #10581: URL: https://github.com/apache/iceberg/pull/10581#issuecomment-2480193779 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Iceberg Hidden Partitioning and Spark SQL Wide Transformation Optimization [iceberg]

2024-11-15 Thread via GitHub
github-actions[bot] closed issue #10187: Iceberg Hidden Partitioning and Spark SQL Wide Transformation Optimization URL: https://github.com/apache/iceberg/issues/10187 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#issuecomment-2480146878 @liurenjie1024 for `set_branch_snapshot` I was missing an easy way to determine the type of a ref. I introduced an enum `ReferenceType`, but if you don't like that, I can also just

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1844670884 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1844669519 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1844668631 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-15 Thread via GitHub
RussellSpitzer commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1844627456 ## core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java: ## @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-15 Thread via GitHub
wypoon commented on code in PR #11520: URL: https://github.com/apache/iceberg/pull/11520#discussion_r1844645606 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java: ## @@ -541,7 +550,19 @@ private static NullabilityHolder newNullabilityHolder(in

Re: [I] [Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog [iceberg]

2024-11-15 Thread via GitHub
dramaticlly commented on issue #11554: URL: https://github.com/apache/iceberg/issues/11554#issuecomment-2480067675 thank you @haizhou-zhao , I realized we actually have similar special handling for IllegalArgumentException per #9225 and I am wondering if it's worth doing the same for Valida

[PR] Bump coverage from 7.6.5 to 7.6.7 [iceberg-python]

2024-11-15 Thread via GitHub
dependabot[bot] opened a new pull request, #1329: URL: https://github.com/apache/iceberg-python/pull/1329 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.6.5 to 7.6.7. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's cha

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-15 Thread via GitHub
szehon-ho commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1844346161 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -341,20 +274,19 @@ private Builder doExecute( } p

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1844602893 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2063 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1844602893 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2063 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1844601246 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2074 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Manifest list encryption [iceberg]

2024-11-15 Thread via GitHub
smaheshwar-pltr commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1844220351 ## core/src/main/java/org/apache/iceberg/encryption/EncryptionUtil.java: ## @@ -70,31 +75,93 @@ public static KeyManagementClient createKmsClient(Map catalogPr

Re: [PR] Spark 3.5: Add DVReaderBenchmark [iceberg]

2024-11-15 Thread via GitHub
aokolnychyi commented on PR #11537: URL: https://github.com/apache/iceberg/pull/11537#issuecomment-2479937213 Thanks, @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Spark 3.5: Add DVReaderBenchmark [iceberg]

2024-11-15 Thread via GitHub
aokolnychyi merged PR #11537: URL: https://github.com/apache/iceberg/pull/11537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Spark 3.5: Adapt PlanningBenchmark for DVs [iceberg]

2024-11-15 Thread via GitHub
aokolnychyi commented on PR #11531: URL: https://github.com/apache/iceberg/pull/11531#issuecomment-2479936456 Thanks, @jbonofre @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Spark 3.5: Adapt PlanningBenchmark for DVs [iceberg]

2024-11-15 Thread via GitHub
aokolnychyi merged PR #11531: URL: https://github.com/apache/iceberg/pull/11531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Spark 3.5: Adapt DeleteFileIndexBenchmark for DVs [iceberg]

2024-11-15 Thread via GitHub
aokolnychyi commented on PR #11529: URL: https://github.com/apache/iceberg/pull/11529#issuecomment-2479935966 Thanks, @jbonofre @nastra! We may look into refactoring some of the benchmark code, but the experience shows it is rarely worth the time. -- This is an automated message fr

Re: [PR] Spark 3.5: Adapt DeleteFileIndexBenchmark for DVs [iceberg]

2024-11-15 Thread via GitHub
aokolnychyi merged PR #11529: URL: https://github.com/apache/iceberg/pull/11529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

[I] `catalog.load_table` raises Invalid JSON error [iceberg-python]

2024-11-15 Thread via GitHub
sandcobainer opened a new issue, #1328: URL: https://github.com/apache/iceberg-python/issues/1328 ### Question Context: So I'm trying to run a simple proof of concept with PyIceberg, Hive Metastore (with an SQL dump as a hive metastore schema) and an S3 bucket of iceberg tables. I se

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2024-11-15 Thread via GitHub
hesham-medhat commented on PR #11039: URL: https://github.com/apache/iceberg/pull/11039#issuecomment-2479805549 Thank you all for your enthusiasm! This is close, it's pending @rdblue's final pass/approval. A little while ago he told me he has been busy nevertheless will get to it as soon as

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-15 Thread via GitHub
huaxingao commented on code in PR #11520: URL: https://github.com/apache/iceberg/pull/11520#discussion_r1844353250 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java: ## @@ -541,7 +550,19 @@ private static NullabilityHolder newNullabilityHolder

Re: [I] Flink: Not Writing [iceberg]

2024-11-15 Thread via GitHub
MarquisC commented on issue #8916: URL: https://github.com/apache/iceberg/issues/8916#issuecomment-2479772050 @belldoor I've always used this reference https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/filesystem/#file-sink -- This is an automated me

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
pvary commented on PR #10996: URL: https://github.com/apache/iceberg/pull/10996#issuecomment-2479763958 > It's been back-ported to hive 3.1, but there's no new release yet. @pvary @nastra thoughts? What about using Hive4 instead? -- This is an automated message from the Apache Git

Re: [PR] API: Add Variant data type [iceberg]

2024-11-15 Thread via GitHub
RussellSpitzer merged PR #11324: URL: https://github.com/apache/iceberg/pull/11324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] API: Add Variant data type [iceberg]

2024-11-15 Thread via GitHub
RussellSpitzer commented on PR #11324: URL: https://github.com/apache/iceberg/pull/11324#issuecomment-2479712954 Sync'd up with Ryan this morning, we are good to go. Thanks @aihuaxu for working on this and @rdblue, @gene-db and @Fokko for reviews! -- This is an automated message

[PR] (WIP) Write DVs in Spark for V3 tables [iceberg]

2024-11-15 Thread via GitHub
amogh-jahagirdar opened a new pull request, #11561: URL: https://github.com/apache/iceberg/pull/11561 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Docs: Use the correct YAML text block indicator to prevent formatting issues [iceberg]

2024-11-15 Thread via GitHub
neodon commented on PR #11552: URL: https://github.com/apache/iceberg/pull/11552#issuecomment-2479693966 Pardon the (probably) unnecessary merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-15 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1844241268 ## crates/iceberg/src/transaction.rs: ## @@ -122,6 +189,387 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append d

Re: [I] Remove Dependency on Hadoop's Filesystem Class from Remove Orphan Files [iceberg]

2024-11-15 Thread via GitHub
rocco408 commented on issue #11541: URL: https://github.com/apache/iceberg/issues/11541#issuecomment-2479622388 I'm happy to take a look, independently and/or with community input, I'll circle-back in the coming days with any progress. -- This is an automated message from the Apache Git S

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-11-15 Thread via GitHub
huaxingao commented on PR #11551: URL: https://github.com/apache/iceberg/pull/11551#issuecomment-2479588343 cc @flyrain @szehon-ho @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[I] ERROR AppendDataExec: Data source write support IcebergBatchWrite(xxx) aborting [iceberg]

2024-11-15 Thread via GitHub
unni-pure opened a new issue, #11562: URL: https://github.com/apache/iceberg/issues/11562 ### Apache Iceberg version 1.6.0 ### Query engine Spark ### Please describe the bug 🐞 I am running a pyspark code to create a iceberg table in hive metastore and stori

Re: [PR] Manifest list encryption [iceberg]

2024-11-15 Thread via GitHub
smaheshwar-pltr commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1844220351 ## core/src/main/java/org/apache/iceberg/encryption/EncryptionUtil.java: ## @@ -70,31 +75,93 @@ public static KeyManagementClient createKmsClient(Map catalogPr

Re: [PR] (WIP) Write DVs in Spark for V3 tables [iceberg]

2024-11-15 Thread via GitHub
amogh-jahagirdar commented on code in PR #11561: URL: https://github.com/apache/iceberg/pull/11561#discussion_r1844213414 ## core/src/main/java/org/apache/iceberg/io/PartitioningDVWriter.java: ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] (WIP) Write DVs in Spark for V3 tables [iceberg]

2024-11-15 Thread via GitHub
amogh-jahagirdar commented on code in PR #11561: URL: https://github.com/apache/iceberg/pull/11561#discussion_r1844211908 ## core/src/main/java/org/apache/iceberg/io/PartitioningDVWriter.java: ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] (WIP) Write DVs in Spark for V3 tables [iceberg]

2024-11-15 Thread via GitHub
amogh-jahagirdar commented on PR #11561: URL: https://github.com/apache/iceberg/pull/11561#issuecomment-2479469983 Still some failing tests, and figuring out a good pattern to extend the existing Delete/Merge/Update tests to run against DVs. Something also worth thinking about for V3 is pre

Re: [I] remove warnings about "Catalog identifiers" if one is not provided [iceberg-python]

2024-11-15 Thread via GitHub
kevinjqliu closed issue #1318: remove warnings about "Catalog identifiers" if one is not provided URL: https://github.com/apache/iceberg-python/issues/1318 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Nested namespace support is broken in 1.7.0 [iceberg]

2024-11-15 Thread via GitHub
bryanck commented on issue #11539: URL: https://github.com/apache/iceberg/issues/11539#issuecomment-2479459903 I can reproduce this if I use Trino w/ Iceberg 1.7(commit `ead6d9f`) connecting to an Iceberg REST catalog running Iceberg 1.6 (using the Tabular image). If I update the image to I

Re: [PR] use the non-deprecated func to parse identifier [iceberg-python]

2024-11-15 Thread via GitHub
kevinjqliu merged PR #1326: URL: https://github.com/apache/iceberg-python/pull/1326 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2024-11-15 Thread via GitHub
k-alkiek commented on PR #11039: URL: https://github.com/apache/iceberg/pull/11039#issuecomment-2479396505 This would be a great addition. Love to see the collaboration with GCP. @rdblue, can we get it across the finish line? -- This is an automated message from the Apache Git Service. To

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-11-15 Thread via GitHub
dwilson1988 commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2479414127 > @dwilson1988 made the suggested changes, there's a deprecation warning on the S3 config EndpointResolver methods that I haven't had time to look into, maybe you could take a look?

Re: [I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2024-11-15 Thread via GitHub
caseyrathbone commented on issue #10808: URL: https://github.com/apache/iceberg/issues/10808#issuecomment-2479411580 Hello @nastra, Can we re-open this ticket? I am still experiencing the 'Byte-buffer read unsupported by com.databricks.common.filesystem.LokiAbfsInputStream' issue us

Re: [PR] feat(FileIO): Adds user extensible FileIO [iceberg-rust]

2024-11-15 Thread via GitHub
Xuanwo commented on PR #699: URL: https://github.com/apache/iceberg-rust/pull/699#issuecomment-2479292509 Thank you @BlakeOrth for this PR, I will visit code and try to find the migrate path. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] API, Core: Replace deprecated ContentFile#path usage with location [iceberg]

2024-11-15 Thread via GitHub
amogh-jahagirdar merged PR #11550: URL: https://github.com/apache/iceberg/pull/11550 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] API, Core: Replace deprecated ContentFile#path usage with location [iceberg]

2024-11-15 Thread via GitHub
amogh-jahagirdar commented on PR #11550: URL: https://github.com/apache/iceberg/pull/11550#issuecomment-2479282957 Thanks @nastra @Fokko @jbonofre for reviewing! I'll go ahead and merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] Core: Serialize `null` when there is no current snapshot [iceberg]

2024-11-15 Thread via GitHub
Fokko opened a new pull request, #11560: URL: https://github.com/apache/iceberg/pull/11560 While reviewing some stuff on the rust end, I noticed that the library can still produce `-1` for the current-snapshot-id which is invalid. -- This is an automated message from the Apache Git Servic

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
Fokko commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843960491 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
manuzhang commented on PR #10996: URL: https://github.com/apache/iceberg/pull/10996#issuecomment-2479011158 6 tests in `iceberg-mr` failed due to [HIVE-21584](https://issues.apache.org/jira/browse/HIVE-21584). ``` TestHiveIcebergWithHiveAutogatherEnable > initializationError FAILED

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-15 Thread via GitHub
nastra commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1843906444 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1200,6 +1200,8 @@ private RESTViewBuilder(SessionContext context, TableIdentifier identi

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
manuzhang commented on PR #10996: URL: https://github.com/apache/iceberg/pull/10996#issuecomment-2478998374 > I expect that there will be users trying to use HiveCatalog with old Hive versions. @pvary We don't expect them to use iceberg 1.8+ with old Hive versions, right? -- This

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
manuzhang commented on code in PR #10996: URL: https://github.com/apache/iceberg/pull/10996#discussion_r1843854663 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java: ## @@ -59,12 +57,12 @@ public class HiveIcebergInputFormat extends MapredIcebergInputFo

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
nastra commented on code in PR #10996: URL: https://github.com/apache/iceberg/pull/10996#discussion_r1843834812 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java: ## @@ -59,12 +57,12 @@ public class HiveIcebergInputFormat extends MapredIcebergInputForma

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
nastra commented on code in PR #10996: URL: https://github.com/apache/iceberg/pull/10996#discussion_r1843836878 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -213,11 +212,11 @@ private static final class IcebergRecordReader extends RecordRea

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
liurenjie1024 commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843798302 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribu

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
liurenjie1024 commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843722934 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2097 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribu

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
Fokko commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843792590 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2074 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-15 Thread via GitHub
nastra commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1843788190 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -127,6 +127,36 @@ public void basicCreateView() { assertThat(catalog().viewExists(ident

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-15 Thread via GitHub
nastra commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1843786671 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -127,6 +127,36 @@ public void basicCreateView() { assertThat(catalog().viewExists(ident

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-15 Thread via GitHub
nastra commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1843786109 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -127,6 +127,36 @@ public void basicCreateView() { assertThat(catalog().viewExists(ident

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-15 Thread via GitHub
nastra commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1843780475 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -871,6 +901,7 @@ public void createOrReplaceView(boolean useCreateOrReplace) { assert

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
liurenjie1024 commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1841689100 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribu

Re: [PR] REST: Docker file for Rest catalog adapter image [iceberg]

2024-11-15 Thread via GitHub
ajantha-bhat commented on code in PR #11283: URL: https://github.com/apache/iceberg/pull/11283#discussion_r1843720848 ## docker/iceberg-rest-adapter-image/README.md: ## @@ -0,0 +1,86 @@ + + +# Iceberg REST Catalog Adapter Test Fixture + +For converting different catalog implemen

[I] Gradle task to update LICENSE and NOTICE on every build for runtime jars [iceberg]

2024-11-15 Thread via GitHub
ajantha-bhat opened a new issue, #11559: URL: https://github.com/apache/iceberg/issues/11559 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 Currently LICENSE and NOTICE info is generated for runtime jars using the plugin

Re: [I] Add `SchemaUpdate` logic to Iceberg-Rust [iceberg-rust]

2024-11-15 Thread via GitHub
vinjai commented on issue #697: URL: https://github.com/apache/iceberg-rust/issues/697#issuecomment-2478643376 Hey @Fokko I would like to give this a shot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Nested namespace support is broken in 1.7.0 [iceberg]

2024-11-15 Thread via GitHub
nastra commented on issue #11539: URL: https://github.com/apache/iceberg/issues/11539#issuecomment-2478640441 Currently I cannot reproduce this issue with the below Trino test: ``` --- a/plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/catalog/rest/TestIcebergRestCatalogN

Re: [I] Nested namespace support is broken in 1.7.0 [iceberg]

2024-11-15 Thread via GitHub
nastra commented on issue #11539: URL: https://github.com/apache/iceberg/issues/11539#issuecomment-2478540063 @mayankvadariya can you provide additional details against which REST server this is running? Are there any reproducible tests in Trino itself to easily run & reproduce this? --

Re: [PR] Docs: Fix level of Deletion Vectors [iceberg]

2024-11-15 Thread via GitHub
liurenjie1024 merged PR #11547: URL: https://github.com/apache/iceberg/pull/11547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ice

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
pvary commented on PR #10996: URL: https://github.com/apache/iceberg/pull/10996#issuecomment-2478468746 I expect that there will be users trying to use HiveCatalog with old Hive versions. Do we want to (could we) throw a meaningful exception when this happens? Maybe we can throw an

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843538624 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2074 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Spec: Add cross-region bucket access property to config [iceberg]

2024-11-15 Thread via GitHub
munendrasn commented on PR #11260: URL: https://github.com/apache/iceberg/pull/11260#issuecomment-2478133424 @nastra Resolved conflicts, please review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
pvary commented on code in PR #10996: URL: https://github.com/apache/iceberg/pull/10996#discussion_r1843517724 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java: ## @@ -770,9 +769,7 @@ public void testIcebergAndHmsTableProperties() throws E

Re: [PR] Remove Hive 2 [iceberg]

2024-11-15 Thread via GitHub
pvary commented on code in PR #10996: URL: https://github.com/apache/iceberg/pull/10996#discussion_r1843512697 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -479,6 +446,17 @@ private CloseableIterable newOrcIterable( return applyResidu

[I] add_files with RestCatalog, S3FileIO [iceberg]

2024-11-15 Thread via GitHub
DongSeungLee opened a new issue, #11558: URL: https://github.com/apache/iceberg/issues/11558 ### Query engine Spark 3.5.3 ### Question for study, i run spark cluster standalone in my local, and i have developed my own IcebergRestCatalog. My IcebergRestCatalog Iceberg

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843391072 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843474749 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843470487 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-15 Thread via GitHub
c-thiel commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1843470080 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

  1   2   >