Re: [PR] Procedure to compute table stats [iceberg]

2024-11-18 Thread via GitHub
nastra commented on code in PR #10986: URL: https://github.com/apache/iceberg/pull/10986#discussion_r1847786178 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestComputeTableStatsProcedure.java: ## @@ -0,0 +1,124 @@ +/* + * Licensed to the Apac

Re: [PR] Parquet: Use native getRowIndexOffset support instead of calculating it [iceberg]

2024-11-18 Thread via GitHub
flyrain commented on PR #11520: URL: https://github.com/apache/iceberg/pull/11520#issuecomment-2484892614 Hi @Fokko, do you have any further feedback? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] flink restore failed with filenotfound [iceberg]

2024-11-18 Thread via GitHub
rahulkr25 commented on issue #6066: URL: https://github.com/apache/iceberg/issues/6066#issuecomment-2484868647 1. Do we have any update here, whether we have planned the resolution for this bug? @jad-grepr @stevenzwu 2. Secondly, as @congd123 mentioned, What is the best approach for rec

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1847748258 ## crates/iceberg/src/spec/manifest.rs: ## @@ -1328,7 +1334,7 @@ mod _serde { Ok(Self { content: value.content as i32,

Re: [PR] Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (#1331) [iceberg-python]

2024-11-18 Thread via GitHub
gitzwz commented on code in PR #1332: URL: https://github.com/apache/iceberg-python/pull/1332#discussion_r1847735535 ## pyiceberg/catalog/hive.py: ## @@ -651,7 +651,7 @@ def list_tables(self, namespace: Union[str, Identifier]) -> List[Identifier]: for table in

Re: [PR] Spark 3.5: Fix NotSerializableException when migrating Spark tables [iceberg]

2024-11-18 Thread via GitHub
manuzhang commented on PR #11157: URL: https://github.com/apache/iceberg/pull/11157#issuecomment-2484823743 @RussellSpitzer Can we move this PR forward? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Spark: 4.0 snapshot support [iceberg]

2024-11-18 Thread via GitHub
huaxingao commented on code in PR #11583: URL: https://github.com/apache/iceberg/pull/11583#discussion_r1847696050 ## spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/IcebergCall.scala: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Spark: 4.0 snapshot support [iceberg]

2024-11-18 Thread via GitHub
pan3793 commented on code in PR #11583: URL: https://github.com/apache/iceberg/pull/11583#discussion_r1847687552 ## spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/IcebergCall.scala: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Spark: 4.0 snapshot support [iceberg]

2024-11-18 Thread via GitHub
pan3793 commented on code in PR #11583: URL: https://github.com/apache/iceberg/pull/11583#discussion_r1847687552 ## spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/IcebergCall.scala: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Spark 3.4: IcebergSource extends SessionConfigSupport [iceberg]

2024-11-18 Thread via GitHub
pan3793 commented on code in PR #7732: URL: https://github.com/apache/iceberg/pull/7732#discussion_r1847679472 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java: ## @@ -1953,6 +1953,65 @@ public void testTableWithInt96Timestamp()

Re: [PR] Build: Bump antlr from 4.9.3 to 4.13.2 [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] commented on PR #10867: URL: https://github.com/apache/iceberg/pull/10867#issuecomment-2484438703 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
ZENOTME commented on PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#issuecomment-2484580413 > @ZENOTME Are you able to run the tests from your IDE? > > When running from RustRover, the test is stuck on starting the containers. When I change the code to exclude the start

Re: [PR] Bump Palentir gradle baseline [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] commented on PR #11012: URL: https://github.com/apache/iceberg/pull/11012#issuecomment-2484438796 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Procedure to compute table stats [iceberg]

2024-11-18 Thread via GitHub
karuppayya commented on code in PR #10986: URL: https://github.com/apache/iceberg/pull/10986#discussion_r1847343083 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestComputeTableStatsProcedure.java: ## @@ -0,0 +1,122 @@ +/* + * Licensed to the

Re: [PR] Ignore tables without table_type parameters while loading all iceberg table from Glue and Hive catalog (#1331) [iceberg-python]

2024-11-18 Thread via GitHub
HonahX commented on code in PR #1332: URL: https://github.com/apache/iceberg-python/pull/1332#discussion_r1847527528 ## pyiceberg/catalog/hive.py: ## @@ -651,7 +651,7 @@ def list_tables(self, namespace: Union[str, Identifier]) -> List[Identifier]: for table in

Re: [PR] Add `view_exists` method to REST Catalog [iceberg-python]

2024-11-18 Thread via GitHub
shiv-io commented on PR #1242: URL: https://github.com/apache/iceberg-python/pull/1242#issuecomment-2484508752 @sungwy just bumping this in case this fell off your radar! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[I] Iceberg 1.7.0 java.lang.IllegalStateException: Connection pool shut down [iceberg]

2024-11-18 Thread via GitHub
lawrencemq opened a new issue, #11582: URL: https://github.com/apache/iceberg/issues/11582 ### Apache Iceberg version 1.7.0 (latest release) ### Query engine Kafka Connect ### Please describe the bug ๐Ÿž I've been using the Tabular connector for some time and

Re: [PR] Bump Palentir gradle baseline [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] closed pull request #11012: Bump Palentir gradle baseline URL: https://github.com/apache/iceberg/pull/11012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Build: Bump antlr from 4.9.3 to 4.13.2 [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] closed pull request #10867: Build: Bump antlr from 4.9.3 to 4.13.2 URL: https://github.com/apache/iceberg/pull/10867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Materialized View Spec [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] commented on PR #11041: URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2484438838 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Build: Bump antlr from 4.9.3 to 4.13.2 [iceberg]

2024-11-18 Thread via GitHub
dependabot[bot] commented on PR #10867: URL: https://github.com/apache/iceberg/pull/10867#issuecomment-2484438750 OK, I won't notify you again about this release, but will get in touch when a new version is available. You can also ignore all major, minor, or patch releases for a dependency

Re: [PR] Core, API, Arrow: Type promotion for int/long to string for V3 tables [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] commented on PR #10991: URL: https://github.com/apache/iceberg/pull/10991#issuecomment-2484438766 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Materialized View Spec [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] closed pull request #11041: Materialized View Spec URL: https://github.com/apache/iceberg/pull/11041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] add aliyun bundle jar [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] commented on PR #10971: URL: https://github.com/apache/iceberg/pull/10971#issuecomment-2484438743 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Core, API, Arrow: Type promotion for int/long to string for V3 tables [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] closed pull request #10991: Core, API, Arrow: Type promotion for int/long to string for V3 tables URL: https://github.com/apache/iceberg/pull/10991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] add aliyun bundle jar [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] closed pull request #10971: add aliyun bundle jar URL: https://github.com/apache/iceberg/pull/10971 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [I] Is there any way on Flink to read newly appended data only (NOT in current Iceberg table snapshot)? [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] commented on issue #9955: URL: https://github.com/apache/iceberg/issues/9955#issuecomment-2484438543 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] AWS credential with [iceberg]

2024-11-18 Thread via GitHub
dyno commented on issue #11581: URL: https://github.com/apache/iceberg/issues/11581#issuecomment-2484392096 we also tried s3 magic committer and there is no expiration problem. it feels like the credentials retrieved from profile is not refreshed in iceberg S3FileIO S3 client. -- This

Re: [I] OR condition does not leverage all parquet metadata (metrics, dictionary, bloom filter) causing inefficient queries [iceberg]

2024-11-18 Thread via GitHub
raphaelauv commented on issue #10029: URL: https://github.com/apache/iceberg/issues/10029#issuecomment-2484383730 could we re-open this issue ? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] File size in bytes tracking with deleted files in expire snapshots [iceberg]

2024-11-18 Thread via GitHub
dramaticlly commented on code in PR #10036: URL: https://github.com/apache/iceberg/pull/10036#discussion_r1847331597 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RichFileInfo.java: ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[PR] Bump moto from 5.0.20 to 5.0.21 [iceberg-python]

2024-11-18 Thread via GitHub
dependabot[bot] opened a new pull request, #1339: URL: https://github.com/apache/iceberg-python/pull/1339 Bumps [moto](https://github.com/getmoto/moto) from 5.0.20 to 5.0.21. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog.

Re: [I] Do not deprecate Botocore Session in upcoming release (0.8) [iceberg-python]

2024-11-18 Thread via GitHub
matteosimone commented on issue #1104: URL: https://github.com/apache/iceberg-python/issues/1104#issuecomment-2484240520 @cshenrik I have the same issue of trying to utilize AWS profile that hits a web service to drive automatically refreshable credentials. Did you find any solution to thi

Re: [PR] Procedure to compute table stats [iceberg]

2024-11-18 Thread via GitHub
karuppayya commented on code in PR #10986: URL: https://github.com/apache/iceberg/pull/10986#discussion_r1847344065 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/ComputeTableStatsProcedure.java: ## @@ -0,0 +1,121 @@ +/* + * Licensed to the Apache Software

Re: [PR] Procedure to compute table stats [iceberg]

2024-11-18 Thread via GitHub
karuppayya commented on code in PR #10986: URL: https://github.com/apache/iceberg/pull/10986#discussion_r1847343494 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/ComputeTableStatsProcedure.java: ## @@ -0,0 +1,121 @@ +/* + * Licensed to the Apache Software

Re: [PR] Procedure to compute table stats [iceberg]

2024-11-18 Thread via GitHub
szehon-ho commented on code in PR #10986: URL: https://github.com/apache/iceberg/pull/10986#discussion_r1847292068 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestComputeTableStatsProcedure.java: ## @@ -0,0 +1,122 @@ +/* + * Licensed to the A

[PR] Bump aiohttp from 3.10.5 to 3.10.11 [iceberg-python]

2024-11-18 Thread via GitHub
dependabot[bot] opened a new pull request, #1338: URL: https://github.com/apache/iceberg-python/pull/1338 [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aiohttp&package-manager=pip&previous-version=3.10.5&new-vers

Re: [PR] 0.8.0 post release steps [iceberg-python]

2024-11-18 Thread via GitHub
kevinjqliu merged PR #1334: URL: https://github.com/apache/iceberg-python/pull/1334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

[I] SortField requires you to specify a transform? [iceberg-python]

2024-11-18 Thread via GitHub
anentropic opened a new issue, #1337: URL: https://github.com/apache/iceberg-python/issues/1337 ### Apache Iceberg version 0.8.0 (latest release) ### Please describe the bug ๐Ÿž It looks like`transform` is intended to be an optional field (?): ```python class Sort

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
Fokko commented on PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#issuecomment-2484102346 @ZENOTME Are you able to run the tests from your IDE? When running from RustRover, the test is stuck on starting the containers. When I change the code to exclude the starting of t

Re: [PR] Kafka Connect: Add config to route to tables using topic name [iceberg]

2024-11-18 Thread via GitHub
mun1r0b0t commented on PR #11313: URL: https://github.com/apache/iceberg/pull/11313#issuecomment-2484098247 Hi folks, is this still being worked on? I have a similar need to route by topic, but the changes in this PR will not address my needs. I need to configure the topic for indivi

Re: [PR] Bump coverage from 7.6.5 to 7.6.7 [iceberg-python]

2024-11-18 Thread via GitHub
Fokko merged PR #1329: URL: https://github.com/apache/iceberg-python/pull/1329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Default to `overwrite` when operation is missing [iceberg]

2024-11-18 Thread via GitHub
Fokko commented on PR #11421: URL: https://github.com/apache/iceberg/pull/11421#issuecomment-2484094018 @manuzhang I'd rather not clutter the spec with ways how to fix "out-of-spec" metadata. I've added a test in case the summary is empty, I think this also clarifies that we only want to fi

Re: [PR] Use `svn mv` instead for releasing [iceberg-go]

2024-11-18 Thread via GitHub
Fokko merged PR #211: URL: https://github.com/apache/iceberg-go/pull/211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1847225254 ## crates/e2e_test/tests/conflict_commit_test.rs: ## @@ -0,0 +1,196 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

[I] How to get rid of the warning [iceberg-python]

2024-11-18 Thread via GitHub
djouallah opened a new issue, #1336: URL: https://github.com/apache/iceberg-python/issues/1336 ### Question using this code with pyiceberg 0.8 ``` tbl = db+"."+"calendar" if not catalog.table_exists(tbl): df=duckdb.sql(""" SELECT cast(unnest(generate_series(cast ('201

Re: [PR] Spark 3.3: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-18 Thread via GitHub
RussellSpitzer commented on PR #11580: URL: https://github.com/apache/iceberg/pull/11580#issuecomment-2484033237 Thanks @pan3793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Spark 3.4: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-18 Thread via GitHub
RussellSpitzer commented on PR #11579: URL: https://github.com/apache/iceberg/pull/11579#issuecomment-2484032296 Merged Thanks @pan3793 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Spark 3.3: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-18 Thread via GitHub
RussellSpitzer merged PR #11580: URL: https://github.com/apache/iceberg/pull/11580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Spark 3.4: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-18 Thread via GitHub
RussellSpitzer merged PR #11579: URL: https://github.com/apache/iceberg/pull/11579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-18 Thread via GitHub
nastra commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1846005241 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1200,6 +1200,8 @@ private RESTViewBuilder(SessionContext context, TableIdentifier identi

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1847168059 ## crates/iceberg/src/transaction.rs: ## @@ -96,6 +109,60 @@ impl<'a> Transaction<'a> { Ok(self) } +fn generate_unique_snapshot_id(&self) -> i64 {

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1847167614 ## crates/iceberg/src/transaction.rs: ## @@ -96,6 +109,60 @@ impl<'a> Transaction<'a> { Ok(self) } +fn generate_unique_snapshot_id(&self) -> i64 {

Re: [I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2024-11-18 Thread via GitHub
caseyrathbone commented on issue #10808: URL: https://github.com/apache/iceberg/issues/10808#issuecomment-2483969761 Excellent, I just confirmed DBRT 16.0 works properly, thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Spark 3.4: IcebergSource extends SessionConfigSupport [iceberg]

2024-11-18 Thread via GitHub
szehon-ho commented on code in PR #7732: URL: https://github.com/apache/iceberg/pull/7732#discussion_r1847150692 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java: ## @@ -1953,6 +1953,65 @@ public void testTableWithInt96Timestamp(

Re: [PR] Spark 3.4: IcebergSource extends SessionConfigSupport [iceberg]

2024-11-18 Thread via GitHub
szehon-ho commented on code in PR #7732: URL: https://github.com/apache/iceberg/pull/7732#discussion_r1847150692 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java: ## @@ -1953,6 +1953,65 @@ public void testTableWithInt96Timestamp(

Re: [PR] Spark 3.4: IcebergSource extends SessionConfigSupport [iceberg]

2024-11-18 Thread via GitHub
szehon-ho commented on code in PR #7732: URL: https://github.com/apache/iceberg/pull/7732#discussion_r1847158756 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java: ## @@ -61,7 +62,8 @@ * The above list is in order of priority. For example: a

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-11-18 Thread via GitHub
huaxingao commented on code in PR #11551: URL: https://github.com/apache/iceberg/pull/11551#discussion_r1847131830 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -45,11 +45,23 @@ public class ColumnarBatchReader extends

[PR] 0.8.0 post release steps [iceberg-python]

2024-11-18 Thread via GitHub
kevinjqliu opened a new pull request, #1334: URL: https://github.com/apache/iceberg-python/pull/1334 This PR addresses post-release steps as documented in https://py.iceberg.apache.org/how-to-release/ including * Update the Github template * Update the integration tests This PR

Re: [PR] Feature: Write to branches [iceberg-python]

2024-11-18 Thread via GitHub
MChamberlin commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2483824116 @kevinjqliu Is there anyway this can be included in the 0.8.0 release? This is an important feature add with only relatively simple refactoring. -- This is an automated message

Re: [PR] Bugfix for incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]

2024-11-18 Thread via GitHub
pvary commented on PR #11576: URL: https://github.com/apache/iceberg/pull/11576#issuecomment-2482286546 @ZhendongBai: Could you please add some unit tests which fail before the patch, and succeed after the fix? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-18 Thread via GitHub
RussellSpitzer commented on PR #11513: URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2483756188 @pvary and I were talking this over a bit, I think we really want to get a stronger division between "Planning" and "Excecution" since the two are very intertwined right now.

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-11-18 Thread via GitHub
singhpk234 commented on code in PR #11551: URL: https://github.com/apache/iceberg/pull/11551#discussion_r1847025083 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -45,11 +45,23 @@ public class ColumnarBatchReader extend

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-11-18 Thread via GitHub
singhpk234 commented on code in PR #11551: URL: https://github.com/apache/iceberg/pull/11551#discussion_r1847025083 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -45,11 +45,23 @@ public class ColumnarBatchReader extend

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-11-18 Thread via GitHub
singhpk234 commented on code in PR #11551: URL: https://github.com/apache/iceberg/pull/11551#discussion_r1847025083 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -45,11 +45,23 @@ public class ColumnarBatchReader extend

Re: [PR] Spark 3.4: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-18 Thread via GitHub
pan3793 commented on PR #11579: URL: https://github.com/apache/iceberg/pull/11579#issuecomment-2483735222 thank you for quick checking, @RussellSpitzer and @singhpk234, I also opened a PR for Spark 3.3 https://github.com/apache/iceberg/pull/11580 -- This is an automated message from the A

[I] .pyiceberg.yaml config files should be loaded from current dir instead of home folder [iceberg-python]

2024-11-18 Thread via GitHub
anentropic opened a new issue, #1333: URL: https://github.com/apache/iceberg-python/issues/1333 ### Feature Request / Improvement https://py.iceberg.apache.org/configuration/#catalogs > There are three ways to pass in configuration: > > - Using the `~/.pyiceberg.yaml` conf

Re: [PR] Core: Serialize `null` when there is no current snapshot [iceberg]

2024-11-18 Thread via GitHub
RussellSpitzer merged PR #11560: URL: https://github.com/apache/iceberg/pull/11560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-18 Thread via GitHub
liurenjie1024 commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1846505256 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2074 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribu

Re: [PR] Bump mkdocstrings from 0.26.2 to 0.27.0 [iceberg-python]

2024-11-18 Thread via GitHub
Fokko merged PR #1324: URL: https://github.com/apache/iceberg-python/pull/1324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Spark 3.4: IcebergSource extends SessionConfigSupport [iceberg]

2024-11-18 Thread via GitHub
pan3793 commented on code in PR #7732: URL: https://github.com/apache/iceberg/pull/7732#discussion_r1846854428 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java: ## @@ -1953,6 +1953,65 @@ public void testTableWithInt96Timestamp()

Re: [PR] Spark 3.4: IcebergSource extends SessionConfigSupport [iceberg]

2024-11-18 Thread via GitHub
pan3793 commented on code in PR #7732: URL: https://github.com/apache/iceberg/pull/7732#discussion_r1846854428 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java: ## @@ -1953,6 +1953,65 @@ public void testTableWithInt96Timestamp()

Re: [PR] Spark 3.4: IcebergSource extends SessionConfigSupport [iceberg]

2024-11-18 Thread via GitHub
pan3793 commented on PR #7732: URL: https://github.com/apache/iceberg/pull/7732#issuecomment-2483411197 rebased on the latest main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Revert "Core: Use encoding/decoding methods for namespaces and deprecate Splitter/Joiner" [iceberg]

2024-11-18 Thread via GitHub
bryanck commented on PR #11574: URL: https://github.com/apache/iceberg/pull/11574#issuecomment-2483436246 I'm not convinced we need to revert this. In my test env I have Trino 465 + 1.7 with nested namespaces enabled, connecting to an Iceberg 1.6 REST catalog. I'm seeing the same error in t

Re: [I] Incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]

2024-11-18 Thread via GitHub
pvary commented on issue #11575: URL: https://github.com/apache/iceberg/issues/11575#issuecomment-2482281011 @ZhendongBai: Could you please check if this issue still exists in the current connector version? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] API, Arrow, Core, Data, Spark: Replace usage of deprecated ContentFile#path API with location API [iceberg]

2024-11-18 Thread via GitHub
amogh-jahagirdar commented on PR #11563: URL: https://github.com/apache/iceberg/pull/11563#issuecomment-2483373763 Thanks @nastra for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] API, Arrow, Core, Data, Spark: Replace usage of deprecated ContentFile#path API with location API [iceberg]

2024-11-18 Thread via GitHub
amogh-jahagirdar merged PR #11563: URL: https://github.com/apache/iceberg/pull/11563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

2024-11-18 Thread via GitHub
pvary commented on code in PR #11513: URL: https://github.com/apache/iceberg/pull/11513#discussion_r1846675235 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -341,20 +274,19 @@ private Builder doExecute( } priva

Re: [PR] Bugfix for incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]

2024-11-18 Thread via GitHub
pvary commented on PR #11576: URL: https://github.com/apache/iceberg/pull/11576#issuecomment-2482760387 @ZhendongBai: `TestHiveCommits` contains plenty of examples where the different error scenarios are tested. I think it should be possible to create one which tests the scenario mentioned

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
ZENOTME commented on PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#issuecomment-2483287118 > @ZENOTME Sorry for leaving this hanging for so long, I'll do another pass today. I wanted to do it, but running into some issues with RustRover. I have to step out for some other iss

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1846727927 ## crates/e2e_test/tests/conflict_commit_test.rs: ## @@ -0,0 +1,196 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

Re: [I] Byte-buffer read unsupported by org.apache.hadoop.fs.BufferedFSInputStream across multiple iceberg versions on 14.3 LTS databricks, Spark 3.5 Scala 2.12 [iceberg]

2024-11-18 Thread via GitHub
amogh-jahagirdar commented on issue #10808: URL: https://github.com/apache/iceberg/issues/10808#issuecomment-2483251029 Hey @caseyrathbone this should be fixed in DBR 16+, could you please try that and if there's still any issues please reach out through DB support? This is a DB specific is

Re: [PR] REST: Docker file for Rest catalog adapter image [iceberg]

2024-11-18 Thread via GitHub
ajantha-bhat commented on PR #11283: URL: https://github.com/apache/iceberg/pull/11283#issuecomment-2482670213 @jbonofre: Do you have any more comments for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
Fokko commented on PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#issuecomment-2483245755 @ZENOTME Sorry for leaving this hanging for so long, I'll do another pass today. I wanted to do it, but running into some issues with RustRover. I have to step out for some other issues,

Re: [PR] Build: Bump nessie from 0.99.0 to 0.100.0 [iceberg]

2024-11-18 Thread via GitHub
nastra merged PR #11567: URL: https://github.com/apache/iceberg/pull/11567 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Bugfix for incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]

2024-11-18 Thread via GitHub
ZhendongBai commented on PR #11576: URL: https://github.com/apache/iceberg/pull/11576#issuecomment-2483206183 > @ZhendongBai: `TestHiveCommits` contains plenty of examples where the different error scenarios are tested. I think it should be possible to create one which tests the scenario me

Re: [I] Consider Using object_store as IO Abstraction [iceberg-rust]

2024-11-18 Thread via GitHub
tustvold commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2483162776 I _think_ that should work, the DataFusion wrapper can just hook the iceberg metadata operations into via that StorageProvider trait, and then use the DataFusion machinery direct

Re: [PR] column to column comparisons for filtering file scans and row data [iceberg]

2024-11-18 Thread via GitHub
github-actions[bot] commented on PR #11152: URL: https://github.com/apache/iceberg/pull/11152#issuecomment-2481696719 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatโ€™s incorrect or this pul

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1846645067 ## crates/iceberg/src/transaction.rs: ## @@ -122,6 +189,387 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append dat

Re: [PR] Remove Hive 2 [iceberg]

2024-11-18 Thread via GitHub
pvary commented on PR #10996: URL: https://github.com/apache/iceberg/pull/10996#issuecomment-2482780914 > The failed tests list [..] > Do we have these testing for Hive 3 on main branch now? I think these tests should be running fine with Java 8. The issue is that with Java 11 we c

Re: [I] iceberg docker-compose quickstart fail [iceberg]

2024-11-18 Thread via GitHub
nastra commented on issue #11544: URL: https://github.com/apache/iceberg/issues/11544#issuecomment-2482143545 It might be because docker is limiting how much memory that container gets. Maybe try running the `iceberg-rest` container in isolation first. Typically it shouldn't need a lot of m

Re: [PR] Flink: Maintenance - RewriteDataFiles [iceberg]

2024-11-18 Thread via GitHub
pvary commented on PR #11497: URL: https://github.com/apache/iceberg/pull/11497#issuecomment-2482774719 @ygrzjh: Seems like an issue with the `S3FileIO` serialization. Do you have any more issue on which field causes the nullpointer? -- This is an automated message from the Apache Git Ser

Re: [I] Consider Using object_store as IO Abstraction [iceberg-rust]

2024-11-18 Thread via GitHub
liurenjie1024 commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2483125592 > > point about the existing Datafusion machinery > > DataFusion provides an [ObjectStoreRegistry](https://docs.rs/datafusion/latest/datafusion/datasource/object_store

Re: [I] Handling Updates on Partition Columns in Iceberg with Flink CDC [iceberg]

2024-11-18 Thread via GitHub
pvary commented on issue #11573: URL: https://github.com/apache/iceberg/issues/11573#issuecomment-2482029220 What are the records generated by the MySQL CDC connector? You are using upsert mode in FlinkSink. In upsert mode when an update happens, Flink expects an unchanged prima

Re: [I] Flink: Not Writing [iceberg]

2024-11-18 Thread via GitHub
a8356555 closed issue #8916: Flink: Not Writing URL: https://github.com/apache/iceberg/issues/8916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issue

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1846625846 ## crates/e2e_test/tests/conflict_commit_test.rs: ## @@ -0,0 +1,196 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-18 Thread via GitHub
Fokko commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1846624096 ## crates/e2e_test/tests/conflict_commit_test.rs: ## @@ -0,0 +1,196 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-18 Thread via GitHub
liurenjie1024 commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1846578408 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribu

Re: [I] iceberg docker-compose quickstart fail [iceberg]

2024-11-18 Thread via GitHub
nastra closed issue #11544: iceberg docker-compose quickstart fail URL: https://github.com/apache/iceberg/issues/11544 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] TableMetadataBuilder [iceberg-rust]

2024-11-18 Thread via GitHub
liurenjie1024 commented on code in PR #587: URL: https://github.com/apache/iceberg-rust/pull/587#discussion_r1846508704 ## crates/iceberg/src/spec/table_metadata_builder.rs: ## @@ -0,0 +1,2070 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribu

Re: [PR] API: Support removeUnusedSpecs in ExpireSnapshots [iceberg]

2024-11-18 Thread via GitHub
advancedxy commented on PR #10755: URL: https://github.com/apache/iceberg/pull/10755#issuecomment-2482921541 @nastra Thanks for reviewing, all your comments should be addressed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

  1   2   >