Re: [I] No module named 'pyiceberg.table.partitioning' [iceberg]

2024-06-14 Thread via GitHub
Shekharrajak commented on issue #10491: URL: https://github.com/apache/iceberg/issues/10491#issuecomment-2167390602 Issue should be part of https://github.com/apache/iceberg-python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Concurrent table scans [iceberg-rust]

2024-06-14 Thread via GitHub
sdd commented on code in PR #373: URL: https://github.com/apache/iceberg-rust/pull/373#discussion_r1639394777 ## crates/iceberg/src/scan.rs: ## @@ -302,13 +262,147 @@ impl TableScan { arrow_reader_builder.build().read(self.plan_files().await?) } +} + +#[derive(De

Re: [I] Iceberg fails ACID verification test [iceberg]

2024-06-14 Thread via GitHub
matthijseikelenboom closed issue #10454: Iceberg fails ACID verification test URL: https://github.com/apache/iceberg/issues/10454 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Iceberg fails ACID verification test [iceberg]

2024-06-14 Thread via GitHub
matthijseikelenboom commented on issue #10454: URL: https://github.com/apache/iceberg/issues/10454#issuecomment-2167457284 Closing as this has been resolved -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Avro referenced in metadata json file is missing [iceberg]

2024-06-14 Thread via GitHub
cgpoh commented on issue #7137: URL: https://github.com/apache/iceberg/issues/7137#issuecomment-2167500208 I'm facing this issue again today. I think the cause of the problem is our Hive Metastore ran out of memory and before it crashes, it created the metadata file and cause the key not fo

Re: [I] Spark: Dropping partition column from old partition table corrupts entire table [iceberg]

2024-06-14 Thread via GitHub
walkkker commented on issue #10234: URL: https://github.com/apache/iceberg/issues/10234#issuecomment-2167528491 > Hi, may I ask the repair method is to traverse all schemas or the partition spec persist with type in metadata? According to #10352 , I assume it traverses all manifests t

Re: [PR] Run revapi workflow on workflow/build system changes [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10485: URL: https://github.com/apache/iceberg/pull/10485#discussion_r1639502893 ## .github/workflows/api-binary-compatibility.yml: ## @@ -27,6 +27,9 @@ on: - 'apache-iceberg-**' pull_request: paths: + - '.github/**' Review Com

Re: [PR] Run revapi workflow on workflow/build system changes [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10485: URL: https://github.com/apache/iceberg/pull/10485#discussion_r1639502893 ## .github/workflows/api-binary-compatibility.yml: ## @@ -27,6 +27,9 @@ on: - 'apache-iceberg-**' pull_request: paths: + - '.github/**' Review Com

Re: [PR] Use all known Scala versions when -DallVersions set [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10478: URL: https://github.com/apache/iceberg/pull/10478#discussion_r1639509324 ## settings.gradle: ## @@ -72,6 +72,7 @@ project(':open-api').name = 'iceberg-open-api' if (null != System.getProperty("allVersions")) { System.setProperty("flin

Re: [PR] Build: Bump Nessie to 0.90.4 [iceberg]

2024-06-14 Thread via GitHub
ajantha-bhat commented on PR #10492: URL: https://github.com/apache/iceberg/pull/10492#issuecomment-2167580787 FYI: Dependabot will raise this automatically. (It checks every sunday) https://github.com/apache/iceberg/blob/b6c949cd86c372a87a4d43a557c19dd310af80d8/.github/dependabot.yml#L34

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639512270 ## gradle/wrapper/gradle-wrapper.properties: ## @@ -1,8 +1,8 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists # checksum was taken from https://g

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639514171 ## baseline.gradle: ## @@ -60,8 +63,7 @@ subprojects { spotless { java { target 'src/main/java/**/*.java', 'src/test/java/**/*.java', 'src/jmh/ja

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1639515495 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639515154 ## aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogCommitFailure.java: ## @@ -531,7 +531,8 @@ private int metadataFileCount(TableMetadata metadata)

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639515485 ## .github/workflows/api-binary-compatibility.yml: ## @@ -27,6 +27,9 @@ on: - 'apache-iceberg-**' pull_request: paths: + - '.github/**' Review Com

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1639518145 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1639519224 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1639527100 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +29,8 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.or

Re: [PR] Build: Bump Nessie to 0.90.4 [iceberg]

2024-06-14 Thread via GitHub
nastra merged PR #10492: URL: https://github.com/apache/iceberg/pull/10492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spark Action to Analyze table [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1639586779 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/NDVSketchGenerator.java: ## @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Run Flink, Spark3 tests on Java 17 too [iceberg]

2024-06-14 Thread via GitHub
findepi commented on PR #10477: URL: https://github.com/apache/iceberg/pull/10477#issuecomment-2167694287 > I know a lot of Spark and Flink users run on Java 17. this is valuable feedback! > this was not an issue, probably because code compiled with old Java typically runs fine

Re: [PR] Run revapi workflow on workflow/build system changes [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10485: URL: https://github.com/apache/iceberg/pull/10485#discussion_r1639599087 ## .github/workflows/api-binary-compatibility.yml: ## @@ -27,6 +27,9 @@ on: - 'apache-iceberg-**' pull_request: paths: + - '.github/**' Review Co

Re: [PR] Use all known Scala versions when -DallVersions set [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10478: URL: https://github.com/apache/iceberg/pull/10478#discussion_r1639601085 ## settings.gradle: ## @@ -72,6 +72,7 @@ project(':open-api').name = 'iceberg-open-api' if (null != System.getProperty("allVersions")) { System.setProperty("fli

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639601901 ## gradle/wrapper/gradle-wrapper.properties: ## @@ -1,8 +1,8 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists # checksum was taken from https://

Re: [I] No module named 'pyiceberg.table.partitioning' [iceberg]

2024-06-14 Thread via GitHub
Shekharrajak commented on issue #10491: URL: https://github.com/apache/iceberg/issues/10491#issuecomment-2167706584 I think you must try ``` >>> from pyiceberg.partitioning import PartitionSpec, PartitionField ``` -- This is an automated message from the Apache Git Service. To

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639609595 ## baseline.gradle: ## @@ -60,8 +63,7 @@ subprojects { spotless { java { target 'src/main/java/**/*.java', 'src/test/java/**/*.java', 'src/jmh/j

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639612184 ## .github/workflows/api-binary-compatibility.yml: ## @@ -27,6 +27,9 @@ on: - 'apache-iceberg-**' pull_request: paths: + - '.github/**' Review Co

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639613179 ## aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogCommitFailure.java: ## @@ -531,7 +531,8 @@ private int metadataFileCount(TableMetadata metadata

Re: [PR] Use all known Scala versions when -DallVersions set [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10478: URL: https://github.com/apache/iceberg/pull/10478#discussion_r1639856434 ## settings.gradle: ## @@ -72,6 +72,7 @@ project(':open-api').name = 'iceberg-open-api' if (null != System.getProperty("allVersions")) { System.setProperty("fli

Re: [PR] Use all known Scala versions when -DallVersions set [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10478: URL: https://github.com/apache/iceberg/pull/10478#discussion_r1639845567 ## settings.gradle: ## @@ -72,6 +72,7 @@ project(':open-api').name = 'iceberg-open-api' if (null != System.getProperty("allVersions")) { System.setProperty("flin

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1639862872 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1639866342 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1639868640 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1639863416 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1639863717 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

Re: [I] JdbcCatalog fails to initialize with MS SQL Server [iceberg]

2024-06-14 Thread via GitHub
elabedo commented on issue #10068: URL: https://github.com/apache/iceberg/issues/10068#issuecomment-2168115988 Thank you @jbonofre , have you the PR links ? you can help you on that :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1639862391 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] Use all known Scala versions when -DallVersions set [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10478: URL: https://github.com/apache/iceberg/pull/10478#discussion_r1639879685 ## settings.gradle: ## @@ -72,6 +72,7 @@ project(':open-api').name = 'iceberg-open-api' if (null != System.getProperty("allVersions")) { System.setProperty("flin

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
nastra commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1639894786 ## baseline.gradle: ## @@ -60,8 +63,7 @@ subprojects { spotless { java { target 'src/main/java/**/*.java', 'src/test/java/**/*.java', 'src/jmh/ja

[PR] Add list_views to rest catalog [iceberg-python]

2024-06-14 Thread via GitHub
ndrluis opened a new pull request, #817: URL: https://github.com/apache/iceberg-python/pull/817 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Remove ContentCache.invalidateAll [iceberg]

2024-06-14 Thread via GitHub
findepi commented on PR #10494: URL: https://github.com/apache/iceberg/pull/10494#issuecomment-2168277975 BTW i did a lot of work on https://github.com/google/guava/issues/1881 problem in Trino. The result of this was the `EvictableCache` library in Trino https://github.com/trinodb/trino

Re: [I] Custom s3 endpoint: Unable to execute HTTP request: Remote host terminated the handshake [iceberg]

2024-06-14 Thread via GitHub
nastra commented on issue #10490: URL: https://github.com/apache/iceberg/issues/10490#issuecomment-2168277444 It complains because it can't access `http://localhost:8181/v1/namespaces/default/tables`. Make sure that the REST server is accessible via that URI -- This is an automated messa

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
slessard commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1640137631 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
slessard commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1640138360 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] #10275 - fix NullPointerException [iceberg]

2024-06-14 Thread via GitHub
slessard commented on code in PR #10284: URL: https://github.com/apache/iceberg/pull/10284#discussion_r1640138994 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -263,6 +265,60 @@ public void testReadColumnFilter2() throws Exception {

[PR] Remove deprecated methods from ContentCache [iceberg]

2024-06-14 Thread via GitHub
findepi opened a new pull request, #10495: URL: https://github.com/apache/iceberg/pull/10495 They are deprecated, unused and not part of API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [Reference PR] [API + Avro] Add default value APIs and Avro implementation [iceberg]

2024-06-14 Thread via GitHub
wmoustafa commented on PR #9502: URL: https://github.com/apache/iceberg/pull/9502#issuecomment-2168482841 @Fokko Could you please take a look since you helped with #9366? FYI @rdblue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [I] Why Iceberg do not support column with default value? [iceberg]

2024-06-14 Thread via GitHub
wmoustafa commented on issue #8527: URL: https://github.com/apache/iceberg/issues/8527#issuecomment-2168484232 PR here https://github.com/apache/iceberg/pull/9502. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Deprecate ContentCache.invalidateAll [iceberg]

2024-06-14 Thread via GitHub
findepi commented on PR #10494: URL: https://github.com/apache/iceberg/pull/10494#issuecomment-2168490131 cc @rizaon @szehon-ho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [I] Support default value semantics in Iceberg [iceberg]

2024-06-14 Thread via GitHub
wmoustafa commented on issue #2039: URL: https://github.com/apache/iceberg/issues/2039#issuecomment-2168491629 Linking the PR. @jbonofre could you help reopening this issue? It is not resolved yet. -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Cast PyArrow schema to `large_*` types [iceberg-python]

2024-06-14 Thread via GitHub
syun64 commented on PR #807: URL: https://github.com/apache/iceberg-python/pull/807#issuecomment-2168510997 @HonahX could I ask for you to merge this in? It'll help unblock me in https://github.com/apache/iceberg-python/pull/786/files -- This is an automated message from the Apache Git Se

[I] Support IsolationLevels [iceberg-python]

2024-06-14 Thread via GitHub
syun64 opened a new issue, #819: URL: https://github.com/apache/iceberg-python/issues/819 ### Feature Request / Improvement Support Isolation Levels from specified snapshot ID https://iceberg.apache.org/docs/latest/spark-configuration/#write-options -- This is an automated m

[PR] Add drop_view to the rest catalog [iceberg-python]

2024-06-14 Thread via GitHub
ndrluis opened a new pull request, #820: URL: https://github.com/apache/iceberg-python/pull/820 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Support building with Java 21 [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10474: URL: https://github.com/apache/iceberg/pull/10474#discussion_r1640228930 ## baseline.gradle: ## @@ -60,8 +63,7 @@ subprojects { spotless { java { target 'src/main/java/**/*.java', 'src/test/java/**/*.java', 'src/jmh/j

Re: [PR] Use all known Scala versions when -DallVersions set [iceberg]

2024-06-14 Thread via GitHub
findepi commented on code in PR #10478: URL: https://github.com/apache/iceberg/pull/10478#discussion_r1640230101 ## settings.gradle: ## @@ -72,6 +72,7 @@ project(':open-api').name = 'iceberg-open-api' if (null != System.getProperty("allVersions")) { System.setProperty("fli

Re: [PR] Deprecate ContentCache.invalidateAll [iceberg]

2024-06-14 Thread via GitHub
rizaon commented on PR #10494: URL: https://github.com/apache/iceberg/pull/10494#issuecomment-2168593729 Hi @findepi, thank you for looking into this. https://github.com/google/guava/issues/1881 also came to my attention recently in an [Impala code review](https://gerrit.cloudera.org/c/208

Re: [PR] Deprecate ContentCache.invalidateAll [iceberg]

2024-06-14 Thread via GitHub
findepi commented on PR #10494: URL: https://github.com/apache/iceberg/pull/10494#issuecomment-2168612952 @rizaon thank you for your feedback! > Although, I think it is less of an issue in Iceberg since Iceberg manifest files are immutable? Good point, I also thought about this

Re: [PR] Bump pydantic from 2.7.3 to 2.7.4 [iceberg-python]

2024-06-14 Thread via GitHub
Fokko merged PR #816: URL: https://github.com/apache/iceberg-python/pull/816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Bump pypa/cibuildwheel from 2.19.0 to 2.19.1 [iceberg-python]

2024-06-14 Thread via GitHub
Fokko merged PR #814: URL: https://github.com/apache/iceberg-python/pull/814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Remove deprecated methods from ContentCache [iceberg]

2024-06-14 Thread via GitHub
szehon-ho commented on PR #10495: URL: https://github.com/apache/iceberg/pull/10495#issuecomment-2168679498 How about removing the next one: public InputFile tryCache(FileIO io, String location, long length) ? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Core: Simplify loadCatalog method call in Iceberg [iceberg]

2024-06-14 Thread via GitHub
Fokko merged PR #10488: URL: https://github.com/apache/iceberg/pull/10488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] Support IsolationLevels [iceberg-python]

2024-06-14 Thread via GitHub
jqin61 commented on issue #819: URL: https://github.com/apache/iceberg-python/issues/819#issuecomment-2168685740 Hi I am interested in working on it! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Cast PyArrow schema to `large_*` types [iceberg-python]

2024-06-14 Thread via GitHub
Fokko merged PR #807: URL: https://github.com/apache/iceberg-python/pull/807 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Upcasting and Downcasting inconsistencies with PyArrow Schema [iceberg-python]

2024-06-14 Thread via GitHub
Fokko closed issue #791: Upcasting and Downcasting inconsistencies with PyArrow Schema URL: https://github.com/apache/iceberg-python/issues/791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Bump mypy-boto3-glue from 1.34.121 to 1.34.126 [iceberg-python]

2024-06-14 Thread via GitHub
Fokko merged PR #815: URL: https://github.com/apache/iceberg-python/pull/815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Support IsolationLevels [iceberg-python]

2024-06-14 Thread via GitHub
syun64 commented on issue #819: URL: https://github.com/apache/iceberg-python/issues/819#issuecomment-2168717631 Some relevant links to the Java implementation - validateNewDataFiles flag -> MergingSnapshotProducer: [validateAddedDataFiles](https://github.com/apache/iceberg/blob/c7d3e

Re: [PR] Remove deprecated methods from ContentCache [iceberg]

2024-06-14 Thread via GitHub
findepi commented on PR #10495: URL: https://github.com/apache/iceberg/pull/10495#issuecomment-2168719835 @szehon-ho `public InputFile tryCache(FileIO io, String location, long length)` is indeed also deprecated, but strictly speaking, removing this method would be breaking iceberg-core api

Re: [PR] Remove deprecated methods from ContentCache [iceberg]

2024-06-14 Thread via GitHub
szehon-ho commented on PR #10495: URL: https://github.com/apache/iceberg/pull/10495#issuecomment-2168763163 Yea Im not familiar with ContentCache but that is strange that return type is private..., yea in any case according to revapi we are breaking core compatibility. -- This is an auto

Re: [PR] Remove deprecated methods from ContentCache [iceberg]

2024-06-14 Thread via GitHub
findepi closed pull request #10495: Remove deprecated methods from ContentCache URL: https://github.com/apache/iceberg/pull/10495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Avoid Upcasting to `pa.large_binary()` [iceberg-python]

2024-06-14 Thread via GitHub
syun64 closed pull request #803: Avoid Upcasting to `pa.large_binary()` URL: https://github.com/apache/iceberg-python/pull/803 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Mr: Optimize schema string retrieval in Iceberg [iceberg]

2024-06-14 Thread via GitHub
amogh-jahagirdar merged PR #10489: URL: https://github.com/apache/iceberg/pull/10489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

[I] Add support for Spark 4.0.0 [iceberg]

2024-06-14 Thread via GitHub
Desmeister opened a new issue, #10497: URL: https://github.com/apache/iceberg/issues/10497 ### Feature Request / Improvement We should explore adding support for Spark 4.0.0. Based on the [Spark 4.0.0 Umbrella JIRA](https://issues.apache.org/jira/browse/SPARK-44111) and [this m

Re: [I] Shoud Flink Iceberg table source implement the LookupTableSource to support temporal table join for flink streaming task? [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] closed issue #2645: Shoud Flink Iceberg table source implement the LookupTableSource to support temporal table join for flink streaming task? URL: https://github.com/apache/iceberg/issues/2645 -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] flink cdc 2 iceberg0.11.1 [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] closed issue #2650: flink cdc 2 iceberg0.11.1 URL: https://github.com/apache/iceberg/issues/2650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [I] flink cdc 2 iceberg0.11.1 [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] commented on issue #2650: URL: https://github.com/apache/iceberg/issues/2650#issuecomment-2168958248 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Shoud Flink Iceberg table source implement the LookupTableSource to support temporal table join for flink streaming task? [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] commented on issue #2645: URL: https://github.com/apache/iceberg/issues/2645#issuecomment-2168958231 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] An error occurred when I started running Iceberg [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] commented on issue #2669: URL: https://github.com/apache/iceberg/issues/2669#issuecomment-2168958298 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: Can not see hive udf in Flink when use iceberg hive catalog [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] closed issue #2653: Flink: Can not see hive udf in Flink when use iceberg hive catalog URL: https://github.com/apache/iceberg/issues/2653 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] hive create table on existing iceberg table faild [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] closed issue #2673: hive create table on existing iceberg table faild URL: https://github.com/apache/iceberg/issues/2673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] hive create table on existing iceberg table faild [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] commented on issue #2673: URL: https://github.com/apache/iceberg/issues/2673#issuecomment-2168958305 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: Can not see hive udf in Flink when use iceberg hive catalog [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] commented on issue #2653: URL: https://github.com/apache/iceberg/issues/2653#issuecomment-2168958266 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Create a docker based demo environment for user testing and community onboarding [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] closed issue #2676: Create a docker based demo environment for user testing and community onboarding URL: https://github.com/apache/iceberg/issues/2676 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] Improve hidden partition support for Hive CREATE TABLE [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] closed issue #2681: Improve hidden partition support for Hive CREATE TABLE URL: https://github.com/apache/iceberg/issues/2681 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Improve hidden partition support for Hive CREATE TABLE [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] commented on issue #2681: URL: https://github.com/apache/iceberg/issues/2681#issuecomment-2168958326 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] An error occurred when I started running Iceberg [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] closed issue #2669: An error occurred when I started running Iceberg URL: https://github.com/apache/iceberg/issues/2669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] How to realize Write Iceberg Tables via Hive? (Ideas share) [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] commented on issue #2685: URL: https://github.com/apache/iceberg/issues/2685#issuecomment-2168958335 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Create a docker based demo environment for user testing and community onboarding [iceberg]

2024-06-14 Thread via GitHub
github-actions[bot] commented on issue #2676: URL: https://github.com/apache/iceberg/issues/2676#issuecomment-2168958311 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-06-14 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1640554789 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2210,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [I] SnapshotTableProcedure to migrate iceberg tables from one namespace to another [iceberg]

2024-06-14 Thread via GitHub
gauthamnair commented on issue #10262: URL: https://github.com/apache/iceberg/issues/10262#issuecomment-2169023746 We are experiencing the same problem: https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1718395195306979 -- This is an automated message from the Apache Git Service

Re: [I] Spark: Dropping partition column from old partition table corrupts entire table [iceberg]

2024-06-14 Thread via GitHub
lurnagao-dahua commented on issue #10234: URL: https://github.com/apache/iceberg/issues/10234#issuecomment-2169036597 > > Hi, may I ask the repair method is to traverse all schemas or the partition spec persist with type in metadata? > > According to #10352 , I assume it traverses all

Re: [I] Fails to insert into iceberg table after dropping a partition column that was used by older partition specs [iceberg]

2024-06-14 Thread via GitHub
lurnagao-dahua closed issue #7415: Fails to insert into iceberg table after dropping a partition column that was used by older partition specs URL: https://github.com/apache/iceberg/issues/7415 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-06-14 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1640748623 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2210,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [PR] Add missing arrow predicate pushdown implementations for `StartsWith`, `NotStartsWith`, `In`, and `NotIn` [iceberg-rust]

2024-06-14 Thread via GitHub
liurenjie1024 commented on code in PR #404: URL: https://github.com/apache/iceberg-rust/pull/404#discussion_r1640748380 ## crates/iceberg/src/arrow/reader.rs: ## @@ -741,42 +742,98 @@ impl<'a> BoundPredicateVisitor for PredicateConverter<'a> { fn starts_with( &m

Re: [I] field-id in avro schema is missing [iceberg-rust]

2024-06-14 Thread via GitHub
liurenjie1024 commented on issue #131: URL: https://github.com/apache/iceberg-rust/issues/131#issuecomment-2169124249 Thanks @ZENOTME for this pr: https://github.com/apache/avro/pull/2650 But I think we should still make this open since it has not been released yet? I think it will be incl

Re: [PR] Core: pushdown data_file.content when filter manifests in entries table [iceberg]

2024-06-14 Thread via GitHub
szehon-ho commented on code in PR #10203: URL: https://github.com/apache/iceberg/pull/10203#discussion_r1640361591 ## core/src/main/java/org/apache/iceberg/BaseEntriesTable.java: ## @@ -94,6 +105,188 @@ static CloseableIterable planFiles( table, manifest, projec

Re: [PR] Core: pushdown data_file.content when filter manifests in entries table [iceberg]

2024-06-14 Thread via GitHub
szehon-ho commented on code in PR #10203: URL: https://github.com/apache/iceberg/pull/10203#discussion_r1640779604 ## core/src/main/java/org/apache/iceberg/BaseEntriesTable.java: ## @@ -94,6 +105,188 @@ static CloseableIterable planFiles( table, manifest, projec

Re: [I] [DISCUSS] Update metadata serially [iceberg]

2024-06-14 Thread via GitHub
pavarotti17 commented on issue #732: URL: https://github.com/apache/iceberg/issues/732#issuecomment-2169147137 is the "commit.retry.num-retries" accumulated throughout all the jobs in a spark application, or its just for each job and count number will reset to 0 after job success? -- Thi

Re: [PR] Build: Update NOTICE to include copyright for 2024 [iceberg]

2024-06-14 Thread via GitHub
szehon-ho commented on PR #10471: URL: https://github.com/apache/iceberg/pull/10471#issuecomment-2169147378 While we are here, let's do the other ones, like azure-bundle NOTICE as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] field-id in avro schema is missing [iceberg-rust]

2024-06-14 Thread via GitHub
ZENOTME commented on issue #131: URL: https://github.com/apache/iceberg-rust/issues/131#issuecomment-2169153738 > Thanks @ZENOTME for this pr: [apache/avro#2650](https://github.com/apache/avro/pull/2650) But I think we should still make this open since it has not been released yet? I think