Re: [PR] Build/Release: Upgrade to RAT 0.16.1 [iceberg]

2024-01-29 Thread via GitHub
nastra merged PR #9579: URL: https://github.com/apache/iceberg/pull/9579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump org.assertj:assertj-core from 3.25.1 to 3.25.2 [iceberg]

2024-01-29 Thread via GitHub
nastra merged PR #9576: URL: https://github.com/apache/iceberg/pull/9576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.2.3 to 5.3.1 [iceberg]

2024-01-29 Thread via GitHub
nastra merged PR #9572: URL: https://github.com/apache/iceberg/pull/9572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spark: Support altering views [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1469223182 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -123,4 +132,17 @@ case class RewriteViewCommands(sp

Re: [PR] Small getting started guide on writes [iceberg-python]

2024-01-29 Thread via GitHub
Fokko commented on code in PR #311: URL: https://github.com/apache/iceberg-python/pull/311#discussion_r1469257727 ## mkdocs/docs/index.md: ## @@ -38,36 +38,129 @@ You can install the latest release version from pypi: pip install "pyiceberg[s3fs,hive]" ``` -Install it directl

Re: [I] Reading large data through Glue Catalog is SLOW [iceberg]

2024-01-29 Thread via GitHub
anechii closed issue #9559: Reading large data through Glue Catalog is SLOW URL: https://github.com/apache/iceberg/issues/9559 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
Fokko commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1469288671 ## pyiceberg/io/pyarrow.py: ## @@ -761,6 +761,32 @@ def primitive(self, primitive: pa.DataType) -> T: """Visit a primitive type.""" +class PreOrderPyArr

Re: [PR] Spark: Support altering views [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1469223182 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -123,4 +132,17 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Support altering views [iceberg]

2024-01-29 Thread via GitHub
nastra commented on PR #9510: URL: https://github.com/apache/iceberg/pull/9510#issuecomment-1914349245 @rdblue I have extracted setting/unsetting view properties into https://github.com/apache/iceberg/pull/9582 -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] gc.enabled property is set to false by default for Apache Iceberg table created in Nessie Catalog [iceberg]

2024-01-29 Thread via GitHub
nastra closed issue #9562: gc.enabled property is set to false by default for Apache Iceberg table created in Nessie Catalog URL: https://github.com/apache/iceberg/issues/9562 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on PR #9565: URL: https://github.com/apache/iceberg/pull/9565#issuecomment-1914388103 @ilyasahsan123 we typically apply such changes on the latest version first (1.18) and then backport them in a separate PR. Could you apply these changes to Flink 1.18 first please? -- T

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
ilyasahsan123 commented on PR #9565: URL: https://github.com/apache/iceberg/pull/9565#issuecomment-1914437348 Sure, I will do it. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Spark: Add support for describing/showing views [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9513: URL: https://github.com/apache/iceberg/pull/9513#discussion_r1469444718 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala: ## @@ -123,6 +132,47 @@ case class Extend

Re: [PR] Spark 3.3: Add RemoveDanglingDeletes action [iceberg]

2024-01-29 Thread via GitHub
zinking commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1469455133 ## api/src/main/java/org/apache/iceberg/DeleteFiles.java: ## @@ -55,6 +55,17 @@ default DeleteFiles deleteFile(DataFile file) { return this; } + /** + * De

Re: [PR] Spark: Add support for describing/showing views [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9513: URL: https://github.com/apache/iceberg/pull/9513#discussion_r1469477447 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -60,17 +65,27 @@ case class RewriteViewCommands(spa

Re: [PR] Spark: Add support for describing/showing views [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9513: URL: https://github.com/apache/iceberg/pull/9513#discussion_r1469484505 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowCreateV2ViewExec.scala: ## @@ -0,0 +1,79 @@ +/* + * Licensed to the Apach

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
ilyasahsan123 commented on PR #9565: URL: https://github.com/apache/iceberg/pull/9565#issuecomment-1914620937 hi @nastra , I've addressed your suggestion. Could you please take another look? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1469014546 ## tests/catalog/test_base.py: ## @@ -330,6 +333,34 @@ def test_create_table(catalog: InMemoryCatalog) -> None: assert catalog.load_table(TEST_TABLE_IDENTIFIER

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9565: URL: https://github.com/apache/iceberg/pull/9565#discussion_r1469628869 ## flink/v1.18/build.gradle: ## @@ -22,6 +22,10 @@ String scalaVersion = System.getProperty("scalaVersion") != null ? System.getPro project(":iceberg-flink:iceberg-f

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9565: URL: https://github.com/apache/iceberg/pull/9565#discussion_r1469634197 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestRowDataWrapper.java: ## @@ -49,12 +49,12 @@ public void testTime() { return;

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9565: URL: https://github.com/apache/iceberg/pull/9565#discussion_r1469635766 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestRowDataWrapper.java: ## @@ -75,8 +75,8 @@ protected void generateAndValidate(Schema schema, RecordWrappe

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9565: URL: https://github.com/apache/iceberg/pull/9565#discussion_r1469638343 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/data/TestStructRowData.java: ## @@ -22,7 +22,7 @@ import org.apache.iceberg.flink.DataGenerator; import org

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9565: URL: https://github.com/apache/iceberg/pull/9565#discussion_r1469640535 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/sink/TestAvroGenericRecordToRowDataMapper.java: ## @@ -18,10 +18,11 @@ */ package org.apache.iceberg.flink

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9565: URL: https://github.com/apache/iceberg/pull/9565#discussion_r1469642860 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/sink/TestAvroGenericRecordToRowDataMapper.java: ## @@ -18,10 +18,11 @@ */ package org.apache.iceberg.flink

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1469643574 ## pyiceberg/schema.py: ## @@ -1221,50 +1221,57 @@ def assign_fresh_schema_ids(schema_or_type: Union[Schema, IcebergType], next_id: class _SetFreshIDs(PreOrderSch

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9565: URL: https://github.com/apache/iceberg/pull/9565#discussion_r1469643916 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/sink/TestRowDataPartitionKey.java: ## @@ -91,10 +92,10 @@ public void testNullPartitionValue() { for (Ro

Re: [PR] Flink v1.16: Switch Flink v1.16 tests to Junit5 (Part 1) [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9565: URL: https://github.com/apache/iceberg/pull/9565#discussion_r1469646130 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/reader/TestArrayBatchRecords.java: ## @@ -50,19 +51,18 @@ private void testArray( fileOff

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1469674294 ## pyiceberg/schema.py: ## @@ -1221,50 +1221,57 @@ def assign_fresh_schema_ids(schema_or_type: Union[Schema, IcebergType], next_id: class _SetFreshIDs(PreOrderSch

Re: [PR] Spark: Support altering views [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1469684751 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckViews.scala: ## @@ -36,6 +38,9 @@ object CheckViews extends (LogicalPlan => Uni

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on PR #305: URL: https://github.com/apache/iceberg-python/pull/305#issuecomment-1914837022 Hi @Fokko - for some reason I don't see an option to respond to this specific Review comment, so I'm just replying here instead: > I would love to get this in with the 0.6.0 rel

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1469706761 ## pyiceberg/io/pyarrow.py: ## @@ -906,6 +932,21 @@ def after_map_value(self, element: pa.Field) -> None: self._field_names.pop() +class _ConvertToIceb

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1469710203 ## pyiceberg/io/pyarrow.py: ## @@ -906,6 +986,76 @@ def after_map_value(self, element: pa.Field) -> None: self._field_names.pop() +class _ConvertToIce

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1469711334 ## pyiceberg/io/pyarrow.py: ## @@ -761,6 +761,32 @@ def primitive(self, primitive: pa.DataType) -> T: """Visit a primitive type.""" +class PreOrderPyAr

Re: [PR] Spark: Support altering views [iceberg]

2024-01-29 Thread via GitHub
nastra commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1469766873 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -60,17 +63,23 @@ case class RewriteViewCommands(spa

[PR] docs: Add community page [iceberg-python]

2024-01-29 Thread via GitHub
Fokko opened a new pull request, #315: URL: https://github.com/apache/iceberg-python/pull/315 Analog to the general Iceberg page. I also think it is good to revive the PyIceberg sync. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] docs: Remove feature support page [iceberg-python]

2024-01-29 Thread via GitHub
Fokko opened a new pull request, #316: URL: https://github.com/apache/iceberg-python/pull/316 With the 0.6.0 release we'll check the last box, and we have feature parity with Java according to this table. I would prefer creating tickets on Github to point out the gaps rather than this page.

Re: [PR] Flink: Added error handling and default logic for Flink version detection [iceberg]

2024-01-29 Thread via GitHub
gjacoby126 commented on PR #9452: URL: https://github.com/apache/iceberg/pull/9452#issuecomment-1915203994 @nastra @pvary , just checking back in. I was wondering if there were any more changes you'd like to see to this patch, or if it's ready for commit? -- This is an automated message f

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-01-29 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1469960212 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -564,14 +569,26 @@ private static String sanitizeDate(int days, int today) { return "(

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-01-29 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1469960791 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -205,38 +208,74 @@ public String toString() { } public static class TimestampType extends Primitiv

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-01-29 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1469961074 ## api/src/test/java/org/apache/iceberg/types/TestConversions.java: ## @@ -93,15 +93,26 @@ public void testByteBufferConversions() { assertThat(Literal.of(1L).to

Re: [I] Add View Support to Trino [iceberg]

2024-01-29 Thread via GitHub
maxpoulain commented on issue #7939: URL: https://github.com/apache/iceberg/issues/7939#issuecomment-1915233640 Hi ! Is there any news on this subject ? Is this issue was moved to Trino project ? -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Spark 3.4: Fix writing of default values in CoW for rows with NULL columns which are unmatched [iceberg]

2024-01-29 Thread via GitHub
amogh-jahagirdar commented on PR #9556: URL: https://github.com/apache/iceberg/pull/9556#issuecomment-1915239004 > Just to be sure, can you set spark.sql.planChangeLog.level to info and execute the failing test to see what rule adds the default value behavior? It is still not clear how a wr

Re: [PR] Spark 3.4: Fix writing of default values in CoW for rows with NULL columns which are unmatched [iceberg]

2024-01-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #9556: URL: https://github.com/apache/iceberg/pull/9556#discussion_r1469971672 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala: ## @@ -214,6 +214,8 @@ object RewriteMergeInt

Re: [PR] Spark 3.4: Fix writing of default values in CoW for rows with NULL columns which are unmatched [iceberg]

2024-01-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #9556: URL: https://github.com/apache/iceberg/pull/9556#discussion_r1469971672 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala: ## @@ -214,6 +214,8 @@ object RewriteMergeInt

Re: [I] Add View Support to Trino [iceberg]

2024-01-29 Thread via GitHub
amogh-jahagirdar closed issue #7939: Add View Support to Trino URL: https://github.com/apache/iceberg/issues/7939 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [I] Add View Support to Trino [iceberg]

2024-01-29 Thread via GitHub
amogh-jahagirdar commented on issue #7939: URL: https://github.com/apache/iceberg/issues/7939#issuecomment-1915260354 Hey @maxpoulain yes, the tracking issue is in Trino https://github.com/trinodb/trino/issues/14120#issuecomment-1803568597 I also have a PR open there for View Support

Re: [PR] docs: Add community page [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #315: URL: https://github.com/apache/iceberg-python/pull/315#discussion_r1470043007 ## mkdocs/docs/community.md: ## @@ -0,0 +1,64 @@ +--- +hide: + - navigation +--- + + + +# Join the community + +Apache Iceberg tracks issues in GitHub and prefers

Re: [PR] docs: Add community page [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #315: URL: https://github.com/apache/iceberg-python/pull/315#discussion_r1470043007 ## mkdocs/docs/community.md: ## @@ -0,0 +1,64 @@ +--- +hide: + - navigation +--- + + + +# Join the community + +Apache Iceberg tracks issues in GitHub and prefers

Re: [PR] docs: Add community page [iceberg-python]

2024-01-29 Thread via GitHub
Fokko commented on code in PR #315: URL: https://github.com/apache/iceberg-python/pull/315#discussion_r1470046433 ## mkdocs/docs/community.md: ## @@ -0,0 +1,64 @@ +--- +hide: + - navigation +--- + + + +# Join the community + +Apache Iceberg tracks issues in GitHub and prefers t

Re: [PR] Small getting started guide on writes [iceberg-python]

2024-01-29 Thread via GitHub
HonahX commented on code in PR #311: URL: https://github.com/apache/iceberg-python/pull/311#discussion_r1470060936 ## mkdocs/docs/index.md: ## @@ -38,36 +38,129 @@ You can install the latest release version from pypi: pip install "pyiceberg[s3fs,hive]" ``` -Install it direct

Re: [PR] Fix writing to local filesystem [iceberg-python]

2024-01-29 Thread via GitHub
Fokko merged PR #301: URL: https://github.com/apache/iceberg-python/pull/301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] docs: Remove feature support page [iceberg-python]

2024-01-29 Thread via GitHub
Fokko merged PR #316: URL: https://github.com/apache/iceberg-python/pull/316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] docs: Remove feature support page [iceberg-python]

2024-01-29 Thread via GitHub
Fokko commented on PR #316: URL: https://github.com/apache/iceberg-python/pull/316#issuecomment-1915379284 Thanks everyone for the quick review 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
Fokko commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470106203 ## pyiceberg/schema.py: ## @@ -1221,50 +1221,57 @@ def assign_fresh_schema_ids(schema_or_type: Union[Schema, IcebergType], next_id: class _SetFreshIDs(PreOrderSche

Re: [PR] Spark 3.4: Support creating views via SQL [iceberg]

2024-01-29 Thread via GitHub
amogh-jahagirdar merged PR #9580: URL: https://github.com/apache/iceberg/pull/9580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470130736 ## pyiceberg/schema.py: ## @@ -1221,50 +1221,57 @@ def assign_fresh_schema_ids(schema_or_type: Union[Schema, IcebergType], next_id: class _SetFreshIDs(PreOrderSch

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470130736 ## pyiceberg/schema.py: ## @@ -1221,50 +1221,57 @@ def assign_fresh_schema_ids(schema_or_type: Union[Schema, IcebergType], next_id: class _SetFreshIDs(PreOrderSch

Re: [PR] InMemory Catalog Implementation [iceberg-python]

2024-01-29 Thread via GitHub
Fokko commented on code in PR #289: URL: https://github.com/apache/iceberg-python/pull/289#discussion_r1469989182 ## pyiceberg/catalog/__init__.py: ## @@ -137,12 +138,19 @@ def load_sql(name: str, conf: Properties) -> Catalog: raise NotInstalledError("SQLAlchemy support

[PR] Build: Bump pydantic from 2.5.3 to 2.6.0 [iceberg-python]

2024-01-29 Thread via GitHub
dependabot[bot] opened a new pull request, #317: URL: https://github.com/apache/iceberg-python/pull/317 Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.5.3 to 2.6.0. Release notes Sourced from https://github.com/pydantic/pydantic/releases";>pydantic's releases.

[PR] Build: Bump mkdocs-material from 9.5.5 to 9.5.6 [iceberg-python]

2024-01-29 Thread via GitHub
dependabot[bot] opened a new pull request, #318: URL: https://github.com/apache/iceberg-python/pull/318 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.5 to 9.5.6. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdo

[PR] Build: Bump pytest from 7.4.4 to 8.0.0 [iceberg-python]

2024-01-29 Thread via GitHub
dependabot[bot] opened a new pull request, #319: URL: https://github.com/apache/iceberg-python/pull/319 Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.4 to 8.0.0. Release notes Sourced from https://github.com/pytest-dev/pytest/releases";>pytest's releases. pyt

[PR] Build: Bump adlfs from 2023.12.0 to 2024.1.0 [iceberg-python]

2024-01-29 Thread via GitHub
dependabot[bot] opened a new pull request, #320: URL: https://github.com/apache/iceberg-python/pull/320 Bumps [adlfs](https://github.com/fsspec/adlfs) from 2023.12.0 to 2024.1.0. Release notes Sourced from https://github.com/fsspec/adlfs/releases";>adlfs's releases. 2024.1.0

[PR] Build: Bump moto from 4.2.13 to 5.0.0 [iceberg-python]

2024-01-29 Thread via GitHub
dependabot[bot] opened a new pull request, #321: URL: https://github.com/apache/iceberg-python/pull/321 Bumps [moto](https://github.com/getmoto/moto) from 4.2.13 to 5.0.0. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 5.0

Re: [PR] Flink: Adds the ability to read from a branch on the Flink Iceberg Source [iceberg]

2024-01-29 Thread via GitHub
stevenzwu commented on code in PR #9547: URL: https://github.com/apache/iceberg/pull/9547#discussion_r1470308941 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestStreamScanSql.java: ## @@ -97,7 +98,38 @@ public void clean() { super.clean(); } + /

Re: [PR] Spark 3.5: Use execution ID in executor cache [iceberg]

2024-01-29 Thread via GitHub
aokolnychyi commented on PR #9583: URL: https://github.com/apache/iceberg/pull/9583#issuecomment-1915715186 I am going to run this a number of times to make sure the new logic is not flaky and then will test it one more time on a cluster. -- This is an automated message from the Apache Gi

Re: [PR] [DO NOT MERGE] New docs switch [iceberg]

2024-01-29 Thread via GitHub
rdblue commented on PR #9520: URL: https://github.com/apache/iceberg/pull/9520#issuecomment-1915715262 @bitsondatadev, thanks! This looks more like what I was expecting! I'm +1 on this when the thread on the dev list concludes. -- This is an automated message from the Apache Git Service.

[PR] Build: Bump pypa/cibuildwheel from 2.16.3 to 2.16.4 [iceberg-python]

2024-01-29 Thread via GitHub
dependabot[bot] opened a new pull request, #322: URL: https://github.com/apache/iceberg-python/pull/322 Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.16.3 to 2.16.4. Release notes Sourced from https://github.com/pypa/cibuildwheel/releases";>pypa/cibuildwhee

[PR] Build: Bump aiohttp from 3.9.1 to 3.9.2 [iceberg-python]

2024-01-29 Thread via GitHub
dependabot[bot] opened a new pull request, #323: URL: https://github.com/apache/iceberg-python/pull/323 Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.1 to 3.9.2. Release notes Sourced from https://github.com/aio-libs/aiohttp/releases";>aiohttp's releases. 3.9

Re: [PR] Flink: Adds the ability to read from a branch on the Flink Iceberg Source [iceberg]

2024-01-29 Thread via GitHub
stevenzwu merged PR #9547: URL: https://github.com/apache/iceberg/pull/9547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
HonahX commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470463965 ## tests/catalog/test_base.py: ## @@ -330,6 +334,34 @@ def test_create_table(catalog: InMemoryCatalog) -> None: assert catalog.load_table(TEST_TABLE_IDENTIFIER

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470488890 ## tests/catalog/test_base.py: ## @@ -330,6 +334,34 @@ def test_create_table(catalog: InMemoryCatalog) -> None: assert catalog.load_table(TEST_TABLE_IDENTIFIER

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
HonahX commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470492242 ## tests/catalog/test_base.py: ## @@ -330,6 +334,34 @@ def test_create_table(catalog: InMemoryCatalog) -> None: assert catalog.load_table(TEST_TABLE_IDENTIFIER

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470494285 ## pyiceberg/schema.py: ## @@ -1221,50 +1221,57 @@ def assign_fresh_schema_ids(schema_or_type: Union[Schema, IcebergType], next_id: class _SetFreshIDs(PreOrderSch

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470494285 ## pyiceberg/schema.py: ## @@ -1221,50 +1221,57 @@ def assign_fresh_schema_ids(schema_or_type: Union[Schema, IcebergType], next_id: class _SetFreshIDs(PreOrderSch

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on PR #305: URL: https://github.com/apache/iceberg-python/pull/305#issuecomment-1915934234 Looks good to merge from my side. I left my closing comments on all the open discussions. Thank you @Fokko @kevinjqliu and @HonahX for your reviews! -- This is an automated message

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
HonahX commented on PR #305: URL: https://github.com/apache/iceberg-python/pull/305#issuecomment-1915934692 Thanks @syun64 for the great work! Thanks @Fokko and @kevinjqliu for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
HonahX merged PR #305: URL: https://github.com/apache/iceberg-python/pull/305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] `create_table` with a PyArrow Schema [iceberg-python]

2024-01-29 Thread via GitHub
syun64 commented on code in PR #305: URL: https://github.com/apache/iceberg-python/pull/305#discussion_r1470494285 ## pyiceberg/schema.py: ## @@ -1221,50 +1221,57 @@ def assign_fresh_schema_ids(schema_or_type: Union[Schema, IcebergType], next_id: class _SetFreshIDs(PreOrderSch

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-29 Thread via GitHub
zeroshade commented on PR #51: URL: https://github.com/apache/iceberg-go/pull/51#issuecomment-1915981210 @wolfeidau Can you add check marks to the appropriate spots in the README for the functionality you're adding? -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-01-29 Thread via GitHub
liurenjie1024 commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1470518241 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## Review Comment: I'm thinking moving this to another crate `iceberg-parquet`, the core crate

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-01-29 Thread via GitHub
BsoBird commented on PR #9546: URL: https://github.com/apache/iceberg/pull/9546#issuecomment-1915987434 @szehon-ho Hi. can you check this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Rest Catalog: Add RESTful AppendFiles data operation [iceberg]

2024-01-29 Thread via GitHub
danielcweeks commented on code in PR #9292: URL: https://github.com/apache/iceberg/pull/9292#discussion_r1470559173 ## core/src/main/java/org/apache/iceberg/MetadataUpdate.java: ## @@ -490,4 +491,21 @@ public void applyTo(ViewMetadata.Builder viewMetadataBuilder) { viewM

Re: [PR] Rest Catalog: Add RESTful AppendFiles data operation [iceberg]

2024-01-29 Thread via GitHub
danielcweeks commented on code in PR #9292: URL: https://github.com/apache/iceberg/pull/9292#discussion_r1470562788 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -112,6 +111,7 @@ public class RESTSessionCatalog extends BaseViewSessionCatalog pr

Re: [PR] Add UnboundSortOrder [iceberg-rust]

2024-01-29 Thread via GitHub
fqaiser94 commented on code in PR #115: URL: https://github.com/apache/iceberg-rust/pull/115#discussion_r1470584482 ## crates/iceberg/src/spec/sort.rs: ## @@ -88,15 +91,106 @@ impl SortOrder { pub fn is_unsorted(&self) -> bool { self.fields.is_empty() } + +

Re: [PR] Add UnboundSortOrder [iceberg-rust]

2024-01-29 Thread via GitHub
fqaiser94 commented on PR #115: URL: https://github.com/apache/iceberg-rust/pull/115#issuecomment-1916069215 Found some time to look at this again :D -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] InMemory Catalog Implementation [iceberg-python]

2024-01-29 Thread via GitHub
kevinjqliu commented on code in PR #289: URL: https://github.com/apache/iceberg-python/pull/289#discussion_r1470604775 ## pyiceberg/catalog/in_memory.py: ## @@ -0,0 +1,222 @@ +import uuid +from typing import ( +Dict, +List, +Optional, +Set, +Union, +) + +from

Re: [PR] Spark 3.5: Fix flaky TestSparkExecutorCache [iceberg]

2024-01-29 Thread via GitHub
aokolnychyi commented on PR #9583: URL: https://github.com/apache/iceberg/pull/9583#issuecomment-1916101647 Re-triggering one more time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Spark 3.5: Fix flaky TestSparkExecutorCache [iceberg]

2024-01-29 Thread via GitHub
aokolnychyi closed pull request #9583: Spark 3.5: Fix flaky TestSparkExecutorCache URL: https://github.com/apache/iceberg/pull/9583 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[PR] Spark 3.5: Fix flaky TestSparkExecutorCache [iceberg]

2024-01-29 Thread via GitHub
aokolnychyi opened a new pull request, #9583: URL: https://github.com/apache/iceberg/pull/9583 This PR adds code to destroy all currently live table broadcasts after initialization to fix #9511. Our tests have the following behavior: ``` Append Append Execute a row-lev

Re: [PR] InMemory Catalog Implementation [iceberg-python]

2024-01-29 Thread via GitHub
kevinjqliu commented on PR #289: URL: https://github.com/apache/iceberg-python/pull/289#issuecomment-1916109508 > Should we also add this catalog to the tests in tests/integration/test_reads.py? @Fokko I had the same idea! Unfortunately, the way integration tests are configured right

Re: [I] Cannot write to local filesystem [iceberg-python]

2024-01-29 Thread via GitHub
kevinjqliu closed issue #299: Cannot write to local filesystem URL: https://github.com/apache/iceberg-python/issues/299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-01-29 Thread via GitHub
Xuanwo commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1470623846 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## Review Comment: The files supported by iceberg are limited, which is not as extensive as catalogs

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-01-29 Thread via GitHub
Xuanwo commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1470624504 ## crates/iceberg/src/io.rs: ## @@ -268,6 +268,16 @@ impl OutputFile { .await?) } +/// Delete file. +pub async fn delete(&self) -> Result<

Re: [PR] docs: Add community page [iceberg-python]

2024-01-29 Thread via GitHub
Fokko merged PR #315: URL: https://github.com/apache/iceberg-python/pull/315 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-01-29 Thread via GitHub
Xuanwo commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1470626858 ## crates/iceberg/src/io.rs: ## @@ -278,7 +288,7 @@ impl OutputFile { } /// Creates output file for writing. -pub async fn writer(&self) -> Result { +

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-01-29 Thread via GitHub
Xuanwo commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1470626858 ## crates/iceberg/src/io.rs: ## @@ -278,7 +288,7 @@ impl OutputFile { } /// Creates output file for writing. -pub async fn writer(&self) -> Result { +

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-01-29 Thread via GitHub
Xuanwo commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1470626858 ## crates/iceberg/src/io.rs: ## @@ -278,7 +288,7 @@ impl OutputFile { } /// Creates output file for writing. -pub async fn writer(&self) -> Result { +

Re: [PR] Build: Bump aiohttp from 3.9.1 to 3.9.2 [iceberg-python]

2024-01-29 Thread via GitHub
Fokko merged PR #323: URL: https://github.com/apache/iceberg-python/pull/323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump adlfs from 2023.12.0 to 2024.1.0 [iceberg-python]

2024-01-29 Thread via GitHub
Fokko merged PR #320: URL: https://github.com/apache/iceberg-python/pull/320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump pydantic from 2.5.3 to 2.6.0 [iceberg-python]

2024-01-29 Thread via GitHub
Fokko merged PR #317: URL: https://github.com/apache/iceberg-python/pull/317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

  1   2   >