Re: [I] bug: The current snapshot with id 0 will be skip. [iceberg-python]

2023-12-20 Thread via GitHub
Fokko commented on issue #232: URL: https://github.com/apache/iceberg-python/issues/232#issuecomment-1865816252 That's indeed a bug, thanks for spotting this @ZENOTME. Are you interested in creating a PR to fix it? -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Core: Close the MetricsReporter when the Catalog is closed. [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9353: URL: https://github.com/apache/iceberg/pull/9353#discussion_r1433669118 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -482,7 +483,14 @@ public boolean removeProperties(Namespace namespace, Set properties) @Overr

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2023-12-20 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1433664382 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,288 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Empty implementation for pauseOrResumeSplits to prevent UnsupportedOperationException [iceberg]

2023-12-20 Thread via GitHub
pvary commented on PR #9308: URL: https://github.com/apache/iceberg/pull/9308#issuecomment-1865765813 Thanks @stevenzwu, @mas-chen, @nastra, @mxm and @rodmeneses for all the patience and the useful comments! -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Flink: Empty implementation for pauseOrResumeSplits to prevent UnsupportedOperationException [iceberg]

2023-12-20 Thread via GitHub
pvary merged PR #9308: URL: https://github.com/apache/iceberg/pull/9308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
pvary commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433654079 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final ConfigOpt

[PR] [1.4.x] Core: Fix missing delete files from transaction (#9354) [iceberg]

2023-12-20 Thread via GitHub
nastra opened a new pull request, #9356: URL: https://github.com/apache/iceberg/pull/9356 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Core: Fix missing delete files from transaction [iceberg]

2023-12-20 Thread via GitHub
nastra merged PR #9354: URL: https://github.com/apache/iceberg/pull/9354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Fix missing delete files from transaction [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9354: URL: https://github.com/apache/iceberg/pull/9354#discussion_r1433651142 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -892,15 +892,18 @@ private void cleanUncommittedAppends(Set committed) { } }

Re: [I] com/fasterxml/jackson/core/exc/StreamConstraintsException [iceberg]

2023-12-20 Thread via GitHub
nastra commented on issue #9355: URL: https://github.com/apache/iceberg/issues/9355#issuecomment-1865735371 @smileyboy2019 this seems to be similar to https://github.com/apache/iceberg/issues/9339. You're missing a dependency at runtime. Can you please post the dependencies you're using?

Re: [I] field-id in avro schema is miss now [iceberg-rust]

2023-12-20 Thread via GitHub
liurenjie1024 commented on issue #131: URL: https://github.com/apache/iceberg-rust/issues/131#issuecomment-1865721222 It would be better to link apache avro rust issue here so that we can track it. -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] Java calls API error [iceberg]

2023-12-20 Thread via GitHub
nastra closed issue #9339: Java calls API error URL: https://github.com/apache/iceberg/issues/9339 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issue

Re: [I] org.apache.flink.table.api.TableException: Required context of factory [iceberg]

2023-12-20 Thread via GitHub
windylcx commented on issue #2814: URL: https://github.com/apache/iceberg/issues/2814#issuecomment-1865708330 flink-1.16.3 the same issue org.apache.flink.table.client.gateway.SqlExecutionException: Could not execute SQL statement. at org.apache.flink.table.client.gateway.l

Re: [I] Required context of factory 'org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory' must not be null. [iceberg]

2023-12-20 Thread via GitHub
windylcx commented on issue #3922: URL: https://github.com/apache/iceberg/issues/3922#issuecomment-1865685012 > When i run bin/sql-client.sh , cause a exception : > > Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set. SLF4J: Class path contains multiple SLF4J bi

Re: [I] Support Page Skipping in Iceberg Parquet Reader [iceberg]

2023-12-20 Thread via GitHub
7mming7 commented on issue #193: URL: https://github.com/apache/iceberg/issues/193#issuecomment-1865683168 @shangxinli ,Doesn't it seem like there's no progress here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Deliver key metadata to parquet encryption [iceberg]

2023-12-20 Thread via GitHub
ggershinsky commented on code in PR #6762: URL: https://github.com/apache/iceberg/pull/6762#discussion_r1433602851 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -125,6 +127,16 @@ public static WriteBuilder write(OutputFile file) { return new WriteB

[I] com/fasterxml/jackson/core/exc/StreamConstraintsException [iceberg]

2023-12-20 Thread via GitHub
smileyboy2019 opened a new issue, #9355: URL: https://github.com/apache/iceberg/issues/9355 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 I used the iceberg 1.4.2 Java API to create a table, which can be created, but an

Re: [I] Remove deprecated import [iceberg-python]

2023-12-20 Thread via GitHub
Rohit-Pujari commented on issue #221: URL: https://github.com/apache/iceberg-python/issues/221#issuecomment-1865599015 sorry for the inconvenience , when i open the broadcast.py file on my machine the issue is already solved there is no need to modify it -- This is an automated messag

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on PR #9341: URL: https://github.com/apache/iceberg/pull/9341#issuecomment-1865561017 running CI check locally and resolving errors. I'll push once I've cleared the errors / need more help! -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Core: Fix missing delete files from transaction [iceberg]

2023-12-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #9354: URL: https://github.com/apache/iceberg/pull/9354#discussion_r1433548763 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -892,15 +892,18 @@ private void cleanUncommittedAppends(Set committed) { }

Re: [PR] Core: Fix missing delete files from transaction [iceberg]

2023-12-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #9354: URL: https://github.com/apache/iceberg/pull/9354#discussion_r1433548763 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -892,15 +892,18 @@ private void cleanUncommittedAppends(Set committed) { }

Re: [PR] Deliver key metadata to parquet encryption [iceberg]

2023-12-20 Thread via GitHub
ggershinsky commented on code in PR #6762: URL: https://github.com/apache/iceberg/pull/6762#discussion_r1433548058 ## data/src/main/java/org/apache/iceberg/data/BaseFileWriterFactory.java: ## @@ -118,7 +118,7 @@ public DataWriter newDataWriter( case PARQUET:

Re: [PR] Deliver key metadata to parquet encryption [iceberg]

2023-12-20 Thread via GitHub
ggershinsky commented on code in PR #6762: URL: https://github.com/apache/iceberg/pull/6762#discussion_r1433532466 ## data/src/main/java/org/apache/iceberg/data/GenericReader.java: ## @@ -126,6 +128,13 @@ private CloseableIterable openFile(FileScanTask task, Schema fileProject

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2023-12-20 Thread via GitHub
stevenzwu commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1433489506 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,288 @@ +/* + * Licensed to the Apache Software Founda

Re: [I] Spark by default loading/considering spark_catalog catalog instead of given catalog. [iceberg]

2023-12-20 Thread via GitHub
vimalKeshu closed issue #9326: Spark by default loading/considering spark_catalog catalog instead of given catalog. URL: https://github.com/apache/iceberg/issues/9326 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Apply Name mapping [iceberg-python]

2023-12-20 Thread via GitHub
HonahX commented on code in PR #219: URL: https://github.com/apache/iceberg-python/pull/219#discussion_r1433405001 ## pyiceberg/io/pyarrow.py: ## @@ -828,7 +933,9 @@ def _task_to_table( schema_raw = metadata.get(ICEBERG_SCHEMA) # TODO: if field_ids are not

Re: [I] Adding MergeInto into the Spark Scala API [iceberg]

2023-12-20 Thread via GitHub
huaxingao commented on issue #3665: URL: https://github.com/apache/iceberg/issues/3665#issuecomment-1865391716 Spark [`mergeInto`](https://github.com/apache/spark/pull/44119) API is merged in 4.0.0. Example usage: ``` spark.table("source") .mergeInto("target", $"source.id"

Re: [PR] Arrow: Set field-id with prefix [iceberg-python]

2023-12-20 Thread via GitHub
HonahX commented on code in PR #227: URL: https://github.com/apache/iceberg-python/pull/227#discussion_r1433377086 ## tests/io/test_pyarrow_visitor.py: ## @@ -209,9 +209,9 @@ def test_pyarrow_variable_binary_to_iceberg() -> None: def test_pyarrow_struct_to_iceberg() -> None:

Re: [PR] feat: Add website layout [iceberg-rust]

2023-12-20 Thread via GitHub
liurenjie1024 commented on code in PR #130: URL: https://github.com/apache/iceberg-rust/pull/130#discussion_r1433366580 ## website/book.toml: ## Review Comment: It would be better to have a `README.md` so that we can know how to generate the book in local environment for p

Re: [PR] Core: Fix missing delete files from transaction [iceberg]

2023-12-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #9354: URL: https://github.com/apache/iceberg/pull/9354#discussion_r1433360860 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -892,15 +892,18 @@ private void cleanUncommittedAppends(Set committed) { }

Re: [PR] Core: Fix missing delete files from transaction [iceberg]

2023-12-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #9354: URL: https://github.com/apache/iceberg/pull/9354#discussion_r1433360860 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -892,15 +892,18 @@ private void cleanUncommittedAppends(Set committed) { }

Re: [PR] Spark SystemFunctions are not pushed down during JOIN [iceberg]

2023-12-20 Thread via GitHub
wypoon commented on code in PR #9233: URL: https://github.com/apache/iceberg/pull/9233#discussion_r1433347625 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSPJWithBucketing.java: ## @@ -0,0 +1,219 @@ +/* + * Licensed to the Apache Software

Re: [PR] API: Fix day partition transform result type [iceberg]

2023-12-20 Thread via GitHub
tdcmeehan commented on code in PR #9345: URL: https://github.com/apache/iceberg/pull/9345#discussion_r1433343161 ## format/spec.md: ## @@ -318,7 +318,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to

Re: [PR] API: Fix day partition transform result type [iceberg]

2023-12-20 Thread via GitHub
tdcmeehan commented on code in PR #9345: URL: https://github.com/apache/iceberg/pull/9345#discussion_r1433343161 ## format/spec.md: ## @@ -318,7 +318,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to

Re: [I] Zorder not support timestampNTZType [iceberg]

2023-12-20 Thread via GitHub
JMin824 commented on issue #9348: URL: https://github.com/apache/iceberg/issues/9348#issuecomment-1865328187 I try to add timestampntz in SparkZOrderUDF.java, but in spark3.4 ntz cant not cast to bigint, anyone can solve it? -- This is an automated message from the Apache Git Service. To

Re: [I] Documentation improvements in regards to time travel [iceberg]

2023-12-20 Thread via GitHub
github-actions[bot] commented on issue #7000: URL: https://github.com/apache/iceberg/issues/7000#issuecomment-1865301197 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Hive ping functionality seems to leak threads [iceberg]

2023-12-20 Thread via GitHub
github-actions[bot] commented on issue #7034: URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1865301176 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark SystemFunctions are not pushed down during JOIN [iceberg]

2023-12-20 Thread via GitHub
wypoon commented on PR #9233: URL: https://github.com/apache/iceberg/pull/9233#issuecomment-1865291274 The main in my local clone is at https://github.com/apache/iceberg/commit/d6eba2a2b6ecf76dc11910d88febca6c488d. I applied the diff from this PR to my local main. The `TestSPJWithBuc

[PR] Build: Bump coverage from 7.3.3 to 7.3.4 [iceberg-python]

2023-12-20 Thread via GitHub
dependabot[bot] opened a new pull request, #231: URL: https://github.com/apache/iceberg-python/pull/231 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.3.3 to 7.3.4. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's chang

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
rodmeneses commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433249536 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Conf

[I] Rest Catalog Support for a Separate OAuth Server URI [iceberg-python]

2023-12-20 Thread via GitHub
syun64 opened a new issue, #230: URL: https://github.com/apache/iceberg-python/issues/230 ### Feature Request / Improvement This follows the implementation of https://github.com/apache/iceberg/issues/8869 on the parent Iceberg repository. Open PR in parent repository currently

[I] Schema IDs Re-Order? [iceberg-python]

2023-12-20 Thread via GitHub
sebpretzer opened a new issue, #229: URL: https://github.com/apache/iceberg-python/issues/229 ### Apache Iceberg version main (development) ### Please describe the bug 🐞 When publishing a schema via `create_table`, the `id`s sometimes re-order. I haven't looked into all

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
mas-chen commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433194420 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Config

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
mas-chen commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433194420 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Config

Re: [PR] API: Fix day partition transform result type [iceberg]

2023-12-20 Thread via GitHub
Fokko commented on code in PR #9345: URL: https://github.com/apache/iceberg/pull/9345#discussion_r1433175326 ## format/spec.md: ## @@ -318,7 +318,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to widt

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
rodmeneses commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433159641 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Conf

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
rodmeneses commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433159641 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Conf

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
rodmeneses commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433159641 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Conf

Re: [PR] API: Fix day partition transform result type [iceberg]

2023-12-20 Thread via GitHub
tdcmeehan commented on PR #9345: URL: https://github.com/apache/iceberg/pull/9345#issuecomment-1865080559 CC: @nastra @szehon-ho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
mas-chen commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433139429 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Config

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
rodmeneses commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433134831 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Conf

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
pvary commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433119614 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceBoundedSql.java: ## @@ -77,4 +100,92 @@ protected List run( String optionStr = Sq

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
mas-chen commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433118149 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkConfigOptions.java: ## @@ -94,7 +94,7 @@ private FlinkConfigOptions() {} public static final Config

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
pvary commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433114878 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceBoundedSql.java: ## @@ -77,4 +100,92 @@ protected List run( String optionStr = Sq

Re: [PR] Arrow: Set field-id with prefix [iceberg-python]

2023-12-20 Thread via GitHub
Fokko commented on code in PR #227: URL: https://github.com/apache/iceberg-python/pull/227#discussion_r1433111604 ## pyiceberg/io/pyarrow.py: ## @@ -720,25 +719,17 @@ def primitive(self, primitive: pa.DataType) -> Optional[T]: def _get_field_id(field: pa.Field) -> Optional

Re: [PR] Arrow: Set field-id with prefix [iceberg-python]

2023-12-20 Thread via GitHub
Fokko commented on code in PR #227: URL: https://github.com/apache/iceberg-python/pull/227#discussion_r1433110576 ## pyiceberg/io/pyarrow.py: ## @@ -720,25 +719,17 @@ def primitive(self, primitive: pa.DataType) -> Optional[T]: def _get_field_id(field: pa.Field) -> Optional

Re: [PR] Arrow: Set field-id with prefix [iceberg-python]

2023-12-20 Thread via GitHub
Fokko commented on code in PR #227: URL: https://github.com/apache/iceberg-python/pull/227#discussion_r1433110149 ## tests/io/test_pyarrow_visitor.py: ## @@ -209,9 +209,9 @@ def test_pyarrow_variable_binary_to_iceberg() -> None: def test_pyarrow_struct_to_iceberg() -> None:

Re: [PR] Arrow: Set field-id with prefix [iceberg-python]

2023-12-20 Thread via GitHub
Fokko commented on code in PR #227: URL: https://github.com/apache/iceberg-python/pull/227#discussion_r1433109602 ## pyiceberg/io/pyarrow.py: ## @@ -154,10 +154,9 @@ ONE_MEGABYTE = 1024 * 1024 BUFFER_SIZE = "buffer-size" ICEBERG_SCHEMA = b"iceberg.schema" -FIELD_ID = "field_i

Re: [PR] Flink: Watermark read options [iceberg]

2023-12-20 Thread via GitHub
pvary commented on code in PR #9346: URL: https://github.com/apache/iceberg/pull/9346#discussion_r1433108950 ## docs/flink-configuration.md: ## @@ -133,7 +133,8 @@ env.getConfig() | max-planning-snapshot-count | connector.iceberg.max-planning-snapshot-count | N/A

Re: [PR] Build: Bump cython from 3.0.6 to 3.0.7 [iceberg-python]

2023-12-20 Thread via GitHub
Fokko merged PR #228: URL: https://github.com/apache/iceberg-python/pull/228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Remove deprecated import [iceberg-python]

2023-12-20 Thread via GitHub
Fokko commented on issue #221: URL: https://github.com/apache/iceberg-python/issues/221#issuecomment-1864980887 @Rohit-Pujari feel free to raise a PR 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2023-12-20 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1433058848 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestMapRangePartitioner.java: ## @@ -0,0 +1,511 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2023-12-20 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1433057435 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestMapRangePartitioner.java: ## @@ -0,0 +1,511 @@ +/* + * Licensed to the Apache Software Founda

Re: [I] Remove deprecated import [iceberg-python]

2023-12-20 Thread via GitHub
Rohit-Pujari commented on issue #221: URL: https://github.com/apache/iceberg-python/issues/221#issuecomment-1864945986 Hi iam Rohit, I am a beginner to open source contribution I could try to solve the above issue -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2023-12-20 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1433038190 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,288 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2023-12-20 Thread via GitHub
stevenzwu commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1433025320 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,288 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1433000616 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -64,11 +66,14 @@ public void testFullProjection() throws Exception

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on code in PR #9341: URL: https://github.com/apache/iceberg/pull/9341#discussion_r1432997327 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkOrcReadMetadataColumns.java: ## @@ -100,7 +99,7 @@ public static Object[] parameters() {

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9341: URL: https://github.com/apache/iceberg/pull/9341#discussion_r1432986527 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkOrcReadMetadataColumns.java: ## @@ -100,7 +99,7 @@ public static Object[] parameters() { re

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9341: URL: https://github.com/apache/iceberg/pull/9341#discussion_r1432986527 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkOrcReadMetadataColumns.java: ## @@ -100,7 +99,7 @@ public static Object[] parameters() { re

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9341: URL: https://github.com/apache/iceberg/pull/9341#discussion_r1432985133 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkOrcReadMetadataColumns.java: ## @@ -100,7 +99,7 @@ public static Object[] parameters() { re

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

2023-12-20 Thread via GitHub
amogh-jahagirdar commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1864845786 I've been discussing with @sabyasachinandy @harshith-bolar-rapido offline, I can reproduce the issue from the minimal reproduction steps here https://github.com/apache/icebe

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432977949 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432971413 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -64,11 +66,14 @@ public void testFullProjection() throws Exception {

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2023-12-20 Thread via GitHub
stevenzwu commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1432960039 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestMapRangePartitioner.java: ## @@ -0,0 +1,511 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432954411 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -64,11 +66,14 @@ public void testFullProjection() throws Exception

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432954411 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -64,11 +66,14 @@ public void testFullProjection() throws Exception

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on PR #9341: URL: https://github.com/apache/iceberg/pull/9341#issuecomment-1864806108 When I run the build check (`./gradlew -DallVersions build -x test -x javadoc -x integrationTest`) locally, build succeeds. BUILD SUCCESSFUL in 5m 29s 496 actionable tas

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432947294 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -64,11 +66,14 @@ public void testFullProjection() throws Exception {

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432931800 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -64,11 +66,14 @@ public void testFullProjection() throws Exception

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432924610 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala: ## @@ -0,0 +1,156 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432919611 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -64,11 +66,14 @@ public void testFullProjection() throws Exception {

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432856893 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -49,7 +50,7 @@ public abstract class TestReadProjection { prote

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432876466 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on PR #9341: URL: https://github.com/apache/iceberg/pull/9341#issuecomment-1864702341 CI failing. Error: ``` > Task :iceberg-spark:iceberg-spark-3.5_2.12:jar /home/runner/work/iceberg/iceberg/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/Te

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432875557 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432856893 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -49,7 +50,7 @@ public abstract class TestReadProjection { prote

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432847330 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/logicalOperators.scala: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Sof

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432844969 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/logicalOperators.scala: ## @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache Sof

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432844267 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432837104 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark: Add support for reading Iceberg views [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9340: URL: https://github.com/apache/iceberg/pull/9340#discussion_r1432837381 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spark Migration to JUnit5 AssertJ: non-parameterized, spark/source directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9342: URL: https://github.com/apache/iceberg/pull/9342#discussion_r1432834544 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestReadProjection.java: ## @@ -49,7 +50,7 @@ public abstract class TestReadProjection { protected a

Re: [PR] Core: Close the MetricsReporter when the Catalog is closed. [iceberg]

2023-12-20 Thread via GitHub
huyuanfeng2018 commented on code in PR #9353: URL: https://github.com/apache/iceberg/pull/9353#discussion_r1432824971 ## .palantir/revapi.yml: ## @@ -886,6 +886,13 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.Ser

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
chinmay-bhat commented on PR #9341: URL: https://github.com/apache/iceberg/pull/9341#issuecomment-1864563722 Changes: 1. update assertJ imports 2. remove cast - int, byte[] 3. update tempDir access - solves build errors 4. I had erroneously included a parameterized file in this PR

Re: [PR] Core: Close the MetricsReporter when the Catalog is closed. [iceberg]

2023-12-20 Thread via GitHub
huyuanfeng2018 commented on code in PR #9353: URL: https://github.com/apache/iceberg/pull/9353#discussion_r1432764950 ## .palantir/revapi.yml: ## @@ -886,6 +886,13 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.Ser

Re: [PR] Core: Close the MetricsReporter when the Catalog is closed. [iceberg]

2023-12-20 Thread via GitHub
huyuanfeng2018 commented on code in PR #9353: URL: https://github.com/apache/iceberg/pull/9353#discussion_r1432764950 ## .palantir/revapi.yml: ## @@ -886,6 +886,13 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.Ser

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9341: URL: https://github.com/apache/iceberg/pull/9341#discussion_r1432758094 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/GenericsHelpers.java: ## @@ -116,90 +116,85 @@ private static void assertEqualsSafe(Type type, Object ex

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9341: URL: https://github.com/apache/iceberg/pull/9341#discussion_r1432757273 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/GenericsHelpers.java: ## @@ -264,86 +259,83 @@ private static void assertEqualsUnsafe(Type type, Object

Re: [PR] Spark Migration to JUnit5 AssertJ - non-parameterized, spark/data directory [iceberg]

2023-12-20 Thread via GitHub
nastra commented on code in PR #9341: URL: https://github.com/apache/iceberg/pull/9341#discussion_r1432755921 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/TestHelpers.java: ## @@ -304,86 +294,77 @@ private static void assertEqualsUnsafe(Type type, Object expe

  1   2   >