Re: [PR] mr：Fix issues 10639 [iceberg]

zhangbutao commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1669851520 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -144,21 +147,32 @@ public List getSplits(JobContext context) { InputFormatC

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669847482 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -581,13 +544,36 @@ public void testFilterWithDateAndTimestamp() throws IOException {

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669847090 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -521,30 +508,7 @@ public void testAsOfTimeOlderThanFirstSnapshot() { "Cannot find

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669845396 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -401,15 +392,15 @@ public void testProjectWithMissingFilterColumn() { } results.for

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669845095 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -378,11 +369,11 @@ public void testProjectWithSchema() { GenericRecord.create(schema)

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669844508 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -326,15 +317,15 @@ private void verifyProjectIdColumn(Iterable results) { Lists.trans

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669842925 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -225,12 +221,11 @@ public void createTables() throws IOException { sharedTable.newAppend(

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669842708 ## data/src/test/java/org/apache/iceberg/data/TestLocalScan.java: ## @@ -195,10 +192,9 @@ private void appendData() throws IOException { sharedTable.newFastAppen

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669838611 ## data/src/test/java/org/apache/iceberg/TestSplitScan.java: ## @@ -53,38 +51,31 @@ public class TestSplitScan { private Table table; private File tableLocatio

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

nastra commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1669835108 ## build.gradle: ## @@ -81,6 +81,7 @@ if (JavaVersion.current() == JavaVersion.VERSION_1_8) { "--add-opens", "java.base/jdk.internal.re

Re: [PR] Support create multiple element ns together for nessie [iceberg]

ajantha-bhat commented on PR #10630: URL: https://github.com/apache/iceberg/pull/10630#issuecomment-2216704059 ACK and agree that it is a grey area in the spec. Maybe good to discuss in iceberg mailing list about standardising it and updating the spec. -- This is an automated message from

Re: [I] Encryption via RewriteFiles [iceberg]

ggershinsky commented on issue #3928: URL: https://github.com/apache/iceberg/issues/3928#issuecomment-2216693281 Thanks @anuragmantri , SGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] mr：Fix issues 10639 [iceberg]

lurnagao-dahua commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1669791684 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -144,21 +147,32 @@ public List getSplits(JobContext context) { InputFor

Re: [PR] Disallow writing empty Manifest files [iceberg-python]

Fokko merged PR #876: URL: https://github.com/apache/iceberg-python/pull/876 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] mr：Fix issues 10639 [iceberg]

lurnagao-dahua commented on PR #10661: URL: https://github.com/apache/iceberg/pull/10661#issuecomment-2216683973 > Could we have a summary in the description why the change is needed? Could we have some tests verifying th fix? Hi, May I ask if test cases are specifically used to verif

Re: [PR] mr：Fix issues 10639 [iceberg]

lurnagao-dahua commented on PR #10661: URL: https://github.com/apache/iceberg/pull/10661#issuecomment-2216677283 > Could we have a summary in the description why the change is needed? Could we have some tests verifying th fix? Thank you for your reply! I will try to write test case

Re: [PR] mr：Fix issues 10639 [iceberg]

lurnagao-dahua commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r1669791684 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -144,21 +147,32 @@ public List getSplits(JobContext context) { InputFor

Re: [PR] feat: add `RollingManifestWriter` [iceberg-python]

felixscherz commented on PR #650: URL: https://github.com/apache/iceberg-python/pull/650#issuecomment-2216648246 @Fokko Thanks for taking a look! Sorry about the formatting, should be fixed now:) -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] mr：Fix issues 10639 [iceberg]

pvary commented on PR #10661: URL: https://github.com/apache/iceberg/pull/10661#issuecomment-2216641499 Could we have some description why the change is needed? Could we have some tests verifying th fix? -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] mr：Fix issues 10639 [iceberg]

pvary commented on code in PR #10661: URL: https://github.com/apache/iceberg/pull/10661#discussion_r166926 ## mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java: ## @@ -144,21 +147,32 @@ public List getSplits(JobContext context) { InputFormatConfig

Re: [I] Spark: read migrate table error [iceberg]

tanvn commented on issue #2896: URL: https://github.com/apache/iceberg/issues/2896#issuecomment-2216609900 @hameizi May I know your Spark and Iceberg version on which this issue has happened? I can not reproduce this issue with Spark 3.4 and Iceberg 1.3.1. -- This is an automated mes

Re: [PR] Add interfaces for Action RemoveExpiredFiles [iceberg]

ajantha-bhat commented on code in PR #10643: URL: https://github.com/apache/iceberg/pull/10643#discussion_r1669739401 ## api/src/main/java/org/apache/iceberg/actions/ActionsProvider.java: ## @@ -70,4 +70,10 @@ default RewritePositionDeleteFiles rewritePositionDeletes(Table tabl

Re: [PR] Bump fastavro from 1.9.4 to 1.9.5 [iceberg-python]

HonahX merged PR #904: URL: https://github.com/apache/iceberg-python/pull/904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Bump moto from 5.0.10 to 5.0.11 [iceberg-python]

HonahX merged PR #903: URL: https://github.com/apache/iceberg-python/pull/903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Support partial deletes [iceberg-python]

HonahX commented on code in PR #569: URL: https://github.com/apache/iceberg-python/pull/569#discussion_r1669726783 ## pyiceberg/table/__init__.py: ## @@ -539,6 +578,81 @@ def overwrite( for data_file in data_files: update_snapshot.append_dat

Re: [PR] mr：Fix issues 10639 [iceberg]

lurnagao-dahua commented on PR #10661: URL: https://github.com/apache/iceberg/pull/10661#issuecomment-2216509309 Hi, May I ask if you can help me check this issue? @pvary @deniskuzZ @nastra i would be very grateful if you have any response! -- This is an automated message from the A

Re: [I] UncheckedSQLException while deleting table [iceberg-python]

ArijitSinghEDA commented on issue #901: URL: https://github.com/apache/iceberg-python/issues/901#issuecomment-2216496437 Hi @kevinjqliu On further investigation, it was a problem with the standalone iceberg server. It has been rectified now. Thank you for taking out your time.

Re: [I] UncheckedSQLException while deleting table [iceberg-python]

ArijitSinghEDA closed issue #901: UncheckedSQLException while deleting table URL: https://github.com/apache/iceberg-python/issues/901 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Implement nan_value_counts && distinct_counts metrics in parquet writer [iceberg-rust]

Xuanwo commented on issue #417: URL: https://github.com/apache/iceberg-rust/issues/417#issuecomment-2216489604 > I can take this up @liurenjie1024 Welcome! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

rahil-c commented on PR #7914: URL: https://github.com/apache/iceberg/pull/7914#issuecomment-2216488472 > The basic issue is we want to make sure we have a test which uses both the supportsPrefix enabled FS and one where it is not enabled to we are sure that both implementations remain corr

Re: [I] Implement nan_value_counts && distinct_counts metrics in parquet writer [iceberg-rust]

liurenjie1024 commented on issue #417: URL: https://github.com/apache/iceberg-rust/issues/417#issuecomment-2216474646 > I can take this up @liurenjie1024 Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Support convert orc timestamptz [iceberg]

tanvn commented on PR #9905: URL: https://github.com/apache/iceberg/pull/9905#issuecomment-2216473669 @snazy @nastra @rdblue I confirmed that this issue is happening on my env (Spark 3.4, Iceberg 1.3.1) as well and this is blocking my team from migrating our Hive tables to Iceberg.

Re: [I] Implement nan_value_counts && distinct_counts metrics in parquet writer [iceberg-rust]

vaibhawvipul commented on issue #417: URL: https://github.com/apache/iceberg-rust/issues/417#issuecomment-2216426309 I can take this up @liurenjie1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Deprecate `oauth/tokens` endpoint [iceberg]

danielcweeks commented on code in PR #10603: URL: https://github.com/apache/iceberg/pull/10603#discussion_r1669603688 ## open-api/rest-catalog-open-api.yaml: ## @@ -134,9 +134,17 @@ paths: post: tags: - OAuth2 API - summary: Get a token using an OAuth2

Re: [PR] reuse docker container to save compute resources [iceberg-rust]

thexiay commented on PR #428: URL: https://github.com/apache/iceberg-rust/pull/428#issuecomment-2216305068 cc @liurenjie1024 all comment is resolved -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] feat: Add cargo machete [iceberg-rust]

liurenjie1024 merged PR #448: URL: https://github.com/apache/iceberg-rust/pull/448 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Add `cargo machete` to check unused dependencies. [iceberg-rust]

liurenjie1024 closed issue #439: Add `cargo machete` to check unused dependencies. URL: https://github.com/apache/iceberg-rust/issues/439 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] feat: Add cargo machete [iceberg-rust]

Xuanwo commented on PR #448: URL: https://github.com/apache/iceberg-rust/pull/448#issuecomment-2216198479 > Cool, let wait for a moment. Apologies, I missed some discussion about using `cargo install` directly. I'm fine with merging this PR first and switching to GitHub Action version

Re: [PR] Deprecate `oauth/tokens` endpoint [iceberg]

jackye1995 commented on code in PR #10603: URL: https://github.com/apache/iceberg/pull/10603#discussion_r1669533209 ## open-api/rest-catalog-open-api.yaml: ## @@ -134,9 +134,17 @@ paths: post: tags: - OAuth2 API - summary: Get a token using an OAuth2 fl

Re: [PR] feat: Add cargo machete [iceberg-rust]

liurenjie1024 commented on PR #448: URL: https://github.com/apache/iceberg-rust/pull/448#issuecomment-2216135691 > > due to ASF infra policy, this third-party github action is not allowed. I've left some suggestion to improve. > > I'm going to submit a ticket to the ASF infra. We can

Re: [PR] Support create multiple element ns together for nessie [iceberg]

zymap commented on PR #10630: URL: https://github.com/apache/iceberg/pull/10630#issuecomment-2216082740 > Considering the potential access control tie up around namespaces, I wouldn't recommend auto creating missing namespaces. The current user may not have permission to drop parent namespa

Re: [PR] PyArrow: Don't enforce the schema [iceberg-python]

syun64 commented on code in PR #902: URL: https://github.com/apache/iceberg-python/pull/902#discussion_r1669524329 ## pyiceberg/io/pyarrow.py: ## @@ -1268,14 +1265,8 @@ def __init__(self, file_schema: Schema): def _cast_if_needed(self, field: NestedField, values: pa.Array

Re: [PR] PyArrow: Don't enforce the schema [iceberg-python]

syun64 commented on code in PR #902: URL: https://github.com/apache/iceberg-python/pull/902#discussion_r1669486866 ## pyiceberg/table/__init__.py: ## @@ -1884,8 +1884,9 @@ def to_arrow_batch_reader(self) -> pa.RecordBatchReader: from pyiceberg.io.pyarrow import projec

Re: [I] ugi not correct in WORKER_POOL [iceberg]

zhangbutao commented on issue #10639: URL: https://github.com/apache/iceberg/issues/10639#issuecomment-2216011499 > @lurnagao-dahua, is this a thread pool in question: [apache/hive@45867be](https://github.com/apache/hive/commit/45867be6cb5308566e4cf16c7b4cf8081085b58c)? cc @zhangbutao >

[I] column type change failed : timestamp without timezone change to timestamp with timezone [iceberg]

lichaohao opened a new issue, #10660: URL: https://github.com/apache/iceberg/issues/10660 ### Query engine iceberg version: 1.3.0 ### Question iceberg api why not support column type change from timestamp withont timezone to timestamp with timezone hello , @Fo

Re: [I] ugi not correct in WORKER_POOL [iceberg]

lurnagao-dahua commented on issue #10639: URL: https://github.com/apache/iceberg/issues/10639#issuecomment-2215854206 > @lurnagao-dahua, what version of Hive are you using? Thank you for your reply！ hive 3.1.3 and I added more information in the description now. -- This is an aut

Re: [PR] Allow writing dataframes that are either a subset of table schema or in arbitrary order [iceberg-python]

kevinjqliu commented on PR #829: URL: https://github.com/apache/iceberg-python/pull/829#issuecomment-2215827324 > First of all, sorry for the late reply. Feel free to ping me more aggressively. No worries at all, I forgot to ping about this PR > How about re-aligning the table

Re: [I] Incremental Changelog Scan [iceberg-python]

github-actions[bot] closed issue #240: Incremental Changelog Scan URL: https://github.com/apache/iceberg-python/issues/240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] Incremental Changelog Scan [iceberg-python]

github-actions[bot] commented on issue #240: URL: https://github.com/apache/iceberg-python/issues/240#issuecomment-2215667242 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

[PR] IO Implementation using Go CDK [iceberg-go]

srilman opened a new pull request, #111: URL: https://github.com/apache/iceberg-go/pull/111 Partially implements #92. The Go CDK has well-maintained implementations for accessing objects stores from S3, Azure, and GCS via a `io/fs.Fs`-like interface. However, their file interface doesn't su

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669424764 ## api/src/test/java/org/apache/iceberg/transforms/TestYears.java: ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669424619 ## api/src/test/java/org/apache/iceberg/transforms/TestYears.java: ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669412029 ## api/src/test/java/org/apache/iceberg/types/TestTypes.java: ## @@ -31,6 +31,8 @@ public void fromPrimitiveString() { assertThat(Types.fromPrimitiveString("B

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669410065 ## api/src/test/java/org/apache/iceberg/transforms/TestTimestamps.java: ## @@ -241,4 +243,54 @@ public void testTimestampsReturnType() { Type hourResultType =

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669416443 ## api/src/test/java/org/apache/iceberg/expressions/TestStringLiteralConversions.java: ## @@ -181,17 +226,30 @@ public void testNegativeStringToTimestampLiteral() {

[PR] Bump fastavro from 1.9.4 to 1.9.5 [iceberg-python]

dependabot[bot] opened a new pull request, #904: URL: https://github.com/apache/iceberg-python/pull/904 Bumps [fastavro](https://github.com/fastavro/fastavro) from 1.9.4 to 1.9.5. Changelog Sourced from https://github.com/fastavro/fastavro/blob/master/ChangeLog";>fastavro's changel

[PR] Bump moto from 5.0.10 to 5.0.11 [iceberg-python]

dependabot[bot] opened a new pull request, #903: URL: https://github.com/apache/iceberg-python/pull/903 Bumps [moto](https://github.com/getmoto/moto) from 5.0.10 to 5.0.11. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 5.

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669413492 ## api/src/test/java/org/apache/iceberg/expressions/TestStringLiteralConversions.java: ## @@ -181,17 +226,30 @@ public void testNegativeStringToTimestampLiteral() {

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669412029 ## api/src/test/java/org/apache/iceberg/types/TestTypes.java: ## @@ -31,6 +31,8 @@ public void fromPrimitiveString() { assertThat(Types.fromPrimitiveString("B

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669410065 ## api/src/test/java/org/apache/iceberg/transforms/TestTimestamps.java: ## @@ -241,4 +243,54 @@ public void testTimestampsReturnType() { Type hourResultType =

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

jacobmarble commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2215448506 > Given that this type is for v3, wouldn't a v2 writer just blindly write this new type if it's being used anywhere and thus break forward compability? @nastra I don't think I u

Re: [PR] Spark Action to Analyze table [iceberg]

karuppayya commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1669391969 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Spark 3.3/3.4: support read of partition metadata column when table is over 1k [iceberg]

dramaticlly commented on PR #10641: URL: https://github.com/apache/iceberg/pull/10641#issuecomment-2215083572 @szehon-ho or @RussellSpitzer if you want to take a look at this back port -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Support Spark Column Stats [iceberg]

huaxingao commented on PR #10659: URL: https://github.com/apache/iceberg/pull/10659#issuecomment-2215051692 cc @szehon-ho @karuppayya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] feat: add `RollingManifestWriter` [iceberg-python]

Fokko commented on code in PR #650: URL: https://github.com/apache/iceberg-python/pull/650#discussion_r1669196117 ## pyiceberg/manifest.py: ## @@ -770,6 +818,81 @@ def add_entry(self, entry: ManifestEntry) -> ManifestWriter: self._writer.write_block([self.prepare_entry

Re: [PR] feat: add `RollingManifestWriter` [iceberg-python]

Fokko commented on code in PR #650: URL: https://github.com/apache/iceberg-python/pull/650#discussion_r1669195441 ## pyiceberg/manifest.py: ## @@ -770,6 +818,81 @@ def add_entry(self, entry: ManifestEntry) -> ManifestWriter: self._writer.write_block([self.prepare_entry

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669176686 ## api/src/test/java/org/apache/iceberg/transforms/TestBucketing.java: ## @@ -112,12 +112,59 @@ public void testSpecValues() { .as("Spec example: hash(2017-11-16

Re: [PR] Rename data_sequence_number to sequence_number in ManifestEntry [iceberg-python]

soumya-ghosh commented on PR #900: URL: https://github.com/apache/iceberg-python/pull/900#issuecomment-2214953159 @kevinjqliu missed to rename a variable, hence CI checks failed in mypy stage, it is fixed now. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Allow writing dataframes that are either a subset of table schema or in arbitrary order [iceberg-python]

Fokko commented on code in PR #829: URL: https://github.com/apache/iceberg-python/pull/829#discussion_r1669089423 ## pyiceberg/table/__init__.py: ## @@ -158,9 +158,11 @@ def _check_schema_compatible(table_schema: Schema, other_schema: "pa.Schema") -> None: """ -Chec

[PR] Support Spark Column Stats [iceberg]

huaxingao opened a new pull request, #10659: URL: https://github.com/apache/iceberg/pull/10659 Co-authored-by: Huaxin Gao Co-authored-by: Karuppayya Rajendran This PR adds the column stats support, so Iceberg can report column stats to Spark engine for CBO. -- This is an automat

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1669075324 ## api/src/test/java/org/apache/iceberg/expressions/TestStringLiteralConversions.java: ## @@ -132,6 +132,51 @@ public void testStringToTimestampLiteral() { .isEq

Re: [PR] Rename data_sequence_number to sequence_number in ManifestEntry [iceberg-python]

soumya-ghosh commented on code in PR #900: URL: https://github.com/apache/iceberg-python/pull/900#discussion_r1669074287 ## pyiceberg/manifest.py: ## @@ -762,10 +762,10 @@ def add_entry(self, entry: ManifestEntry) -> ManifestWriter: if ( (entry.status ==

Re: [PR] Rename data_sequence_number to sequence_number in ManifestEntry [iceberg-python]

kevinjqliu commented on code in PR #900: URL: https://github.com/apache/iceberg-python/pull/900#discussion_r1669065944 ## tests/utils/test_manifest.py: ## @@ -66,7 +66,7 @@ def test_read_manifest_entry(generated_manifest_entry_file: str) -> None: assert manifest_entry.st

Re: [PR] Rename data_sequence_number to sequence_number in ManifestEntry [iceberg-python]

soumya-ghosh commented on code in PR #900: URL: https://github.com/apache/iceberg-python/pull/900#discussion_r1669065871 ## pyiceberg/manifest.py: ## @@ -762,10 +762,10 @@ def add_entry(self, entry: ManifestEntry) -> ManifestWriter: if ( (entry.status ==

Re: [PR] Rename data_sequence_number to sequence_number in ManifestEntry [iceberg-python]

soumya-ghosh commented on code in PR #900: URL: https://github.com/apache/iceberg-python/pull/900#discussion_r1669055951 ## tests/utils/test_manifest.py: ## @@ -66,7 +66,7 @@ def test_read_manifest_entry(generated_manifest_entry_file: str) -> None: assert manifest_entry.

Re: [PR] Rename data_sequence_number to sequence_number in ManifestEntry [iceberg-python]

kevinjqliu commented on code in PR #900: URL: https://github.com/apache/iceberg-python/pull/900#discussion_r1668994905 ## pyiceberg/manifest.py: ## @@ -762,10 +762,10 @@ def add_entry(self, entry: ManifestEntry) -> ManifestWriter: if ( (entry.status == M

Re: [PR] PyArrow: Don't enforce the schema [iceberg-python]

Fokko commented on PR #902: URL: https://github.com/apache/iceberg-python/pull/902#issuecomment-2214713703 I'm aware of the failing CI. Looking into this. It looks like we can automatically cast in the `RecordBatchReader`. -- This is an automated message from the Apache Git Service. To re

Re: [PR] Add interfaces for Action RemoveExpiredFiles [iceberg]

huaxingao commented on code in PR #10643: URL: https://github.com/apache/iceberg/pull/10643#discussion_r1668984923 ## api/src/main/java/org/apache/iceberg/actions/ActionsProvider.java: ## @@ -70,4 +70,10 @@ default RewritePositionDeleteFiles rewritePositionDeletes(Table table)

Re: [I] How to use equality delete in Iceberg v2 table [iceberg]

tanvn commented on issue #6196: URL: https://github.com/apache/iceberg/issues/6196#issuecomment-2214708473 > @singhpk234 is it worth documenting this so that it's clearer for users? Actually it took me a while to find this issue to confirm that Equality deletes are not supported in S

Re: [PR] PyArrow: Don't enforce the schema [iceberg-python]

kevinjqliu commented on code in PR #902: URL: https://github.com/apache/iceberg-python/pull/902#discussion_r1668984296 ## pyiceberg/io/pyarrow.py: ## @@ -1170,7 +1167,7 @@ def project_table( if len(tables) < 1: return pa.Table.from_batches([], schema=schema_to_pya

Re: [PR] Add interfaces for Action RemoveExpiredFiles [iceberg]

huaxingao commented on code in PR #10643: URL: https://github.com/apache/iceberg/pull/10643#discussion_r1668979778 ## api/src/main/java/org/apache/iceberg/actions/RemoveExpiredFiles.java: ## @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Allow writing dataframes that are either a subset of table schema or in arbitrary order [iceberg-python]

kevinjqliu commented on PR #829: URL: https://github.com/apache/iceberg-python/pull/829#issuecomment-2214693217 r? @fokko / @HonahX -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] UncheckedSQLException while deleting table [iceberg-python]

kevinjqliu commented on issue #901: URL: https://github.com/apache/iceberg-python/issues/901#issuecomment-2214690259 Thanks for the report! Do you mind adding a code snippet to help reproduce this issue? It looks like this is related to the SQL Catalog `drop_table` function ht

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668938237 ## data/src/test/java/org/apache/iceberg/TestSplitScan.java: ## @@ -32,59 +35,50 @@ import org.apache.iceberg.io.FileAppender; import org.apache.iceberg.reloc

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668937599 ## build.gradle: ## @@ -81,6 +81,7 @@ if (JavaVersion.current() == JavaVersion.VERSION_1_8) { "--add-opens", "java.base/jdk.inte

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668937119 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java: ## @@ -75,7 +75,8 @@ public class TestIcebergSourceContinuo

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668936598 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/SplitHelpers.java: ## @@ -113,7 +113,7 @@ public static List createSplitsFromTransientHadoop

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668936219 ## data/src/test/java/org/apache/iceberg/data/GenericAppenderHelper.java: ## @@ -45,34 +44,19 @@ public class GenericAppenderHelper { private final Table t

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668934081 ## data/src/test/java/org/apache/iceberg/data/avro/TestGenericReadProjection.java: ## @@ -32,8 +32,7 @@ public class TestGenericReadProjection extends TestRead

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668934458 ## data/src/test/java/org/apache/iceberg/parquet/TestGenericMergingMetrics.java: ## @@ -18,25 +18,22 @@ */ package org.apache.iceberg.parquet; +import java

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668933800 ## data/src/test/java/org/apache/iceberg/data/TestMetricsRowGroupFilterTypes.java: ## @@ -214,81 +218,101 @@ public void createParquetInputFile(List records) t

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668933222 ## data/src/test/java/org/apache/iceberg/data/TestDataFileIndexStatsFilters.java: ## @@ -68,9 +66,11 @@ public class TestDataFileIndexStatsFilters { private

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668932686 ## data/src/test/java/org/apache/iceberg/data/GenericAppenderHelper.java: ## @@ -45,34 +44,19 @@ public class GenericAppenderHelper { private final Table t

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668932233 ## data/src/test/java/org/apache/iceberg/data/DataTestHelpers.java: ## @@ -85,14 +84,14 @@ private static void assertEquals(Type type, Object expected, Object

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668931884 ## data/src/test/java/org/apache/iceberg/data/DataTestHelpers.java: ## @@ -57,7 +56,7 @@ public static void assertEquals(Types.ListType list, List expected, Li

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668930945 ## data/src/test/java/org/apache/iceberg/TestMergingMetrics.java: ## @@ -192,16 +187,16 @@ public void verifyRandomlyGeneratedRecordsMetric() throws Exception

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668930719 ## data/src/test/java/org/apache/iceberg/io/TestWriterMetrics.java: ## @@ -98,9 +100,9 @@ public TestWriterMetrics(FileFormat fileFormat) { protected abstr

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668931619 ## data/src/test/java/org/apache/iceberg/data/DataTestHelpers.java: ## @@ -44,7 +43,7 @@ public static void assertEquals(Types.StructType struct, Record expect

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668931222 ## data/src/test/java/org/apache/iceberg/TestSplitScan.java: ## @@ -32,59 +35,50 @@ import org.apache.iceberg.io.FileAppender; import org.apache.iceberg.reloc

Re: [PR] Data: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

attilakreiner commented on code in PR #10657: URL: https://github.com/apache/iceberg/pull/10657#discussion_r1668929997 ## data/src/test/java/org/apache/iceberg/TestMergingMetrics.java: ## @@ -110,22 +110,17 @@ public abstract class TestMergingMetrics { MAP_FIELD_2,

  1   2   >