[PR] Spark: Include catalog name in view errors [iceberg]

nastra opened a new pull request, #9807: URL: https://github.com/apache/iceberg/pull/9807 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] [AWS] S3FileIO - Add Cross-Region Bucket Access [iceberg]

nastra commented on code in PR #9804: URL: https://github.com/apache/iceberg/pull/9804#discussion_r1502143236 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestS3FileIOProperties.java: ## @@ -439,6 +449,7 @@ public void testApplyS3ServiceConfigurations() { ArgumentCapt

Re: [PR] [AWS][Docs] S3FileIO - Add Cross-Region Bucket Access. [iceberg]

nastra closed pull request #9805: [AWS][Docs] S3FileIO - Add Cross-Region Bucket Access. URL: https://github.com/apache/iceberg/pull/9805 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [AWS][Docs] S3FileIO - Add Cross-Region Bucket Access. [iceberg]

nastra commented on PR #9805: URL: https://github.com/apache/iceberg/pull/9805#issuecomment-1963473902 can you include this into #9804 please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] feat(FileScanTask): partial execute impl for parquet [iceberg-rust]

sdd commented on PR #207: URL: https://github.com/apache/iceberg-rust/pull/207#issuecomment-1963466600 Hi @liurenjie1024 - Thanks for the review. I agree with your conclusions. I'll take a closer look at the Java and Python implementations. The framework approach sounds sensible - happy to

Re: [PR] feat(FileScanTask): partial execute impl for parquet [iceberg-rust]

liurenjie1024 commented on PR #207: URL: https://github.com/apache/iceberg-rust/pull/207#issuecomment-1963461557 The process of reading into arrow record batch stream seems more complex than I expected, you can refer [the codes in python](https://github.com/apache/iceberg-python/blob/6c8ea0

Re: [PR] feat(FileScanTask): partial execute impl for parquet [iceberg-rust]

liurenjie1024 commented on code in PR #207: URL: https://github.com/apache/iceberg-rust/pull/207#discussion_r1502118976 ## crates/iceberg/src/spec/manifest.rs: ## @@ -855,6 +855,11 @@ impl ManifestEntry { self.data_file.content } +/// Content type of this man

Re: [PR] test: add FileIO s3 test [iceberg-rust]

liurenjie1024 commented on code in PR #220: URL: https://github.com/apache/iceberg-rust/pull/220#discussion_r1502118268 ## crates/iceberg/Cargo.toml: ## @@ -62,6 +62,8 @@ urlencoding = { workspace = true } uuid = { workspace = true } [dev-dependencies] +iceberg_test_utils =

Re: [PR] test: add FileIO s3 test [iceberg-rust]

liurenjie1024 commented on code in PR #220: URL: https://github.com/apache/iceberg-rust/pull/220#discussion_r1502116767 ## crates/iceberg/testdata/file_io_s3/docker-compose.yaml: ## @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contrib

Re: [PR] Flink 1.18: Fix iceberg source plan parallelism not effective. [iceberg]

pvary commented on PR #9761: URL: https://github.com/apache/iceberg/pull/9761#issuecomment-1963404740 @Reo-LEI: Do we have a backport pull request for Flink 1.16/1.17? Thanks, Peter -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Partition Evolution [iceberg-python]

amogh-jahagirdar commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1502013528 ## tests/catalog/test_hive.py: ## @@ -277,7 +277,7 @@ def test_create_table(table_schema_simple: Schema, hive_database: HiveDatabase, )

Re: [PR] Partition Evolution [iceberg-python]

amogh-jahagirdar commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1502013528 ## tests/catalog/test_hive.py: ## @@ -277,7 +277,7 @@ def test_create_table(table_schema_simple: Schema, hive_database: HiveDatabase, )

Re: [PR] Cleanup conftest, remove LocalOutputFile [iceberg-python]

HonahX merged PR #468: URL: https://github.com/apache/iceberg-python/pull/468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

[PR] Build: Ignore major version update in dependabot [iceberg]

manuzhang opened a new pull request, #9806: URL: https://github.com/apache/iceberg/pull/9806 Major version updates like https://github.com/apache/iceberg/pull/9799 and https://github.com/apache/iceberg/pull/9798 are usually breaking and can't auto merged with dependabot. This PR pro

Re: [PR] Build: Free disk space before running action in Spark CI [iceberg]

manuzhang commented on PR #9786: URL: https://github.com/apache/iceberg/pull/9786#issuecomment-1963192717 @Fokko and @singhpk234 please take a look at your convenience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Cleanup conftest, remove LocalOutputFile [iceberg-python]

kevinjqliu commented on PR #468: URL: https://github.com/apache/iceberg-python/pull/468#issuecomment-1963191731 thanks@HonahX, I just ran `make lint`! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] [AWS][Docs] S3FileIO - Add Cross-Region Bucket Access. [iceberg]

ebelgasmi12 opened a new pull request, #9805: URL: https://github.com/apache/iceberg/pull/9805 [AWS][Docs] S3FileIO - Add Cross-Region Bucket Access. Added corresponding configuration to the Documentation. Resolves #9785 CC @nastra -- This is an automated message fro

Re: [PR] Bin-pack Writes Operation into multiple parquet files, and parallelize writing `WriteTask`s [iceberg-python]

kevinjqliu commented on PR #444: URL: https://github.com/apache/iceberg-python/pull/444#issuecomment-1963143389 We rely on `Table.to_batches()` to produce smaller `RecordBatch`s from `Table` object which we then use to bin-pack. Depending on how the table was constructed, `.to_batches()` m

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

rdblue commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1501944749 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java: ## @@ -20,39 +20,33 @@ import org.apache.iceberg.io.InputFile; import org.apache.iceberg.io

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

rdblue commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1501944749 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java: ## @@ -20,39 +20,33 @@ import org.apache.iceberg.io.InputFile; import org.apache.iceberg.io

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

rdblue commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1501943394 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java: ## @@ -20,39 +20,33 @@ import org.apache.iceberg.io.InputFile; import org.apache.iceberg.io

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501932950 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,274 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501932441 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,274 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501931992 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,279 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [I] Error when creating a table `InvalidObjectException` [iceberg]

github-actions[bot] commented on issue #1533: URL: https://github.com/apache/iceberg/issues/1533#issuecomment-1963115272 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Add in support for vectorized parquet reads on data files with row position (_pos) in the metadata column [iceberg]

github-actions[bot] commented on issue #1540: URL: https://github.com/apache/iceberg/issues/1540#issuecomment-1963115290 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Reading snapshot of table uses current schema [iceberg]

github-actions[bot] commented on issue #1501: URL: https://github.com/apache/iceberg/issues/1501#issuecomment-1963115263 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Update version-hint.txt atomically [iceberg]

github-actions[bot] commented on issue #1496: URL: https://github.com/apache/iceberg/issues/1496#issuecomment-1963115229 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Read custom metadata on iceberg snapshots via spark options [iceberg]

github-actions[bot] commented on issue #1500: URL: https://github.com/apache/iceberg/issues/1500#issuecomment-1963115243 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Slow parallel operations fail to commit [iceberg]

github-actions[bot] commented on issue #1286: URL: https://github.com/apache/iceberg/issues/1286#issuecomment-1963115148 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Slow parallel operations fail to commit [iceberg]

github-actions[bot] closed issue #1286: Slow parallel operations fail to commit URL: https://github.com/apache/iceberg/issues/1286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Iceberg Datasource Writer Should Automatically Prune Identity Transform Partition Columns [iceberg]

github-actions[bot] closed issue #1281: Iceberg Datasource Writer Should Automatically Prune Identity Transform Partition Columns URL: https://github.com/apache/iceberg/issues/1281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Add a github action to label PRs with the relevant subprojects that are affected [iceberg]

github-actions[bot] closed issue #1277: Add a github action to label PRs with the relevant subprojects that are affected URL: https://github.com/apache/iceberg/issues/1277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] Add a github action to label PRs with the relevant subprojects that are affected [iceberg]

github-actions[bot] commented on issue #1277: URL: https://github.com/apache/iceberg/issues/1277#issuecomment-1963115114 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: Implement Flink InputFormat and integrate it to FlinkCatalog [iceberg]

github-actions[bot] commented on issue #1275: URL: https://github.com/apache/iceberg/issues/1275#issuecomment-1963115101 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: Implement Flink InputFormat and integrate it to FlinkCatalog [iceberg]

github-actions[bot] closed issue #1275: Flink: Implement Flink InputFormat and integrate it to FlinkCatalog URL: https://github.com/apache/iceberg/issues/1275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type ReferenceEquality. [iceberg]

github-actions[bot] commented on issue #1250: URL: https://github.com/apache/iceberg/issues/1250#issuecomment-1963115083 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] How to use `iceberg.mr.filter.expression` in the IcebergInputFormat? [iceberg]

github-actions[bot] commented on issue #1193: URL: https://github.com/apache/iceberg/issues/1193#issuecomment-1963115069 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type ReferenceEquality. [iceberg]

github-actions[bot] closed issue #1250: [ErrorProne] Fix outstanding error prone warnings of type ReferenceEquality. URL: https://github.com/apache/iceberg/issues/1250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] How to use `iceberg.mr.filter.expression` in the IcebergInputFormat? [iceberg]

github-actions[bot] closed issue #1193: How to use `iceberg.mr.filter.expression` in the IcebergInputFormat? URL: https://github.com/apache/iceberg/issues/1193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Error when creating table 'is not a directory or unable to create one' [iceberg]

github-actions[bot] commented on issue #1163: URL: https://github.com/apache/iceberg/issues/1163#issuecomment-1963115054 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Error when creating table 'is not a directory or unable to create one' [iceberg]

github-actions[bot] closed issue #1163: Error when creating table 'is not a directory or unable to create one' URL: https://github.com/apache/iceberg/issues/1163 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Avoid rewriting big files in RewriteDataFilesAction [iceberg]

github-actions[bot] commented on issue #1159: URL: https://github.com/apache/iceberg/issues/1159#issuecomment-1963115039 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Avoid rewriting big files in RewriteDataFilesAction [iceberg]

github-actions[bot] closed issue #1159: Avoid rewriting big files in RewriteDataFilesAction URL: https://github.com/apache/iceberg/issues/1159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501930633 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,274 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501930283 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501930153 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,274 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501928527 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501926569 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,274 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1501926170 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1501924946 ## format/spec.md: ## @@ -1314,6 +1331,24 @@ Default values are added to struct fields in v3. Types `timestamp_ns` and `timestamptz_ns` are added in v3. +Writing V3

[PR] test: add file io s3 test [iceberg-rust]

odysa opened a new pull request, #220: URL: https://github.com/apache/iceberg-rust/pull/220 Add minio docker-compose file for test. Add tests using the minio docker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1501924360 ## format/spec.md: ## @@ -1314,6 +1330,24 @@ Default values are added to struct fields in v3. Types `timestamp_ns` and `timestamptz_ns` are added in v3. +Writing V3

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1501924078 ## format/spec.md: ## @@ -1134,10 +1148,9 @@ In some cases partition specs are stored using only the field list instead of th The `field-id` property was added for e

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1501923473 ## format/spec.md: ## @@ -1128,16 +1142,14 @@ Each partition field in the fields list is stored as an object. See the table fo |**`month`**|`JSON string: "month"`|`"mo

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1501923252 ## format/spec.md: ## @@ -1134,10 +1148,9 @@ In some cases partition specs are stored using only the field list instead of th The `field-id` property was added for e

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1501922331 ## format/spec.md: ## @@ -301,7 +303,7 @@ Tables are configured with a **partition spec** that defines how to produce a tu * A **transform** that is applied to the s

Re: [PR] Improve error message in case of a mismatch [iceberg-python]

Fokko merged PR #352: URL: https://github.com/apache/iceberg-python/pull/352 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1501904263 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + des

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1501904263 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + des

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1501904443 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,24 @@ components: explode: false example: "vended-credentials,remote-signing" +page-toke

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1501904263 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + des

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1501904263 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + des

[PR] Update table metadata throughout transaction [iceberg-python]

Fokko opened a new pull request, #471: URL: https://github.com/apache/iceberg-python/pull/471 This PR add support for updating the table metadata throughout the transaction. This way, if a schema is first evolved, and then a snapshot is created based on the latest schema, it will be

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501901195 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -77,8 +78,47 @@ public void write(SinkRecord record) { }

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501900737 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -77,8 +78,47 @@ public void write(SinkRecord record) { }

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501900020 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/BaseDeltaTaskWriter.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

fqaiser94 commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501880384 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -77,8 +78,47 @@ public void write(SinkRecord record) { }

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

fqaiser94 commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501878010 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

fqaiser94 commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501882436 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -77,8 +78,47 @@ public void write(SinkRecord record) { }

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

fqaiser94 commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501877753 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/BaseDeltaTaskWriter.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Fo

[PR] Fix `InMemoryCatalog` Catalog commit operation [iceberg-python]

anupam-saini opened a new pull request, #470: URL: https://github.com/apache/iceberg-python/pull/470 1. Adding a new schema to the InMemoryCatalog catalog results in removing the old schema as can be seen by the assertions of `test_commit_table` ([link](https://github.com/apache/iceberg-pyt

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501873385 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,35 @@ public void showViews() throws NoSuchTableExce

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501873385 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,35 @@ public void showViews() throws NoSuchTableExce

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue merged PR #9675: URL: https://github.com/apache/iceberg/pull/9675 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501871931 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -149,4 +167,20 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501871639 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,20 @@ case class RewriteViewCommands(s

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501871166 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [PR] Spark: Detect temp functions in views [iceberg]

rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501870940 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -471,6 +471,39 @@ public void readFromViewReferencingGlobalTempVie

Re: [PR] Spark 3.4,3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501870626 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,35 @@ public void showViews() throws NoSuchTableExce

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on PR #9641: URL: https://github.com/apache/iceberg/pull/9641#issuecomment-1963015425 A few comments/questions, but overall looks good. You might want to double check all the access levels for classes as I think we're generally overexposing. -- This is an automated

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501869002 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/UnpartitionedDeltaWriter.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Soft

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501868946 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordWrapper.java: ## @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501868749 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Operation.java: ## @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501868634 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -77,8 +78,47 @@ public void write(SinkRecord record) { }

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501868634 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -77,8 +78,47 @@ public void write(SinkRecord record) { }

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501867687 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -52,20 +51,22 @@ public IcebergWriter(Table table, String ta

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501865059 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordProjection.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501864940 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501864346 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

danielcweeks commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1501863990 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/PartitionedDeltaWriter.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Softwa

[I] Cannot find constructor for interface org.apache.parquet.column.page.PageWriteStore? [iceberg]

williamliu111 opened a new issue, #9802: URL: https://github.com/apache/iceberg/issues/9802 ### Query engine _No response_ ### Question xception in thread "main" java.lang.ExceptionInInitializerError at org.apache.iceberg.parquet.Parquet$WriteBuilder.build(Par

Re: [I] Parallel Table.append [iceberg-python]

bigluck commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1962855117 Hey @kevinjqliu , we're currently debugging the issue on Slack, but I thought it would be helpful to report our findings here as well. In my tests, the pyarrow table is generate