Re: [PR] Sort Order update [iceberg-python]

Fokko commented on code in PR #476: URL: https://github.com/apache/iceberg-python/pull/476#discussion_r1503784888 ## pyiceberg/table/metadata.py: ## @@ -226,6 +226,10 @@ def schema_by_id(self, schema_id: int) -> Optional[Schema]: """Get the schema by schema_id."""

Re: [I] Support metadata compaction [iceberg-python]

HonahX commented on issue #270: URL: https://github.com/apache/iceberg-python/issues/270#issuecomment-1965963701 Based on offline discussion with @Fokko, I will focus on implementing the `MergeAppend` which supports these keys - `commit.manifest-merge.enabled` - `commit.manifest.min-co

Re: [PR] AWS: Revert DynamoDb deprecation for 1.5.0 [iceberg]

nastra commented on PR #9815: URL: https://github.com/apache/iceberg/pull/9815#issuecomment-1965963686 for other reviewers: The original deprecation was supposed to be done for 1.6.0 but happened to land on main while an RC was cancelled and so the deprecation was updated to 1.5.0. In o

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

nastra merged PR #9787: URL: https://github.com/apache/iceberg/pull/9787 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] ci: Ignore RUSTSEC-2023-0071 for no actions to take [iceberg-rust]

Fokko merged PR #222: URL: https://github.com/apache/iceberg-rust/pull/222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] test: add FileIO s3 test [iceberg-rust]

Fokko commented on PR #220: URL: https://github.com/apache/iceberg-rust/pull/220#issuecomment-1965933411 Thanks @odysa for working on this! Great to have this in, thanks @liurenjie1024 for the review 👍 -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] test: add FileIO s3 test [iceberg-rust]

Fokko merged PR #220: URL: https://github.com/apache/iceberg-rust/pull/220 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

ajantha-bhat commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1965932224 @SreeramGarlapati , @namrathamyske: Can you also comment that salesforce is using it and mention the problems with Glue catalog in details as a reply to the mailing list discussion?

Re: [PR] feat: Add expression builder and display. [iceberg-rust]

liurenjie1024 commented on PR #169: URL: https://github.com/apache/iceberg-rust/pull/169#issuecomment-1965897689 Sorry for late reply, I have fixed comments, PTAL cc @Fokko @Xuanwo @ZENOTME -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] Allow non-string typed values in table properties [iceberg-python]

kevinjqliu commented on code in PR #469: URL: https://github.com/apache/iceberg-python/pull/469#discussion_r1503717843 ## tests/catalog/test_sql.py: ## @@ -851,3 +852,39 @@ def test_concurrent_commit_table(catalog: SqlCatalog, table_schema_simple: Schem # This one shou

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

nastra commented on PR #9787: URL: https://github.com/apache/iceberg/pull/9787#issuecomment-1965887088 thanks for the reviews @Fokko, @amogh-jahagirdar, @rdblue. I'll go ahead and merge this so that we can do another RC with this fix -- This is an automated message from the Apache Git Ser

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

namrathamyske commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1965884032 @geruh it was decided in sync to move all catalogs to another repo if i am not wrong, can we do that instead of removing the catalog altogether -- This is an automated message fro

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

nastra commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1503707724 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,43 @@ public void showViews() throws NoSuchTableExce

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg-python]

HonahX commented on PR #475: URL: https://github.com/apache/iceberg-python/pull/475#issuecomment-1965871433 Ah, just saw this comment: https://github.com/apache/iceberg/pull/9783#issuecomment-1965781361. So maybe we can wait for a while to see how the discussion goes, -- This is an autom

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg-python]

HonahX commented on code in PR #475: URL: https://github.com/apache/iceberg-python/pull/475#discussion_r1503700463 ## pyiceberg/catalog/dynamodb.py: ## @@ -81,6 +82,10 @@ ITEM = "Item" +@deprecated( +deprecated_in="0.6.0", +removed_in="1.0.0", +) Review Comment:

Re: [I] Cannot find constructor for interface org.apache.parquet.column.page.PageWriteStore? [iceberg]

manuzhang commented on issue #9802: URL: https://github.com/apache/iceberg/issues/9802#issuecomment-1965869982 It looks you have two versions of parquet in your dependencies, 1.11.1 and 1.12.0 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] feat: Add expression builder and display. [iceberg-rust]

liurenjie1024 commented on code in PR #169: URL: https://github.com/apache/iceberg-rust/pull/169#discussion_r1503693670 ## crates/iceberg/src/spec/values.rs: ## @@ -70,6 +79,571 @@ pub enum PrimitiveLiteral { Decimal(i128), } +/// Literal associated with its type. The va

Re: [I] Why does FlinkSink writes position deletes in append-mode if identifier fields are specified? [iceberg]

pvary commented on issue #9773: URL: https://github.com/apache/iceberg/issues/9773#issuecomment-1965862091 What do you mean by > FlinkSink in append-mode Could you please share your sink definition? The code snippet highlighted by you is part of the `BaseEqualityDeltaWriter`

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

SreeramGarlapati commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1965781361 @geruh - we (Salesforce DataCloud team) use DynamoDB catalog. AWS is the biggest cloud out there & dynamoDb is the most reliable aws's no-sql db as mgd. svc. we started with

Re: [I] reset catalog can't reslove s3 domain name [iceberg]

dongsupkim-onepredict commented on issue #7709: URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1965763850 Is there anyone who solved this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] feat: Add expression builder and display. [iceberg-rust]

liurenjie1024 commented on code in PR #169: URL: https://github.com/apache/iceberg-rust/pull/169#discussion_r1503606792 ## crates/iceberg/src/spec/values.rs: ## @@ -70,6 +79,571 @@ pub enum PrimitiveLiteral { Decimal(i128), } +/// Literal associated with its type. The va

Re: [PR] [Bug Fix] cast None `current-snapshot-id` as -1 for Backwards Compatibility [iceberg-python]

syun64 commented on PR #473: URL: https://github.com/apache/iceberg-python/pull/473#issuecomment-1965729096 > Thanks for the great catch @syun64 ! My understanding is that we need to write `current_snapshot_id` to `-1` when serializing the new metadata object to JSON. Would it be better to

Re: [PR] Update table metadata throughout transaction [iceberg-python]

HonahX commented on code in PR #471: URL: https://github.com/apache/iceberg-python/pull/471#discussion_r1503532245 ## pyiceberg/table/__init__.py: ## @@ -219,68 +220,41 @@ def property_as_int(properties: Dict[str, str], property_name: str, default: Opt class Transaction:

Re: [I] EMR 6.10.0 Cannot migrate a table from a non-Iceberg Spark Session Catalog. Found spark_catalog [iceberg]

tomtongue commented on issue #7317: URL: https://github.com/apache/iceberg/issues/7317#issuecomment-1965718945 You can use `snapshot` for the Glue Data Catalog. Can you try setting `catalog.db.tbl` for your target table? -- This is an automated message from the Apache Git Service. To resp

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

fqaiser94 commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1503273341 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordProjection.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Found

[PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg-python]

geruh opened a new pull request, #475: URL: https://github.com/apache/iceberg-python/pull/475 The deprecation of the dynamo catalog in python is stemming from the discussion in the [community sync](https://youtu.be/uAQVGd5zV4I?si=cj0xpfprgvJIGYIm&t=1323), we want to reduce the scope of sup

Re: [PR] Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction [iceberg]

dramaticlly commented on PR #9724: URL: https://github.com/apache/iceberg/pull/9724#issuecomment-1965684365 > Okay, I did one pass and here are my high-level notes: > > * We should use `RewriteFiles` instead of `DeleteFiles`, changes in `DeleteFiles` should be reverted. > * I don't

Re: [PR] feat: Add expression builder and display. [iceberg-rust]

liurenjie1024 commented on code in PR #169: URL: https://github.com/apache/iceberg-rust/pull/169#discussion_r150386 ## crates/iceberg/src/expr/predicate.rs: ## @@ -20,43 +20,89 @@ //! `a > 10` is a predicate expression, and it evaluates to `true` if `a` is greater than `10

Re: [PR] feat: Add expression builder and display. [iceberg-rust]

liurenjie1024 commented on code in PR #169: URL: https://github.com/apache/iceberg-rust/pull/169#discussion_r1503555188 ## crates/iceberg/src/expr/mod.rs: ## @@ -18,12 +18,15 @@ //! This module contains expressions. mod term; + +use std::fmt::{Display, Formatter}; pub use t

Re: [PR] [Bug Fix] cast None `current-snapshot-id` as -1 for Backwards Compatibility [iceberg-python]

HonahX commented on PR #473: URL: https://github.com/apache/iceberg-python/pull/473#issuecomment-1965634235 Thanks for the great catch @syun64 ! My understanding is that we need to write `current_snapshot_id` to `-1` when serializing the new metadata object to JSON. Would it be better to di

Re: [PR] feat(FileScanTask): partial execute impl for parquet [iceberg-rust]

liurenjie1024 commented on PR #207: URL: https://github.com/apache/iceberg-rust/pull/207#issuecomment-1965612136 > Thanks for those links, Renjie, they were very useful.. I've taken a deeper look at `ArrowReader` and the python code. I'll submit an improved design here tomorrow morning (UK)

[PR] core: Add data sequence number as derived column to files metadata table [iceberg]

dramaticlly opened a new pull request, #9813: URL: https://github.com/apache/iceberg/pull/9813 This PR add data_sequence_number as derived/virtual column on all files metadata table, enables query like ``` SELECT data_sequence_number FROM iceberg.db.table.files ``` without chang

Re: [PR] objstore: Replace IO interface with objstore.Bucket [iceberg-go]

thorfour commented on PR #66: URL: https://github.com/apache/iceberg-go/pull/66#issuecomment-1965584472 > This seems pretty interesting to me, not sure how I feel about it yet. But that said, I wouldn't want `LoadFS` to return an `objstore.Bucket` because that would tie us to them as an ext

Re: [PR] feat: glue table creation with some docs on testing [iceberg-go]

zeroshade commented on code in PR #59: URL: https://github.com/apache/iceberg-go/pull/59#discussion_r1503479405 ## table/metadata.go: ## @@ -399,3 +400,32 @@ func (m *MetadataV2) UnmarshalJSON(b []byte) error { m.preValidate() return m.validate() } + +func NewMe

Re: [PR] feat: glue table creation with some docs on testing [iceberg-go]

zeroshade commented on code in PR #59: URL: https://github.com/apache/iceberg-go/pull/59#discussion_r1503478251 ## docs/cfn/AWS_TESTING.md: ## @@ -0,0 +1,74 @@ + + +# AWS integration testing + Review Comment: why `/docs/cfn/` ? What is `cfn`? -- This is an automated mess

Re: [PR] feat: glue table creation with some docs on testing [iceberg-go]

zeroshade commented on code in PR #59: URL: https://github.com/apache/iceberg-go/pull/59#discussion_r1503477206 ## catalog/catalog.go: ## @@ -185,3 +197,33 @@ func TableNameFromIdent(ident table.Identifier) string { func NamespaceFromIdent(ident table.Identifier) table.Identifi

Re: [PR] feat: glue table creation with some docs on testing [iceberg-go]

zeroshade commented on code in PR #59: URL: https://github.com/apache/iceberg-go/pull/59#discussion_r1503476723 ## catalog/catalog.go: ## @@ -185,3 +197,33 @@ func TableNameFromIdent(ident table.Identifier) string { func NamespaceFromIdent(ident table.Identifier) table.Identifi

Re: [PR] objstore: Replace IO interface with objstore.Bucket [iceberg-go]

zeroshade commented on PR #66: URL: https://github.com/apache/iceberg-go/pull/66#issuecomment-1965567918 This seems pretty interesting to me, not sure how I feel about it yet. But that said, I wouldn't want `LoadFS` to return an `objstore.Bucket` because that would tie us to them as an exte

Re: [I] class not found exception emr 6.1.0, iceberg-spark-0.9.1,iceberg-spark-runtime-0.9.1 java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.DataSourceV2 [iceberg]

github-actions[bot] commented on issue #1622: URL: https://github.com/apache/iceberg/issues/1622#issuecomment-1965559461 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Implements the Flink source based on the new FLIP-27 interface [iceberg]

github-actions[bot] commented on issue #1626: URL: https://github.com/apache/iceberg/issues/1626#issuecomment-1965559532 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Iceberg table via Parquet files on S3 implementation [iceberg]

github-actions[bot] commented on issue #1621: URL: https://github.com/apache/iceberg/issues/1621#issuecomment-1965559398 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] The AvroSchemaUtil will convert iceberg's optional map into confusing union. [iceberg]

github-actions[bot] commented on issue #1578: URL: https://github.com/apache/iceberg/issues/1578#issuecomment-1965559323 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spark Predicate Pushdown for Metadata Table Scans [iceberg]

github-actions[bot] commented on issue #1552: URL: https://github.com/apache/iceberg/issues/1552#issuecomment-1965559258 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Ignore rewrite commits while validating copy-on-write deletes in OverwriteFiles [iceberg]

github-actions[bot] commented on issue #1547: URL: https://github.com/apache/iceberg/issues/1547#issuecomment-1965559131 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Consider adding extra validation for metrics config when creating a table [iceberg]

github-actions[bot] commented on issue #1548: URL: https://github.com/apache/iceberg/issues/1548#issuecomment-1965559198 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] numBuckets can cause division by zero [iceberg]

github-actions[bot] commented on issue #1543: URL: https://github.com/apache/iceberg/issues/1543#issuecomment-1965559058 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Capture the template parameter in the tests for TestIcebergInputFormat#Factory [iceberg]

github-actions[bot] commented on issue #1542: URL: https://github.com/apache/iceberg/issues/1542#issuecomment-1965558992 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Should we consider adding a github action to lint YAML files for correctness / proper style? [iceberg]

github-actions[bot] commented on issue #1300: URL: https://github.com/apache/iceberg/issues/1300#issuecomment-1965558661 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type UnusedVariable. [iceberg]

github-actions[bot] commented on issue #1291: URL: https://github.com/apache/iceberg/issues/1291#issuecomment-1965558596 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Should we consider adding a github action to lint YAML files for correctness / proper style? [iceberg]

github-actions[bot] closed issue #1300: Should we consider adding a github action to lint YAML files for correctness / proper style? URL: https://github.com/apache/iceberg/issues/1300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type UnusedVariable. [iceberg]

github-actions[bot] closed issue #1291: [ErrorProne] Fix outstanding error prone warnings of type UnusedVariable. URL: https://github.com/apache/iceberg/issues/1291 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type NarrowingCompoundAssignment. [iceberg]

github-actions[bot] commented on issue #1290: URL: https://github.com/apache/iceberg/issues/1290#issuecomment-1965558537 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] [ErrorProne] Fix outstanding error prone warnings of type NarrowingCompoundAssignment. [iceberg]

github-actions[bot] closed issue #1290: [ErrorProne] Fix outstanding error prone warnings of type NarrowingCompoundAssignment. URL: https://github.com/apache/iceberg/issues/1290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

szehon-ho commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1503456330 ## format/spec.md: ## @@ -1170,9 +1170,9 @@ Each sort field in the fields list is stored as an object with the following pro | required | required | required | **`d

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

rdblue commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1503435600 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,279 @@ components: type: integer format: int64 +BooleanTypeValue: + type: bo

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on PR #9661: URL: https://github.com/apache/iceberg/pull/9661#issuecomment-1965505699 Merging this. Thanks, @szehon-ho! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1503432867 ## format/spec.md: ## @@ -1170,9 +1170,9 @@ Each sort field in the fields list is stored as an object with the following pro | required | required | required | **`dire

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue merged PR #9661: URL: https://github.com/apache/iceberg/pull/9661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

rdblue commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1503431015 ## format/spec.md: ## @@ -1170,9 +1170,9 @@ Each sort field in the fields list is stored as an object with the following pro | required | required | required | **`dire

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1503409864 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,43 @@ public void showViews() throws NoSuchTableExce

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1503408840 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,43 @@ public void showViews() throws NoSuchTableExce

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1503408646 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,43 @@ public void showViews() throws NoSuchTableExce

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1503408071 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,43 @@ public void showViews() throws NoSuchTableExce

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1503407151 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1335,6 +1335,43 @@ public void showViews() throws NoSuchTableExce

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

geruh commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1503405472 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,279 @@ components: type: integer format: int64 +BooleanTypeValue: + type: boo

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

szehon-ho commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1503400947 ## format/spec.md: ## @@ -1134,10 +1148,9 @@ In some cases partition specs are stored using only the field list instead of th The `field-id` property was added fo

Re: [PR] feat(FileScanTask): partial execute impl for parquet [iceberg-rust]

sdd commented on PR #207: URL: https://github.com/apache/iceberg-rust/pull/207#issuecomment-1965420027 Thanks for those links, Renjie, they were very useful.. I've taken a deeper look at `ArrowReader` and the python code. I'll submit an improved design here tomorrow morning (UK) with some `

Re: [PR] [AWS] S3FileIO - Add Cross-Region Bucket Access [iceberg]

geruh commented on code in PR #9804: URL: https://github.com/apache/iceberg/pull/9804#discussion_r1503279807 ## docs/docs/aws.md: ## @@ -500,6 +500,13 @@ spark-sql --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCata For more details on using S3 Access Gran

Re: [PR] Make issued_token_type optional to support OAuth2 Client Credential Flow [iceberg-python]

flyrain commented on code in PR #466: URL: https://github.com/apache/iceberg-python/pull/466#discussion_r1503212827 ## pyiceberg/catalog/rest.py: ## @@ -157,7 +157,7 @@ class TokenResponse(IcebergBaseModel): access_token: str = Field() token_type: str = Field() ex

Re: [PR] Make issued_token_type optional to support OAuth2 Client Credential Flow [iceberg-python]

flyrain commented on code in PR #466: URL: https://github.com/apache/iceberg-python/pull/466#discussion_r1503212827 ## pyiceberg/catalog/rest.py: ## @@ -157,7 +157,7 @@ class TokenResponse(IcebergBaseModel): access_token: str = Field() token_type: str = Field() ex

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

fqaiser94 commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1502911936 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

nastra commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1503123700 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1456,6 +1456,43 @@ public void showViews() throws NoSuchTableExce

Re: [PR] [AWS] S3FileIO - Add Cross-Region Bucket Access [iceberg]

elmehdibelgasmi commented on code in PR #9804: URL: https://github.com/apache/iceberg/pull/9804#discussion_r1503103822 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -356,6 +356,16 @@ public class S3FileIOProperties implements Serializable { p

Re: [PR] [AWS] S3FileIO - Add Cross-Region Bucket Access [iceberg]

jackye1995 commented on code in PR #9804: URL: https://github.com/apache/iceberg/pull/9804#discussion_r1503096129 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestS3FileIOProperties.java: ## @@ -439,6 +449,7 @@ public void testApplyS3ServiceConfigurations() { Argument

Re: [PR] [AWS] S3FileIO - Add Cross-Region Bucket Access [iceberg]

jackye1995 commented on code in PR #9804: URL: https://github.com/apache/iceberg/pull/9804#discussion_r1503094756 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -356,6 +356,16 @@ public class S3FileIOProperties implements Serializable { public

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1503083082 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] [AWS] S3FileIO - Add Cross-Region Bucket Access [iceberg]

amogh-jahagirdar commented on code in PR #9804: URL: https://github.com/apache/iceberg/pull/9804#discussion_r1503057978 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestS3FileIOProperties.java: ## @@ -439,6 +449,7 @@ public void testApplyS3ServiceConfigurations() { Ar

[PR] [Bug Fix] cast None `current-snapshot-id` as -1 for Backwards Compatibility [iceberg-python]

syun64 opened a new pull request, #473: URL: https://github.com/apache/iceberg-python/pull/473 The existing PyIceberg `cleanup_snapshot_id` validator creates tables that are not backward compatible. On table creation, the existing behavior in Java is to create tables with current_sna

Re: [PR] Spark 3.4, 3.5: Use current namespace for SHOW VIEWS cmd [iceberg]

amogh-jahagirdar commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1502940098 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1456,6 +1456,43 @@ public void showViews() throws NoSuc

Re: [I] Writing Equality Deletes using Iceberg Java API [iceberg]

robinsinghstudios commented on issue #9808: URL: https://github.com/apache/iceberg/issues/9808#issuecomment-1964530610 > @robinsinghstudios i have done [similar implementation here ](https://github.com/memiiso/debezium-server-iceberg/tree/master/debezium-server-iceberg-sink/src/main/java/io/

Re: [I] EMR 6.10.0 Cannot migrate a table from a non-Iceberg Spark Session Catalog. Found spark_catalog [iceberg]

wstevenson-r7 commented on issue #7317: URL: https://github.com/apache/iceberg/issues/7317#issuecomment-1964454910 For what it's worth I see the same issue with `snapshot`, any progress on this? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Core: Make CatalogHandlers.commit public [iceberg]

N-o-Z commented on PR #9789: URL: https://github.com/apache/iceberg/pull/9789#issuecomment-1964374624 > @N-o-Z can you elaborate please what exactly you're trying to achieve? Are you trying to implement multi-table commits on the server? For `COMMIT_TRANSACTION` the approach is very simplis

Re: [PR] Core: Make CatalogHandlers.commit public [iceberg]

nastra commented on PR #9789: URL: https://github.com/apache/iceberg/pull/9789#issuecomment-1964365614 @N-o-Z can you elaborate please what exactly you're trying to achieve? Are you trying to implement multi-table commits on the server? For `COMMIT_TRANSACTION` the approach is very sim

Re: [PR] Spark 3.4: Include catalog name in view errors [iceberg]

nastra merged PR #9810: URL: https://github.com/apache/iceberg/pull/9810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Allow non-string typed values in table properties [iceberg-python]

Fokko commented on code in PR #469: URL: https://github.com/apache/iceberg-python/pull/469#discussion_r1502723574 ## pyiceberg/table/metadata.py: ## @@ -178,6 +178,12 @@ class TableMetadataCommonFields(IcebergBaseModel): to be used for arbitrary metadata. For example, commi

Re: [PR] enable set hadoop ugi for hive catalog [iceberg-python]

j7nhai commented on PR #472: URL: https://github.com/apache/iceberg-python/pull/472#issuecomment-1964326603 > @j7nhai this change seems reasonable to me. Can you check the type annotations? updated. -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

ggershinsky commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1502705557 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java: ## @@ -20,39 +20,33 @@ import org.apache.iceberg.io.InputFile; import org.apache.icebe

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

ggershinsky commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1502719019 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java: ## @@ -20,39 +20,33 @@ import org.apache.iceberg.io.InputFile; import org.apache.icebe

Re: [PR] Fix `InMemoryCatalog` Catalog commit operation [iceberg-python]

Fokko merged PR #470: URL: https://github.com/apache/iceberg-python/pull/470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] enable set hadoop ugi for hive catalog [iceberg-python]

Fokko commented on code in PR #472: URL: https://github.com/apache/iceberg-python/pull/472#discussion_r1502712949 ## pyiceberg/catalog/hive.py: ## @@ -130,17 +130,22 @@ class _HiveClient: _transport: TTransport _client: Client +_ugi: list -def __init__(self

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

ggershinsky commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1502705557 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java: ## @@ -20,39 +20,33 @@ import org.apache.iceberg.io.InputFile; import org.apache.icebe

[PR] enable set hadoop ugi for hive catalog. [iceberg-python]

j7nhai opened a new pull request, #472: URL: https://github.com/apache/iceberg-python/pull/472 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Build: Free disk space before running action in Spark CI [iceberg]

manuzhang commented on PR #9786: URL: https://github.com/apache/iceberg/pull/9786#issuecomment-1964179087 @ajantha-bhat It took around two minutes per action run. I suppose actions are run in parallel so that's also the overall increase time? ![CleanShot 2024-02-26 at 21 40 35@2x](https:

Re: [PR] Core: Make CatalogHandlers.commit public [iceberg]

N-o-Z commented on PR #9789: URL: https://github.com/apache/iceberg/pull/9789#issuecomment-1964153139 > I believe the purpose was that `CatalogHandlers.commit(..)` should only be used through `CatalogHandlers.updateTable(...)`. Is there a particular issue that prevents you from going throug

Re: [PR] Flink: backport #9761 to Flink 1.16 and 1.17 for fix iceberg source plan parallelism not effective. [iceberg]

nastra merged PR #9811: URL: https://github.com/apache/iceberg/pull/9811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Flink: backport #9762 to Flink 1.16 and 1.17 for fix continuous enumerator lost enumeration history state when restore from checkpoint. [iceberg]

nastra merged PR #9812: URL: https://github.com/apache/iceberg/pull/9812 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Free disk space before running action in Spark CI [iceberg]

ajantha-bhat commented on PR #9786: URL: https://github.com/apache/iceberg/pull/9786#issuecomment-1964061257 @manuzhang: Very nice to see this addition. Have we benchmarked how long it took to clean it up and overall increase in CI time with this? -- This is an automated message from the

Re: [PR] Flink: Supports specifying comment for iceberg fields in create table and addcolumn syntax using flinksql [iceberg]

huyuanfeng2018 commented on code in PR #9606: URL: https://github.com/apache/iceberg/pull/9606#discussion_r1502526761 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkSchemaUtil.java: ## @@ -68,6 +73,42 @@ public static Schema convert(TableSchema schema) { r

Re: [I] Writing Equality Deletes using Iceberg Java API [iceberg]

ismailsimsek commented on issue #9808: URL: https://github.com/apache/iceberg/issues/9808#issuecomment-1963949626 @robinsinghstudios i have done [similar implementation here ](https://github.com/memiiso/debezium-server-iceberg/tree/master/debezium-server-iceberg-sink/src/main/java/io/debeziu

Re: [PR] Docs: Sync contributing page / refer to website for contributing [iceberg]

nastra merged PR #9776: URL: https://github.com/apache/iceberg/pull/9776 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

  1   2   >