Re: [I] EPIC: Rust Based Compaction [iceberg-rust]

2024-09-11 Thread via GitHub
sdd commented on issue #624: URL: https://github.com/apache/iceberg-rust/issues/624#issuecomment-2342825385 I will be working on adding support for positional and identity deletes to the table scans next. Hopefully this will help towards this goal. -- This is an automated message from the

Re: [PR] Core: Allow servers to express supported endpoints via endpoint field in ConfigResponse [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10929: URL: https://github.com/apache/iceberg/pull/10929#discussion_r1753345154 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contri

Re: [PR] Core: Allow servers to express supported endpoints via endpoint field in ConfigResponse [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10929: URL: https://github.com/apache/iceberg/pull/10929#discussion_r1753354122 ## core/src/main/java/org/apache/iceberg/rest/responses/ConfigResponse.java: ## @@ -153,8 +173,18 @@ public Builder withOverrides(Map overridesToAdd) { return

Re: [PR] Core: Allow servers to express supported endpoints via endpoint field in ConfigResponse [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10929: URL: https://github.com/apache/iceberg/pull/10929#discussion_r1753360024 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contri

Re: [PR] Core: Allow servers to express supported endpoints via endpoint field in ConfigResponse [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10929: URL: https://github.com/apache/iceberg/pull/10929#discussion_r1753366181 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contri

Re: [PR] Core: Allow servers to express supported endpoints via endpoint field in ConfigResponse [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10929: URL: https://github.com/apache/iceberg/pull/10929#discussion_r1753408304 ## core/src/main/java/org/apache/iceberg/rest/ResourcePaths.java: ## @@ -26,6 +26,56 @@ public class ResourcePaths { private static final Joiner SLASH = Joiner.on(

Re: [PR] Core: Allow servers to express supported endpoints via endpoint field in ConfigResponse [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10929: URL: https://github.com/apache/iceberg/pull/10929#discussion_r1753475044 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contri

Re: [PR] WIP: Upgrade google-java-format [iceberg]

2024-09-11 Thread via GitHub
nastra closed pull request #0: WIP: Upgrade google-java-format URL: https://github.com/apache/iceberg/pull/0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Docs: Clarified Partition Transform [iceberg]

2024-09-11 Thread via GitHub
nastra commented on PR #8337: URL: https://github.com/apache/iceberg/pull/8337#issuecomment-2342944753 @jasonmfehr can you please fix the conflicts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[I] KC Integration tests occasionally fail [iceberg]

2024-09-11 Thread via GitHub
nastra opened a new issue, #1: URL: https://github.com/apache/iceberg/issues/1 ### Apache Iceberg version main (development) ### Query engine Kafka Connect ### Please describe the bug 🐞 ``` > Task :iceberg-kafka-connect:iceberg-kafka-connect-runti

Re: [PR] Build: Upgrade google-java-format to latest version [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #11050: URL: https://github.com/apache/iceberg/pull/11050#discussion_r1753546454 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergCommitter.java: ## @@ -1441,5 +1441,6 @@ public int hashCode() { public TypeSerializer

Re: [I] Why call deleteKey for Insert and Update After in Flink BaseDeltaTaskWriter? [iceberg]

2024-09-11 Thread via GitHub
SML0127 commented on issue #11081: URL: https://github.com/apache/iceberg/issues/11081#issuecomment-2343005989 @pvary Thx pvary! I have one question. As I understand under code, EqualityDeletes.filter() function don't filter delete file that has same seq number with data file.

Re: [PR] Build: Upgrade google-java-format to 1.22.0 [iceberg]

2024-09-11 Thread via GitHub
nastra merged PR #11050: URL: https://github.com/apache/iceberg/pull/11050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Flink: Maintenance - Lock remover [iceberg]

2024-09-11 Thread via GitHub
pvary commented on PR #11010: URL: https://github.com/apache/iceberg/pull/11010#issuecomment-2343218837 @stevenzwu: I didn't get answer for my question on the Flink dev list about the output of the PostCommitTopology. See: https://lists.apache.org/thread/28qb1q6b7kz30dqjdbsw855osx4t38s7

Re: [PR] Core: Remove one comment from FastAppend [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10995: URL: https://github.com/apache/iceberg/pull/10995#discussion_r1753988942 ## core/src/test/java/org/apache/iceberg/TestFastAppend.java: ## @@ -252,11 +252,36 @@ public void testFailure() { assertThat(new File(newManifest.path())).doesNo

Re: [PR] Core: Remove one comment from FastAppend [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10995: URL: https://github.com/apache/iceberg/pull/10995#discussion_r1754002089 ## core/src/main/java/org/apache/iceberg/FastAppend.java: ## @@ -24,20 +24,14 @@ import java.util.Set; import org.apache.iceberg.encryption.EncryptedOutputFile; imp

Re: [PR] Spark 3.5: Fix TestTagDDL only testing with SparkSessionCatalog [iceberg]

2024-09-11 Thread via GitHub
manuzhang commented on code in PR #11048: URL: https://github.com/apache/iceberg/pull/11048#discussion_r1754081441 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceTagExec.scala: ## @@ -48,7 +48,8 @@ case class CreateOrRe

Re: [PR] Spark 3.5: Fix TestTagDDL only testing with SparkSessionCatalog [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #11048: URL: https://github.com/apache/iceberg/pull/11048#discussion_r1754101092 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceTagExec.scala: ## @@ -48,7 +48,8 @@ case class CreateOrRepla

Re: [PR] Spark 3.5: Fix TestTagDDL only testing with SparkSessionCatalog [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #11048: URL: https://github.com/apache/iceberg/pull/11048#discussion_r1754106397 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java: ## @@ -45,19 +43,9 @@ public class TestTagDDL extends ExtensionsTest

Re: [PR] Spark 3.5: Fix TestTagDDL only testing with SparkSessionCatalog [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #11048: URL: https://github.com/apache/iceberg/pull/11048#discussion_r1754106397 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java: ## @@ -45,19 +43,9 @@ public class TestTagDDL extends ExtensionsTest

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-09-11 Thread via GitHub
nastra closed pull request #9940: OpenAPI: Express server capabilities via /config endpoint URL: https://github.com/apache/iceberg/pull/9940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-09-11 Thread via GitHub
nastra commented on PR #9940: URL: https://github.com/apache/iceberg/pull/9940#issuecomment-2343350495 closing this in favor of #10928 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[I] bug: FileScanTask project_field_ids order could be inconsistent with the RecordBatch schema [iceberg-rust]

2024-09-11 Thread via GitHub
chenzl25 opened a new issue, #627: URL: https://github.com/apache/iceberg-rust/issues/627 As we know, `FileScanTask` has two fields `project_field_ids` and `schema`. I think the `RecordBatch` from the reader of this `FileScanTask` should always follow the schema specified in `FileScanTask`.

Re: [PR] Spark 3.5: Fix TestTagDDL only testing with SparkSessionCatalog [iceberg]

2024-09-11 Thread via GitHub
manuzhang closed pull request #11048: Spark 3.5: Fix TestTagDDL only testing with SparkSessionCatalog URL: https://github.com/apache/iceberg/pull/11048 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Spark 3.5: Fix TestTagDDL only testing with SparkSessionCatalog [iceberg]

2024-09-11 Thread via GitHub
manuzhang commented on code in PR #11048: URL: https://github.com/apache/iceberg/pull/11048#discussion_r1754220227 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java: ## @@ -45,19 +43,9 @@ public class TestTagDDL extends ExtensionsT

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-09-11 Thread via GitHub
manuzhang commented on PR #10811: URL: https://github.com/apache/iceberg/pull/10811#issuecomment-2343454242 cc @nastra help review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10811: URL: https://github.com/apache/iceberg/pull/10811#discussion_r1754328006 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java: ## @@ -419,5 +421,13 @@ public void testFaultToleranceOnWrite() throws IOEx

Re: [PR] DRAFT - Issue 10275 - Reward support for nulls [iceberg]

2024-09-11 Thread via GitHub
slessard commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1753042474 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -262,6 +263,89 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] Bug Fix: Use historical partition field name [iceberg-python]

2024-09-11 Thread via GitHub
ndrluis commented on code in PR #1161: URL: https://github.com/apache/iceberg-python/pull/1161#discussion_r1754541566 ## pyiceberg/table/update/spec.py: ## @@ -280,7 +280,7 @@ def _partition_field(self, transform_key: Tuple[int, Transform[Any, Any]], name: for fiel

Re: [PR] refactor(python): Expose transform as a submodule for pyiceberg_core [iceberg-rust]

2024-09-11 Thread via GitHub
Xuanwo commented on PR #628: URL: https://github.com/apache/iceberg-rust/pull/628#issuecomment-2343720482 Hi, @sungwy, would you like to take a review? I believe this way is more pythonic. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] Why call deleteKey for Insert and Update After in Flink BaseDeltaTaskWriter? [iceberg]

2024-09-11 Thread via GitHub
pvary commented on issue #11081: URL: https://github.com/apache/iceberg/issues/11081#issuecomment-2343745305 @SML0127: I would suggest to write some unit test, or chose an existing one to test the behaviour. I have not seen complains about this yet. -- This is an automated message fro

Re: [PR] fix: SIGSEGV when describe empty table [iceberg-go]

2024-09-11 Thread via GitHub
zeroshade commented on PR #145: URL: https://github.com/apache/iceberg-go/pull/145#issuecomment-2343773844 @alex-kar we could probably leverage "testable examples" https://go.dev/blog/examples with go and just use `// Output:` stuff to point out our expected output. -- This is an automat

Re: [PR] Kafka Connect: separate CI workflow [iceberg]

2024-09-11 Thread via GitHub
bryanck commented on code in PR #11075: URL: https://github.com/apache/iceberg/pull/11075#discussion_r1754685987 ## kafka-connect/kafka-connect-runtime/src/integration/java/org/apache/iceberg/connect/TestContext.java: ## @@ -51,6 +52,7 @@ public class TestContext { private Te

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1754684549 ## open-api/rest-catalog-open-api.yaml: ## @@ -2747,6 +2747,81 @@ components: uuid: type: string +ADLSCredentials: + type: object +

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1754694285 ## open-api/rest-catalog-open-api.yaml: ## @@ -2747,6 +2747,81 @@ components: uuid: type: string +ADLSCredentials: + type: object +

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1754684549 ## open-api/rest-catalog-open-api.yaml: ## @@ -2747,6 +2747,81 @@ components: uuid: type: string +ADLSCredentials: + type: object +

Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]

2024-09-11 Thread via GitHub
JanKrl commented on issue #1093: URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2343901577 > Adding some context to the Avro decoder build process. > > We use Poetry to build the Avro decoder via this script > > https://github.com/apache/iceberg-python/

Re: [PR] OpenAPI: Use %2E as namespace separator instead of %1F [iceberg]

2024-09-11 Thread via GitHub
nastra closed pull request #10839: OpenAPI: Use %2E as namespace separator instead of %1F URL: https://github.com/apache/iceberg/pull/10839 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] OpenAPI: Use %2E as namespace separator instead of %1F [iceberg]

2024-09-11 Thread via GitHub
nastra commented on PR #10839: URL: https://github.com/apache/iceberg/pull/10839#issuecomment-2343901990 closing this in favor of https://github.com/apache/iceberg/pull/10877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-09-11 Thread via GitHub
ajantha-bhat commented on PR #10176: URL: https://github.com/apache/iceberg/pull/10176#issuecomment-2343906548 > I just learned that there is an ongoing effort to support writes with the internal Iceberg object model, which will nicely solve this problem for us (given that it is already in

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-09-11 Thread via GitHub
jwtryg commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1754878148 ## catalog/catalog.go: ## @@ -146,9 +147,9 @@ type Catalog interface { // identifiers containing the information required to load the table via that catalog.

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-09-11 Thread via GitHub
jwtryg commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1754883143 ## catalog/catalog.go: ## @@ -146,9 +147,9 @@ type Catalog interface { // identifiers containing the information required to load the table via that catalog.

Re: [I] Add support for building runtime jars containing the sources [iceberg]

2024-09-11 Thread via GitHub
jshmchenxi closed issue #1865: Add support for building runtime jars containing the sources URL: https://github.com/apache/iceberg/issues/1865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] API, Core: Add manifestPath API to ContentFile which will return the path to a manifest from which the content file resides in [iceberg]

2024-09-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #11044: URL: https://github.com/apache/iceberg/pull/11044#discussion_r1754922979 ## core/src/main/java/org/apache/iceberg/InheritableMetadataFactory.java: ## @@ -46,11 +49,14 @@ static class BaseInheritableMetadata implements Inheritable

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-09-11 Thread via GitHub
jwtryg commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1754949535 ## partitions.go: ## @@ -117,6 +117,10 @@ func (ps PartitionSpec) Equals(other PartitionSpec) bool { return ps.id == other.id && slices.Equal(ps.fields, other.f

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-09-11 Thread via GitHub
jwtryg commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1754951127 ## table/metadata.go: ## @@ -80,20 +86,512 @@ type Metadata interface { SnapshotByName(name string) *Snapshot // CurrentSnapshot returns the table's cur

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-09-11 Thread via GitHub
jwtryg commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1754957335 ## table/metadata.go: ## @@ -80,20 +86,512 @@ type Metadata interface { SnapshotByName(name string) *Snapshot // CurrentSnapshot returns the table's cur

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-09-11 Thread via GitHub
jwtryg commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1754962455 ## table/metadata.go: ## @@ -80,20 +86,512 @@ type Metadata interface { SnapshotByName(name string) *Snapshot // CurrentSnapshot returns the table's cur

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-09-11 Thread via GitHub
jwtryg commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1754963661 ## table/metadata.go: ## @@ -80,20 +86,512 @@ type Metadata interface { SnapshotByName(name string) *Snapshot // CurrentSnapshot returns the table's cur

Re: [PR] refactor(python): Expose transform as a submodule for pyiceberg_core [iceberg-rust]

2024-09-11 Thread via GitHub
Xuanwo commented on PR #628: URL: https://github.com/apache/iceberg-rust/pull/628#issuecomment-2344001364 > There's an outstanding issue with pyo3 that prevents the functions defined within a module to be imported like below: Oh, I see the problem. Let me take a look. -- This is an

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-09-11 Thread via GitHub
jwtryg commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1754973597 ## table/metadata.go: ## @@ -370,26 +876,26 @@ func (c *commonMetadata) validate() error { func (c *commonMetadata) Version() int { return c.FormatVersion } -type

Re: [PR] refactor(python): Expose transform as a submodule for pyiceberg_core [iceberg-rust]

2024-09-11 Thread via GitHub
Xuanwo commented on PR #628: URL: https://github.com/apache/iceberg-rust/pull/628#issuecomment-2344019903 Hi, @sungwy, I fixed this issue and add a test for it. Please take a look, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] KC Integration tests occasionally fail [iceberg]

2024-09-11 Thread via GitHub
nastra closed issue #1: KC Integration tests occasionally fail URL: https://github.com/apache/iceberg/issues/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Kafka Connect: separate CI workflow [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #11075: URL: https://github.com/apache/iceberg/pull/11075#discussion_r1755050156 ## kafka-connect/kafka-connect-runtime/src/integration/java/org/apache/iceberg/connect/TestContext.java: ## @@ -51,6 +52,7 @@ public class TestContext { private Tes

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.60.1 to 1.61.2 [iceberg-go]

2024-09-11 Thread via GitHub
Xuanwo merged PR #144: URL: https://github.com/apache/iceberg-go/pull/144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.30.4 to 1.30.5 [iceberg-go]

2024-09-11 Thread via GitHub
nastra commented on PR #141: URL: https://github.com/apache/iceberg-go/pull/141#issuecomment-2344084492 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.30.4 to 1.30.5 [iceberg-go]

2024-09-11 Thread via GitHub
dependabot[bot] commented on PR #141: URL: https://github.com/apache/iceberg-go/pull/141#issuecomment-2344084514 Looks like github.com/aws/aws-sdk-go-v2 is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.30.4 to 1.30.5 [iceberg-go]

2024-09-11 Thread via GitHub
dependabot[bot] commented on PR #141: URL: https://github.com/apache/iceberg-go/pull/141#issuecomment-2344084630 Looks like this PR is closed. If you re-open it, I'll rebase it, as long as no-one else has edited it. -- This is an automated message from the Apache Git Service. To respo

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.30.4 to 1.30.5 [iceberg-go]

2024-09-11 Thread via GitHub
dependabot[bot] closed pull request #141: build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.30.4 to 1.30.5 URL: https://github.com/apache/iceberg-go/pull/141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.30 to 1.27.33 [iceberg-go]

2024-09-11 Thread via GitHub
Xuanwo merged PR #140: URL: https://github.com/apache/iceberg-go/pull/140 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.29 to 1.17.32 [iceberg-go]

2024-09-11 Thread via GitHub
dependabot[bot] closed pull request #143: build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.29 to 1.17.32 URL: https://github.com/apache/iceberg-go/pull/143 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.29 to 1.17.32 [iceberg-go]

2024-09-11 Thread via GitHub
dependabot[bot] commented on PR #143: URL: https://github.com/apache/iceberg-go/pull/143#issuecomment-2344096285 Looks like github.com/aws/aws-sdk-go-v2/credentials is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Spark 3.5: Don't change table distribution when only altering local order [iceberg]

2024-09-11 Thread via GitHub
szehon-ho commented on code in PR #10774: URL: https://github.com/apache/iceberg/pull/10774#discussion_r1755110580 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala: ## @@ -226,11 +226,13 @@ class I

Re: [PR] Spark: Add RewriteTablePath action interface [iceberg]

2024-09-11 Thread via GitHub
nastra commented on code in PR #10920: URL: https://github.com/apache/iceberg/pull/10920#discussion_r1755114815 ## core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePathActionResult.java: ## @@ -0,0 +1,57 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Spark 3.5: Don't change table distribution when only altering local order [iceberg]

2024-09-11 Thread via GitHub
szehon-ho commented on code in PR #10774: URL: https://github.com/apache/iceberg/pull/10774#discussion_r1755120322 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala: ## @@ -226,11 +226,13 @@ class I

Re: [PR] build(deps): bump github.com/hamba/avro/v2 from 2.23.0 to 2.25.1 [iceberg-go]

2024-09-11 Thread via GitHub
Xuanwo commented on PR #142: URL: https://github.com/apache/iceberg-go/pull/142#issuecomment-2344131523 Hi, @zeroshade, the tests failed for: ```shell === RUN TestScanner/test_partitioned_by_identity_GreaterThanEqual(term=Reference(name='ts'),_literal=2023-03-05T00:00:00+00:00)

Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu commented on issue #1093: URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2344154285 Great to hear. did you clean the env manually or use `make clean`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]

2024-09-11 Thread via GitHub
JanKrl commented on issue #1093: URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2344182977 > Great to hear. did you clean the env manually or use `make clean`? I use venv, so I removed `.venv` directory and created it again. -- This is an automated message f

Re: [PR] refactor(python): Expose transform as a submodule for pyiceberg_core [iceberg-rust]

2024-09-11 Thread via GitHub
kevinjqliu commented on code in PR #628: URL: https://github.com/apache/iceberg-rust/pull/628#discussion_r1755178187 ## bindings/python/src/transform.rs: ## @@ -15,24 +15,55 @@ // specific language governing permissions and limitations // under the License. +use arrow::array

Re: [I] `ALTER TABLE ... DROP COLUMN` allows dropping a column used by old PartitionSpecs [iceberg]

2024-09-11 Thread via GitHub
hashhar commented on issue #4563: URL: https://github.com/apache/iceberg/issues/4563#issuecomment-2344223755 cc: @rdblue Seems this is important. See https://github.com/apache/iceberg/pull/5707#issuecomment-1980199024 for why this is important. -- This is an automated message from the Ap

Re: [I] Mixed usage of snapshotCreationTs, metadataCommitTs & tableAccessTs when using REST Catalog [iceberg]

2024-09-11 Thread via GitHub
haizhou-zhao commented on issue #11103: URL: https://github.com/apache/iceberg/issues/11103#issuecomment-2344239473 I realized I've been putting many thought threads that doesn't directly related to this problem, making it hard for people to read. The problem really is: the current R

[I] [bug?] [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu opened a new issue, #1162: URL: https://github.com/apache/iceberg-python/issues/1162 ### Apache Iceberg version main (development) ### Please describe the bug 🐞 On Mac, anyone having issue running `make test-integration` on the current `main` branch? I

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-09-11 Thread via GitHub
flyrain commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1755210069 ## open-api/rest-catalog-open-api.yaml: ## @@ -3103,6 +3103,81 @@ components: uuid: type: string +ADLSCredentials: + type: object +

Re: [PR] refactor(python): Expose transform as a submodule for pyiceberg_core [iceberg-rust]

2024-09-11 Thread via GitHub
Xuanwo commented on code in PR #628: URL: https://github.com/apache/iceberg-rust/pull/628#discussion_r1755216295 ## bindings/python/src/transform.rs: ## @@ -15,24 +15,55 @@ // specific language governing permissions and limitations // under the License. +use arrow::array::{m

Re: [PR] Remove `initial_change` when dealing with table updates [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu commented on PR #950: URL: https://github.com/apache/iceberg-python/pull/950#issuecomment-2344286460 @HonahX do you mind taking a look at this when you get a chance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] refactor(python): Expose transform as a submodule for pyiceberg_core [iceberg-rust]

2024-09-11 Thread via GitHub
kevinjqliu commented on code in PR #628: URL: https://github.com/apache/iceberg-rust/pull/628#discussion_r1755222868 ## bindings/python/src/transform.rs: ## @@ -15,24 +15,55 @@ // specific language governing permissions and limitations // under the License. +use arrow::array

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-09-11 Thread via GitHub
dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1755259576 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return Closeabl

Re: [PR] refactor(python): Expose transform as a submodule for pyiceberg_core [iceberg-rust]

2024-09-11 Thread via GitHub
Xuanwo commented on PR #628: URL: https://github.com/apache/iceberg-rust/pull/628#issuecomment-2344348879 Thank you @sungwy and @kevinjqliu for the quick review. We can implement more transforms in following PRs. And I will merge this one first 😘 -- This is an automated message from the A

Re: [PR] refactor(python): Expose transform as a submodule for pyiceberg_core [iceberg-rust]

2024-09-11 Thread via GitHub
Xuanwo merged PR #628: URL: https://github.com/apache/iceberg-rust/pull/628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Spark 3.4: Action to compute table stats [iceberg]

2024-09-11 Thread via GitHub
huaxingao commented on code in PR #11106: URL: https://github.com/apache/iceberg/pull/11106#discussion_r1755274103 ## spark/v3.3/build.gradle: ## @@ -59,6 +59,7 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") { implementation project(':icebe

Re: [PR] scan: fix error when reading an empty table [iceberg-rust]

2024-09-11 Thread via GitHub
sdd commented on PR #608: URL: https://github.com/apache/iceberg-rust/pull/608#issuecomment-2344385358 We've been very selective when it comes to adding new values to `ErrorKind`. I'd personally go for `Unexpected` here - but maybe @liurenjie1024 or @Xuanwo can confirm what would be best.

Re: [I] [bug?] cannot run integration test [iceberg-python]

2024-09-11 Thread via GitHub
soumya-ghosh commented on issue #1162: URL: https://github.com/apache/iceberg-python/issues/1162#issuecomment-2344408943 Been facing the same issue since yesterday when I rebased latest main branch to my dev branch. Just realized that tests are not running on main branch itself. I did

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2024-09-11 Thread via GitHub
hesham-medhat commented on code in PR #11039: URL: https://github.com/apache/iceberg/pull/11039#discussion_r1755319360 ## build.gradle: ## @@ -647,20 +647,43 @@ project(':iceberg-gcp') { implementation project(':iceberg-common') implementation project(':iceberg-core')

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2024-09-11 Thread via GitHub
hesham-medhat commented on code in PR #11039: URL: https://github.com/apache/iceberg/pull/11039#discussion_r1755322285 ## .gitignore: ## @@ -66,3 +66,6 @@ metastore_db/ # Spark/metastore files spark-warehouse/ derby.log + +# BigQuery/metastore files +gcp/db_folder/ Review Co

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2024-09-11 Thread via GitHub
hesham-medhat commented on code in PR #11039: URL: https://github.com/apache/iceberg/pull/11039#discussion_r1755320886 ## build.gradle: ## @@ -647,20 +647,43 @@ project(':iceberg-gcp') { implementation project(':iceberg-common') implementation project(':iceberg-core')

Re: [PR] Remove unnecessary _ensure_tables_exist method [iceberg-python]

2024-09-11 Thread via GitHub
isc-patrick commented on PR #1155: URL: https://github.com/apache/iceberg-python/pull/1155#issuecomment-2344422315 I can certainly add tests, but that is really testing the Metadata.create_all() function in SQLAlchemy and not pyiceberg code. I think that CREATE TABLE IF NOT EXISTS requires

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-09-11 Thread via GitHub
aniket-s-kulkarni commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1755358145 ## open-api/rest-catalog-open-api.yaml: ## @@ -3103,6 +3103,81 @@ components: uuid: type: string +ADLSCredentials: + type: ob

[I] [BUG] `Catalog.list_tables()` inconsistency between docstring and signature [iceberg-python]

2024-09-11 Thread via GitHub
dataders opened a new issue, #1163: URL: https://github.com/apache/iceberg-python/issues/1163 ### Apache Iceberg version 0.7.1 (latest release) ### Please describe the bug 🐞 the docstring for `catalog.list_tables()` says > If namespace not provided, will list all table

Re: [PR] API, Core: Add manifestLocation API to ContentFile which will return the path to a manifest from which the content file resides in [iceberg]

2024-09-11 Thread via GitHub
rdblue commented on code in PR #11044: URL: https://github.com/apache/iceberg/pull/11044#discussion_r1755367412 ## core/src/main/java/org/apache/iceberg/BaseFile.java: ## @@ -240,6 +242,10 @@ public void setDataSequenceNumber(Long dataSequenceNumber) { this.dataSequenceNum

Re: [I] Cannot parse missing long: current-snapshot-id [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu closed issue #1164: Cannot parse missing long: current-snapshot-id URL: https://github.com/apache/iceberg-python/issues/1164 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Cannot parse missing long: current-snapshot-id [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu commented on issue #1164: URL: https://github.com/apache/iceberg-python/issues/1164#issuecomment-2344451634 Also from slack, Sung: This is actually an issue with some of the older Java applications making the incorrect assumption that the current_snapshot_id is a requir

[I] Cannot parse missing long: current-snapshot-id [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu opened a new issue, #1164: URL: https://github.com/apache/iceberg-python/issues/1164 ### Apache Iceberg version None ### Please describe the bug 🐞 From slack, Hi! We’ve noticed that after creating an empty table in the Glue catalog with PyIceberg, init

Re: [I] Spark configuration for amazon access key and secret key with glue catalog for apache Iceberg is not honoring [iceberg]

2024-09-11 Thread via GitHub
HemantMarve commented on issue #10078: URL: https://github.com/apache/iceberg/issues/10078#issuecomment-2344456352 @clamar14 , @nastra Try after adding below property: `.config("spark.sql.catalog.AwsDataCatalog.client.region","us-south")` `.config("spark.sql.catalog.your_catalog

Re: [I] [BUG] `Catalog.list_tables()` inconsistency between docstring and signature [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu commented on issue #1163: URL: https://github.com/apache/iceberg-python/issues/1163#issuecomment-2344471589 Thanks for reporting this! I noticed it too when reviewing #1140. I think the spec only allows listing tables in a namespace. https://github.com/apache/iceberg/

Re: [I] [bug?] cannot run integration test [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu commented on issue #1162: URL: https://github.com/apache/iceberg-python/issues/1162#issuecomment-2344477343 Thanks for confirming. I see the same issue https://gist.github.com/kevinjqliu/c8310b6253beab52cce93391df03bfe4 And only for commits at and after `1971fcfe0875eeb200

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-09-11 Thread via GitHub
dimas-b commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1755413798 ## open-api/rest-catalog-open-api.yaml: ## @@ -3129,6 +3204,11 @@ components: - `s3.secret-access-key`: secret for credentials that provide access to data

Re: [PR] DRAFT - Issue 10275 - Reward support for nulls [iceberg]

2024-09-11 Thread via GitHub
slessard commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1720293723 ## arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java: ## @@ -262,6 +263,89 @@ public void testReadColumnFilter2() throws Exception {

Re: [PR] OpenAPI: Standardize credentials in loadTable/loadView responses [iceberg]

2024-09-11 Thread via GitHub
dimas-b commented on code in PR #10722: URL: https://github.com/apache/iceberg/pull/10722#discussion_r1755423596 ## open-api/rest-catalog-open-api.yaml: ## @@ -3129,6 +3204,11 @@ components: - `s3.secret-access-key`: secret for credentials that provide access to data

Re: [PR] Remove unnecessary _ensure_tables_exist method [iceberg-python]

2024-09-11 Thread via GitHub
kevinjqliu commented on PR #1155: URL: https://github.com/apache/iceberg-python/pull/1155#issuecomment-2344509223 please do! we want to ensure that this change does not break new and existing DB integrations. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] DRAFT - Issue 10275 - Add support for null vectors [iceberg]

2024-09-11 Thread via GitHub
slessard commented on PR #10953: URL: https://github.com/apache/iceberg/pull/10953#issuecomment-2344515203 > @sl255051 could you please update the PR title to reflect the changes that are being done here? I updated the PR title and description -- This is an automated message from t

  1   2   3   >