Re: [PR] Core: Support IncrementalChangelogScan with deletes. [iceberg]

2024-03-06 Thread via GitHub
manuzhang commented on PR #6182: URL: https://github.com/apache/iceberg/pull/6182#issuecomment-1982827028 I just submitted https://github.com/apache/iceberg/pull/9888 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Core: Support IncrementalChangelogScan with deletes [iceberg]

2024-03-06 Thread via GitHub
manuzhang opened a new pull request, #9888: URL: https://github.com/apache/iceberg/pull/9888 Pick up #6182 with an alternative implementation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9878: URL: https://github.com/apache/iceberg/pull/9878#discussion_r1515696765 ## docs/docs/spark-ddl.md: ## @@ -566,3 +566,85 @@ Tags can be removed via the `DROP TAG` sql ```sql ALTER TABLE prod.db.sample DROP TAG `historical-tag` ``` +

Re: [PR] Implement basic Parquet data file reading capability [iceberg-rust]

2024-03-06 Thread via GitHub
sdd commented on PR #207: URL: https://github.com/apache/iceberg-rust/pull/207#issuecomment-1982809816 @Fokko are you able to take a look at this one yet please? 😁 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9878: URL: https://github.com/apache/iceberg/pull/9878#discussion_r1515692131 ## docs/docs/spark-ddl.md: ## @@ -566,3 +566,85 @@ Tags can be removed via the `DROP TAG` sql ```sql ALTER TABLE prod.db.sample DROP TAG `historical-tag` ``` + +### I

Re: [I] Spark query pushdown error in SQL query optimization phase [iceberg]

2024-03-06 Thread via GitHub
burriad closed issue #9881: Spark query pushdown error in SQL query optimization phase URL: https://github.com/apache/iceberg/issues/9881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Spark query pushdown error in SQL query optimization phase [iceberg]

2024-03-06 Thread via GitHub
burriad commented on issue #9881: URL: https://github.com/apache/iceberg/issues/9881#issuecomment-1982761923 Sorry, I tried to provide a stripped down version of the code but I cannot reproduce it anymore. -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Allow setting non-string typed values in `set_properties` [iceberg-python]

2024-03-06 Thread via GitHub
HonahX commented on code in PR #504: URL: https://github.com/apache/iceberg-python/pull/504#discussion_r1515657781 ## pyiceberg/table/__init__.py: ## @@ -294,17 +295,21 @@ def upgrade_table_version(self, format_version: Literal[1, 2]) -> Transaction: return self -

[PR] Adjust site links to absolute from site_url [iceberg]

2024-03-06 Thread via GitHub
bitsondatadev opened a new pull request, #9887: URL: https://github.com/apache/iceberg/pull/9887 Avoid internal [MkDocs relative linking](https://www.mkdocs.org/user-guide/writing-your-docs/#linking-to-pages) using the [mkdocs-site-urls](https://github.com/OctoPrint/mkdocs-site-urls) plugi

Re: [PR] fix: Remove deprecated methods to pass ci [iceberg-rust]

2024-03-06 Thread via GitHub
Fokko merged PR #234: URL: https://github.com/apache/iceberg-rust/pull/234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9878: URL: https://github.com/apache/iceberg/pull/9878#discussion_r1515655248 ## docs/docs/spark-ddl.md: ## @@ -566,3 +566,85 @@ Tags can be removed via the `DROP TAG` sql ```sql ALTER TABLE prod.db.sample DROP TAG `historical-tag` ``` + +### I

Re: [PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9878: URL: https://github.com/apache/iceberg/pull/9878#discussion_r1515645687 ## docs/docs/spark-ddl.md: ## @@ -566,3 +566,85 @@ Tags can be removed via the `DROP TAG` sql ```sql ALTER TABLE prod.db.sample DROP TAG `historical-tag` ``` + +### I

Re: [PR] fix: Remove deprecated methods to pass ci [iceberg-rust]

2024-03-06 Thread via GitHub
liurenjie1024 commented on PR #234: URL: https://github.com/apache/iceberg-rust/pull/234#issuecomment-1982616229 cc @Xuanwo @ZENOTME @Fokko PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[PR] fix: Remove deprecated methods to pass ci [iceberg-rust]

2024-03-06 Thread via GitHub
liurenjie1024 opened a new pull request, #234: URL: https://github.com/apache/iceberg-rust/pull/234 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [I] [feature request] easier API to set table properties [iceberg-python]

2024-03-06 Thread via GitHub
HonahX commented on issue #502: URL: https://github.com/apache/iceberg-python/issues/502#issuecomment-1982550509 Hi @kevinjqliu @syun64 I think currently we can also set properties using dictionaries, just need `**` in front of the dictionary: ```python properties: Dict[str, str] = {..

Re: [PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
bitsondatadev commented on code in PR #9878: URL: https://github.com/apache/iceberg/pull/9878#discussion_r1514954244 ## docs/docs/spark-ddl.md: ## @@ -566,3 +566,85 @@ Tags can be removed via the `DROP TAG` sql ```sql ALTER TABLE prod.db.sample DROP TAG `historical-tag` ``` +

Re: [PR] Add support for providing output-spec-id during rewrite - spark 3.5 [iceberg]

2024-03-06 Thread via GitHub
RussellSpitzer commented on PR #9803: URL: https://github.com/apache/iceberg/pull/9803#issuecomment-1982473177 Thanks @himadripal for this great improvement, Our general policy is once one Spark Version is in you open another PR to all of the other currently supported Branches with the same

Re: [PR] Add support for providing output-spec-id during rewrite - spark 3.5 [iceberg]

2024-03-06 Thread via GitHub
RussellSpitzer merged PR #9803: URL: https://github.com/apache/iceberg/pull/9803 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceb

Re: [PR] Add support for providing output-spec-id during rewrite - spark 3.5 [iceberg]

2024-03-06 Thread via GitHub
RussellSpitzer commented on code in PR #9803: URL: https://github.com/apache/iceberg/pull/9803#discussion_r1515587628 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java: ## @@ -1463,6 +1465,130 @@ public void testSnapshotProperty()

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1515586577 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -308,6 +308,10 @@ public List listNamespaces(Namespace namespace) throws NoSuchN

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1515586056 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -380,6 +389,10 @@ public boolean removeProperties(Namespace namespace, Set prope

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-06 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1515539847 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-06 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1515525033 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-06 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1515525033 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Foundati

[PR] Kevinjqliu/set non string properties [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu opened a new pull request, #504: URL: https://github.com/apache/iceberg-python/pull/504 Requires #503 to be merged first. This PR modifies `Transaction`'s `set_properties` API to accept table properties as type `Dict[str, Any]`, either as dictionary or kwargs. For

[PR] [WIP] POC of runtime module [iceberg-rust]

2024-03-06 Thread via GitHub
odysa opened a new pull request, #233: URL: https://github.com/apache/iceberg-rust/pull/233 Create a `runtime` module exposing functions like `spawn` Example usage: ```rust let tasks = manifest_list .entries() .iter() .map(|manife

Re: [PR] Data, Spark, Flink: Migrate tests that depend on TableTestBase to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
nk1506 commented on PR #9862: URL: https://github.com/apache/iceberg/pull/9862#issuecomment-1982292262 > I would suggest to either focus on a subset of tests in `iceberg-core` or on `iceberg-data` @nastra , Most of the changes are coming here because of change in `WriterTestBase` and

[PR] Set table properties with dictionary [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu opened a new pull request, #503: URL: https://github.com/apache/iceberg-python/pull/503 Resolves #502 With this PR, table properties can be set with `set_properties` transaction by passing in a dictionary ``` with table.transaction() as transaction: transactio

Re: [PR] feat: add `UnboundPredicate::negate()` [iceberg-rust]

2024-03-06 Thread via GitHub
liurenjie1024 merged PR #228: URL: https://github.com/apache/iceberg-rust/pull/228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Discuss: Design of `TableMetadataBuilder`. [iceberg-rust]

2024-03-06 Thread via GitHub
liurenjie1024 commented on issue #232: URL: https://github.com/apache/iceberg-rust/issues/232#issuecomment-1982235386 > When `add_schema` has been called, both `schemas` and `current_schema_id` will be updated? The same question to `partition_specs`. Yes, the `add_schema` process cons

Re: [I] Discuss: Design of `TableMetadataBuilder`. [iceberg-rust]

2024-03-06 Thread via GitHub
Xuanwo commented on issue #232: URL: https://github.com/apache/iceberg-rust/issues/232#issuecomment-1982224663 When `add_schema` has been called, both `schemas` and `current_schema_id` will be updated? The same question to `partition_specs`. -- This is an automated message from the Apache

[I] Discuss: Design of `TableMetadataBuilder`. [iceberg-rust]

2024-03-06 Thread via GitHub
liurenjie1024 opened a new issue, #232: URL: https://github.com/apache/iceberg-rust/issues/232 # Problem statement `TableMetadataBuilder` is useful in modifying/creating `TableMetadata`, and is a core data structure of transaction api. There are already some efforts to create one usi

Re: [I] Discuss: Design of `TableMetadataBuilder`. [iceberg-rust]

2024-03-06 Thread via GitHub
liurenjie1024 commented on issue #232: URL: https://github.com/apache/iceberg-rust/issues/232#issuecomment-1982200217 cc @Xuanwo @ZENOTME @Fokko @JanKaul @y0psolo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Move field into place when adding during schema evolution [iceberg]

2024-03-06 Thread via GitHub
hiloboy0119 commented on code in PR #8409: URL: https://github.com/apache/iceberg/pull/8409#discussion_r1515373674 ## core/src/main/java/org/apache/iceberg/schema/UnionByNameVisitor.java: ## @@ -90,6 +94,15 @@ public Boolean struct( Types.NestedField field = field

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
tomtongue commented on PR #9849: URL: https://github.com/apache/iceberg/pull/9849#issuecomment-1982188699 @nastra Thank you! Reflected the minor comment. If there's any part that I should change, please let me know. -- This is an automated message from the Apache Git Service. To respond t

Re: [I] A logo for iceberg rust! [iceberg-rust]

2024-03-06 Thread via GitHub
liurenjie1024 commented on issue #216: URL: https://github.com/apache/iceberg-rust/issues/216#issuecomment-1982178258 Looks cool! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Build: Bump Spark 3.5 to 3.5.1 [iceberg]

2024-03-06 Thread via GitHub
manuzhang commented on PR #9832: URL: https://github.com/apache/iceberg/pull/9832#issuecomment-1982140277 Any more comments for this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] fix minor version for striclty libs versions [iceberg]

2024-03-06 Thread via GitHub
artyrian opened a new pull request, #9886: URL: https://github.com/apache/iceberg/pull/9886 https://github.com/apache/iceberg/pull/9484 and there is also a confilict with jackson 2.15.4 and spark 3.5 on gradle 8.4: `java.lang.IllegalArgumentException: Unsupported class file maj

Re: [PR] revert apache 9726 [iceberg]

2024-03-06 Thread via GitHub
artyrian closed pull request #9882: revert apache 9726 URL: https://github.com/apache/iceberg/pull/9882 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Spark <> Iceberg bug integration test [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar closed issue #482: Spark <> Iceberg bug integration test URL: https://github.com/apache/iceberg-python/issues/482 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar merged PR #501: URL: https://github.com/apache/iceberg-python/pull/501 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr.

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar commented on PR #501: URL: https://github.com/apache/iceberg-python/pull/501#issuecomment-1982123757 Sweet, thanks @kevinjqliu! I'm going to go ahead and merge this now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [I] [feature request] easier API to set table properties [iceberg-python]

2024-03-06 Thread via GitHub
syun64 commented on issue #502: URL: https://github.com/apache/iceberg-python/issues/502#issuecomment-1982120843 I think this is a great idea @kevinjqliu . Most of the table properties have `.` in the key anyways, and hence we can't pass them as kwargs anyways. Changing the input parameter

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
syun64 commented on PR #501: URL: https://github.com/apache/iceberg-python/pull/501#issuecomment-1982096927 Great idea @kevinjqliu ! Thanks for adding this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-06 Thread via GitHub
stevenzwu commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1515180358 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,368 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Move field into place when adding during schema evolution [iceberg]

2024-03-06 Thread via GitHub
hiloboy0119 commented on PR #8409: URL: https://github.com/apache/iceberg/pull/8409#issuecomment-1982085238 @amogh-jahagirdar sorry for the long delay. Finally had time to write tests for the various edge cases and fix a few things. The examples of schema evolution are all shown in t

Re: [I] Flink Table Without Timestamp Cannot Query By Spark [iceberg]

2024-03-06 Thread via GitHub
github-actions[bot] commented on issue #1914: URL: https://github.com/apache/iceberg/issues/1914#issuecomment-1982079679 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] The database deletion operation was performed, but the hdfs file directory was not deleted. [iceberg]

2024-03-06 Thread via GitHub
github-actions[bot] commented on issue #1886: URL: https://github.com/apache/iceberg/issues/1886#issuecomment-1982079661 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on code in PR #501: URL: https://github.com/apache/iceberg-python/pull/501#discussion_r1515301588 ## tests/integration/test_writes.py: ## @@ -355,6 +355,26 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert [row.

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #501: URL: https://github.com/apache/iceberg-python/pull/501#discussion_r1515300598 ## tests/integration/test_writes.py: ## @@ -355,6 +355,26 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar commented on PR #501: URL: https://github.com/apache/iceberg-python/pull/501#issuecomment-1982063186 Thanks @kevinjqliu I think this change makes sense. I don't think there's ever a reason on the Python side where we want to have the spark caching enabled. On the Iceberg Ja

Re: [PR] Bin-pack Writes Operation into multiple parquet files, and parallelize writing `WriteTask`s [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on PR #444: URL: https://github.com/apache/iceberg-python/pull/444#issuecomment-1982045743 thanks for the review @HonahX. I've rebased off `main` and addressed your comments. I also added more tests after figuring out the fix for #482 -- This is an automated m

Re: [I] A logo for iceberg rust! [iceberg-rust]

2024-03-06 Thread via GitHub
caldempsey commented on issue #216: URL: https://github.com/apache/iceberg-rust/issues/216#issuecomment-1982004560 ![image](https://github.com/apache/iceberg-rust/assets/8885269/ea2a8b46-15fb-4aba-9a06-49c58bc0b821) -- This is an automated message from the Apache Git Service. To respo

Re: [I] A logo for iceberg rust! [iceberg-rust]

2024-03-06 Thread via GitHub
caldempsey commented on issue #216: URL: https://github.com/apache/iceberg-rust/issues/216#issuecomment-1982002725 I'd prefer a letter I with the RUST gears surrounding it ⚙️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-06 Thread via GitHub
fqaiser94 commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1515260098 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Founda

Re: [I] [feature request] easier API to set table properties [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on issue #502: URL: https://github.com/apache/iceberg-python/issues/502#issuecomment-1981976531 Another workaround can be to use ``` with tbl.transaction() as transaction: transaction._apply((SetPropertiesUpdate(updates=properties),)) ``` -- This i

Re: [PR] Bin-pack Writes Operation into multiple parquet files, and parallelize writing `WriteTask`s [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on code in PR #444: URL: https://github.com/apache/iceberg-python/pull/444#discussion_r1515227170 ## tests/integration/test_writes.py: ## @@ -424,6 +424,30 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert [row.

Re: [I] Can't import pyiceberg.catalog [iceberg-python]

2024-03-06 Thread via GitHub
leonidmakarovsky commented on issue #500: URL: https://github.com/apache/iceberg-python/issues/500#issuecomment-1981884781 My python version is 3.9.7. My colleague told me that pyiceberg works with 3.9.11 and later. It didn't work with 3.9.7 in his env as well. I'm trying to upgrade Python

Re: [I] Can't import pyiceberg.catalog [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on issue #500: URL: https://github.com/apache/iceberg-python/issues/500#issuecomment-1981880634 seems like an issue related to `pydantic` library. What version of it do you have installed? Here's my env setup using `make install`: ``` (venv) ➜ iceberg-pyth

[PR] Core: Mark 502 and 504 failures as retryable to the exponential retry strategy [iceberg]

2024-03-06 Thread via GitHub
amogh-jahagirdar opened a new pull request, #9885: URL: https://github.com/apache/iceberg/pull/9885 Bad gateway or gateway timeouts should be able to be safely retried. A bad gateway could be caused by a load balancer or proxy between the client and the actual server, when there's some netw

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
jaychia commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1515142038 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft Review Comment: I was able to successfully build the site with `mkdocs serve`! ``

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
jaychia commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1515139832 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft + +[Daft](www.getdaft.io) is a distributed query engine written in Python and Rust, two

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
jaychia commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1515138488 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft + +[Daft](www.getdaft.io) is a distributed query engine written in Python and Rust, two

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on PR #9839: URL: https://github.com/apache/iceberg/pull/9839#issuecomment-1981669062 Does all of these optional parameters need to be added in the response as well? @flyrain -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1514845097 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -178,15 +178,18 @@ public void initialize(String name, Map unresolved) { ConfigRes

[I] Can't import pyiceberg.catalog [iceberg-python]

2024-03-06 Thread via GitHub
leonidmakarovsky opened a new issue, #500: URL: https://github.com/apache/iceberg-python/issues/500 ### Apache Iceberg version 0.6.0 (latest release) ### Please describe the bug 🐞 Running on Mac OS Monterey 12.6.7, Spyder IDE (Python 3.9.7 64-bit | Qt 5.9.7 | PyQt5 5.9.2

Re: [I] Make the OAuth2 request audience configurable [iceberg-python]

2024-03-06 Thread via GitHub
flyrain commented on issue #479: URL: https://github.com/apache/iceberg-python/issues/479#issuecomment-1981523086 This is fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [I] Make the OAuth2 request audience configurable [iceberg-python]

2024-03-06 Thread via GitHub
flyrain closed issue #479: Make the OAuth2 request audience configurable URL: https://github.com/apache/iceberg-python/issues/479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Make optional oauth configurable [iceberg-python]

2024-03-06 Thread via GitHub
flyrain merged PR #486: URL: https://github.com/apache/iceberg-python/pull/486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1514955976 ## aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java: ## @@ -207,7 +212,13 @@ private AuthSession authSession() { t

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514932065 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -122,11 +124,20 @@ public void initialize(Map newProperties) { close(); // close and disc

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514931349 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -219,7 +230,7 @@ DelegateFileIO io(String location) { @VisibleForTesting String implFro

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514921889 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -122,11 +124,20 @@ public void initialize(Map newProperties) { close(); // close and

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514921889 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -122,11 +124,20 @@ public void initialize(Map newProperties) { close(); // close and

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-06 Thread via GitHub
zachdisc commented on PR #9731: URL: https://github.com/apache/iceberg/pull/9731#issuecomment-1981412777 R4 Renamed `sort` to `clusterBy` - more honest and aligns with the core API as well. Have not made progress on using the custom UDF for both standard partition column ord

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1514880737 ## core/src/test/java/org/apache/iceberg/TestFindFiles.java: ## @@ -184,33 +181,33 @@ public void testCaseSensitivity() { .withMetadataMatching(Expressions.

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514874407 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -219,7 +230,7 @@ DelegateFileIO io(String location) { @VisibleForTesting String imp

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514871547 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -219,7 +230,7 @@ DelegateFileIO io(String location) { @VisibleForTesting String imp

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
tomtongue commented on PR #9849: URL: https://github.com/apache/iceberg/pull/9849#issuecomment-1981376570 @nastra Update with your comments. When you have time, could you review the new commit? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1514845097 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -178,15 +178,18 @@ public void initialize(String name, Map unresolved) { ConfigRes

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1514845097 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -178,15 +178,18 @@ public void initialize(String name, Map unresolved) { ConfigRes

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1981347201 @ajantha-bhat I'm fixing yeah. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1981329539 Testcase failed, probably because of `schemeToFileIo.clear(); ` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Make optional oauth configurable [iceberg-python]

2024-03-06 Thread via GitHub
himadripal commented on PR #486: URL: https://github.com/apache/iceberg-python/pull/486#issuecomment-1981317217 @flyrain and @Fokko thank you for reviewing this. Please take a look again when you can. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1981255063 @ajantha-bhat or also new IO implementations, interesting in Europe 😄 (like wasabi, leviia, ovh store, scaleway store, ...) -- This is an automated message from the Apache Git Service

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514759756 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -122,6 +124,15 @@ public void initialize(Map newProperties) { close(); // close and di

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1514762594 ## core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java: ## @@ -1357,27 +1344,27 @@ private void testPositionDeletesBaseTableFilter(boolean transactiona

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1514762022 ## core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java: ## @@ -1357,27 +1344,27 @@ private void testPositionDeletesBaseTableFilter(boolean transactiona

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
adutra commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514756075 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -53,20 +53,22 @@ public class ResolvingFileIO implements HadoopConfigurable, DelegateFileIO {

[PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre opened a new pull request, #9884: URL: https://github.com/apache/iceberg/pull/9884 This PR adds a way to load custom schemes (in addition of the default ones) in `ResolvingFileIO` using properties prefixed with `resolving-io.schemes.`. This closes #9883 -- This is an

Re: [PR] revert apache 9726 [iceberg]

2024-03-06 Thread via GitHub
nastra commented on PR #9882: URL: https://github.com/apache/iceberg/pull/9882#issuecomment-1981000150 @artyrian can you please add some context why this revert is necessary? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] revert apache 9726 [iceberg]

2024-03-06 Thread via GitHub
artyrian opened a new pull request, #9882: URL: https://github.com/apache/iceberg/pull/9882 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [I] Spark query pushdown error in SQL query optimization phase [iceberg]

2024-03-06 Thread via GitHub
RussellSpitzer commented on issue #9881: URL: https://github.com/apache/iceberg/issues/9881#issuecomment-1980957406 Could you share the query as well as the table schema? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514509086 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -308,6 +308,10 @@ public List listNamespaces(Namespace namespace) throws NoSuchNamespa

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514502547 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -380,6 +389,10 @@ public boolean removeProperties(Namespace namespace, Set properties)

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514493828 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -380,6 +389,10 @@ public boolean removeProperties(Namespace namespace, Set prope

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514489991 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -308,6 +308,10 @@ public List listNamespaces(Namespace namespace) throws NoSuchN

[I] Spark query pushdown error in SQL query optimization phase [iceberg]

2024-03-06 Thread via GitHub
burriad opened a new issue, #9881: URL: https://github.com/apache/iceberg/issues/9881 ### Apache Iceberg version 1.4.3 (latest release) ### Query engine Spark ### Please describe the bug 🐞 When upgrading from Iceberg version 1.3.1 to 1.4.3 we get the followi

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
adutra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514477117 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -308,6 +308,10 @@ public List listNamespaces(Namespace namespace) throws NoSuchNamespa

Re: [I] About /v1/config REST API endpoint [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on issue #9880: URL: https://github.com/apache/iceberg/issues/9880#issuecomment-1980862061 maybe cc: @nastra, @danielcweeks, @rdblue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

  1   2   >