Re: [PR] Views, Spark: Add support for Materialized Views; Integrate with Spark SQL [iceberg]

2024-03-13 Thread via GitHub
bennychow commented on code in PR #9830: URL: https://github.com/apache/iceberg/pull/9830#discussion_r1522639054 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -557,6 +568,88 @@ public View loadView(Identifier ident) throws NoSuchViewExcepti

Re: [PR] Add local nightly build to test current docs changes [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9943: URL: https://github.com/apache/iceberg/pull/9943#discussion_r1522639709 ## site/README.md: ## @@ -60,7 +60,12 @@ The static Iceberg site pages are Markdown files that live at `/site/docs/*.md`. ``` ### Building the versioned docs -The I

Re: [PR] Docs: Enhance Spark pages [iceberg]

2024-03-13 Thread via GitHub
nastra commented on PR #9920: URL: https://github.com/apache/iceberg/pull/9920#issuecomment-1993710672 Can we just have one single PR to fix all the broken links for the current version of the docs (`main`)? We would then also have to fix broken links for 1.4.0 / 1.4.1 / 1.4.2 / 1.4.3 / 1.5

Re: [I] Implement `ManifestEvaluator` to prune mainfest files. [iceberg-rust]

2024-03-13 Thread via GitHub
waynexia commented on issue #253: URL: https://github.com/apache/iceberg-rust/issues/253#issuecomment-1993720022 I suppose this issue has been taken according to https://github.com/apache/iceberg-rust/pull/241#issuecomment-1987506207? -- This is an automated message from the Apache Git S

Re: [I] Open-API: How about adding table change detection in Iceberg Catalog [iceberg]

2024-03-13 Thread via GitHub
nastra commented on issue #9942: URL: https://github.com/apache/iceberg/issues/9942#issuecomment-1993720097 There's an option that you can use to only fetch the latest snapshot via `snapshot-loading-mode=refs` (defaults to `all`) when the table is loaded instead of loading all snapshots.

Re: [PR] Website: Add release schedule on the releases page [iceberg]

2024-03-13 Thread via GitHub
jbonofre commented on code in PR #9666: URL: https://github.com/apache/iceberg/pull/9666#discussion_r1522661888 ## site/docs/releases.md: ## @@ -77,7 +77,11 @@ Apache Iceberg 1.4.3 was released on December 27, 2023. The main issue it solves - Core: Expired Snapshot files in a

Re: [I] Implement `ManifestEvaluator` to prune mainfest files. [iceberg-rust]

2024-03-13 Thread via GitHub
waynexia commented on issue #253: URL: https://github.com/apache/iceberg-rust/issues/253#issuecomment-1993720686 I suppose this issue has been taken according to https://github.com/apache/iceberg-rust/pull/241#issuecomment-1987506207? -- This is an automated message from the Apache Git S

Re: [PR] Parquet: Support Page Skipping in Iceberg Parquet Reader [iceberg]

2024-03-13 Thread via GitHub
iflytek-hmwang5 commented on PR #1566: URL: https://github.com/apache/iceberg/pull/1566#issuecomment-1993728664 @shangxinli Hi, any updates on it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Views, Spark: Add support for Materialized Views; Integrate with Spark SQL [iceberg]

2024-03-13 Thread via GitHub
wmoustafa commented on code in PR #9830: URL: https://github.com/apache/iceberg/pull/9830#discussion_r1522670211 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -557,6 +568,88 @@ public View loadView(Identifier ident) throws NoSuchViewExcepti

Re: [PR] Docs: update document release steps [iceberg]

2024-03-13 Thread via GitHub
ajantha-bhat commented on code in PR #9921: URL: https://github.com/apache/iceberg/pull/9921#discussion_r1522674850 ## site/docs/how-to-release.md: ## @@ -311,74 +311,9 @@ Create a PR in the `iceberg` repo to make revapi run on the new release. For an ### Documentation Release

Re: [PR] Python: Add support for Python 3.12 [iceberg-python]

2024-03-13 Thread via GitHub
Fokko commented on PR #35: URL: https://github.com/apache/iceberg-python/pull/35#issuecomment-1993737373 @steinsgateted can you rebase the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] support python 3.12 [iceberg-python]

2024-03-13 Thread via GitHub
Fokko commented on PR #254: URL: https://github.com/apache/iceberg-python/pull/254#issuecomment-1993737812 @MehulBatra can you rebase? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] [Proposal] Iceberg Materialized View Spec [iceberg]

2024-03-13 Thread via GitHub
wmoustafa commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1993743018 Linking the implementation PR: https://github.com/apache/iceberg/pull/9830 for one of the options. It uses the option of view + separate Iceberg table linked from the view metadata

Re: [PR] Basic manifest encryption [iceberg]

2024-03-13 Thread via GitHub
ggershinsky commented on code in PR #8252: URL: https://github.com/apache/iceberg/pull/8252#discussion_r1522685844 ## core/src/main/java/org/apache/iceberg/ManifestWriter.java: ## @@ -52,13 +56,14 @@ public abstract class ManifestWriter> implements FileAp private long deleted

Re: [PR] Sql catalog [iceberg-rust]

2024-03-13 Thread via GitHub
JanKaul commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1522705902 ## crates/catalog/sql/src/catalog.rs: ## @@ -0,0 +1,397 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Basic manifest encryption [iceberg]

2024-03-13 Thread via GitHub
ggershinsky commented on code in PR #8252: URL: https://github.com/apache/iceberg/pull/8252#discussion_r1522695327 ## core/src/main/java/org/apache/iceberg/ManifestWriter.java: ## @@ -289,8 +294,11 @@ protected ManifestContent content() { static class V1Writer extends Manifes

Re: [PR] Basic manifest encryption [iceberg]

2024-03-13 Thread via GitHub
ggershinsky commented on code in PR #8252: URL: https://github.com/apache/iceberg/pull/8252#discussion_r1522717043 ## core/src/main/java/org/apache/iceberg/ManifestWriter.java: ## @@ -289,8 +294,11 @@ protected ManifestContent content() { static class V1Writer extends Manifes

Re: [PR] Docs: Enhance Spark pages [iceberg]

2024-03-13 Thread via GitHub
manuzhang commented on PR #9920: URL: https://github.com/apache/iceberg/pull/9920#issuecomment-1993781522 Yes, after merging in #9943 and #9939, I will submit a fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] [docs] Fix broken links to config reference [iceberg]

2024-03-13 Thread via GitHub
manuzhang commented on PR #9939: URL: https://github.com/apache/iceberg/pull/9939#issuecomment-1993782090 @nastra please help approve the workflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] reset catalog can't reslove s3 domain name [iceberg]

2024-03-13 Thread via GitHub
sg-c0de commented on issue #7709: URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1993790181 > Is there anyone who solved this issue? Try to add env variable to the rest container: CATALOG_S3_PATH__STYLE__ACCESS: true It'll be [converted](https://github.com/tab

Re: [PR] Improve the InMemory Catalog Implementation [iceberg-python]

2024-03-13 Thread via GitHub
Fokko merged PR #289: URL: https://github.com/apache/iceberg-python/pull/289 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-03-13 Thread via GitHub
Fokko commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1522734243 ## pyiceberg/table/__init__.py: ## @@ -1091,7 +,7 @@ def append(self, df: pa.Table) -> None: _check_schema(self.schema(), other_schema=df.schema)

Re: [PR] Sql catalog [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1522752571 ## crates/catalog/sql/src/catalog.rs: ## @@ -0,0 +1,397 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agree

Re: [I] Implement `ManifestEvaluator` to prune mainfest files. [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on issue #253: URL: https://github.com/apache/iceberg-rust/issues/253#issuecomment-1993820561 > I suppose this issue has been taken according to [#241 (comment)](https://github.com/apache/iceberg-rust/pull/241#issuecomment-1987506207)? Not yet, there are some d

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1522763680 ## crates/iceberg/src/static_table.rs: ## @@ -0,0 +1,146 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agre

Re: [I] Loading table from metadata file directly. [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on issue #246: URL: https://github.com/apache/iceberg-rust/issues/246#issuecomment-1993843083 Hi, @a-agmon I've reviewed the pr and it looks great! I've left small comments to improve it. -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1993846294 @marvinlanhenke Sure, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
a-agmon commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1522788180 ## crates/iceberg/src/static_table.rs: ## @@ -0,0 +1,146 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
a-agmon commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1522787615 ## crates/iceberg/src/static_table.rs: ## @@ -0,0 +1,146 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1522791407 ## crates/iceberg/src/static_table.rs: ## @@ -0,0 +1,146 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agre

Re: [I] Proxy support unavailable for iceberg spark client [iceberg]

2024-03-13 Thread via GitHub
sravanscb commented on issue #9174: URL: https://github.com/apache/iceberg/issues/9174#issuecomment-1993875583 Hi @syun64 Yes, there should be a way to properly pass the proxy configs to the iceberg client and it be utilized by the code. I tried several ways - at the system level and sp

Re: [PR] Add local nightly build to test current docs changes [iceberg]

2024-03-13 Thread via GitHub
manuzhang commented on PR #9943: URL: https://github.com/apache/iceberg/pull/9943#issuecomment-1993874414 Thanks @bitsondatadev, I will test locally later today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
snazy commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1522811023 ## open-api/rest-catalog-open-api.yaml: ## @@ -1559,6 +1578,22 @@ components: type: string description: Properties that should be use

Re: [PR] Add local nightly build to test current docs changes [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9943: URL: https://github.com/apache/iceberg/pull/9943#discussion_r1522813072 ## site/dev/common.sh: ## @@ -85,6 +85,26 @@ assert_not_empty () { fi } +# Creates a 'nightly' version of the documentation that points to the current versioned d

Re: [PR] Docs: Enhance Spark pages [iceberg]

2024-03-13 Thread via GitHub
nastra commented on PR #9920: URL: https://github.com/apache/iceberg/pull/9920#issuecomment-1993885092 I don't think we want to merge #9939 because it only fixes some links. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Add local nightly build to test current docs changes [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9943: URL: https://github.com/apache/iceberg/pull/9943#discussion_r1522870334 ## site/dev/common.sh: ## @@ -85,6 +85,26 @@ assert_not_empty () { fi } +# Creates a 'nightly' version of the documentation that points to the current versioned d

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1522904160 ## open-api/rest-catalog-open-api.yaml: ## @@ -61,6 +61,14 @@ security: - OAuth2: [catalog] - BearerAuth: [] +tags: + - name: views +description: Requires s

Re: [PR] Rename stat_table to table_exists in Catalog trait [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on PR #257: URL: https://github.com/apache/iceberg-rust/pull/257#issuecomment-1994012956 Let's wait to see if @Xuanwo @Fokko has other concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[PR] Fix `rewrite_position_delete_files` result file set [iceberg]

2024-03-13 Thread via GitHub
bk-mz opened a new pull request, #9945: URL: https://github.com/apache/iceberg/pull/9945 The SparkBinPackPositionDeletesRewriter module disables Adaptive Query Execution (AQE) to correct issues with partitioning. However, if the parent Spark session is configured with a `spark.sql.sh

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1522920303 ## aws/src/main/resources/s3-signer-open-api.yaml: ## @@ -56,13 +56,18 @@ servers: description: Optional prefix to be appended to all routes default: "

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1522939008 ## open-api/rest-catalog-open-api.yaml: ## @@ -1559,6 +1578,22 @@ components: type: string description: Properties that should be us

[PR] Docs: Update site docs [iceberg]

2024-03-13 Thread via GitHub
ajantha-bhat opened a new pull request, #9946: URL: https://github.com/apache/iceberg/pull/9946 - Resolve Nessie version for site docs - Add release notes link to template -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Docs: Update site docs [iceberg]

2024-03-13 Thread via GitHub
ajantha-bhat commented on code in PR #9946: URL: https://github.com/apache/iceberg/pull/9946#discussion_r1522943830 ## site/docs/how-to-release.md: ## @@ -294,6 +294,8 @@ evolution. This release can be downloaded from: https://www.apache.org/dyn/closer.cgi/iceberg// +Relea

Re: [PR] Rename stat_table to table_exists in Catalog trait [iceberg-rust]

2024-03-13 Thread via GitHub
Xuanwo commented on code in PR #257: URL: https://github.com/apache/iceberg-rust/pull/257#discussion_r1522945260 ## crates/iceberg/src/catalog/mod.rs: ## @@ -83,7 +83,7 @@ pub trait Catalog: Debug + Sync + Send { async fn drop_table(&self, table: &TableIdent) -> Result<()>;

Re: [PR] Docs: Update site docs [iceberg]

2024-03-13 Thread via GitHub
ajantha-bhat commented on code in PR #9946: URL: https://github.com/apache/iceberg/pull/9946#discussion_r1522945276 ## site/mkdocs.yml: ## @@ -79,6 +79,7 @@ markdown_extensions: extra: icebergVersion: '1.5.0' + nessieVersion: '0.77.1' Review Comment: NessieVersion was

Re: [PR] Rename stat_table to table_exists in Catalog trait [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #257: URL: https://github.com/apache/iceberg-rust/pull/257#discussion_r1522969785 ## crates/iceberg/src/catalog/mod.rs: ## @@ -83,7 +83,7 @@ pub trait Catalog: Debug + Sync + Send { async fn drop_table(&self, table: &TableIdent) -> Resu

Re: [PR] Docs: Update site docs [iceberg]

2024-03-13 Thread via GitHub
Fokko merged PR #9946: URL: https://github.com/apache/iceberg/pull/9946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Docs: Update site docs [iceberg]

2024-03-13 Thread via GitHub
Fokko commented on code in PR #9946: URL: https://github.com/apache/iceberg/pull/9946#discussion_r1522982032 ## site/mkdocs.yml: ## @@ -79,6 +79,7 @@ markdown_extensions: extra: icebergVersion: '1.5.0' + nessieVersion: '0.77.1' Review Comment: Great catch! -- This

Re: [PR] Implement Display instead of ToString [iceberg-rust]

2024-03-13 Thread via GitHub
lewiszlw commented on PR #256: URL: https://github.com/apache/iceberg-rust/pull/256#issuecomment-1994137128 > @lewiszlw Thanks for this pr. LGTM, it would be better to add some ut. A bit weird to test Display implementation. -- This is an automated message from the Apache Git Servic

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-03-13 Thread via GitHub
Fokko commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1522768023 ## tests/integration/test_writes.py: ## @@ -355,6 +355,44 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert [row.delet

Re: [PR] Implement Display instead of ToString [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 merged PR #256: URL: https://github.com/apache/iceberg-rust/pull/256 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Support merge manifests on writes (MergeAppend) [iceberg-python]

2024-03-13 Thread via GitHub
Fokko commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1522750042 ## pyiceberg/table/__init__.py: ## @@ -2697,6 +2810,9 @@ def __init__(self, transaction: Transaction, io: FileIO) -> None: def fast_append(self) -> FastAppendF

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1523089270 ## open-api/rest-catalog-open-api.yaml: ## @@ -1559,6 +1578,22 @@ components: type: string description: Properties that should be us

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1523104924 ## open-api/rest-catalog-open-api.yaml: ## @@ -61,6 +61,14 @@ security: - OAuth2: [catalog] - BearerAuth: [] +tags: Review Comment: For a capability like `ta

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1523089270 ## open-api/rest-catalog-open-api.yaml: ## @@ -1559,6 +1578,22 @@ components: type: string description: Properties that should be us

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nk1506 commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523107552 ## core/src/test/java/org/apache/iceberg/TableMetadataParserTest.java: ## @@ -37,30 +40,24 @@ import org.apache.iceberg.io.OutputFile; import org.apache.iceberg.reloca

Re: [PR] Add `table_exists` method to Catalog [iceberg-python]

2024-03-13 Thread via GitHub
anupam-saini commented on PR #512: URL: https://github.com/apache/iceberg-python/pull/512#issuecomment-1994228272 > This looks good @anupam-saini, thanks for working on this. One more request, could you also update the docs? Otherwise people won't be able to find this :) Good point :

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1522784655 ## crates/iceberg/src/spec/arrow.rs: ## Review Comment: We already have an `arrow` module in root, how about putting this into that module? ##

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nk1506 commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523117875 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssignU

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nk1506 commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523118198 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssignU

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nk1506 commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523122080 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssignU

Re: [I] Implement `ManifestEvaluator` to prune mainfest files. [iceberg-rust]

2024-03-13 Thread via GitHub
marvinlanhenke commented on issue #253: URL: https://github.com/apache/iceberg-rust/issues/253#issuecomment-1994259576 @liurenjie1024 Is it correct that we need to implement the `ProjectionEvaluator` beforehand in order to provide the `partition_filters` and extend `TableScan` with ro

[I] [Flink] CTAS data isn't returned in Flink query [iceberg]

2024-03-13 Thread via GitHub
rmoff opened a new issue, #9947: URL: https://github.com/apache/iceberg/issues/9947 ### Query engine Flink 1.18.1 Iceberg 1.5.0 ### Question I populate an Iceberg table using `CREATE …AS SELECT FROM` as a streaming query. On disk I see datafiles written. However,

[I] Do we need to clean the file if close return fail in FileWriter [iceberg-rust]

2024-03-13 Thread via GitHub
ZENOTME opened a new issue, #260: URL: https://github.com/apache/iceberg-rust/issues/260 For FileWriter close fail, it may result in a garbage file. I look through how java process it, and seems it provided a interface like `abort` so user can call abort if it close fail. But our writer wil

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
tomtongue commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523140669 ## core/src/test/java/org/apache/iceberg/TableMetadataParserTest.java: ## @@ -37,30 +40,24 @@ import org.apache.iceberg.io.OutputFile; import org.apache.iceberg.rel

[I] Flink SQl with Iceberg snapshots doesn't react if table has upsert [iceberg]

2024-03-13 Thread via GitHub
VidakM opened a new issue, #9948: URL: https://github.com/apache/iceberg/issues/9948 ### Query engine Flink 1.17.2 with Iceberg 1.4.2 libraries ### Question I have a few Iceberg v2 tables defined and a Flink job that reads them in a streaming fashion before transforming

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
tomtongue commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523140693 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssi

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
tomtongue commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523140693 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssi

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
tomtongue commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523143873 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssi

Re: [I] Do we need to clean the file if close return fail in FileWriter [iceberg-rust]

2024-03-13 Thread via GitHub
Xuanwo commented on issue #260: URL: https://github.com/apache/iceberg-rust/issues/260#issuecomment-1994292504 > For FileWriter close fail, it may result in a garbage file. I assume that `FileWriter` just about writing a single file. For most object storage services (like s3, gc

Re: [I] Implement `ManifestEvaluator` to prune mainfest files. [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on issue #253: URL: https://github.com/apache/iceberg-rust/issues/253#issuecomment-1994325107 > @liurenjie1024 Is it correct that we need to implement the `ProjectionEvaluator` beforehand in order to provide the `partition_filters` and extend `TableScan` with row_fil

Re: [I] [Flink] CTAS data isn't returned in Flink query [iceberg]

2024-03-13 Thread via GitHub
rmoff commented on issue #9947: URL: https://github.com/apache/iceberg/issues/9947#issuecomment-1994334466 CTAS definitely works in some cases: ``` Flink SQL> CREATE TABLE iceberg_test WITH ( > 'connector' = 'iceberg', > 'catalog-type'='hive', > 'catalog-name'='dev',

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523212522 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssignU

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523217221 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssignU

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523220281 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1121,33 +1117,21 @@ private static void assertEqualsAddSortOrder( private static void

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523221222 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1158,20 +1142,13 @@ private static void assertEqualsSetStatistics( BlobMeta

Re: [PR] Fix `rewrite_position_delete_files` result file set [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9945: URL: https://github.com/apache/iceberg/pull/9945#discussion_r1523241967 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkBinPackPositionDeletesRewriter.java: ## @@ -64,6 +64,8 @@ class SparkBinPackPositionDeletesRewrit

Re: [I] Manage root of URL in manifests in one place per entire table. [iceberg]

2024-03-13 Thread via GitHub
PigNatovsky commented on issue #9936: URL: https://github.com/apache/iceberg/issues/9936#issuecomment-1994396413 Thank You a lot! Will try to find other proposals, it's worth to work on this. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Manage root of URL in manifests in one place per entire table. [iceberg]

2024-03-13 Thread via GitHub
PigNatovsky closed issue #9936: Manage root of URL in manifests in one place per entire table. URL: https://github.com/apache/iceberg/issues/9936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Add issue template and docs for iceberg proposals [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9932: URL: https://github.com/apache/iceberg/pull/9932#discussion_r1523253714 ## .github/ISSUE_TEMPLATE/iceberg_proposal.yml: ## @@ -0,0 +1,50 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [I] Implement `ManifestEvaluator` to prune mainfest files. [iceberg-rust]

2024-03-13 Thread via GitHub
marvinlanhenke commented on issue #253: URL: https://github.com/apache/iceberg-rust/issues/253#issuecomment-1994398862 I just browsed the issues, you've already listed the dependencies here: linking this for ref: https://github.com/apache/iceberg-rust/issues/153 -- This is an autom

[PR] Implement prune column for schema [iceberg-rust]

2024-03-13 Thread via GitHub
Dysprosium0626 opened a new pull request, #261: URL: https://github.com/apache/iceberg-rust/pull/261 Aiming for https://github.com/apache/iceberg-rust/issues/251. Basically implement `PruneColumn` based on [Java Implementation](https://github.com/apache/iceberg/blob/c07f2aabc0a1d02f068ecf

Re: [PR] Add issue template and docs for iceberg proposals [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9932: URL: https://github.com/apache/iceberg/pull/9932#discussion_r1523258200 ## site/docs/contribute.md: ## @@ -46,6 +46,34 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is posted for visibility a

Re: [PR] Add issue template and docs for iceberg proposals [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9932: URL: https://github.com/apache/iceberg/pull/9932#discussion_r1523260018 ## .github/ISSUE_TEMPLATE/iceberg_proposal.yml: ## @@ -0,0 +1,50 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] HIVE-28021: escape percent symbol [iceberg]

2024-03-13 Thread via GitHub
tthorpeIBM commented on code in PR #9667: URL: https://github.com/apache/iceberg/pull/9667#discussion_r1523262102 ## mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java: ## @@ -966,4 +966,41 @@ private String getCurrentSnapshotForHiveCatalogTabl

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1523104924 ## open-api/rest-catalog-open-api.yaml: ## @@ -61,6 +61,14 @@ security: - OAuth2: [catalog] - BearerAuth: [] +tags: Review Comment: For a capability like `ta

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1523281874 ## open-api/rest-catalog-open-api.yaml: ## @@ -80,12 +88,14 @@ paths: " All REST clients should first call this route to get catalog configuration

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
nastra commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1523294867 ## open-api/rest-catalog-open-api.yaml: ## @@ -114,7 +124,8 @@ paths: }, "defaults": { "clients": "4" -

[I] hdfs storeage purge table not move to trash [iceberg]

2024-03-13 Thread via GitHub
mengshangxun opened a new issue, #9949: URL: https://github.com/apache/iceberg/issues/9949 ### Apache Iceberg version 1.5.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 if hdfs trash is enabled,when drop iceberg table with purge , ic

[PR] hdfs file delete check trash configration [iceberg]

2024-03-13 Thread via GitHub
mengshangxun opened a new pull request, #9950: URL: https://github.com/apache/iceberg/pull/9950 1、delete file check current hdfs configration 2、if trash is enabled ,move to trash,else drictly delete -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] hdfs storeage purge table not move to trash [iceberg]

2024-03-13 Thread via GitHub
mengshangxun commented on issue #9949: URL: https://github.com/apache/iceberg/issues/9949#issuecomment-1994544889 i am try repair this issue. #9950 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] [WIP] Add Data Files from Parquet Files [iceberg-python]

2024-03-13 Thread via GitHub
syun64 commented on PR #506: URL: https://github.com/apache/iceberg-python/pull/506#issuecomment-1994561440 > We will replace file_path based partition inference with parquet metadata footer based partition inference. Currently we only support IdentityPartitions, and we can infer the partit

[PR] docs: Add links checker [iceberg]

2024-03-13 Thread via GitHub
Fokko opened a new pull request, #9951: URL: https://github.com/apache/iceberg/pull/9951 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [PR] docs: Add links checker [iceberg]

2024-03-13 Thread via GitHub
Fokko closed pull request #9951: docs: Add links checker URL: https://github.com/apache/iceberg/pull/9951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [PR] docs: Add links checker [iceberg]

2024-03-13 Thread via GitHub
Fokko closed pull request #9951: docs: Add links checker URL: https://github.com/apache/iceberg/pull/9951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [I] feat: Implement expression rewriter to remove `NOT` operator. [iceberg-rust]

2024-03-13 Thread via GitHub
marvinlanhenke commented on issue #150: URL: https://github.com/apache/iceberg-rust/issues/150#issuecomment-1994688748 @liurenjie1024 @sdd ...I'm a little lost what's left to be done here - sorry for the confusion. We already have ```rust pub fn negate(self) -> Predicate

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nk1506 commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523500668 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -1036,43 +1038,39 @@ public void assertEquals( private static void assertEqualsAssignU

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
nk1506 commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523509055 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -38,15 +40,13 @@ import org.apache.iceberg.util.Pair; import org.apache.iceberg.view.Immu

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
tomtongue commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523512147 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -38,15 +40,13 @@ import org.apache.iceberg.util.Pair; import org.apache.iceberg.view.I

[PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
ZENOTME opened a new pull request, #262: URL: https://github.com/apache/iceberg-rust/pull/262 This PR init TableMetadataBuilder. Complete #250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

  1   2   3   >