Re: [I] Loading table from metadata file directly. [iceberg-rust]

2024-03-13 Thread via GitHub
Xuanwo commented on issue #246: URL: https://github.com/apache/iceberg-rust/issues/246#issuecomment-1996678394 > But The motivation behind Is that we want to do it in a type safe way without runtime check. Do you think it's a good idea to provide both ways? -- This is an automated

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-03-13 Thread via GitHub
snazy commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1524318797 ## open-api/rest-catalog-open-api.yaml: ## @@ -1559,6 +1578,22 @@ components: type: string description: Properties that should be use

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-13 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1524317705 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,373 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-13 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1524316147 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,373 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r152431 ## open-api/rest-catalog-open-api.yaml: ## @@ -2838,6 +2978,63 @@ components: additionalProperties: type: string +PreplanTableRequest: +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1524316116 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-13 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1524316147 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,373 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] [WIP] Add Data Files from Parquet Files [iceberg-python]

2024-03-13 Thread via GitHub
Fokko commented on PR #506: URL: https://github.com/apache/iceberg-python/pull/506#issuecomment-1996646115 @syun64 I'm all for it if it works, but I see a lot of issues with inferring it from the Hive path. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1524314654 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
waynexia commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1524148294 ## crates/iceberg/src/arrow.rs: ## @@ -106,3 +114,560 @@ impl ArrowReader { ProjectionMask::all() } } + +/// A post order arrow schema visitor. +///

Re: [I] Cast from UUIDLiteral to other types? [iceberg-python]

2024-03-13 Thread via GitHub
Fokko commented on issue #522: URL: https://github.com/apache/iceberg-python/issues/522#issuecomment-1996627258 @sebpretzer Thanks for raising this, and it looks like you already found the solution. What PyIceberg is doing; when you create a predicate it will accept everything. Once you run

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
a-agmon commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1524304240 ## crates/iceberg/src/table.rs: ## @@ -63,3 +64,99 @@ impl Table { TableScanBuilder::new(self) } } + +/// `StaticTable` is a read-only table struct th

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1524301299 ## open-api/rest-catalog-open-api.yaml: ## @@ -2838,6 +2978,63 @@ components: additionalProperties: type: string +PreplanTableRequest: +

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-13 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1524300998 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,373 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-13 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1524300363 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,373 @@ +/* + * Licensed to the Apache Software Foundation

Re: [I] Loading table from metadata file directly. [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on issue #246: URL: https://github.com/apache/iceberg-rust/issues/246#issuecomment-1996611762 > Should simply returning an error for modification operations be acceptable? Yeah, that's one approach. But The motivation behind Is that we want to do it in a type s

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-13 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1524298729 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,373 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-13 Thread via GitHub
pvary commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1524298729 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,373 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
marvinlanhenke commented on PR #262: URL: https://github.com/apache/iceberg-rust/pull/262#issuecomment-1996606877 @ZENOTME thanks so much for your work - this will definitely unblock me. > We can left the binding to other pr, and forbid table creation with partition spec and sort orde

Re: [I] Loading table from metadata file directly. [iceberg-rust]

2024-03-13 Thread via GitHub
Xuanwo commented on issue #246: URL: https://github.com/apache/iceberg-rust/issues/246#issuecomment-1996603012 From the discussion: https://apache-iceberg.slack.com/archives/C05HTENMJG4/p1710392414486259 > The reason we need StaticTable Is that we don't want to allow modidication on

Re: [PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
marvinlanhenke commented on code in PR #262: URL: https://github.com/apache/iceberg-rust/pull/262#discussion_r1524291618 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -275,6 +277,82 @@ impl TableMetadata { } } +/// Manipulating table metadata. +pub struct TableMeta

Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]

2024-03-13 Thread via GitHub
pvary commented on issue #9954: URL: https://github.com/apache/iceberg/issues/9954#issuecomment-1996601635 @okayhooni: I was struggling with the same issue in my recent tests, and ended up creating my own test extension which encapsulated the catalog(sql) and catalog loader (java) creation

[PR] Add the build from source section [iceberg-go]

2024-03-13 Thread via GitHub
git-hulk opened a new pull request, #70: URL: https://github.com/apache/iceberg-go/pull/70 It will be better to add the build instruction for the newcomers even though it might be straight for Go develop to find where's main.go file. By the way, I'm wondering if adding the Makefile

Re: [I] Is there any way on Flink to read newly appended data only (NOT in current Iceberg table snapshot)? [iceberg]

2024-03-13 Thread via GitHub
pvary commented on issue #9955: URL: https://github.com/apache/iceberg/issues/9955#issuecomment-1996587247 @okayhooni: IIUC, INCREMENTAL_FROM_LATEST_SNAPSHOT is the reading mode you are looking for. @dyzcs: There is an option for Flink Iceberg readers to read the splits in commit ord

Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]

2024-03-13 Thread via GitHub
dyzcs commented on issue #9954: URL: https://github.com/apache/iceberg/issues/9954#issuecomment-1996566044 ``` //1. create TableLoader configuration Configuration conf = new Configuration(); //2. create catalog and set table store path Catalog catalog = new HiveCatalog(conf,"p

Re: [I] Is there any way on Flink to read newly appended data only (NOT in current Iceberg table snapshot)? [iceberg]

2024-03-13 Thread via GitHub
dyzcs commented on issue #9955: URL: https://github.com/apache/iceberg/issues/9955#issuecomment-1996547964 @okayhooni I'm not sure if Iceberg has a similar development plan, but overall, Iceberg won't imitate Kafka's related features. I understand Iceberg's positioning as a table format.

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
a-agmon commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1524244075 ## crates/iceberg/src/table.rs: ## @@ -63,3 +64,99 @@ impl Table { TableScanBuilder::new(self) } } + +/// `StaticTable` is a read-only table struct th

Re: [PR] Docs: Enhance create_changelog_view usage [iceberg]

2024-03-13 Thread via GitHub
manuzhang commented on PR #9889: URL: https://github.com/apache/iceberg/pull/9889#issuecomment-1996506484 @flyrain Thanks for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
a-agmon commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1524244075 ## crates/iceberg/src/table.rs: ## @@ -63,3 +64,99 @@ impl Table { TableScanBuilder::new(self) } } + +/// `StaticTable` is a read-only table struct th

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
Xuanwo commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1524239432 ## crates/iceberg/src/table.rs: ## @@ -63,3 +64,99 @@ impl Table { TableScanBuilder::new(self) } } + +/// `StaticTable` is a read-only table struct tha

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
Xuanwo commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1524241367 ## crates/iceberg/src/table.rs: ## @@ -63,3 +64,99 @@ impl Table { TableScanBuilder::new(self) } } + +/// `StaticTable` is a read-only table struct tha

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
viirya commented on PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#issuecomment-1996389392 The tests are updated using json and with all primitive types. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1524228159 ## open-api/rest-catalog-open-api.yaml: ## @@ -2838,6 +2978,63 @@ components: additionalProperties: type: string +PreplanTableRequest: +

Re: [I] Is there any way on Flink to read newly appended data only (NOT in current Iceberg table snapshot)? [iceberg]

2024-03-13 Thread via GitHub
okayhooni commented on issue #9955: URL: https://github.com/apache/iceberg/issues/9955#issuecomment-1996382016 @dyzcs , Thank you for kind reply! Did you mean the two parts you mentioned have to be implemented on Iceberg flink connector for my use case..? (= theoritically possible, bu

[PR] [wip] cast to pyarrow schema [iceberg-python]

2024-03-13 Thread via GitHub
kevinjqliu opened a new pull request, #523: URL: https://github.com/apache/iceberg-python/pull/523 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [I] Handle Arrow `large_string` data type [iceberg-python]

2024-03-13 Thread via GitHub
kevinjqliu commented on issue #520: URL: https://github.com/apache/iceberg-python/issues/520#issuecomment-1996377106 I can think of 2 options. 1. Add Arrow `LargeString` as an Iceberg data type. Map 1:1 with Arrow data type. The physical representation will still be backed by string.

Re: [PR] Add issue template and docs for iceberg proposals [iceberg]

2024-03-13 Thread via GitHub
danielcweeks commented on code in PR #9932: URL: https://github.com/apache/iceberg/pull/9932#discussion_r1524217993 ## .github/ISSUE_TEMPLATE/iceberg_proposal.yml: ## @@ -0,0 +1,50 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Add issue template and docs for iceberg proposals [iceberg]

2024-03-13 Thread via GitHub
danielcweeks commented on code in PR #9932: URL: https://github.com/apache/iceberg/pull/9932#discussion_r1524216124 ## .github/ISSUE_TEMPLATE/iceberg_proposal.yml: ## @@ -0,0 +1,50 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #262: URL: https://github.com/apache/iceberg-rust/pull/262#discussion_r1524213529 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -275,6 +277,82 @@ impl TableMetadata { } } +/// Manipulating table metadata. +pub struct TableMetad

Re: [PR] Add issue template and docs for iceberg proposals [iceberg]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #9932: URL: https://github.com/apache/iceberg/pull/9932#discussion_r1524208288 ## .github/ISSUE_TEMPLATE/iceberg_proposal.yml: ## @@ -0,0 +1,50 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor licens

Re: [I] Is there any way on Flink to read newly appended data only (NOT in current Iceberg table snapshot)? [iceberg]

2024-03-13 Thread via GitHub
dyzcs commented on issue #9955: URL: https://github.com/apache/iceberg/issues/9955#issuecomment-1996358615 Hi @okayhooni regarding this issue, I would like to express my opinion. I am currently exploring the use of Iceberg as a replacement for Kafka in **near real-time** processing. The pro

Re: [PR] Add issue template and docs for iceberg proposals [iceberg]

2024-03-13 Thread via GitHub
danielcweeks commented on code in PR #9932: URL: https://github.com/apache/iceberg/pull/9932#discussion_r1524167782 ## site/docs/contribute.md: ## @@ -46,6 +46,34 @@ The Iceberg community prefers to receive contributions as [Github pull requests] * If a PR is posted for visibi

Re: [PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
ZENOTME commented on code in PR #262: URL: https://github.com/apache/iceberg-rust/pull/262#discussion_r1524159851 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -275,6 +277,82 @@ impl TableMetadata { } } +/// Manipulating table metadata. +pub struct TableMetadataBui

Re: [PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #262: URL: https://github.com/apache/iceberg-rust/pull/262#discussion_r1524157344 ## crates/iceberg/src/spec/partition.rs: ## @@ -133,6 +133,25 @@ impl UnboundPartitionSpec { pub fn builder() -> UnboundPartitionSpecBuilder { Un

Re: [I] Flink CDC job getting failed due to G1 old gc and large checkpointing time [iceberg]

2024-03-13 Thread via GitHub
dyzcs commented on issue #2900: URL: https://github.com/apache/iceberg/issues/2900#issuecomment-1996323592 Hi @ayush-san I think there are two possible reasons for OOM, one of which is as shown in the figure below. Therefore, according to the partition fields of the Iceberg table, perform K

Re: [PR] Add UnboundSortOrder [iceberg-rust]

2024-03-13 Thread via GitHub
ZENOTME commented on code in PR #115: URL: https://github.com/apache/iceberg-rust/pull/115#discussion_r1524152882 ## crates/iceberg/src/spec/sort.rs: ## @@ -88,15 +91,106 @@ impl SortOrder { pub fn is_unsorted(&self) -> bool { self.fields.is_empty() } + +/

Re: [PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
ZENOTME commented on PR #262: URL: https://github.com/apache/iceberg-rust/pull/262#issuecomment-1996316028 > We can left the binding to other pr, and forbid table creation with partition spec and sort order for now, it's up to you. Let's forbid them now. I think I'm not clear with th

Re: [PR] Views, Spark: Add support for Materialized Views; Integrate with Spark SQL [iceberg]

2024-03-13 Thread via GitHub
manuzhang commented on code in PR #9830: URL: https://github.com/apache/iceberg/pull/9830#discussion_r1524151096 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceTagExec.scala: ## @@ -40,7 +40,8 @@ case class CreateOrRepl

Re: [PR] Views, Spark: Add support for Materialized Views; Integrate with Spark SQL [iceberg]

2024-03-13 Thread via GitHub
manuzhang commented on code in PR #9830: URL: https://github.com/apache/iceberg/pull/9830#discussion_r1524141148 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/MaterializedViewUtil.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [I] feat: Implement expression projection evaluator. [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 closed issue #151: feat: Implement expression projection evaluator. URL: https://github.com/apache/iceberg-rust/issues/151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] feat: Implement expression projection evaluator. [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on issue #151: URL: https://github.com/apache/iceberg-rust/issues/151#issuecomment-1996315067 Duplicate with #152 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] [Proposal] Iceberg Materialized View Spec [iceberg]

2024-03-13 Thread via GitHub
manuzhang commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1996303300 Thanks @wmoustafa. It does show the simplicity of this option. Would you mind rebasing on latest main branch such that people can try out? -- This is an automated message from th

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
viirya commented on PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#issuecomment-1996259061 Thanks. Addressed all above comments except for tests. I will update the tests for all primitive types and using json. -- This is an automated message from the Apache Git Service. To

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
viirya commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1524112078 ## crates/iceberg/src/spec/arrow.rs: ## @@ -0,0 +1,553 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
viirya commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1524111935 ## crates/iceberg/src/spec/arrow.rs: ## @@ -0,0 +1,553 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [I] feat: Implement expression rewriter to remove `NOT` operator. [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 closed issue #150: feat: Implement expression rewriter to remove `NOT` operator. URL: https://github.com/apache/iceberg-rust/issues/150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] add rewrite_not [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 merged PR #263: URL: https://github.com/apache/iceberg-rust/pull/263 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on PR #262: URL: https://github.com/apache/iceberg-rust/pull/262#issuecomment-1996240458 cc @marvinlanhenke @Fokko @Xuanwo PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] feat: init TableMetadataBuilder [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #262: URL: https://github.com/apache/iceberg-rust/pull/262#discussion_r1524080929 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -275,6 +277,78 @@ impl TableMetadata { } } +/// Manipulating table metadata. +pub struct TableMetad

Re: [PR] Parquet: Refactor BasePageIterator to add initRepetitionLevelsReader [iceberg]

2024-03-13 Thread via GitHub
wgtmac commented on PR #9751: URL: https://github.com/apache/iceberg/pull/9751#issuecomment-1996227121 @nastra Gentle ping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] feat: Implement expression rewriter to remove `NOT` operator. [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on issue #150: URL: https://github.com/apache/iceberg-rust/issues/150#issuecomment-1996208007 Hi, @marvinlanhenke I think your pr is right about this. `rewrite_not` is based on the `negate` api. -- This is an automated message from the Apache Git Service. To respon

[I] Is there any way on Flink to read newly appended data only (NOT in current Iceberg table snapshot)? [iceberg]

2024-03-13 Thread via GitHub
okayhooni opened a new issue, #9955: URL: https://github.com/apache/iceberg/issues/9955 ### Query engine Flink ### Question When I ingest from iceberg table to iceberg table with Flink, I want to start sink table ingestion with newly added records from current snapshot,

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
viirya commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1524077331 ## crates/iceberg/src/spec/arrow.rs: ## @@ -0,0 +1,553 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

[I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]

2024-03-13 Thread via GitHub
okayhooni opened a new issue, #9954: URL: https://github.com/apache/iceberg/issues/9954 ### Query engine Flink ### Question I tried to implement Flink application(source: Iceberg, sink: Iceberg). and use both Flink's datastream API and Table API for our convention for st

Re: [PR] feat (static table): implement a read-only table struct loaded from metadata [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #259: URL: https://github.com/apache/iceberg-rust/pull/259#discussion_r1524075802 ## crates/iceberg/src/table.rs: ## @@ -63,3 +64,98 @@ impl Table { TableScanBuilder::new(self) } } + +/// `StaticTable` is a read-only table str

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
liurenjie1024 commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1524074891 ## crates/iceberg/src/spec/arrow.rs: ## @@ -0,0 +1,553 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreem

Re: [I] Remove calls to the deprecated Metrics constructors [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #2044: URL: https://github.com/apache/iceberg/issues/2044#issuecomment-1996166783 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Partition data to path, and path to data conversions are not consistent with each other [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #2043: URL: https://github.com/apache/iceberg/issues/2043#issuecomment-1996166752 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Out of order fields are corrupting data with Insert Into SQL on Spark 3.0 [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #2041: URL: https://github.com/apache/iceberg/issues/2041#issuecomment-1996166729 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Iceberg specific files (manifest files, manifest list file, metadata file) are not created, only Flink specific manifest file is created [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #2033: URL: https://github.com/apache/iceberg/issues/2033#issuecomment-1996166676 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink : table column comment missing [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #1995: URL: https://github.com/apache/iceberg/issues/1995#issuecomment-1996166616 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Can we compact data files by some partition? [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #1973: URL: https://github.com/apache/iceberg/issues/1973#issuecomment-1996166559 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] no 0.10.0 for iceberg-hive [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #1992: URL: https://github.com/apache/iceberg/issues/1992#issuecomment-1996166599 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink: Commit the existing manifests to iceberg txn in IcebergFilesCommitter. [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #1959: URL: https://github.com/apache/iceberg/issues/1959#issuecomment-1996166542 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Partial data ingestion to Iceberg in failing with Spark 3.0.x [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #2040: URL: https://github.com/apache/iceberg/issues/2040#issuecomment-1996166715 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Support default value semantics in Iceberg [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #2039: URL: https://github.com/apache/iceberg/issues/2039#issuecomment-1996166698 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] The data generated by Actions cannot be serialized [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #2028: URL: https://github.com/apache/iceberg/issues/2028#issuecomment-199613 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink : The Iceberg metadata is not updated after the SavePoint recovery task [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #2023: URL: https://github.com/apache/iceberg/issues/2023#issuecomment-1996166636 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink : query large iceberg tables slowly [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #1983: URL: https://github.com/apache/iceberg/issues/1983#issuecomment-1996166579 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Read options to read append and delete with overwrite snapshots in Incremental reads [iceberg]

2024-03-13 Thread via GitHub
github-actions[bot] commented on issue #1949: URL: https://github.com/apache/iceberg/issues/1949#issuecomment-1996166517 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Fix pagination description based on new feedback [iceberg]

2024-03-13 Thread via GitHub
danielcweeks commented on code in PR #9917: URL: https://github.com/apache/iceberg/pull/9917#discussion_r1524040792 ## open-api/rest-catalog-open-api.yaml: ## @@ -1610,13 +1610,27 @@ components: PageToken: description: -An opaque token which allows clients

Re: [PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-13 Thread via GitHub
tomtongue commented on code in PR #9927: URL: https://github.com/apache/iceberg/pull/9927#discussion_r1523512147 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -38,15 +40,13 @@ import org.apache.iceberg.util.Pair; import org.apache.iceberg.view.I

Re: [PR] Fix pagination description based on new feedback [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9917: URL: https://github.com/apache/iceberg/pull/9917#discussion_r1523982295 ## open-api/rest-catalog-open-api.yaml: ## @@ -1610,13 +1610,27 @@ components: PageToken: description: -An opaque token which allows clients to ma

Re: [PR] Fix pagination description based on new feedback [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9917: URL: https://github.com/apache/iceberg/pull/9917#discussion_r1523980933 ## open-api/rest-catalog-open-api.yaml: ## @@ -1610,13 +1610,27 @@ components: PageToken: description: -An opaque token which allows clients to ma

[I] Cast from UUIDLiteral to other types? [iceberg-python]

2024-03-13 Thread via GitHub
sebpretzer opened a new issue, #522: URL: https://github.com/apache/iceberg-python/issues/522 ### Apache Iceberg version 0.6.0 (latest release) ### Please describe the bug 🐞 Apologies for not fully finding the root cause, tracing the issue was a bit difficult for me, and

Re: [PR] Fix pagination description based on new feedback [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9917: URL: https://github.com/apache/iceberg/pull/9917#discussion_r1523964022 ## open-api/rest-catalog-open-api.yaml: ## @@ -1610,13 +1610,27 @@ components: PageToken: description: -An opaque token which allows clients to ma

Re: [PR] Fix pagination description based on new feedback [iceberg]

2024-03-13 Thread via GitHub
rahil-c commented on code in PR #9917: URL: https://github.com/apache/iceberg/pull/9917#discussion_r1523964589 ## open-api/rest-catalog-open-api.yaml: ## @@ -1610,13 +1610,27 @@ components: PageToken: description: -An opaque token which allows clients to ma

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1523844406 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/nam

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1523844779 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,113 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/nam

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-13 Thread via GitHub
viirya commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1523819574 ## crates/iceberg/src/spec/arrow.rs: ## @@ -0,0 +1,553 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [I] [Flink] CTAS data isn't returned in Flink query [iceberg]

2024-03-13 Thread via GitHub
pvary commented on issue #9947: URL: https://github.com/apache/iceberg/issues/9947#issuecomment-1995468731 I would guess that there is no checkpointing enabled in your case. The data files are written, but not added/committed to the Iceberg table, which happens on checkpoints, or at the end

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

2024-03-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #9953: URL: https://github.com/apache/iceberg/pull/9953#discussion_r1523786976 ## api/src/main/java/org/apache/iceberg/io/FileIO.java: ## @@ -49,25 +49,25 @@ default InputFile newInputFile(String path, long length) { default InputFile

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

2024-03-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #9953: URL: https://github.com/apache/iceberg/pull/9953#discussion_r1523777029 ## api/src/main/java/org/apache/iceberg/io/FileIO.java: ## @@ -49,25 +49,25 @@ default InputFile newInputFile(String path, long length) { default InputFile

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

2024-03-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #9953: URL: https://github.com/apache/iceberg/pull/9953#discussion_r1523777029 ## api/src/main/java/org/apache/iceberg/io/FileIO.java: ## @@ -49,25 +49,25 @@ default InputFile newInputFile(String path, long length) { default InputFile

Re: [PR] Fix pagination description based on new feedback [iceberg]

2024-03-13 Thread via GitHub
rdblue commented on code in PR #9917: URL: https://github.com/apache/iceberg/pull/9917#discussion_r1523742808 ## open-api/rest-catalog-open-api.yaml: ## @@ -1610,13 +1610,27 @@ components: PageToken: description: -An opaque token which allows clients to mak

Re: [PR] Fix pagination description based on new feedback [iceberg]

2024-03-13 Thread via GitHub
rdblue commented on code in PR #9917: URL: https://github.com/apache/iceberg/pull/9917#discussion_r1523740390 ## open-api/rest-catalog-open-api.yaml: ## @@ -1610,13 +1610,27 @@ components: PageToken: description: -An opaque token which allows clients to mak

Re: [PR] Fix pagination description based on new feedback [iceberg]

2024-03-13 Thread via GitHub
rdblue commented on code in PR #9917: URL: https://github.com/apache/iceberg/pull/9917#discussion_r1523737728 ## open-api/rest-catalog-open-api.yaml: ## @@ -1610,13 +1610,27 @@ components: PageToken: description: -An opaque token which allows clients to mak

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

2024-03-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #9953: URL: https://github.com/apache/iceberg/pull/9953#discussion_r1523734039 ## api/src/main/java/org/apache/iceberg/io/FileIO.java: ## @@ -51,23 +51,23 @@ default InputFile newInputFile(DataFile file) { file.keyMetadata() == n

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1523734524 ## open-api/rest-catalog-open-api.yaml: ## @@ -2838,6 +2978,63 @@ components: additionalProperties: type: string +PreplanTableRequest: +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-13 Thread via GitHub
rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1523734188 ## open-api/rest-catalog-open-api.yaml: ## @@ -2838,6 +2978,63 @@ components: additionalProperties: type: string +PreplanTableRequest: +

  1   2   3   >