ZENOTME commented on code in PR #78:
URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1357818399
##
crates/iceberg/src/catalog/rest.rs:
##
@@ -0,0 +1,900 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.
ZENOTME commented on PR #79:
URL: https://github.com/apache/iceberg-rust/pull/79#issuecomment-1760940516
For now, it still not been completed.
It only complete the basic design and I want to make sure whether the design
well first.
If it looks well, I will complete it and add
ZENOTME opened a new pull request, #79:
URL: https://github.com/apache/iceberg-rust/pull/79
This PR prepare to support read/write Manifest.
related issue: #36
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
ajantha-bhat commented on issue #8824:
URL: https://github.com/apache/iceberg/issues/8824#issuecomment-1760921944
Looks like it is a regression : https://github.com/apache/iceberg/pull/4859
It seems we tried fixing it long time back but didn't fix properly.
--
This is an automated mess
ajantha-bhat opened a new issue, #8824:
URL: https://github.com/apache/iceberg/issues/8824
PR: https://github.com/apache/iceberg/pull/8822
Build:
https://github.com/apache/iceberg/actions/runs/6499875599/job/17655030618?pr=8822
```
TestRemoveOrphanFilesAction3 > orphanedFile
liurenjie1024 commented on code in PR #78:
URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1357791341
##
crates/iceberg/Cargo.toml:
##
@@ -41,20 +41,24 @@ either = "1"
futures = "0.3"
itertools = "0.11"
lazy_static = "1"
+log = "^0.4"
murmur3 = "0.5.2"
once_
atifiu commented on PR #6252:
URL: https://github.com/apache/iceberg/pull/6252#issuecomment-1760825445
@huaxingao What can be the possible reasons for aggregate pushdown to not
work when using filters, if you can give me some idea/hint I will try to look
into it further.
--
This is an au
atifiu commented on issue #8045:
URL: https://github.com/apache/iceberg/issues/8045#issuecomment-1760787847
@szehon-ho Thanks for the fix. I am facing the same issue on iceberg 1.3.0
while trying to remove delete files using proc `rewrite_position_delete_files`
. Reason why I have remove de
Xuanwo commented on code in PR #78:
URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1357738084
##
crates/iceberg/Cargo.toml:
##
@@ -41,20 +41,24 @@ either = "1"
futures = "0.3"
itertools = "0.11"
lazy_static = "1"
+log = "^0.4"
murmur3 = "0.5.2"
once_cell =
liurenjie1024 commented on code in PR #78:
URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1357732820
##
crates/iceberg/Cargo.toml:
##
@@ -41,20 +41,24 @@ either = "1"
futures = "0.3"
itertools = "0.11"
lazy_static = "1"
+log = "^0.4"
murmur3 = "0.5.2"
once_
liurenjie1024 commented on issue #77:
URL: https://github.com/apache/iceberg-rust/issues/77#issuecomment-1760723094
> Another solution is pass struct type as another parameter when we need it,
e.g.
I prefer this approach. It's weird for me to store types with values, and we
can alway
Xuanwo commented on code in PR #78:
URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1357725856
##
crates/iceberg/src/catalog/rest.rs:
##
@@ -0,0 +1,912 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.
liurenjie1024 commented on PR #78:
URL: https://github.com/apache/iceberg-rust/pull/78#issuecomment-1760707832
cc @JanKaul @Xuanwo @Fokko @ZENOTME PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
liurenjie1024 opened a new pull request, #78:
URL: https://github.com/apache/iceberg-rust/pull/78
In this pr we add initial support for rest, which finished simple rest apis.
Complex apis such as create table, update table, commits which be added in
following pr so that we can make e
Zhanxiao-Ma commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1760704067
> Sometimes we need to do similar thing in Flink Source, and we ended up
creating our own comparator for this which compares Iceberg splits (which are a
wrapper above ScanTasks).
Zhanxiao-Ma opened a new issue, #8802:
URL: https://github.com/apache/iceberg/issues/8802
### Query engine
Iceberg java api(Version 0.14.1)
### Question
I want the data read from iceberg to be in the same order every time. But I
can't find an attribute that would make Fi
Zhanxiao-Ma commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1760700741
> Currently there is no way to order the scan task. The planning side
specifically makes sure that even the planning could be done by parallel
threads (reading manifests files pa
Zhanxiao-Ma closed issue #8802: How to read data in the order in which files
are commited?
URL: https://github.com/apache/iceberg/issues/8802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spec
puchengy opened a new issue, #64:
URL: https://github.com/apache/iceberg-python/issues/64
### Apache Iceberg version
None
### Please describe the bug 🐞
```
tasks = table.scan(row_filter="dt='2023-08-20' AND view_type=1 AND hr='00'
").plan_files()
```
Only filter
MarsKT commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1760659443
> Thanks @pvary, I have a maybe naive question @MarsKT
>
> > want the data read from iceberg to be in the same order every time.
>
> Can I ask what's driving this need?
ZENOTME commented on issue #77:
URL: https://github.com/apache/iceberg-rust/issues/77#issuecomment-1760655157
cc @JanKaul @Fokko @Xuanwo @liurenjie1024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
zinking commented on PR #8807:
URL: https://github.com/apache/iceberg/pull/8807#issuecomment-1760653401
> wondering if we could benefit from reads in general as well ?
yep, like mentioned in the distributed planning work: when metadata becomes
big, hand crafted parallel code is n
github-actions[bot] closed issue #7231: Add outputFile() for FileAppender
URL: https://github.com/apache/iceberg/issues/7231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
github-actions[bot] commented on issue #7231:
URL: https://github.com/apache/iceberg/issues/7231#issuecomment-1760563302
This issue has been closed because it has not received any activity in the
last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Gi
dramaticlly commented on issue #8702:
URL: https://github.com/apache/iceberg/issues/8702#issuecomment-1760455077
data compaction only change physical files layout but not the data visible
to users. Consider you originally have 1000 records with 10 duplicates, after
deduplication it would be
flyrain commented on PR #8799:
URL: https://github.com/apache/iceberg/pull/8799#issuecomment-1760415613
I will take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
Fokko opened a new pull request, #62:
URL: https://github.com/apache/iceberg-python/pull/62
We should only use this in the table module.
Follow up on
https://github.com/apache/iceberg-python/pull/60#discussion_r1355656751
--
This is an automated message from the Apache Git Service.
Fokko opened a new pull request, #61:
URL: https://github.com/apache/iceberg-python/pull/61
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-m
Fokko merged PR #59:
URL: https://github.com/apache/iceberg-python/pull/59
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap
singhpk234 commented on PR #8807:
URL: https://github.com/apache/iceberg/pull/8807#issuecomment-1760327041
interesting this is an approach which impala folks took too :
-
https://docs.google.com/document/d/1WF_UOanQ61RUuQlM4LaiRWI0YXpPKZ2VEJ8gyJdDyoY/edit#
wondering if we could be
Fokko commented on code in PR #8819:
URL: https://github.com/apache/iceberg/pull/8819#discussion_r1357327064
##
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsCoordinator.java:
##
@@ -340,7 +340,7 @@ private void unregisterSubtaskGateway(int
Fokko commented on code in PR #8730:
URL: https://github.com/apache/iceberg/pull/8730#discussion_r1357291336
##
format/spec.md:
##
@@ -443,13 +443,13 @@ The schema of a manifest file is a struct called
`manifest_entry` with the follo
| _optional_ | _optional_ | **`132 split_o
bitsondatadev commented on code in PR #272:
URL: https://github.com/apache/iceberg-docs/pull/272#discussion_r1357238803
##
landing-page/content/common/roadmap.md:
##
@@ -22,28 +22,36 @@ disableSidebar: true
# Roadmap Overview
-This roadmap outlines projects that the Iceberg
bitsondatadev commented on PR #272:
URL: https://github.com/apache/iceberg-docs/pull/272#issuecomment-1760168587
Hey all, I'm looping back to this today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
wypoon commented on PR #8805:
URL: https://github.com/apache/iceberg/pull/8805#issuecomment-1760074698
@rdblue @aokolnychyi as @rbalamohan indicated that he's not working on
https://github.com/apache/iceberg/pull/6432 anymore, I have taken it up here. I
rebased it on master and resolved the
PickBas commented on issue #8810:
URL: https://github.com/apache/iceberg/issues/8810#issuecomment-1760006978
@Fokko I have changed everywhere in core/* module from `size()` to
`isEmpty()` except _ContentFileParser.java_. In order to move away from
`.size() > 0` it is required to add the `is
ZENOTME commented on issue #77:
URL: https://github.com/apache/iceberg-rust/issues/77#issuecomment-1759997715
I find that our struct value didn't include type info. Do we want include
type info in it?
1. If we include info in struct, the struct value may look like
```
struc
ZENOTME opened a new issue, #77:
URL: https://github.com/apache/iceberg-rust/issues/77
Use lookup will make the memory cost if we have multiple struct with same
type.
One solve way is to use `Arc` in Struct. I try this design in
https://github.com/icelake-io/icelake/pull/136.
PickBas commented on issue #8810:
URL: https://github.com/apache/iceberg/issues/8810#issuecomment-1759962395
@Fokko Will be done. Could you assign the issue to me, if you don't mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Gi
PickBas commented on PR #8813:
URL: https://github.com/apache/iceberg/pull/8813#issuecomment-1759958867
@Fokko Sure, will be done. PR per module works for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abo
Fokko commented on PR #8813:
URL: https://github.com/apache/iceberg/pull/8813#issuecomment-1759955441
Thanks for opening this PR @PickBas. There are a couple more in the
codebase. What do you think of doing a PR per module? So we keep it manageable.
In this case everything in `core/*`?
--
Fokko commented on issue #8810:
URL: https://github.com/apache/iceberg/issues/8810#issuecomment-1759954274
There are quite a few:
```
./core/src/main/java/org/apache/iceberg/BaseDistributedDataScan.java:
boolean mayHaveEqualityDeletes = deleteManifests.size() > 0 &&
mayHaveEqua
paulpaul1076 commented on issue #8809:
URL: https://github.com/apache/iceberg/issues/8809#issuecomment-1759723529
@RussellSpitzer provided this code which achieves the same:
```
foreachBatch (batch_df, batch_id) => {
val lastBatch =
Spark3Util.loadIcebergTable(spark,"db.timezo
Fokko commented on PR #8722:
URL: https://github.com/apache/iceberg/pull/8722#issuecomment-1759635303
Thanks @jbonofre for taking the lead on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
Fokko merged PR #8722:
URL: https://github.com/apache/iceberg/pull/8722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach
jbonofre commented on PR #8722:
URL: https://github.com/apache/iceberg/pull/8722#issuecomment-1759631883
`master` branch has been renamed to `main`.
@Fokko @nastra if you can merge this PR when you have time, it would be
great. Thanks !
--
This is an automated message from the Apa
dependabot[bot] closed pull request #8778: Build: Bump
com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.22.0
URL: https://github.com/apache/iceberg/pull/8778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
dependabot[bot] commented on PR #8778:
URL: https://github.com/apache/iceberg/pull/8778#issuecomment-1759625459
Superseded by #8812.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
dependabot[bot] opened a new pull request, #8812:
URL: https://github.com/apache/iceberg/pull/8812
Bumps
[com.palantir.baseline:gradle-baseline-java](https://github.com/palantir/gradle-baseline)
from 4.42.0 to 5.24.0.
Release notes
Sourced from https://github.com/palantir/gradle-b
dependabot[bot] closed pull request #8734: Build: Bump
org.springframework:spring-web from 5.3.9 to 6.0.12
URL: https://github.com/apache/iceberg/pull/8734
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
dependabot[bot] commented on PR #8734:
URL: https://github.com/apache/iceberg/pull/8734#issuecomment-1759624056
Superseded by #8811.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
dependabot[bot] opened a new pull request, #8811:
URL: https://github.com/apache/iceberg/pull/8811
Bumps
[org.springframework:spring-web](https://github.com/spring-projects/spring-framework)
from 5.3.9 to 6.0.13.
Release notes
Sourced from https://github.com/spring-projects/spring
Humbedooh closed pull request #3686: Aliyun: Add iceberg-aliyun document
URL: https://github.com/apache/iceberg/pull/3686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To uns
Humbedooh closed pull request #3392: Hive: Bug when runing SQL with multiple
table join.
URL: https://github.com/apache/iceberg/pull/3392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
Humbedooh closed pull request #2894: Update the comment content of
'commit.status-check.total-timeout-ms'
URL: https://github.com/apache/iceberg/pull/2894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
Humbedooh closed pull request #2680: Core: Add RocksDBStructLikeMap
URL: https://github.com/apache/iceberg/pull/2680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscr
Humbedooh closed pull request #3539: Hive: Allow to create external table to
access the iceberg table managed in hive catalog
URL: https://github.com/apache/iceberg/pull/3539
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and us
Humbedooh closed pull request #2434: Core: DeleteMarker to mark row as deleted
URL: https://github.com/apache/iceberg/pull/2434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
nk1506 commented on PR #8798:
URL: https://github.com/apache/iceberg/pull/8798#issuecomment-1759595831
> nit: `Adopt` -> `Adapt` in title?
>
> I believe the removed test case is worth keeping.
Since `CONF_NESSIE_CLIENT_BUILDER_IMPL` has been deprecated, replacing it
with `CONF_
nk1506 commented on PR #8804:
URL: https://github.com/apache/iceberg/pull/8804#issuecomment-1759573960
@nastra , Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
ajantha-bhat commented on code in PR #272:
URL: https://github.com/apache/iceberg-docs/pull/272#discussion_r1356790636
##
landing-page/content/common/roadmap.md:
##
@@ -22,28 +22,36 @@ disableSidebar: true
# Roadmap Overview
-This roadmap outlines projects that the Iceberg
Fokko opened a new issue, #8810:
URL: https://github.com/apache/iceberg/issues/8810
### Feature Request / Improvement
Suggestion by IDEA:
I think this is nice because `isEmpty
pvary merged PR #282:
URL: https://github.com/apache/iceberg-docs/pull/282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap
ajantha-bhat commented on code in PR #8798:
URL: https://github.com/apache/iceberg/pull/8798#discussion_r1356770832
##
nessie/src/test/java/org/apache/iceberg/nessie/TestCustomNessieClient.java:
##
@@ -78,30 +77,11 @@ public void testNonExistentCustomClient() {
dimas-b commented on code in PR #8798:
URL: https://github.com/apache/iceberg/pull/8798#discussion_r1356768108
##
nessie/src/test/java/org/apache/iceberg/nessie/TestCustomNessieClient.java:
##
@@ -78,30 +77,11 @@ public void testNonExistentCustomClient() {
ayushtkn commented on PR #282:
URL: https://github.com/apache/iceberg-docs/pull/282#issuecomment-1759537789
Tried building locally to validate, no link is broken, Attaching screenshot
https://github.com/apache/iceberg-docs/assets/25608848/205e05af-1eb8-41f3-b0e4-7cfb66c80c58";>
--
T
ayushtkn opened a new pull request, #282:
URL: https://github.com/apache/iceberg-docs/pull/282
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe,
paulpaul1076 opened a new issue, #8809:
URL: https://github.com/apache/iceberg/issues/8809
### Feature Request / Improvement
Delta lake has an interesting feature which you can read about here:
https://docs.delta.io/latest/delta-streaming.html#idempotent-table-writes-in-foreachbatch
ajantha-bhat commented on code in PR #8701:
URL: https://github.com/apache/iceberg/pull/8701#discussion_r1356710076
##
kafka-connect/kafka-connect-events/src/main/java/org/apache/iceberg/connect/events/CommitCompletePayload.java:
##
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apach
jbonofre commented on PR #8586:
URL: https://github.com/apache/iceberg/pull/8586#issuecomment-1759269677
Awesome ! Thanks, I'm dealing with the ASF record now ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
nastra merged PR #8586:
URL: https://github.com/apache/iceberg/pull/8586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac
nastra commented on PR #8586:
URL: https://github.com/apache/iceberg/pull/8586#issuecomment-1759247885
thanks @jbonofre
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
amogh-jahagirdar commented on PR #6252:
URL: https://github.com/apache/iceberg/pull/6252#issuecomment-1759179813
My mistake, yes you can have format version 2 and have copy on write. The
remaining issue is why you are even seeing delete files if CoW is set. That
seems to be the fundamental
ajantha-bhat commented on code in PR #8701:
URL: https://github.com/apache/iceberg/pull/8701#discussion_r1356434493
##
kafka-connect/kafka-connect-events/src/main/java/org/apache/iceberg/connect/events/CommitCompletePayload.java:
##
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apach
ajantha-bhat commented on code in PR #8797:
URL: https://github.com/apache/iceberg/pull/8797#discussion_r1356348310
##
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java:
##
@@ -109,6 +110,8 @@ public InternalRow[] call(InternalRow
ajantha-bhat commented on PR #8798:
URL: https://github.com/apache/iceberg/pull/8798#issuecomment-1759048641
cc: @dimas-b, @snazy
Dependabot is raising PRs for Nessie bumps now. This is a follow up for the
latest bump.
--
This is an automated message from the Apache Git Service.
ajantha-bhat commented on PR #8807:
URL: https://github.com/apache/iceberg/pull/8807#issuecomment-1759043430
Is running rewrite_position_delete before running rewrite_data_files not
helping in this scenario?
--
This is an automated message from the Apache Git Service.
To respond to the m
77 matches
Mail list logo
