mayankasthana closed issue #8910: Spark Delete Orphan Files on executors
URL: https://github.com/apache/iceberg/issues/8910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
liurenjie1024 commented on PR #82:
URL: https://github.com/apache/iceberg-rust/pull/82#issuecomment-1776370231
cc @Fokko @Xuanwo @JanKaul PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
liurenjie1024 opened a new issue, #85:
URL: https://github.com/apache/iceberg-rust/issues/85
I think we can still test them? They are encoded in bytes when
converting from literal to raw literal?
_Originally posted by @liurenjie1024 in
https://github.com/apache/iceberg-
liurenjie1024 commented on code in PR #82:
URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1369463348
##
crates/iceberg/src/spec/values.rs:
##
@@ -996,10 +1028,593 @@ mod timestamptz {
}
}
+mod _serde {
+use std::collections::BTreeMap;
+
+use crat
ZENOTME commented on code in PR #82:
URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1369459787
##
crates/iceberg/src/spec/values.rs:
##
@@ -996,10 +1028,593 @@ mod timestamptz {
}
}
+mod _serde {
+use std::collections::BTreeMap;
+
+use crate::{
+
liurenjie1024 commented on code in PR #82:
URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1369450219
##
crates/iceberg/src/spec/values.rs:
##
@@ -996,10 +1028,593 @@ mod timestamptz {
}
}
+mod _serde {
+use std::collections::BTreeMap;
+
+use crat
dimas-b commented on code in PR #8857:
URL: https://github.com/apache/iceberg/pull/8857#discussion_r1369427545
##
nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java:
##
@@ -323,28 +417,13 @@ public void renameTable(TableIdentifier from,
TableIdentifier to)
dimas-b commented on code in PR #8857:
URL: https://github.com/apache/iceberg/pull/8857#discussion_r1369419740
##
nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java:
##
@@ -181,133 +185,223 @@ public IcebergTable table(TableIdentifier
tableIdentifier) {
github-actions[bot] closed issue #7186: Enforce variable order in checkstyle
URL: https://github.com/apache/iceberg/issues/7186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
github-actions[bot] commented on issue #7186:
URL: https://github.com/apache/iceberg/issues/7186#issuecomment-1776250781
This issue has been closed because it has not received any activity in the
last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Gi
github-actions[bot] closed issue #7187: Refactoring metadata location and
adding API to get data and metadata location
URL: https://github.com/apache/iceberg/issues/7187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
github-actions[bot] commented on issue #7187:
URL: https://github.com/apache/iceberg/issues/7187#issuecomment-1776250753
This issue has been closed because it has not received any activity in the
last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Gi
tombarti commented on issue #8911:
URL: https://github.com/apache/iceberg/issues/8911#issuecomment-1775911356
Thanks for taking the time to explain, that all makes sense now!
I can see that #7886 in Iceberg 1.4.0 could be handful for the other avenue
you are suggesting!
--
This is
RussellSpitzer commented on issue #8911:
URL: https://github.com/apache/iceberg/issues/8911#issuecomment-1775880284
Iceberg uses the Datasource API from Spark, so we only see filters and
expressions that Spark decides to pass through to us. In this case "substring"
is just not an expression
tombarti commented on issue #8911:
URL: https://github.com/apache/iceberg/issues/8911#issuecomment-1775871158
Thanks for the quick reply @RussellSpitzer, so what you are saying is that
this really should be implemented in Spark and once it is, there is nothing
much to do on the Iceberg side
RussellSpitzer commented on issue #8911:
URL: https://github.com/apache/iceberg/issues/8911#issuecomment-1775822458
I think this could be done but the fix would have to be in Spark I believe.
Spark needs to be able to convert Substring => StartsWith on their end
--
This is an automated me
mayankasthana closed issue #8910: Spark Delete Orphan Files on executors
URL: https://github.com/apache/iceberg/issues/8910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
mayankasthana commented on issue #8910:
URL: https://github.com/apache/iceberg/issues/8910#issuecomment-1775630259
Duplicate of https://github.com/apache/iceberg/issues/8862 so closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
stevenzwu commented on code in PR #7798:
URL: https://github.com/apache/iceberg/pull/7798#discussion_r1368952270
##
api/src/main/java/org/apache/iceberg/SortKey.java:
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor l
stevenzwu commented on code in PR #7798:
URL: https://github.com/apache/iceberg/pull/7798#discussion_r1368952270
##
api/src/main/java/org/apache/iceberg/SortKey.java:
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor l
abehnia closed pull request #83: fix(avro-schema): fix avro map schema
generation
URL: https://github.com/apache/iceberg-rust/pull/83
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific com
RussellSpitzer commented on code in PR #7798:
URL: https://github.com/apache/iceberg/pull/7798#discussion_r1368918819
##
api/src/main/java/org/apache/iceberg/StructTransform.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more
RussellSpitzer commented on code in PR #7798:
URL: https://github.com/apache/iceberg/pull/7798#discussion_r1368916210
##
api/src/main/java/org/apache/iceberg/SortKey.java:
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contribu
rakesh-das08 commented on issue #8881:
URL: https://github.com/apache/iceberg/issues/8881#issuecomment-1775511123
Few observations:
1) There are 2 ways to work with branches: WAP or Branch Identifiers. IMO
working with branch identifiers is much intuitive and easier. You can refer to
a
erikcw commented on issue #7137:
URL: https://github.com/apache/iceberg/issues/7137#issuecomment-1775505118
I'm also experiencing this issue. I'm on Trino version 430.
```
çorg.apache.iceberg.exceptions.RuntimeIOException: Failed to open file:
s3://MY.BUCKET/parquet/campaigns-pre-
ajantha-bhat commented on PR #8906:
URL: https://github.com/apache/iceberg/pull/8906#issuecomment-1775495652
> I believe we typically don't update RevAPI to patch versions (at least we
haven't done so for 1.2.1 or 1.3.1 in the past)
It is because revAPI expects a tag from the main bra
nastra commented on PR #8906:
URL: https://github.com/apache/iceberg/pull/8906#issuecomment-1775485225
I believe we typically don't update RevAPI to patch versions (at least we
haven't done so for 1.2.1 or 1.3.1 in the past)
--
This is an automated message from the Apache Git Service.
To
nastra closed pull request #8905: Github: Update template to 1.4.1
URL: https://github.com/apache/iceberg/pull/8905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscri
nastra commented on PR #8905:
URL: https://github.com/apache/iceberg/pull/8905#issuecomment-1775480518
oops I didn't realize this updates a different file than
https://github.com/apache/iceberg/pull/8886. LGTM
--
This is an automated message from the Apache Git Service.
To respond to the
nastra closed pull request #8905: Github: Update template to 1.4.1
URL: https://github.com/apache/iceberg/pull/8905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscri
nastra commented on PR #8905:
URL: https://github.com/apache/iceberg/pull/8905#issuecomment-1775478787
I'll close this out since this was already done by
https://github.com/apache/iceberg/pull/8886
--
This is an automated message from the Apache Git Service.
To respond to the message, ple
lovromazgon commented on issue #24:
URL: https://github.com/apache/iceberg-go/issues/24#issuecomment-1775468566
Thank you @zeroshade!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
nastra merged PR #8886:
URL: https://github.com/apache/iceberg/pull/8886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac
cccs-jc commented on issue #8902:
URL: https://github.com/apache/iceberg/issues/8902#issuecomment-1775412513
yes I would skip over all the files of a rewrite snapshot.
also I've noticed here when you calculate the `latestOffset` you call
`skippedManifestIndexesFromSnapshot`.
h
zeroshade commented on issue #24:
URL: https://github.com/apache/iceberg-go/issues/24#issuecomment-1775397551
I'll try to add the checklist to the readme in the next day or so to provide
more information and a roadmap.
--
This is an automated message from the Apache Git Service.
To respon
ajantha-bhat opened a new pull request, #8909:
URL: https://github.com/apache/iceberg/pull/8909
- Extend `NessieCatalog` with `BaseMetastoreViewCatalog`.
- Move some common code to `NessieUtil` for reusing between tables and views.
- Generalize `ViewCatalogTests` to reuse for Nessie cat
singhpk234 commented on issue #8902:
URL: https://github.com/apache/iceberg/issues/8902#issuecomment-1775255912
ACK @cccs-jc let me take a closer looks ideally yes, what you are suggesting
does makes sense and we should hop over all the files added via overwrite
snapshot in one go.
--
T
josgarmar14 opened a new issue, #8908:
URL: https://github.com/apache/iceberg/issues/8908
### Apache Iceberg version
1.0.0
### Query engine
Spark
### Please describe the bug 🐞
Facing issue when creating a new iceberg snapshot with new schema.
**Code:**
jbonofre commented on PR #7105:
URL: https://github.com/apache/iceberg/pull/7105#issuecomment-1775210294
I took a new complete look on this PR. @ajantha-bhat isolated specs change
in this PR, which is a good approach (to focus only on specs and not the impact
on impl).
As this PR:
- f
jbonofre commented on PR #8906:
URL: https://github.com/apache/iceberg/pull/8906#issuecomment-1775193170
FYI, related to `revapi`, I'm working on
https://github.com/palantir/gradle-revapi/issues/612.
--
This is an automated message from the Apache Git Service.
To respond to the message, p
onemriganka opened a new pull request, #102:
URL: https://github.com/apache/iceberg-python/pull/102
fixes / #101
HEY, I HAVE ADDED A SIMPLE CONTRIBUTING.MD FILE. HOPE IT WILL HELP BEGINEERS
--
This is an automated message from the Apache Git Service.
To respond to the message, please l
nk1506 opened a new pull request, #8907:
URL: https://github.com/apache/iceberg/pull/8907
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mai
Fokko commented on issue #101:
URL: https://github.com/apache/iceberg-python/issues/101#issuecomment-1775158682
Hey @onemriganka Thanks for raising this. We have a contributing section on
the docs: https://py.iceberg.apache.org/contributing/ Feel free to improve that
in any way.
--
This
onemriganka commented on issue #101:
URL: https://github.com/apache/iceberg-python/issues/101#issuecomment-1775155054
attempted #101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
onemriganka opened a new issue, #101:
URL: https://github.com/apache/iceberg-python/issues/101
### Feature Request / Improvement
I THINK HAVING A CONTRIBUTING.MD FILE WILL HELP BEGINNERS TO CONTRIBUTE TO
THIS PROJECT
--
This is an automated message from the Apache Git Service.
To r
Fokko commented on PR #87:
URL: https://github.com/apache/iceberg-python/pull/87#issuecomment-1775119725
@jayceslesar It looks like that the ruff checker does not implement all the
rules. Are you interested in adding Refurb directly?
--
This is an automated message from the Apache Git Ser
Fokko merged PR #93:
URL: https://github.com/apache/iceberg-python/pull/93
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap
Fokko closed pull request #87: Add Refurb to ruff
URL: https://github.com/apache/iceberg-python/pull/87
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
Fokko commented on code in PR #93:
URL: https://github.com/apache/iceberg-python/pull/93#discussion_r1368606470
##
.pre-commit-config.yaml:
##
@@ -70,3 +70,11 @@ repos:
]
additional_dependencies:
- tomli==2.0.1
+ - repo: https://github.com/ikamens
ZENOTME commented on code in PR #82:
URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1368605517
##
crates/iceberg/src/spec/values.rs:
##
@@ -1309,4 +1928,256 @@ mod tests {
&Type::Primitive(PrimitiveType::String),
);
}
+
+#[test]
+
Fokko merged PR #96:
URL: https://github.com/apache/iceberg-python/pull/96
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap
bitsondatadev commented on PR #8659:
URL: https://github.com/apache/iceberg/pull/8659#issuecomment-1775020283
Yeah, I think the projects make sense to have a better visual of progress
broken down by phase. The first phase was a bunch of research and discussion on
the design, so that would h
Fokko commented on PR #8659:
URL: https://github.com/apache/iceberg/pull/8659#issuecomment-1774998904
@bitsondatadev Yes, I'm so excited to see this happening. Great work here! 🙌
I've created a project
[here](https://github.com/orgs/apache/projects/300/views/1). Maybe jus a label
mi
Fokko opened a new pull request, #289:
URL: https://github.com/apache/iceberg-docs/pull/289
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-m
bitsondatadev commented on PR #8659:
URL: https://github.com/apache/iceberg/pull/8659#issuecomment-1774978383
> @bitsondatadev have you created a project to track progress and to-do
list? Maybe others can help and pick up some tasks as well.
I want to but I don't have the ability to s
Fokko opened a new pull request, #288:
URL: https://github.com/apache/iceberg-docs/pull/288
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-m
Fokko commented on PR #287:
URL: https://github.com/apache/iceberg-docs/pull/287#issuecomment-1774970805
The changes are from master:
https://github.com/apache/iceberg/pull/8678/files#diff-4680d52dc70590abc27b56e8da794ddae43ef39ffdd0099b0d6d3802a12eb74f
(Maybe good to publish those anyway?)
Fokko closed pull request #287: Release: 1.4.1 common pages
URL: https://github.com/apache/iceberg-docs/pull/287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe,
Fokko opened a new pull request, #287:
URL: https://github.com/apache/iceberg-docs/pull/287
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-m
Fokko opened a new pull request, #8906:
URL: https://github.com/apache/iceberg/pull/8906
I don't think this is necessary since the API doesn't change between minor
versions, but it is part of the release process :)
--
This is an automated message from the Apache Git Service.
To respond to
Fokko commented on PR #285:
URL: https://github.com/apache/iceberg-docs/pull/285#issuecomment-1774940533
Thanks @nastra 🙌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
T
Fokko merged PR #285:
URL: https://github.com/apache/iceberg-docs/pull/285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap
jbonofre commented on PR #8895:
URL: https://github.com/apache/iceberg/pull/8895#issuecomment-1774841047
As it's "big" update, let me do some tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
jbonofre commented on PR #8898:
URL: https://github.com/apache/iceberg/pull/8898#issuecomment-1774837636
jaxb update can be challenging with the jakarta namespace change. I will
work on the jakarta namespace support in a specific PR.
--
This is an automated message from the Apache Git Ser
jbonofre commented on PR #8899:
URL: https://github.com/apache/iceberg/pull/8899#issuecomment-1774835791
I'm checking this one as part of the gradle (and revapi related issue)
update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
jbonofre commented on PR #8894:
URL: https://github.com/apache/iceberg/pull/8894#issuecomment-1774832069
I will work on this update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368378892
##
core/src/main/java/org/apache/iceberg/rest/RESTViewOperations.java:
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more
nastra closed issue #8850: doc: The doc for catalog is missing.
URL: https://github.com/apache/iceberg/issues/8850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscrib
nastra merged PR #284:
URL: https://github.com/apache/iceberg-docs/pull/284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.a
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368356669
##
open-api/rest-catalog-open-api.yaml:
##
@@ -2033,6 +2537,68 @@ components:
metadata-location:
type: string
+CreateViewRequest:
+ type:
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368352618
##
open-api/rest-catalog-open-api.yaml:
##
@@ -1860,6 +2320,26 @@ components:
items:
type: string
+AddViewVersionUpdate:
+ all
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368347557
##
core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java:
##
@@ -971,4 +996,261 @@ public void commitTransaction(SessionContext context,
List commits)
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368337508
##
open-api/rest-catalog-open-api.yaml:
##
@@ -1056,6 +1056,357 @@ paths:
}
}
+ /v1/{prefix}/namespaces/{namespace}/views:
+parame
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368330924
##
core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java:
##
@@ -943,11 +968,11 @@ private static Cache
newSessionCache(Map pr
.build();
}
-
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368329224
##
open-api/rest-catalog-open-api.yaml:
##
@@ -2033,6 +2537,68 @@ components:
metadata-location:
type: string
+CreateViewRequest:
+ type:
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368326061
##
core/src/main/java/org/apache/iceberg/rest/RESTViewOperations.java:
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368323650
##
open-api/rest-catalog-open-api.yaml:
##
@@ -2033,6 +2537,68 @@ components:
metadata-location:
type: string
+CreateViewRequest:
+ type:
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1368319133
##
open-api/rest-catalog-open-api.yaml:
##
@@ -1056,6 +1056,357 @@ paths:
}
}
+ /v1/{prefix}/namespaces/{namespace}/views:
+parame
Zhangg7723 commented on issue #8663:
URL: https://github.com/apache/iceberg/issues/8663#issuecomment-1774648837
rewrite data files and position delete files with Spark, expire snapshots
and clean orphan files periodically
--
This is an automated message from the Apache Git Service.
To res
mstebelev opened a new issue, #8904:
URL: https://github.com/apache/iceberg/issues/8904
### Apache Iceberg version
1.3.1
### Query engine
Spark
### Please describe the bug 🐞
Hi. After upgrading spark to 3.4 I sometimes come across the following
problem: whi
liurenjie1024 commented on code in PR #82:
URL: https://github.com/apache/iceberg-rust/pull/82#discussion_r1368190040
##
crates/iceberg/src/avro/schema.rs:
##
@@ -431,14 +456,35 @@ impl AvroSchemaVisitor for AvroSchemaToSchema {
AvroSchema::Date => Type::Primitive(P
81 matches
Mail list logo
