frankliee commented on code in PR #701:
URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1591903957
##
pyiceberg/catalog/hive.py:
##
@@ -111,6 +122,13 @@
HIVE2_COMPATIBLE = "hive.hive2-compatible"
HIVE2_COMPATIBLE_DEFAULT = False
+DEFAULT_LOCK_CHECK_MIN_WAI
HonahX commented on code in PR #703:
URL: https://github.com/apache/iceberg-python/pull/703#discussion_r1591829611
##
pyiceberg/transforms.py:
##
@@ -349,6 +356,13 @@ def dedup_name(self) -> str:
def preserves_order(self) -> bool:
return True
+@abstractmethod
a-agmon commented on issue #338:
URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2097495570
Thanks @zeodtr ,
We can certainly cache the manifest schema, and also recurse on the fields
read from the file.
Implementing a more efficient reader is also possible but ind
dependabot[bot] closed pull request #708: Build: Bump jinja2 from 3.1.3 to 3.1.4
URL: https://github.com/apache/iceberg-python/pull/708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific co
dependabot[bot] commented on PR #708:
URL: https://github.com/apache/iceberg-python/pull/708#issuecomment-2097489594
Looks like jinja2 is up-to-date now, so this is no longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Gi
HonahX merged PR #707:
URL: https://github.com/apache/iceberg-python/pull/707
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg
HonahX merged PR #706:
URL: https://github.com/apache/iceberg-python/pull/706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg
HonahX merged PR #699:
URL: https://github.com/apache/iceberg-python/pull/699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg
HonahX commented on code in PR #703:
URL: https://github.com/apache/iceberg-python/pull/703#discussion_r1591829611
##
pyiceberg/transforms.py:
##
@@ -349,6 +356,13 @@ def dedup_name(self) -> str:
def preserves_order(self) -> bool:
return True
+@abstractmethod
oku95 commented on issue #8655:
URL: https://github.com/apache/iceberg/issues/8655#issuecomment-209746
Thanks will check, does this behavior by default on in all version of
iceberg ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
HonahX merged PR #697:
URL: https://github.com/apache/iceberg-python/pull/697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg
HonahX merged PR #696:
URL: https://github.com/apache/iceberg-python/pull/696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg
HonahX commented on code in PR #704:
URL: https://github.com/apache/iceberg-python/pull/704#discussion_r1591786106
##
tests/integration/test_writes/test_writes.py:
##
@@ -833,3 +833,13 @@ def test_sanitize_character_partitioned(catalog: Catalog)
-> None:
)
assert le
HonahX commented on code in PR #701:
URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1591782640
##
pyiceberg/catalog/hive.py:
##
@@ -111,6 +122,13 @@
HIVE2_COMPATIBLE = "hive.hive2-compatible"
HIVE2_COMPATIBLE_DEFAULT = False
+DEFAULT_LOCK_CHECK_MIN_WAIT_T
HonahX commented on code in PR #701:
URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1591768312
##
pyiceberg/catalog/hive.py:
##
@@ -356,11 +371,46 @@ def _create_lock_request(self, database_name: str,
table_name: str) -> LockReque
return lock_requ
frankliee commented on code in PR #701:
URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1591771902
##
pyiceberg/catalog/hive.py:
##
@@ -111,6 +122,13 @@
HIVE2_COMPATIBLE = "hive.hive2-compatible"
HIVE2_COMPATIBLE_DEFAULT = False
+DEFAULT_LOCK_CHECK_MIN_WAI
HonahX commented on code in PR #701:
URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1591764194
##
pyiceberg/catalog/hive.py:
##
@@ -111,6 +122,13 @@
HIVE2_COMPATIBLE = "hive.hive2-compatible"
HIVE2_COMPATIBLE_DEFAULT = False
+DEFAULT_LOCK_CHECK_MIN_WAIT_T
manuzhang commented on issue #8655:
URL: https://github.com/apache/iceberg/issues/8655#issuecomment-2097170237
I think you need to check with AWS support. It looks to be related to Glue
arg parser.
--
This is an automated message from the Apache Git Service.
To respond to the message, ple
zeodtr commented on issue #338:
URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2097101287
@a-agmon My concerns are as follows:
1. The `manifest_file_schema_fields` hashmap should be calculated only once
in an application's lifetime (for performance).
2. There are m
nastra commented on PR #9008:
URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2097095003
@jacobmarble sorry for the delay here. I'm traveling this week and should be
able to get to this PR after the Iceberg summit
--
This is an automated message from the Apache Git Service.
flyrain commented on code in PR #10256:
URL: https://github.com/apache/iceberg/pull/10256#discussion_r1591603492
##
core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java:
##
@@ -215,6 +215,12 @@ public void initialize(String name, Map
unresolved) {
this.paths
dimas-b opened a new issue, #10279:
URL: https://github.com/apache/iceberg/issues/10279
### Apache Iceberg version
1.5.0
### Query engine
Spark
### Please describe the bug 🐞
What a table is partitioned by a column that contains `#` in the name, some
inserts
flyrain commented on code in PR #10256:
URL: https://github.com/apache/iceberg/pull/10256#discussion_r1591587764
##
core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java:
##
@@ -215,6 +215,12 @@ public void initialize(String name, Map
unresolved) {
this.paths
MehulBatra commented on issue #705:
URL: https://github.com/apache/iceberg-python/issues/705#issuecomment-2096955905
I would love to take a shot at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
flyrain commented on issue #10205:
URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2096945405
I think we are talking about the same thing here. I don't think there is a
need to change the REST spec.
--
This is an automated message from the Apache Git Service.
To respond t
dependabot[bot] opened a new pull request, #708:
URL: https://github.com/apache/iceberg-python/pull/708
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4.
Release notes
Sourced from https://github.com/pallets/jinja/releases";>jinja2's releases.
3.1.4
Thi
dependabot[bot] opened a new pull request, #707:
URL: https://github.com/apache/iceberg-python/pull/707
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4.
Release notes
Sourced from https://github.com/pallets/jinja/releases";>jinja2's releases.
3.1.4
Thi
gupteaj commented on issue #352:
URL: https://github.com/apache/iceberg-rust/issues/352#issuecomment-2096739753
@Fokko , @liurenjie1024
what changes are we looking for ?
Since rust is using option to hold snapshot id internally, but writing as -1
for V2 manifest files.
Does it need
dependabot[bot] opened a new pull request, #706:
URL: https://github.com/apache/iceberg-python/pull/706
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3.
Release notes
Sourced from https://github.com/pallets/werkzeug/releases";>werkzeug's
releases.
3
danielcweeks commented on issue #10205:
URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2096712578
@flyrain This is already spelled out in the [REST
Spec](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L78).
The warehouse property is just a
pvary commented on code in PR #10179:
URL: https://github.com/apache/iceberg/pull/10179#discussion_r1591425847
##
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/BaseDeltaTaskWriter.java:
##
@@ -38,7 +38,7 @@
import org.apache.iceberg.relocated.com.google.common.c
pvary commented on code in PR #10179:
URL: https://github.com/apache/iceberg/pull/10179#discussion_r1591411561
##
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestSinkV2Committer.java:
##
@@ -0,0 +1,808 @@
+/*
+ * Licensed to the Apache Software Founda
pvary commented on code in PR #10179:
URL: https://github.com/apache/iceberg/pull/10179#discussion_r1591405651
##
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/CachingTableSupplier.java:
##
@@ -33,7 +34,8 @@
* table loader should be used carefully when used wit
flyrain commented on issue #10205:
URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2096591207
`iceberg.rest-catalog.warehouse` normally points to a location like
`s3://my_bucket/warehouse_location`. Not sure Trino community is OK to use it
as the catalog name. Even if that
rodmeneses commented on code in PR #10179:
URL: https://github.com/apache/iceberg/pull/10179#discussion_r1591350292
##
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/IcebergManifestOutputFileFactory.java:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apach
HonahX merged PR #702:
URL: https://github.com/apache/iceberg-python/pull/702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg
rodmeneses commented on code in PR #10179:
URL: https://github.com/apache/iceberg/pull/10179#discussion_r1591338007
##
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/writer/BaseDeltaTaskWriter.java:
##
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foun
syun64 commented on PR #614:
URL: https://github.com/apache/iceberg-python/pull/614#issuecomment-2096517390
Hi @HonahX could we get your help in triggering this workflow to see if the
CI succeeds?
--
This is an automated message from the Apache Git Service.
To respond to the message, plea
kevinjqliu commented on issue #674:
URL: https://github.com/apache/iceberg-python/issues/674#issuecomment-2096498100
Looks like we're able to write (append/overwrite) a subset of schema, here's
a PR with a test
https://github.com/apache/iceberg-python/pull/704
This feature is a si
kevinjqliu opened a new pull request, #704:
URL: https://github.com/apache/iceberg-python/pull/704
#674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsub
jkolash opened a new issue, #10277:
URL: https://github.com/apache/iceberg/issues/10277
### Feature Request / Improvement
We are using the hadoop catalog and have encountered tables written by a 3rd
party that are encoding the latest-version.text field in a value higher than
supporte
ggershinsky commented on code in PR #5544:
URL: https://github.com/apache/iceberg/pull/5544#discussion_r1591261035
##
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (AS
syun64 opened a new pull request, #703:
URL: https://github.com/apache/iceberg-python/pull/703
Support writes with TimeTransforms.
TimeTransforms are can be supported natively within pyarrow as pyarrow
compute functions, without requiring conversions back and forth between Arrow
and
slessard opened a new issue, #10275:
URL: https://github.com/apache/iceberg/issues/10275
### Apache Iceberg version
1.5.1 (latest release)
### Query engine
Other
### Please describe the bug 🐞
I am writing a compatibility layer for Teradata so that it can acc
jacobmarble commented on PR #9008:
URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2096395819
> @epgif can you please address the test failures?
@nastra do you intend to review this pull request further?
--
This is an automated message from the Apache Git Service.
To r
oku95 commented on issue #8655:
URL: https://github.com/apache/iceberg/issues/8655#issuecomment-2096066510
Hi @manuzhang
Getting similar error in AWS Glue 4.0 Spark env
```
24/05/06 00:49:40 ERROR Executor: Exception in task 1.0 in stage 11.0 (TID
20)
java.lang.IllegalStateExcep
lurnagao-dahua commented on PR #10210:
URL: https://github.com/apache/iceberg/pull/10210#issuecomment-2095887589
@Fokko Hi, I am very much looking forward to and grateful for your review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
HonahX merged PR #610:
URL: https://github.com/apache/iceberg-python/pull/610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg
javrasya commented on issue #10274:
URL: https://github.com/apache/iceberg/issues/10274#issuecomment-2095823012
An interesting finding, once there is any update on the main branch after
the schema is evolved, this no longer becomes a problem.
I ran the following to trigger a dummy upd
ggershinsky commented on code in PR #5544:
URL: https://github.com/apache/iceberg/pull/5544#discussion_r1590856537
##
core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java:
##
@@ -146,21 +172,92 @@ protected void disableRefresh() {
}
protected String wr
ggershinsky commented on code in PR #5544:
URL: https://github.com/apache/iceberg/pull/5544#discussion_r1590852470
##
core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java:
##
@@ -146,21 +172,92 @@ protected void disableRefresh() {
}
protected String wr
ggershinsky commented on code in PR #5544:
URL: https://github.com/apache/iceberg/pull/5544#discussion_r1590852470
##
core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java:
##
@@ -146,21 +172,92 @@ protected void disableRefresh() {
}
protected String wr
sdd commented on code in PR #347:
URL: https://github.com/apache/iceberg-rust/pull/347#discussion_r1590836313
##
crates/iceberg/src/expr/visitors/inclusive_metrics_evaluator.rs:
##
@@ -0,0 +1,2255 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more cont
SamRaza356 commented on issue #6619:
URL: https://github.com/apache/iceberg/issues/6619#issuecomment-2095687386
There is another issue the files present in metadata(.json & .avro) files
are having original path locations that causes an issue if original table is
dropped. So what we can do i
Fokko commented on issue #10251:
URL: https://github.com/apache/iceberg/issues/10251#issuecomment-2095665215
Thanks everyone for testing! Make sure to share your findings on the
devlist: https://lists.apache.org/thread/j6kszdofsbxyv7cckmkd898hn2zfngpr
--
This is an automated message from
ZENOTME commented on issue #341:
URL: https://github.com/apache/iceberg-rust/issues/341#issuecomment-2095637090
Thanks! @Dysprosium0626 Sorry for replying late. Our original idea here is
to construct the delete schema outside the EqualityDeleteFileWriter.
```
let equality_ids = vec![
ZENOTME commented on issue #341:
URL: https://github.com/apache/iceberg-rust/issues/341#issuecomment-2095633576
Thanks! @Dysprosium0626 Sorry for replying late. Our original idea here is
to construct the delete schema outside the EqualityDeleteFileWriter.
```
let equality_ids = vec![
wForget commented on issue #10251:
URL: https://github.com/apache/iceberg/issues/10251#issuecomment-2095570106
> Feel free to test the Iceberg 1.5.2 staged artifacts
Kyuubi CI with iceberg 1.5.2 RC0 also works fine.
https://github.com/apache/kyuubi/pull/6361
--
This is an automated
javrasya opened a new issue, #10274:
URL: https://github.com/apache/iceberg/issues/10274
### Apache Iceberg version
1.4.3
### Query engine
Spark
### Please describe the bug 🐞
We have added to columns in a nested struct field by using Iceberg Java API.
I can
chenwyi2 commented on code in PR #9447:
URL: https://github.com/apache/iceberg/pull/9447#discussion_r1590740653
##
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteManifestsProcedure.java:
##
@@ -118,4 +126,15 @@ private InternalRow[] toOutputRows(Rewrit
osscm commented on issue #10205:
URL: https://github.com/apache/iceberg/issues/10205#issuecomment-2095492489
thanks @flyrain
using `namespace` to include `catalog-name` can also work.
though then what will be the use of `catalog-id`
Though IMO, as Spark and Trino using hive'
liurenjie1024 commented on issue #124:
URL: https://github.com/apache/iceberg-rust/issues/124#issuecomment-2095481603
> > How do you feel starting with one task for one manifest file
>
> you mean:
>
> * spawn a new task for each manifest, load the manifest
(entry.load_manifest(
Fokko merged PR #10267:
URL: https://github.com/apache/iceberg/pull/10267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa
Fokko merged PR #10160:
URL: https://github.com/apache/iceberg/pull/10160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa
Fokko commented on issue #124:
URL: https://github.com/apache/iceberg-rust/issues/124#issuecomment-2095455322
> so if we have a manifest_list with e.g. 5 entries, 1 is pruned
(ManifestEvaluator) we'd effectively spawn 4 tasks, to load the manifest and
handle all the data files; is this corr
Fokko commented on issue #338:
URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2095451994
I think creating a field-id to a field-name map is a good (interim) solution.
Keep in mind that the next Avro release is planned for this week:
https://lists.apache.org/thread
marvinlanhenke commented on issue #124:
URL: https://github.com/apache/iceberg-rust/issues/124#issuecomment-2095442660
> How do you feel starting with one task for one manifest file
you mean:
- spawn a new task for each manifest, load the manifest
(entry.load_manifest(...).await?)
manuzhang commented on PR #10160:
URL: https://github.com/apache/iceberg/pull/10160#issuecomment-2095436514
@pvary @nastra any more comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
Fokko commented on issue #124:
URL: https://github.com/apache/iceberg-rust/issues/124#issuecomment-2095389830
With Iceberg, the manifests are written to a target size (8 megabyte) by
default. Each manifest is bound to the same schema and partition, so you can
re-use the evaluators here. I w
adutra commented on code in PR #10256:
URL: https://github.com/apache/iceberg/pull/10256#discussion_r1590651661
##
core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java:
##
@@ -215,6 +215,12 @@ public void initialize(String name, Map
unresolved) {
this.paths =
adutra commented on code in PR #10256:
URL: https://github.com/apache/iceberg/pull/10256#discussion_r1590646919
##
core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java:
##
@@ -215,6 +215,12 @@ public void initialize(String name, Map
unresolved) {
this.paths =
71 matches
Mail list logo
