swat1234 opened a new issue, #8713:
URL: https://github.com/apache/iceberg/issues/8713
Iceberg tables not compressing parquet file in s3. When the below Table
parameters are used for the Compression the file size is increasing in
comparison with uncompression. Can some one please assist on
nastra commented on code in PR #8552:
URL: https://github.com/apache/iceberg/pull/8552#discussion_r1345331616
##
core/src/test/java/org/apache/iceberg/view/TestViewMetadataParser.java:
##
@@ -308,4 +322,57 @@ public void
replaceViewMetadataWithMultipleSQLsForDialect() throws Ex
nastra commented on code in PR #8648:
URL: https://github.com/apache/iceberg/pull/8648#discussion_r1345339487
##
api/src/main/java/org/apache/iceberg/view/UpdateViewLocation.java:
##
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more
nastra commented on code in PR #8648:
URL: https://github.com/apache/iceberg/pull/8648#discussion_r1345349415
##
core/src/main/java/org/apache/iceberg/view/SetViewLocation.java:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more co
nastra commented on PR #8648:
URL: https://github.com/apache/iceberg/pull/8648#issuecomment-1746330496
thanks for the reviews @rdblue and @amogh-jahagirdar, I've adjusted the code
accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please lo
gzagarwal commented on issue #7094:
URL: https://github.com/apache/iceberg/issues/7094#issuecomment-1746338892
> okay let me pick, I am working on iceberg-aws
Shall i assume current test cases are working? on my local system they are
not working so asking this question.
Is there a
nastra commented on issue #8713:
URL: https://github.com/apache/iceberg/issues/8713#issuecomment-1746347077
I see that you configured `"write.metadata.compression-codec": "gzip"` but
this is for table metadata files being compressed, not individual data files.
Also any particular reason to
nastra commented on issue #7094:
URL: https://github.com/apache/iceberg/issues/7094#issuecomment-1746356259
@gzagarwal yes the tests should all be working. What issue are you seeing?
https://github.com/apache/iceberg/blob/a3aff95f9e60962240b94242e24a778760bdd1d9/CONTRIBUTING.md
and https://
swat1234 commented on issue #8713:
URL: https://github.com/apache/iceberg/issues/8713#issuecomment-1746360361
I am are trying to reduce the storage space of the files by applying Snappy
or Gzip compression. I can see metadata is getting compression to gzip but not
the data files. Could you
nastra commented on issue #8713:
URL: https://github.com/apache/iceberg/issues/8713#issuecomment-1746404632
I would probably start first by reducing the amount of random table
properties being set.
As I mentioned earlier, the one that matters in your case is
`write.parquet.compression-c
jbonofre commented on issue #8485:
URL: https://github.com/apache/iceberg/issues/8485#issuecomment-1746467781
FYI, I tested `revapi` with Gradle 8.3 (on my PR). Here's the test I did:
* I added `void test();` method in `SessionCatalog`
* I added the corresponding `public void test() {}`
nastra merged PR #8648:
URL: https://github.com/apache/iceberg/pull/8648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac
gzagarwal commented on issue #7094:
URL: https://github.com/apache/iceberg/issues/7094#issuecomment-1746492707
> @gzagarwal yes the tests should all be working. What issue are you seeing?
https://github.com/apache/iceberg/blob/a3aff95f9e60962240b94242e24a778760bdd1d9/CONTRIBUTING.md
and htt
ajantha-bhat commented on issue #8485:
URL: https://github.com/apache/iceberg/issues/8485#issuecomment-1746502923
> With Gradle 8.3, revapi doesn't fail, it doesn't detect the API change.
Yes. Thats what we have observed with Gradle 8.2 also.
Maybe we need to raise an issue to revA
gyfora commented on code in PR #8653:
URL: https://github.com/apache/iceberg/pull/8653#discussion_r1345500193
##
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##
@@ -18,72 +18,39 @@
*/
package org.apache.iceberg.flink.sink;
-import static org
nk1506 commented on code in PR #8709:
URL: https://github.com/apache/iceberg/pull/8709#discussion_r1345504986
##
docs/spark-queries.md:
##
@@ -357,6 +381,31 @@ SELECT * FROM prod.db.table.all_data_files;
|
0|s3://.../dt=20210103/0-0-26222098-032f-472b-8ea5-651a55b2121
ajantha-bhat commented on code in PR #8709:
URL: https://github.com/apache/iceberg/pull/8709#discussion_r1345506789
##
docs/spark-queries.md:
##
@@ -357,6 +381,31 @@ SELECT * FROM prod.db.table.all_data_files;
|
0|s3://.../dt=20210103/0-0-26222098-032f-472b-8ea5-651a5
JanKaul opened a new pull request, #74:
URL: https://github.com/apache/iceberg-rust/pull/74
This is a draft PR to implement some functionality for an in memory catalog.
The in memory catalog is supposed to simplify tests.
Additionally this PR serves as a way to test the requirements f
nk1506 commented on issue #8670:
URL: https://github.com/apache/iceberg/issues/8670#issuecomment-1746511694
@RussellSpitzer , I want to look into it and fix it accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
jbonofre commented on issue #8485:
URL: https://github.com/apache/iceberg/issues/8485#issuecomment-1746517112
I think the problem is more on gradle or a mix with gradle and revapi gradle
plugin.
I'm doing a bisect on gradle to identify the change causing the issue.
--
This is an
gyfora commented on code in PR #8653:
URL: https://github.com/apache/iceberg/pull/8653#discussion_r1345500193
##
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:
##
@@ -18,72 +18,39 @@
*/
package org.apache.iceberg.flink.sink;
-import static org
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1345527136
##
core/src/main/java/org/apache/iceberg/catalog/BaseSessionCatalog.java:
##
@@ -30,8 +30,10 @@
import org.apache.iceberg.exceptions.NamespaceNotEmptyException;
import
JanKaul opened a new issue, #75:
URL: https://github.com/apache/iceberg-rust/issues/75
Iceberg catalogs that make use of a `*.metadata.json` file to store the
table metadata require the `metadata_location` and the `TableMetadata` of a
Table to perform an `update_table` operation ([see
here
ajantha-bhat commented on issue #8714:
URL: https://github.com/apache/iceberg/issues/8714#issuecomment-1746561829
cc: @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
Tavisca-vinayak-bhadage commented on issue #8155:
URL: https://github.com/apache/iceberg/issues/8155#issuecomment-1746581142
This compareWithFileList would be good solution for AWS S3 based iceberg
tables also. As we are facing below exception with default remove orphan file
implementation
nk1506 commented on issue #8670:
URL: https://github.com/apache/iceberg/issues/8670#issuecomment-1746623860
@hwfff , could you please share the stack-trace if handy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
swat1234 commented on issue #8713:
URL: https://github.com/apache/iceberg/issues/8713#issuecomment-1746699625
We tried with only write.parquet.compression-codec parameter set to snappy,
gzip but it is not working. Instead of compressing, the size is getting
increased.
--
This is an autom
RussellSpitzer commented on issue #8713:
URL: https://github.com/apache/iceberg/issues/8713#issuecomment-1746708682
If you are only trying with sub kilobyte files the results will be bad. You
have some amortized costs there and most of the file (footers) will not be
compressed. Try with lar
gyfora commented on code in PR #8653:
URL: https://github.com/apache/iceberg/pull/8653#discussion_r1345697890
##
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/SinkBase.java:
##
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+
gyfora commented on code in PR #8653:
URL: https://github.com/apache/iceberg/pull/8653#discussion_r1345702939
##
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java:
##
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under on
gyfora commented on code in PR #8653:
URL: https://github.com/apache/iceberg/pull/8653#discussion_r1345705474
##
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java:
##
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under on
gyfora commented on code in PR #8653:
URL: https://github.com/apache/iceberg/pull/8653#discussion_r1345702939
##
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java:
##
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under on
gyfora commented on code in PR #8653:
URL: https://github.com/apache/iceberg/pull/8653#discussion_r1345706466
##
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java:
##
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under on
jayceslesar commented on issue #39:
URL: https://github.com/apache/iceberg-python/issues/39#issuecomment-1746873989
This is confirmed an upstream bug in pyarrow 13.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
amogh-jahagirdar commented on issue #8713:
URL: https://github.com/apache/iceberg/issues/8713#issuecomment-1746920616
+1 to @RussellSpitzer point. These files seem way too small for compression
to play a significant role and be meaningful. Compression is most noticeable on
significant amoun
Fokko opened a new pull request, #40:
URL: https://github.com/apache/iceberg-python/pull/40
For V1 and V2 there are some differences that are hard to enforce without
this:
- `1: snapshot_id` is required for V1, optional for V2
- `105: block_size_in_bytes` needs to be written for V1
Fokko commented on code in PR #40:
URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1345991299
##
pyiceberg/manifest.py:
##
@@ -262,15 +346,13 @@ class DataFile(Record):
"split_offsets",
"equality_ids",
"sort_order_id",
-"spec_
shreyanshR7 opened a new pull request, #8715:
URL: https://github.com/apache/iceberg/pull/8715
#7154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1346067694
##
core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java:
##
@@ -568,4 +649,9 @@ private static TableIdentifier
identFromPathVars(Map pathVars) {
retu
nastra commented on code in PR #7913:
URL: https://github.com/apache/iceberg/pull/7913#discussion_r1346082143
##
open-api/rest-catalog-open-api.yaml:
##
@@ -1014,6 +1014,357 @@ paths:
}
}
+ /v1/{prefix}/namespaces/{namespace}/views:
+parame
nastra opened a new pull request, #8716:
URL: https://github.com/apache/iceberg/pull/8716
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mai
Fokko opened a new pull request, #41:
URL: https://github.com/apache/iceberg-python/pull/41
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-m
shreyanshR7 commented on issue #7154:
URL: https://github.com/apache/iceberg/issues/7154#issuecomment-1747275843
@nastra I tried to implement the above method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL ab
nk1506 commented on code in PR #8715:
URL: https://github.com/apache/iceberg/pull/8715#discussion_r1346198636
##
api/src/test/java/org/apache/iceberg/metrics/TestDefaultTimer.java:
##
@@ -104,7 +106,7 @@ public void measureRunnable() {
Runnable runnable =
() -> {
shreyanshR7 commented on PR #8715:
URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1747371647
Oh i see, the code uses a while loop checks the current time until the
condition is met.But its asked to replace Thread.sleep method with awaitility,
should i implement your suggestio
thesquelched closed pull request #2563: Parquet: Support filter operations on
int96 timestamps
URL: https://github.com/apache/iceberg/pull/2563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the sp
mderoy commented on PR #2722:
URL: https://github.com/apache/iceberg/pull/2722#issuecomment-1747635549
@hankfanchiu this is awesome. the change in performance by doing this is
exponential... any chance on reviving this? or did the external issues put a
damper on this? maybe we can do this v
aokolnychyi commented on code in PR #8659:
URL: https://github.com/apache/iceberg/pull/8659#discussion_r1346544156
##
docs-new/site/releases.md:
##
@@ -0,0 +1,777 @@
+---
+title: "Releases"
+---
+
+
+## Downloads
+
+The latest version of Iceberg is [{{ icebergVersion
}}](https:
aokolnychyi commented on PR #8659:
URL: https://github.com/apache/iceberg/pull/8659#issuecomment-1747740766
My primary concern of moving the docs into the main repo was versioning and
pollution. It seems like `git worktree` should solve that. I deployed this
locally, it seems pretty straigh
bitsondatadev commented on PR #8659:
URL: https://github.com/apache/iceberg/pull/8659#issuecomment-1747804852
> My primary concern of moving the docs into the main repo was versioning
and pollution. It seems like `git worktree` should solve that. I deployed this
locally, it seems pretty str
github-actions[bot] commented on issue #7297:
URL: https://github.com/apache/iceberg/issues/7297#issuecomment-1747821140
This issue has been automatically marked as stale because it has been open
for 180 days with no activity. It will be closed in next 14 days if no further
activity occurs.
Ge opened a new issue, #8719:
URL: https://github.com/apache/iceberg/issues/8719
### Feature Request / Improvement
Consider the following continuous insertion into a keyless table:
```
SET 'execution.checkpointing.interval' = '10 s';
SET 'sql-client.execution.result-mo
Ge opened a new issue, #8720:
URL: https://github.com/apache/iceberg/issues/8720
### Query engine
Flink 1.17.1
### Question
According to https://iceberg.apache.org/docs/latest/flink/#flink-to-iceberg,
Iceberg does not handle Flink's `null`. Can you please describe th
nk1506 commented on code in PR #8715:
URL: https://github.com/apache/iceberg/pull/8715#discussion_r1346743526
##
flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceContinuous.java:
##
@@ -325,7 +328,7 @@ public void testSpecificSnapshotTimestamp() t
nk1506 commented on PR #8709:
URL: https://github.com/apache/iceberg/pull/8709#issuecomment-1747996344
@szehon-ho , Please review and share the feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above t
nk1506 commented on PR #8706:
URL: https://github.com/apache/iceberg/pull/8706#issuecomment-1747996641
@nastra , please look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
shreyanshR7 commented on PR #8715:
URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1748006574
Thanks for the suggestion @nk1506 ,I'll update that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
UR
jacobmarble commented on PR #8683:
URL: https://github.com/apache/iceberg/pull/8683#issuecomment-1748044374
@Fokko - friendly reminder to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th
shreyanshR7 commented on PR #8715:
URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1748046446
I've made the changes@nk1506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
pvary commented on PR #2722:
URL: https://github.com/apache/iceberg/pull/2722#issuecomment-1748085546
@mderoy: I am still here to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the speci
pvary commented on issue #8718:
URL: https://github.com/apache/iceberg/issues/8718#issuecomment-1748095935
Is this for a V2 table?
I have seen deleting rows working using V2 table, Java code with the stream
API, but I yet to try out SQL.
--
This is an automated message from the Apache
Ge commented on issue #8718:
URL: https://github.com/apache/iceberg/issues/8718#issuecomment-1748157735
Yes, this is a V2 table. I added the DDL to the description now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and u
nastra merged PR #8706:
URL: https://github.com/apache/iceberg/pull/8706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac
nastra closed issue #1601: Spark SQL Extensions: Document all stored procedures
URL: https://github.com/apache/iceberg/issues/1601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
ashutosh-roy commented on code in PR #8707:
URL: https://github.com/apache/iceberg/pull/8707#discussion_r1346869623
##
dell/src/test/java/org/apache/iceberg/dell/mock/ecs/EcsS3MockRule.java:
##
@@ -178,4 +163,16 @@ public String bucket() {
public String randomObjectName() {
nastra commented on PR #8715:
URL: https://github.com/apache/iceberg/pull/8715#issuecomment-1748190639
The goal of https://github.com/apache/iceberg/issues/7154 is to convert
`Thread.sleep` usages to Awaitility where it makes sense. We don't want to
blindly just replace all `Thread.sleep` u
paulpaul1076 opened a new issue, #8721:
URL: https://github.com/apache/iceberg/issues/8721
### Apache Iceberg version
1.3.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Spark fails to write the dataframe with new schema after updatin
67 matches
Mail list logo
