findepi commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2585320268
Given that many approvals, it looks ready to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above t
findepi commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2585320135
It seems we're in agreement. The deadlock must be resolved. This PR
supposedly avoids any deadlocks at the cost of increased memory usage, which is
fair. I think we should not remove me
stevenzwu commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2584586732
@findepi the deadlock issue is probably the worse of the two. the deadlock
issue probably needs to be addressed urgently.
--
This is an automated message from the Apache Git Service
findepi commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2584202338
IIRC the OOM problem was a real production problem (cc @raunaqmorarka
@dekimir @losipiuk), so I am not convinced it's OK to restore it.
--
This is an automated message from the Apache
RussellSpitzer commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2584026138
> > As noted in my comment to Piotr, I think this is a fix to the deadlock
but I think it may be better to just remove the yielding behavior all together
>
> Would this res
findepi commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2583924725
> As noted in my comment to Piotr, I think this is a fix to the deadlock but
I think it may be better to just remove the yielding behavior all together
Would this restore OOM prob
sopel39 commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2579996415
> @sopel39 I assume you tried this change and it avoided deadlock problem.
Has this been tested with the OOM scenario for large manifest files?
We tested in staging env. So far co
stevenzwu commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2578314363
BTW, I like the new direction that @RussellSpitzer outlined. using byte size
(instead of number of elements) is more intuitive and easier to calculate a
good default to cap memory foo
RussellSpitzer commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2575944738
@findepi I'm on board with this but I want to make sure you are also happy
with this. I'm unsure of whether having the ability to yield before files will
really help memory press
sopel39 commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1905856982
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
@Over
RussellSpitzer commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2569914295
From a discussion I had with @sopel39 today;
I think we can go forward this solution but I think it will basically
re-introduce the memory usage issue that we saw previousl
RussellSpitzer commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1901098696
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
RussellSpitzer commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1901095259
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +263,21 @@ private static class Task implements
Supplier>>, Closeable {
sopel39 commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2551861574
rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubsc
sopel39 commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1890160296
##
core/src/test/java/org/apache/iceberg/util/TestParallelIterable.java:
##
@@ -171,38 +171,55 @@ public void limitQueueSize() {
}
@Test
- public void queueS
findepi commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1888440219
##
core/src/test/java/org/apache/iceberg/util/TestParallelIterable.java:
##
@@ -171,38 +171,55 @@ public void limitQueueSize() {
}
@Test
- public void queueS
findepi commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1888434738
##
core/src/test/java/org/apache/iceberg/util/TestParallelIterable.java:
##
@@ -171,38 +171,55 @@ public void limitQueueSize() {
}
@Test
- public void queueS
sopel39 commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1887631464
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
@Over
osscm commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1887173077
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
@Overri
sopel39 commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1886884393
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
@Over
osscm commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1885784288
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
@Overri
sopel39 commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1884535343
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
@Over
RussellSpitzer commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1884527019
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
sopel39 commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1884520727
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
@Over
osscm commented on code in PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#discussion_r1884308287
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -257,17 +257,17 @@ private static class Task implements
Supplier>>, Closeable {
@Overri
sopel39 commented on PR #11781:
URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2541648505
cc @findepi @RussellSpitzer @osscm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
26 matches
Mail list logo
