github-actions[bot] closed issue #10054: Spark rewrite Files Action OOM
URL: https://github.com/apache/iceberg/issues/10054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
github-actions[bot] commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2603392960
This issue has been closed because it has not received any activity in the
last 14 days since being marked as 'stale'
--
This is an automated message from the Apache
github-actions[bot] commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2571807122
This issue has been automatically marked as stale because it has been open
for 180 days with no activity. It will be closed in next 14 days if no further
activity occur
manuzhang commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2218139056
I've created a [draft PR](https://github.com/apache/iceberg/pull/10667)
which has been verified in our environment.
--
This is an automated message from the Apache Git Service.
pdames commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2121567721
Any updates here @Zhanxiao-Ma? Would love to take a look at what you've
implemented if you've got a pending PR to link back to this issue, and see if
there's an opportunity to work
manuzhang commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2100175845
> I have implemented a disk-based map to solve this problem. Is this what
Iceberg expects? If so, I will submit the code.
@Zhanxiao-Ma I think it will be valuable to the commun
nk1506 commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2041538414
> @nk1506 Echoing Russell's comments, how many small files are there in your
OOM case? How much memory do you set up?
I didn't use spark-engine for compaction. I was using Jav
Zhanxiao-Ma commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2041085307
> @nk1506 Echoing Russell's comments, how many small files are there in your
OOM case? How much memory do you set up?
@RussellSpitzer I believe increasing memory is not a
manuzhang commented on issue #10054:
URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2024928341
It's not forbidden to delete too many records but could increase memory
required in the driver. If you are using position deletes, there's
`rewrite_position_delete_files`. As for
