BsoBird opened a new issue, #9327:
URL: https://github.com/apache/iceberg/issues/9327
### Apache Iceberg version
1.4.2 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
SPARK 3.4.1.
```
Caused by: java.io.FileNotFoundException: File does not exist:
/iceberg-catalog/warehouse/dwd/b_std_category/data/00012-4526569-b66acfb2-bea0-46af-a6c8-01d9d1731b35-00001.orc
```
我们发现,在一些情况下,ICEBERG表可能会出现FileNotFoundException.
事情的经过是这样子的:
1.我们每天凌晨1点会开始执行8个表的MERGE INTO操作. 在操作完成后,会执行如下三个操作:
```
CALL xxx.system.rewrite_manifests('dwd.b_std_category', false);
CALL xxx.system.remove_orphan_files(table => 'dwd.b_std_category');
CALL xxx.system.expire_snapshots(table => 'dwd.b_std_category', retain_last
=> 10);
```
2.今天执行MERGE操作时,发生了OOM.导致容器被杀死
```
Job aborted due to stage failure: Authorized committer (attemptNumber=0,
stage=415011, partition=1392) failed; but task commit success, data duplication
may happen. reason=ExecutorLostFailure(3759,true,Some(Container killed by YARN
for exceeding physical memory limits. 40.1 GB of 40 GB physical memory used.
Consider boosting spark.executor.memoryOverhead.))
```
3.发生OOM时,dwd.b_std_category 表正在执行这一句指令:
```
CALL xxx.system.expire_snapshots(table => 'dwd.b_std_category', retain_last
=> 10);
```
4.当我们重新恢复SPARK任务后,我们发现dwd.b_std_category表无法被读取.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
