BharadwajaD opened a new issue, #13737:
URL: https://github.com/apache/iceberg/issues/13737
### Query engine
Query Engine: iceberg-spark-runtime-3.4_2.12:1.4.2
### Question
I am using sparkActions expireSnapshot call to clean up my iceberg table,
which has around 250 GB of data. While running this spark job with 2 executors
(each has 8GB memory) and 1 driver (16GB memory), I am facing
`TaskMemoryManager: Failed to allocate a page error.`
Can anyone please help me in finding root cause of this error and ways to
resolve this.
Sharing relevant code snippets:
```java
SparkConf sparkConf = new SparkConf();
sparkConf.setAppName("zephyrCleanUpJob");
// Configure Spark for HDFS access
sparkConf.set("spark.hadoop.fs.defaultFS", config.getHdfsNamenode());
sparkConf.set("spark.hadoop.fs.hdfs.impl",
"org.apache.hadoop.hdfs.DistributedFileSystem");
sparkConf.set("spark.hadoop.fs.file.impl",
"org.apache.hadoop.fs.LocalFileSystem");
sparkConf.set("spark.hadoop.fs.hdfs.impl.disable.cache", "true");
// Iceberg configurations
sparkConf.set("spark.sql.catalog.local",
"org.apache.iceberg.spark.SparkCatalog");
sparkConf.set("spark.sql.catalog.local.type", "hadoop");
sparkConf.set("spark.sql.catalog.local.warehouse",
config.getHdfsWarehousePath());
SparkSession sparkSession = SparkSession.builder()
.config(sparkConf)
.appName("zephyrCleanUpJob")
.getOrCreate();
// Expire old snapshots
SparkActions sparkActions = SparkActions.get(sparkSession);
ExpireSnapshots.Result result = sparkActions.expireSnapshots(table)
.option("stream-results", "true")
.expireOlderThan(expirationTime)
.execute();
```
Error logs
```25/08/04 16:31:10 INFO BlockManagerInfo: Removed broadcast_10_piece0 on
spark-cb474f9875d04f71-driver-svc.tns-postorder-playground.svc:7079 in memory
(size: 26.5 KiB, fre
25/08/04 16:31:24 WARN TaskMemoryManager: Failed to allocate a page
(67108848 bytes), try again.
25/08/04 16:31:25 WARN TaskMemoryManager: Failed to allocate a page
(67108848 bytes), try again.
25/08/04 16:31:25 WARN TaskMemoryManager: Failed to allocate a page
(67108848 bytes), try again.
25/08/04 16:31:26 WARN TaskMemoryManager: Failed to allocate a page
(67108848 bytes), try again.```
Thanks in advance !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
