BharadwajaD commented on issue #13737:
URL: https://github.com/apache/iceberg/issues/13737#issuecomment-3157344315
- The logs are from driver and executor logs contains heartbeat failure
errors ( because driver is kind of down)
- Sharing ExpireSnapShotSparkAction source code, screenshot from spark ui
and driver logs:
```java
public Dataset<FileInfo> expireFiles() {
if (this.expiredFileDS == null) {
TableMetadata originalMetadata = this.ops.current();
org.apache.iceberg.ExpireSnapshots expireSnapshots =
this.table.expireSnapshots();
for(long id : this.expiredSnapshotIds) {
expireSnapshots = expireSnapshots.expireSnapshotId(id);
}
if (this.expireOlderThanValue != null) {
expireSnapshots =
expireSnapshots.expireOlderThan(this.expireOlderThanValue);
}
if (this.retainLastValue != null) {
expireSnapshots =
expireSnapshots.retainLast(this.retainLastValue);
}
expireSnapshots.cleanExpiredFiles(false).commit();
TableMetadata updatedMetadata = this.ops.refresh();
Dataset<FileInfo> validFileDS = this.fileDS(updatedMetadata);
// line 1
Set<Long> deletedSnapshotIds =
this.findExpiredSnapshotIds(originalMetadata, updatedMetadata);
Dataset<FileInfo> deleteCandidateFileDS =
this.fileDS(originalMetadata, deletedSnapshotIds); // line 2
this.expiredFileDS = deleteCandidateFileDS.except(validFileDS);
}
return this.expiredFileDS;
}
private Dataset<FileInfo> fileDS(TableMetadata metadata, Set<Long>
snapshotIds) {
Table staticTable = this.newStaticTable(metadata, this.table.io());
return this.contentFileDS(staticTable,
snapshotIds).union(this.manifestDS(staticTable,
snapshotIds)).union(this.manifestListDS(staticTable,
snapshotIds)).union(this.statisticsFileDS(staticTable, snapshotIds)); // line 3
}
```
<img width="1885" height="730" alt="Image"
src="https://github.com/user-attachments/assets/f7452edc-6584-40ed-9f87-d62d079cc1e6";
/>
<img width="1632" height="715" alt="Image"
src="https://github.com/user-attachments/assets/53da1643-8e6f-4b69-9c7c-a86d50b44072";
/>
- Here I am getting Page error just after completing stage 5.
- It looks like `line 1` is completed but `line 2` is causing problem.
- Please correct me if I am wrong and please suggest any configs or anything
if I am missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
