github-actions[bot] commented on code in PR #69043:
URL: https://github.com/apache/doris/pull/69043#discussion_r4269905055
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java:
##########
@@ -311,7 +337,31 @@ Map<String, FileContent> buildDeleteFileContentMap(Table
icebergTable) {
throw new DorisConnectorException("Failed to build delete file
content map: " + e.getMessage(), e);
}
lastDeleteManifestReadCount = reads;
- return deleteFileContentByPath;
+ return fileKindByPath;
+ }
+
+ /**
+ * Tells a manifest or manifest list that {@link #buildFileKindMap} did
not see apart by its file name. The
+ * expiration refreshes the table when it commits, so it also expires
snapshots that another writer committed
+ * after the map was built. SnapshotProducer names a manifest list
+ * {@code snap-<snapshot id>-<attempt>-<commit UUID>.avro} and a manifest
{@code <commit UUID>-m<N>.avro};
+ * Spark's {@code rewrite_manifests} names a manifest {@code
optimized-m-<UUID>.avro}. Returns null for any
+ * other file. A delete file of such a snapshot still counts as a data
file, as its name does not tell the
+ * two apart.
+ */
+ @VisibleForTesting
+ static FileKind metadataFileKindByName(String path) {
+ String fileName = path.substring(path.lastIndexOf('/') + 1);
+ if (!fileName.endsWith(".avro")) {
+ return null;
+ }
+ if (fileName.startsWith("snap-")) {
+ return FileKind.MANIFEST_LIST;
+ }
+ if (fileName.contains("-m-") ||
MANIFEST_FILE_NAME.matcher(fileName).matches()) {
Review Comment:
[P2] Keep Avro data files out of the manifest-name fallback
`buildFileKindMap` inventories metadata and delete paths, so an ordinary
data file from an already-loaded snapshot also takes this fallback. A valid
Avro file named `s3://b/db1/t/data/sales-m1.avro` matches the new
`-m<digits>.avro` rule. Append it, delete it in the next snapshot, append a
survivor, and expire with `older_than=4102444800000, retain_last=1`: the
callback counts it in `deleted_manifest_files_count` rather than
`deleted_data_files_count` (one fewer data file and one extra manifest). The
previous filename check counted this path as data. Preserve known data-file
identity or constrain the fallback to verified metadata paths, and add this
collision to the test coverage. This occurs without a concurrent writer and is
distinct from the existing refresh thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
