Doris-Breakwater commented on issue #69664: URL: https://github.com/apache/doris/issues/69664#issuecomment-6434041160
The report is consistent with a cleanup gap for Azure HNS at `a626232f5cc1f456117ffe9abca2b3d0578735d4`. I verified the code paths at that commit; I have not independently rerun the query against an HNS account. The reported correct result and removal of all data files indicate a directory-metadata accumulation problem; this report does not establish query correctness or data-loss impact. The listing slowdown is reported but not yet quantified. Verified code behavior: - [`SpillFile::gc()`](https://github.com/apache/doris/blob/a626232f5cc1f456117ffe9abca2b3d0578735d4/be/src/exec/spill/spill_file.cpp#L61-L125) deletes the spill-file directory through its pinned filesystem. [`S3FileSystem::delete_directory_impl()`](https://github.com/apache/doris/blob/a626232f5cc1f456117ffe9abca2b3d0578735d4/be/src/io/fs/s3_file_system.cpp#L222-L237) normalizes the deletion prefix to `directory/` and deletes listed descendants. It never separately deletes the exact key `directory`. Consequently, a successful prefix deletion can leave the directory entry described here and will not enqueue a retry. - [`AzureObjStorageClient::list_objects_page()` and `delete_object()`](https://github.com/apache/doris/blob/a626232f5cc1f456117ffe9abca2b3d0578735d4/common/cpp/obj-client/azure_obj_storage_client.cpp#L354-L495) use Blob listing with the supplied prefix and exact-key Blob deletion respectively. Microsoft documents that HNS listings include directories, and that Blob deletion removes a directory only when it is empty. This supports the proposed exact-directory deletion after its children are gone. [Azure API behavior](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues#blob-storage-apis). - [`QueryContext` teardown](https://github.com/apache/doris/blob/a626232f5cc1f456117ffe9abca2b3d0578735d4/be/src/runtime/query_context.cpp#L231-L268) calls query-directory cleanup, but [`delete_query_spill_directory()`](https://github.com/apache/doris/blob/a626232f5cc1f456117ffe9abca2b3d0578735d4/be/src/exec/spill/spill_file_manager.cpp#L499-L538) immediately returns for remote stores. Thus removing only the spill-file directories would still leave the query directory. These two omissions explain the reported seven file directories plus one query directory, given the reported HNS entries. Suggested implementation direction: 1. Add Azure-aware removal of the exact directory entry after successful descendant cleanup. Retain the `/` boundary for listing: simply dropping it could match sibling names. Keep repeated deletion idempotent and propagate real failures to the existing retry mechanism. Preserve flat-namespace S3 behavior and the BE spill root. 2. Arrange query-directory cleanup after no spill users can publish more parts, retaining retries until failed child cleanup and parent cleanup both finish. Pin every filesystem/vault used by that query: [`RemoteSpillDataDir::ensure_ready()`](https://github.com/apache/doris/blob/a626232f5cc1f456117ffe9abca2b3d0578735d4/be/src/exec/spill/remote_spill_data_dir.cpp#L83-L139) may select a different filesystem for later files, while query bookkeeping currently records only `SpillDataDir*`. Merely removing the early return could therefore clean only the currently selected vault. Preserve pending-delete byte accounting when parent and child retries overlap. 3. If query cleanup recursively processes remaining directories, handle them from children to parents. The generic list-and-batch-delete helper does not impose that ordering; an HNS parent cannot be deleted while children remain. Treat this as a fix-design requirement, not an additional failure demonstrated by this report. Regression coverage should include real HNS directories, multiple spill files/parts, repeated cleanup, injected child and parent deletion failures followed by GC, cancellation, sibling-prefix isolation, and a vault change during a query. Assert that query-owned entries disappear and the BE root remains; also retain S3 and non-HNS Azure coverage. The existing [`FailedDeletionKeepsCapacityCharged` test](https://github.com/apache/doris/blob/a626232f5cc1f456117ffe9abca2b3d0578735d4/be/test/vec/spill/spill_file_s3_test.cpp#L1563-L1609) explicitly expects no remote query-directory requests, so HNS behavior needs additional coverage. Useful remaining evidence for a reproducible regression: the minimal SQL/data generator and relevant memory/spill settings; a sanitized before/after listing with exact names, directory flags/metadata and sizes; and the query ID with `[spill][delete]`/retry and query-deconstruction logs plus listing timestamps. These would distinguish live leftovers from an observation before teardown or a listing that includes deleted entries. For the SDK directory-delete check, include the API/SDK version and response status/error code. The stated Doris commit is already sufficient for code review; a query profile is not needed to establish this cleanup mechanism. Breakwater-GitHub-Analysis-Slot: slot_81ce8c156e15 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
