eubnara opened a new pull request, #8700:
URL: https://github.com/apache/hadoop/pull/8700
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
See YARN-11987 for the full analysis. Summary:
When LocalizerRunner cleans up after an aborted localization, it batches the
leftover dirs as baseDirs with an empty relative path, so `delete_as_user()`
sets `needs_tt_user = 1` and `delete_path()` performs the final top-level rmdir
*only* as the NM user. That works for NM-owned parents (`usercache/<user>`),
but always fails with EACCES for private filecache entries
(`usercache/<user>/filecache/<id>`), whose parent is owned by the run-as user
with mode 0710. Every aborted localization logs a `DeleteAsUser ... exit code:
255` error and leaks an empty directory that is never retried.
This PR makes `delete_path()` try the final rmdir as the current (run-as)
user first, falling back to `rmdir_as_nm()` on EACCES/EPERM when
`needs_tt_user` is set:
* NM-owned parents keep working exactly as before (rmdir as user fails, NM
fallback succeeds).
* User-owned parents are fixed (rmdir as user succeeds immediately).
* No privilege escalation: the NM-user rmdir already existed; the fix only
tries the less-privileged user first. The fallback is restricted to
EACCES/EPERM because other errors (ENOTEMPTY, EBUSY, ...) would fail
identically for the NM user.
An alternative would be to fix the caller (one FileDeletionTask per path,
with the path as subDir), but batching several absolute paths into one
container-executor invocation is only possible through baseDirs, so that costs
one exec per path and leaves the same trap for any other caller. Fixing
`delete_path()` covers all callers in one place.
### How was this patch tested?
Added `test_delete_dir_in_user_owned_parent` to test-container-executor.c.
The failure mode needs the final rmdir to run as a non-root NM user different
from the directory owner, so the test only exercises the bug in the root
invocation mode with two distinct users:
```bash
# build native, then as root:
./test-container-executor <nm-user> <run-as-user> # e.g. yarn nobody
```
Before the fix this fails the new test (rmdir_as_nm gets EACCES); after the
fix it passes. Notably, without the fix even the *existing* `test_delete_user`
fails in this two-user root mode — its empty-subdir scenario deletes a dir
under the run-as user's `appcache`, which hits the same EACCES:
```
Testing delete_user
rmdir of .../local-1/usercache/nobody/appcache/app_3/ failed - Permission
denied
FAIL: directory not deleted
```
Non-root runs (cetest in CI) skip the privileged part gracefully, same as
the existing test_delete_user.
Also verified manually with the exact NM invocation against a directory laid
out like a private filecache entry:
```bash
install -d -o nobody -g nobody -m 755
/data/01/yarn/nm/usercache/alice/filecache/999999
sudo -u yarn $HADOOP_YARN_HOME/bin/container-executor nobody alice 3 ""
/data/01/yarn/nm/usercache/alice/filecache/999999
```
Before: exit 255, `rmdir of ... failed - Permission denied`, empty dir
remains. After: exit 0, dir removed. The preserved NM-owned-parent case (`... 3
"" /data/01/yarn/nm/usercache/alice`) still succeeds.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
Generative AI: Contains content generated by Claude Code (Anthropic
Claude). The change was human-reviewed and verified on a production
cluster, and complies with the ASF Generative Tooling Guidance
(https://www.apache.org/legal/generative-tooling.html).
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
