Where to report: root fails to edit other users file in sticky bit directory
Hi Debian team, I'm sorry to contact you here, but I ran into an IMO extremely important bug where I don't know which package is actually responsible. Even the root user is not permitted to write to an existing file that is owned by another user within a sticky bit directory: --- 2020-12-04 14:16:58 root@micha:/tmp# whoami root 2020-12-04 14:17:07 root@micha:/tmp# ls -dl drwxrwxrwt 5 root root 100 Dec 4 14:17 . 2020-12-04 14:17:10 root@micha:/tmp# > testfile 2020-12-04 14:17:13 root@micha:/tmp# chown nobody testfile 2020-12-04 14:17:17 root@micha:/tmp# chmod 0777 testfile 2020-12-04 14:17:21 root@micha:/tmp# ls -l testfile -rwxrwxrwx 1 nobody root 0 Dec 4 14:17 testfile 2020-12-04 14:17:23 root@micha:/tmp# > testfile bash: testfile: Permission denied 2020-12-04 14:17:26 root@micha:/tmp# rm -v testfile removed 'testfile' 2020-12-04 14:17:31 root@micha:/tmp# ls -l testfile ls: cannot access 'testfile': No such file or directory --- - The sticky bid should only prevent removal of a file by other users, not writing to it. root of course should always have permission to do both. - This issue happens only on Debian Bullseye, independent from Linux version and architecture (tested on x86_64 Debian with Linux 5.8 as well as Raspberry Pi OS with Linux 5.4). - It happens as well on at least tmpfs and ext4 file systems, so I assume all file systems. - Not only shell redirects are affected, but all ways to write the file. I first observed it during a Python pip install as root user. - It does not depend on either the shell or the way root was invoked. bash, dash, sudo, direct root login all behave the same. Please advice me who to forward this to or which package is responsible/involved. Best regards, Micha
Re: Where to report: root fails to edit other users file in sticky bit directory
Sorry for the late reply guys, I was not subscribed to the list, just got the address from the bug report instructions in case the related package could not be identified, and thought I get an answer directly to my mailbox xD: https://www.debian.org/Bugs/Reporting.en.html Back to topic. I totally forgot to make clear that it is an issue on Debian Bullseye only. On Buster this would have been recognised much earlier ;). @Tomas: --- root@VM-Bullseye:/tmp# id uid=0(root) gid=0(root) groups=0(root) root@VM-Bullseye:/tmp# >> testfile -bash: testfile: Permission denied root@VM-Bullseye:/tmp# dash -c '>> testfile' dash: 1: cannot create testfile: Permission denied --- As said, the issue was first recognised during a Python module install, so in a totally different environment. My shell is bash as well, but I can reproduce the issue with any shell or any other language to write to a file in sticky bit directories. @Greg --- root@VM-Bullseye:/tmp# id uid=0(root) gid=0(root) groups=0(root) root@VM-Bullseye:~# cd /tmp root@VM-Bullseye:/tmp# df -T Filesystem Type 1K-blocks Used Available Use% Mounted on udev devtmpfs 1016164 0 1016164 0% /dev /dev/sda1 ext4 8189368 584548 7169108 8% / tmpfs tmpfs 1018568 0 1018568 0% /dev/shm tmpfs tmpfs 407428 5504401924 2% /run tmpfs tmpfs 5120 0 5120 0% /run/lock tmpfs tmpfs 4096 0 4096 0% /sys/fs/cgroup tmpfs tmpfs 1018880 37432981448 4% /tmp tmpfs tmpfs51200 8 51192 1% /var/log root@VM-Bullseye:/tmp# cd /root root@VM-Bullseye:~# mkdir testdir root@VM-Bullseye:~# chmod 1777 testdir root@VM-Bullseye:~# > testdir/testfile root@VM-Bullseye:~# chown www-data testdir/testfile root@VM-Bullseye:~# > testdir/testfile -bash: testdir/testfile: Permission denied --- The issue is not related to the file system and not related to the kernel version. As said I tested in on RPi with their current stable Linux 5.4 (now with a second recent update) and on Bullseye with now as well two kernel versions. But to be sure, I'll downgrade the kernel to a previous major version, probably from Buster backports, just to be sure. This is a bug clearly, I'm just not sure, if kernel and file system and shell/environment does not play a role, which part of the OS does, i.e. is sitting between userland programs and kernel/fs driver to verify UNIX permissions. Best regards, Micha
Re: Where to report: root fails to edit other users file in sticky bit directory
Am 08.12.2020 um 16:15 schrieb Greg Wooledge: It *has* to be related to the kernel. Where else would the permissions (capabilities) be applied? That is exactly what I am wondering about. Those are pretty minimal Debian install, no SELinux and no AppArmor actively installed, only what is pulled in by core system requirements: --- root@VM-Bullseye:~# echo $(apt-mark showmanual) apt bash-completion bzip2 ca-certificates cron curl dropbear gnupg grub-pc haveged htop ifupdown iputils-ping kmod linux-image-4.19.0-13-amd64 nano p7zip parted procps psmisc sudo systemd-sysv systemd-timesyncd tiny-initramfs tzdata udev unzip vmtouch wget whiptail --- But there during boot, AppArmor is sort of loaded: --- [1.051669] AppArmor: AppArmor initialized [1.328558] AppArmor: AppArmor Filesystem Enabled [1.782311] AppArmor: AppArmor sha1 policy hashing enabled [2.753821] systemd[1]: systemd 247.1-3 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid) --- libapparmor1 is pulled in by systemd. There was an update today from +b1 to +b2 version, but that did not make a difference. But you didn't try it with buster's kernel yet? I tried now the current Bullseye kernel, current Buster backports kernel and current Buster kernel, it is definitely not kernel-related. For that matter, how did you run this shell process? I see "VM" in your shell prompt, so I'm guessing it's something far more complex than "I booted my bullseye system, and logged in on the console as root". It is exactly that simple and can be replicated on all sort of machines (Raspberry Pi, VirtualBox VM, VMware and x86_64 notebook). The VM is just most simple to test with. Is there any possibility that the way you invoked your shell process is messing with the capabilities(7) in such a way that you lost (or never got) CAP_FOWNER? --- root@VM-Bullseye:~# capsh --print Current: =ep Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read Ambient set = Current IAB: Securebits: 00/0x0/1'b0 secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) secure-no-ambient-raise: no (unlocked) uid=0(root) euid=0(root) gid=0(root) groups=0(root) Guessed mode: UNCERTAIN (0) --- cap_fowner is listed. If it's not that, then someone's going to have to try to reproduce your results, which would mean running a post-stable kernel, which means "not me". Jep, based on the way the list mail address was shown on the Debian bug report page, I was actually hoping to reach official maintainers, but this seems to be more an end-user support list? (Also... "their current stable Linux 5.4" -- Is this even DEBIAN?! If this is a Raspbian or something, you're posting on the wrong mailing list.) Raspberry Pi is an ARM SoC, and as such, best featured by the official kernel provided by the manufacturer: https://github.com/raspberrypi/linux I'm running official Debian userland packages but with this kernel/bootloader/firmware provided by their repository: http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/ But it is irrelevant in this regards but only shows clearer the the Linux version, hardware and CPU architecture doesn't play a role. I'm guessing actually that it is something with systemd and/or AppArmor. I'll try to switch to SysV init and purge the AppArmor library completely. Best regards, Micha
Re: Where to report: root fails to edit other users file in sticky bit directory
Am 08.12.2020 um 16:55 schrieb to...@tuxteam.de: @Tomas: --- root@VM-Bullseye:/tmp# id uid=0(root) gid=0(root) groups=0(root) root@VM-Bullseye:/tmp# >> testfile -bash: testfile: Permission denied root@VM-Bullseye:/tmp# dash -c '>> testfile' dash: 1: cannot create testfile: Permission denied Oh, I see. Strange error message though -- I'd expect dash to try to open the file in append mode, not to `create' it. Thanks for reporting back, cheers - t I was also wondering about that, but dash prints this message regardless if the file exists already or not, if permissions are not there. Note that creating a new file works fine, removing a file as well, only writing to an existing file fails, regardless of shell or originating program. I now also tested to switch from systemd to SysV init, which allowed to remove libapparmor1. I even purged udev (which breaks network) to get rid of any systemd package, but the issue remains. Strangely the three AppArmor init messages remain: -- [1.051669] AppArmor: AppArmor initialized [1.328558] AppArmor: AppArmor Filesystem Enabled [1.782311] AppArmor: AppArmor sha1 policy hashing enabled -- I think this is fixed part of grub or the kernel then. However, on RPi, with different kernel and different bootloader, those messages are not present, but the issue is. So I'm still completely out of ideas. Btw, if someone knows an "official" Debian mailing list with maintainers/developers, where this can be reported, that might be helpful. If it is clearly related to a certain package, things are easy to address correctly, but in this case I have no clue. I think best to debug is if someone with deeper insights is replicating this on an own system. But of course I'll go on trying/testing ideas, that come to my mind of thrown in here. Best regards, Micha
Re: Where to report: root fails to edit other users file in sticky bit directory
Good idea, although it would have been recognised much earlier during
nearly every normal system operation:
---
root@micha:/tmp# set -o | grep noclobber
noclobber off
---
strace is a fantastic idea, but it seems to fail on the lowest level
"EACCES (Permission denied)":
---
root@VM-Bullseye:~# strace touch testdir/testfile
execve("/usr/bin/touch", ["touch", "testdir/testfile"], 0x7ffc117f64c8
/* 15 vars */) = 0
brk(NULL) = 0x562ade0be000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=10074, ...}) = 0
mmap(NULL, 10074, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd0797bd000
close(3)= 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@n\2\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1839792, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fd0797bb000
mmap(NULL, 1852680, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7fd0795f6000
mprotect(0x7fd07961b000, 1662976, PROT_NONE) = 0
mmap(0x7fd07961b000, 1355776, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7fd07961b000
mmap(0x7fd079766000, 303104, PROT_READ,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17) = 0x7fd079766000
mmap(0x7fd0797b1000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ba000) = 0x7fd0797b1000
mmap(0x7fd0797b7000, 13576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fd0797b7000
close(3)= 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fd0795f4000
arch_prctl(ARCH_SET_FS, 0x7fd0797bc5c0) = 0
mprotect(0x7fd0797b1000, 12288, PROT_READ) = 0
mprotect(0x562adcd52000, 4096, PROT_READ) = 0
mprotect(0x7fd0797ea000, 4096, PROT_READ) = 0
munmap(0x7fd0797bd000, 10074) = 0
brk(NULL) = 0x562ade0be000
brk(0x562ade0df000) = 0x562ade0df000
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1827600, ...}) = 0
mmap(NULL, 1827600, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd079435000
close(3)= 0
openat(AT_FDCWD, "testdir/testfile",
O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = -1 EACCES (Permission denied)
utimensat(AT_FDCWD, "testdir/testfile", NULL, 0) = 0
close(1)= 0
close(2)= 0
exit_group(0) = ?
+++ exited with 0 +++
---
I disabled file system caching by remounting rootfs with "sync" option,
but that didn't help either. Other file systems with UNIX permissions
are outstanding: F2FS, Btrfs, XFS, NTFS with 3g driver, will try that as
well.
I btw reported the issue to the testing list now, which seems to be the
better place to report unspecific issues with Bullseye. Although the
list is not very active :\. Let's see, my mail did not appear yet:
https://lists.debian.org/debian-testing/2020/12/threads.html
Re: Where to report: root fails to edit other users file in sticky bit directory
I think the topic with the modes just applies when the file is actually created, hence not existing yet. However, aiming for a method to edit a file without O_CREAT, and voila: --- root@VM-Bullseye:~# cat testdir/testfile 143 root@VM-Bullseye:~# sed -i 's/1/5/' testdir/testfile root@VM-Bullseye:~# cat testdir/testfile 543 root@VM-Bullseye:~# > testdir/testfile -bash: testdir/testfile: Permission denied --- You nailed it! Now checking the kernel patch: --- root@VM-Bullseye:~# sysctl fs.protected_regular fs.protected_regular = 2 root@VM-Bullseye:~# sysctl fs.protected_regular=0 fs.protected_regular = 0 root@VM-Bullseye:~# > testdir/testfile root@VM-Bullseye:~# cat testdir/testfile root@VM-Bullseye:~# --- Bingo! Okay, so this is a security feature with the aim to prevent accidental writes to a file owned by a potential attacker. I.e. since the directory is world-writeable, an attacker can place a file that is knows is written to by another user/victim. The victim actually aims to create a new file (or write to it's own existing one), but now writes to the prepared attackers file, which can then do whatever (s)he wants to do with it as of ownership, to break the victims process or place different information inside (then originally stored), like poisoning a cache and such. Yeah, somehow a thinkable scenario, although at least with systemd there is PrivateTmp=true for this, and critical/sensitive information should simply otherwise not be stored into /tmp root, a restricted sub directory or /run or something like that. But the biggest problem is that, while this effectively breaks sharing files (writeable) between multiple users/processes, as many methods include O_CREAT, it is not documented anywhere related to /tmp mount and sticky bit, so it is mostly unexpected, isn't it? The results are errors which are hard to debug, in my case it took me a full day to get to the essence of the issue, while first I reviewed and re-tested and changed back and forth a Python script (of some software I use, not my own). And, root should IMO still be allowed to do it, as practically it can anyway via chown. root can as well remove any file, regardless of sticky bit, so this would be consistent. Relevant package is then: procps Ah, btw on Buster, by default it's disabled: --- 2020-12-08 21:39:47 root@VM-Buster:~# sysctl fs.protected_regular fs.protected_regular = 0 --- Many thanks for looking deeper at O_CREAT guys, I likely wouldn't have searched into this direction :)! Best regards and stay healthy, Micha
Re: Where to report: root fails to edit other users file in sticky bit directory
Am 08.12.2020 um 22:03 schrieb Greg Wooledge: Unexpected clobberin' by root is the actual *worst* case that this security feature is trying to protect against. Imagine this hypothetical scenario: 1) Process A running as regular user alice creates a temp file in /tmp, owned by alice. 2) Dumb process B running as root decides to use the exact same temp file name that A is using. It opens the file naively, and writes some stuff to it, intending to read it back later. 3) Process A rewinds the file and writes new stuff to it. 4) Process B reads process A's data from the file, thinking it was going to read its own data. 5) Process B calculates the wrong thing, and turns off the life support system. I don't know how realistic this is, or how closely it mirrors the thought processes of the people who came up with the modification. But if you're going to make this modification at all, it needs to affect root, or else it's pointless. Such a (custom) process should simply not run as root, should create an own temporary sub directory with limited permissions and write it's own files there. If really something is written to the world-writeable /tmp directory root, it must be expected that it can be read and written to or created by other users. Btw. the way the security feature is implemented, explicitly allows anyone writing to the file (as long as UNIX permissions are given), so it does not totally rule out your scenario, but only when process B uses a O_CREAT access method. But it could instead check if the file exists first, and otherwise write to it without O_CREAT. But I got the point, as especially root processes are likely more sensitive and create larger damage when manipulated (accidentally or intentionally) such a way. Your unique situation, where user A is putting a file in /tmp for some reason, and then user R is supposed to edit it in place, is really weird. I don't know why you're putting this important shared file in /tmp, but you might want to move it to a more appropriate place. There is no reason to use the /tmp root (without sub directory) aside of expecting that the file is potentially shared, IMO. If it must only be accessed by the same process/user, it should be placed elsewhere, at least into a sub directory or a private /tmp. Sharing files through /tmp is probably rare, but /tmp it is for temporary files, it's by default a tmpfs (relevant when one wants to minimize file system writes, e.g. drive spin up or SD cards with short life cycle, or runs a R/O system), and it's the only FHS directory that is world-writeable by default. So I cannot imagine any better place to share temporary information through files cross-process/user then /tmp :D. So yeah, basically the feature prevents users, who do not know what they are doing, from rare but possible errors and attacks, but it limits users, who know that they are doing, from rarely relevant possibilities xD. I think it is all fine as long as it's well documented, i.e. part of docs and man pages that deal with the sticky bit, like man chmod(1). Best regards, Micha
Re: Where to report: root fails to edit other users file in sticky bit directory
Please note that it by default appears on Bullseye only. See that last mails regarding this issue, the related changed sysfs setting has identified already do: --- sysctl fs.protected_regular=2 --- and retry the steps, which will then fail. --- sysctl fs.protected_regular=0 --- to revert to default until up to Buster. From Bullseye on that defaults to "2" which prevents any user from write + "O_CREAT" to a file within a world-writeable + sticky bit directory (1777) that is owned by another user. Best regards, Micha
Re: Where to report: root fails to edit other users file in sticky bit directory
Your right /tmp is not a tmpfs by default on Debian. I though it was, maybe being too much used to it as this is configured by default on our images. /dev/shm or /run would work better then, although /run IMO is more aimed for non-temporary files, relevant through the whole runtime of the related process or system, while /tmp is more aimed for short-term temporary files. However it was/is not my choice to use /tmp in the particular case and out of my control. Background, if interested: --- In my particular case where I ran into the issue, it is a 3rd party software that chooses to store information to /tmp/ via sub process which is then read by originating process. Indeed the circumstance is a bid special, the solution is even commented as an "ugly" one within the code, but no better method had been found until now. The software collects information about the Python environment by doing a dummy module install (that is aimed to fail). The dummy modules installer collects info about where it was installed to (venv/user-level/system-level, ...) and writes that to /tmp/ to be read by the originating Python script. It has an internal updater which invokes pip the "correct" way based on the /tmp/ content (e.g. --user flag or not). I want the software to run as system service with an own limited user, but aim to have it installed as global/system Python module to /usr/local/lib|bin instead of in --user mode into its own UNIX user home directory, or as vnev, to prevent doubled Python module installs (target is small embedded systems with potentially limited disk space), easier access/maintenance for the real (login) user and clear separation between data/config and the Python modules. To retain the functionality of the internal updater, it requires specifically limited "sudo pip install " permissions to self-upgrade and do it's Python environment collection dummy install that is out of my control. The problem is now that it pre-creates /tmp/ as its limited service user but since pip is called via sudo, the dummy installer then tries and fails to write to that file as root user. --- I am actually sort of Linux distributor, more precisely developing Debian-based images mostly for ARM SBCs together with a tool set to easily install and configure system a bunch of software titles. See the domain of my mail address. If I was a system administrator only, of course, why should I feel forced to follow any standard, but as a software developer I clearly disagree with you! The FHS is exactly what software developers should then be able to count on and respect themselves (to allow others count on it), which directories are present and used in the same (at least similar) way by the distributions own software packages as well as most 3rd party software installers. Without FHS one would quickly have all sorts of files messed across a bunch of different directory structures and sub structures, making it impossible for admins to follow any logic to find certain types of files, like software and system configuration files, variable data, temporary files etc, also to know where files are expected to survive a reboot and where not (potentially tmpfs) etc. Without FHS (or any other standard across Linux/UNIX systems) we had a mess, so FHS is no "crap" but very valuable and important, IMO. But that is a discussion that has nothing to do with the originating issue, which has been identified thanks to your help! :) It's now a bid others and whether the default has actually been changed intentionally, and if so more like "why not" or if there have been actual concerns, serious enough to change a default, which implies different and probably unexpected behaviour. And if so, if there is a plan to have this documented a bid wider. Best regards and stay healthy, Micha
