[Focal/FIPS][PATCH 0/4] Fix kernel BUG/Oops errors from modprobe while the DRBG has not yet initialized (focal/fips-updates)
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1981487 Title: kernel BUG/Oops errors from modprobe while the DRBG has not yet initialized (focal/fips-updates) Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: Invalid Status in linux source package in Focal: In Progress Status in linux source package in Jammy: Invalid Bug description: [Impact] * The Focal FIPS kernel in fips-updates hits kernel BUG/Oops errors during boot with the FIPS OpenSSL library installed (but those don't cause issues), when it runs modprobe with request_module() when looking up crypto algorithms/modules. * The modprobe command happens to call the OpenSSL library, and the FIPS version of OpenSSL calls getrandom(), and the FIPS kernel calls the DRBG for that, BUT it's _not yet_ initialized that early during boot when the kernel can run modprobe via request_module(). (e.g., IPv6 initialization time.) * The issue impacts the kernels in fips-updates only, per: "UBUNTU: SAUCE: random: Use Crypto API DRBG for urandom in FIPS mode" which exists in Focal, but not in Xenial/Bionic/Jammy. [Fix] * The issue only happens with the crypto algorithms, even if they're built-in (i.e., modprobe is not needed), so the fix approach is to set CRYPTO_NOLOAD while the DRBG isn't yet initialized; this skips request_module() calls. * The fix doesn't change the list and details of algorithms as in /proc/crypto (e.g., name, driver, module, priority) by the time the DRBG is initialized / initramfs started, so even though behavior changes, the net effect doesn't. * (Note: it's not possible to just use an initcall level earlier than rootfs_initcall() so modprobe isn't there, because fips_drbg_init() must run _after_ module_init() level so that crypto_rng_reset() works, even though .) [Test Steps] * Install the kernel and openssl from fips-updates, boot with fips=1, check dmesg for BUG/Oops errors: $ sudo apt install linux-image-fips libssl1.1 # fips-updates $ sudo vim /etc/default/grub # append fips=1 boot option $ sudo update-grub && sudo reboot $ sudo dmesg | grep BUG: * Check/store the /proc/crypto file for comparisons. You can boot with break=top as well, to check that as early as possible, and copy into /run/initramfs/ then exit, to get it later in the rootfs. [Regression Potential] * The fix skips calls to request_module()/modprobe on early boot until the FIPS DRBG is initialized (only in the crypto subsystem) thus potential regressions would show up as early boot kernel errors messages or crypto algorithms not up after system booted. (These should have no impact, theoretically, as it currently doesn't run anyway, because of BUG/Oops.) * The fix also adds a panic() call in request_module() if other/non-crypto kernel code attempts to load a module while the DRBG isn't ready yet (so we catch that earlier). It's possible to use a boot option to disable that behavior, but it disables FIPS mode for strictness. [Original Bug Description] $ sudo apt install --yes linux-image-fips # fips-updates $ sudo vim /etc/default/grub # fips=1 $ sudo update-grub && sudo reboot $ uname -r 5.4.0-1056-fips $ cat /proc/cmdline ... fips=1 No errors with the original/non-FIPS openssl, because it does NOT call getrandom(): $ dmesg | grep -c BUG: 0 $ dpkg -s libssl1.1 | grep ^Version: Version: 1.1.1f-1ubuntu2.15 $ strace -e getrandom modprobe --version kmod version 27 +ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL +++ exited with 0 +++ But if you install the FIPS openssl, it calls getrandom(), then BUG/Oops happen: $ sudo apt install libssl1.1 # updates initramfs $ dpkg -s libssl1.1 | grep ^Version: Version: 1.1.1f-1ubuntu2.fips.13.1 $ strace -e getrandom modprobe --version getrandom("\xc4\x84\x26\x25\x6f\xd4\xed\x38\xdf\xa9\x67\xee\x15\x1c\xe3\x98", 16, GRND_NONBLOCK) = 16 getrandom("\xa1\xd6\x67\x3e\xe4\x90\xb3\x8b\xdf\xe6\x34\x2a\xa7\x50\xbc\x2f", 16, GRND_NONBLOCK) = 16 getrandom("\xf1\x3e\xe4\x27\x9d\x47\x8c\x4b\x8a\x39\x8c\xe1\x2e\xee\xfa\x45", 16, GRND_NONBLOCK) = 16 getrandom("\xfb\x34\x18\x44\xd8\x23\x4c\x87\x13\x2e\x6b\x03\x79\xa7\x99\xf8", 16, 0) = 16 getrandom("\xdd\x83\xa7\x02\x10\x51\x2b\x4f\x21\x6b\xc1\xf1\x0d\xe7\x44\xb7", 16, 0) = 16 kmod version 27 +ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL +++ exited with 0 +++ $ sudo reboot $ dmesg | grep -c BUG: 22 $ dmesg ... [ 1.595759] NET: Registered protocol family 10 [ 1.600256] BUG: kernel NULL pointer dereference, address: 0000000000000038 ... [ 1.603829] CPU: 2 PID: 137 Comm: modprobe Not tainted 5.4.0-1056-fips #64-Ubuntu [ 1.603829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014 [ 1.603829] RIP: 0010:urandom_read+0x268/0x480 ... [ 1.603829] Call Trace: [ 1.603829] __x64_sys_getrandom+0x7f/0x130 [ 1.603829] do_syscall_64+0x57/0x190 [ 1.603829] entry_SYSCALL_64_after_hwframe+0x44/0xa9 ... All BUG/Oops errors are the same: $ dmesg | grep BUG: | sed 's/^.*BUG:/BUG:/' | uniq -c 22 BUG: kernel NULL pointer dereference, address: 0000000000000038 And they stop after the DRBG is initialized: [ 3.651566] random: DRBG (drbg_nopr_ctr_aes256) initialized! [Fix Impact Analysis] The patches add 3 module options to configure the behavior of the components involved in the issue/fix. They can be seen with dynamic debug in the kernel cmdline: dyndbg="func crypto_alg_mod_lookup +p; func crypto_lookup_template +p; func __request_module +p" Also add break=top, so we can copy /proc/crypto and dmesg right after DRBG is initialized (when initramfs is started). Let's compare the 4 possible combination (first is 'no functional change'): random.fips_urandom_drbg_enable=0 random.fips_urandom_drbg_crypto_noload=0 random.fips_urandom_drbg_modprobe_panic=0 random.fips_urandom_drbg_enable=1 random.fips_urandom_drbg_crypto_noload=0 random.fips_urandom_drbg_modprobe_panic=0 random.fips_urandom_drbg_enable=1 random.fips_urandom_drbg_crypto_noload=1 random.fips_urandom_drbg_modprobe_panic=0 random.fips_urandom_drbg_enable=1 random.fips_urandom_drbg_crypto_noload=1 random.fips_urandom_drbg_modprobe_panic=1 @ break=top time enable=$(cat /sys/module/random/parameters/fips_urandom_drbg_enable) noload=$(cat /sys/module/random/parameters/fips_urandom_drbg_crypto_noload) panic=$(cat /sys/module/random/parameters/fips_urandom_drbg_modprobe_panic) suffix="enable${enable}_noload${noload}_panic${panic}" cat /proc/crypto > /run/initramfs/proc-crypto.$suffix dmesg > /run/initramfs/dmesg.$suffix exit @ login time sudo -s cp /run/initramfs/{proc-crypto.*,dmesg.*} . reboot # next possible combination There's no difference in the list/details of loaded crypto algorithms at all, with any combination: # md5sum proc-crypto.enable*_noload*_panic* 0b91bd619078fa342c6b4da039cb1582 proc-crypto.enableN_noloadN_panicN 0b91bd619078fa342c6b4da039cb1582 proc-crypto.enableY_noloadN_panicN 0b91bd619078fa342c6b4da039cb1582 proc-crypto.enableY_noloadY_panicN 0b91bd619078fa342c6b4da039cb1582 proc-crypto.enableY_noloadY_panicY The only option that hits BUG/Oops errors is when you disable the fix (ie, don't set CRYPTO_NOLOAD): # grep -c BUG: dmesg.enable*_noload*_panic* dmesg.enableN_noloadN_panicN:0 dmesg.enableY_noloadN_panicN:22 dmesg.enableY_noloadY_panicN:0 dmesg.enableY_noloadY_panicY:0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1981487/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
