** Description changed:
+ [Impact]
+
+ On Ubuntu 22.04 LTS, a forced boot-time check can send the machine to
emergency mode after an unclean shutdown leaves metadata changes in an XFS
journal.
+ The affected path requires fsck.mode=force and fsck.repair=yes on the kernel
command line.
+
+ Jammy's fsck.xfs runs xfs_repair -e, which exits with status 2 when the
journal must be replayed by mounting the filesystem.
+ fsck.xfs converts that result to an uncorrected-filesystem error without
replaying the journal, so systemd fails the filesystem check and the dependent
mount.
+
+ This path became reachable after the update for LP#2081163 fixed forced XFS
checks under Ubuntu's default shell.
+ The proposed update backports upstream commit
79ba1e15d80eba3aff4396f44629eb8960722d36. Please check original description.
+ When repair is explicitly requested, fsck.xfs temporarily mounts and unmounts
the filesystem to replay the journal, then runs xfs_repair -e again.
+
+ [Test Plan]
+
+ Attach a blank 2 GiB disk as /dev/vdb and create an XFS filesystem.
+
+ sudo apt install xfsprogs
+ sudo mkfs.xfs -f /dev/vdb
+ sudo mkdir -p /mnt/lp2101888
+ sudo mount /dev/vdb /mnt/lp2101888
+ sudo blkid -s UUID -o value /dev/vdb
+
+ Run the following metadata workload and, while it is running, force the VM
off from the host.
+ Before mounting the disk again, make two copies for the archive and proposed
tests.
+
+ sudo sh -c 'i=0; while :; do
+ d=/mnt/lp2101888/work-$((i % 256))
+ mkdir -p "$d"; echo "$i" > "$d/a"; mv -f "$d/a" "$d/b"
+ i=$((i + 1))
+ done'
+
+ Prepare two identical clean Jammy VM root disks.
+ On both, create /mnt/lp2101888, add the data disk UUID to fstab with pass
number 2, and boot with fsck.mode=force fsck.repair=yes.
+
+ UUID=<uuid> /mnt/lp2101888 xfs defaults 0 2
+
+ 1. With xfsprogs 5.13.0-1ubuntu2.1 from jammy-updates, attach the first dirty
disk and boot.
+ The filesystem check must fail with status 4, /mnt/lp2101888 must remain
unmounted, and the VM must enter emergency mode without replaying the log.
+
+ 2. Install xfsprogs from patched pkg on the second root disk, attach the
other dirty disk, and boot.
+ The journal must show "Replaying log for /dev/vdb" followed by xfs_repair
phases 1 through 7.
+ The filesystem check must return 0, /mnt/lp2101888 must be mounted, and the
system must reach the running state with no failed unit.
+
+ [Where problems could occur]
+
+ The new path runs only when fsck.xfs is invoked non-interactively with both
-f and -y and xfs_repair -e returns 2 for a dirty log. It temporarily mounts
the filesystem at /tmp/repair_mnt to replay the log.
+ A regression could replay the log without -y, fail the boot-time check, or
leave /tmp/repair_mnt mounted.
+
+ [Original Description]
+
Description: Ubuntu 22.04 LTS
Release: 22.04
Package: xfsprogs 5.13.0-1ubuntu2.1
Kernel cmdline containing: fsck.mode=force fsck.repair=yes
[Issue]
LP #2081163
(https://bugs.launchpad.net/ubuntu/+source/xfsprogs/+bug/2081163)
backported a fix for fsck.xfs not being run during system startup:
- commit 19dde7fac0f38af2990e367ef4dd8ec512920c12
- Author: Gerald Yang <[email protected]>
- Date: Tue Aug 13 15:25:51 2024 +0800
+ commit 19dde7fac0f38af2990e367ef4dd8ec512920c12
+ Author: Gerald Yang <[email protected]>
+ Date: Tue Aug 13 15:25:51 2024 +0800
- fsck.xfs: fix fsck.xfs run by different shells when fsck.mode=force
is set
-
- When fsck.mode=force is specified in the kernel command line, fsck.xfs
- is executed during the boot process. However, when the default shell
is
- not bash, $PS1 should be a different value, consider the following
script:
- cat ps1.sh
- echo "$PS1"
-
- run ps1.sh with different shells:
- ash ./ps1.sh
- $
- bash ./ps1.sh
-
- dash ./ps1.sh
- $
- ksh ./ps1.sh
-
- zsh ./ps1.sh
-
- On systems like Ubuntu, where dash is the default shell during the
boot
- process to improve startup speed. This results in FORCE being
incorrectly
- set to false and then xfs_repair is not invoked:
- if [ -n "$PS1" -o -t 0 ]; then
- FORCE=false
- fi
-
- Other distros may encounter this issue too if the default shell is set
- to anoother shell.
-
- Check "-t 0" is enough to determine if we are in interactive mode, and
- xfs_repair is invoked as expected regardless of the shell used.
-
- Fixes: 04a2d5dc ("fsck.xfs: allow forced repairs using xfs_repair")
- Signed-off-by: Gerald Yang <[email protected]>
- Reviewed-by: Darrick J. Wong <[email protected]>
+ fsck.xfs: fix fsck.xfs run by different shells when
+ fsck.mode=force is set
+ When fsck.mode=force is specified in the kernel command line, fsck.xfs
+ is executed during the boot process. However, when the default shell
is
+ not bash, $PS1 should be a different value, consider the following
script:
+ cat ps1.sh
+ echo "$PS1"
- That fix was originally applied upstream for 6.11.0, but it appears to
implicitly depend on an earlier commit from xfsprogs 6.1.0:
+ run ps1.sh with different shells:
+ ash ./ps1.sh
+ $
+ bash ./ps1.sh
- commit 79ba1e15d80eba3aff4396f44629eb8960722d36
- Author: Srikanth C S <[email protected]>
- Date: Tue Dec 13 22:45:43 2022 +0530
+ dash ./ps1.sh
+ $
+ ksh ./ps1.sh
- fsck.xfs: mount/umount xfs fs to replay log before running
+ zsh ./ps1.sh
+
+ On systems like Ubuntu, where dash is the default shell during the
boot
+ process to improve startup speed. This results in FORCE being
incorrectly
+ set to false and then xfs_repair is not invoked:
+ if [ -n "$PS1" -o -t 0 ]; then
+ FORCE=false
+ fi
+
+ Other distros may encounter this issue too if the default shell is set
+ to anoother shell.
+
+ Check "-t 0" is enough to determine if we are in interactive mode, and
+ xfs_repair is invoked as expected regardless of the shell used.
+
+ Fixes: 04a2d5dc ("fsck.xfs: allow forced repairs using xfs_repair")
+ Signed-off-by: Gerald Yang <[email protected]>
+ Reviewed-by: Darrick J. Wong <[email protected]>
+
+ That fix was originally applied upstream for 6.11.0, but it appears to
+ implicitly depend on an earlier commit from xfsprogs 6.1.0:
+
+ commit 79ba1e15d80eba3aff4396f44629eb8960722d36
+ Author: Srikanth C S <[email protected]>
+ Date: Tue Dec 13 22:45:43 2022 +0530
+
+ fsck.xfs: mount/umount xfs fs to replay log before running
xfs_repair
- After a recent data center crash, we had to recover root filesystems
- on several thousands of VMs via a boot time fsck. Since these
- machines are remotely manageable, support can inject the kernel
- command line with 'fsck.mode=force fsck.repair=yes' to kick off
- xfs_repair if the machine won't come up or if they suspect there
- might be deeper issues with latent errors in the fs metadata, which
- is what they did to try to get everyone running ASAP while
- anticipating any future problems. But, fsck.xfs does not address the
- journal replay in case of a crash.
+ After a recent data center crash, we had to recover root filesystems
+ on several thousands of VMs via a boot time fsck. Since these
+ machines are remotely manageable, support can inject the kernel
+ command line with 'fsck.mode=force fsck.repair=yes' to kick off
+ xfs_repair if the machine won't come up or if they suspect there
+ might be deeper issues with latent errors in the fs metadata, which
+ is what they did to try to get everyone running ASAP while
+ anticipating any future problems. But, fsck.xfs does not address the
+ journal replay in case of a crash.
- fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is
- possible that when the machine crashes, the fs is in inconsistent
- state with the journal log not yet replayed. This can drop the machine
- into the rescue shell because xfs_fsck.sh does not know how to clean
the
- log. Since the administrator told us to force repairs, address the
- deficiency by cleaning the log and rerunning xfs_repair.
+ fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is
+ possible that when the machine crashes, the fs is in inconsistent
+ state with the journal log not yet replayed. This can drop the machine
+ into the rescue shell because xfs_fsck.sh does not know how to clean
the
+ log. Since the administrator told us to force repairs, address the
+ deficiency by cleaning the log and rerunning xfs_repair.
- Run xfs_repair -e when fsck.mode=force and repair=auto or yes.
- Replay the logs only if fsck.mode=force and fsck.repair=yes. For
- other option -fa and -f drop to the rescue shell if repair detects
- any corruptions.
+ Run xfs_repair -e when fsck.mode=force and repair=auto or yes.
+ Replay the logs only if fsck.mode=force and fsck.repair=yes. For
+ other option -fa and -f drop to the rescue shell if repair detects
+ any corruptions.
- Signed-off-by: Srikanth C S <[email protected]>
- Reviewed-by: Carlos Maiolino <[email protected]>
- Signed-off-by: Carlos Maiolino <[email protected]>
+ Signed-off-by: Srikanth C S <[email protected]>
+ Reviewed-by: Carlos Maiolino <[email protected]>
+ Signed-off-by: Carlos Maiolino <[email protected]>
Now that xfs_repair is actually invoked when fsck.mode=force is set,
it's possible for xfs_repair to fail at boot time and drop into a rescue
shell.
This behaviour is seen with xfsprogs 5.13.0-1ubuntu2.1 in Jammy. It
would presumably also affect 5.3.0-1ubuntu2.1 in Focal, but I haven't
tried it personally.
-
[Potential Solution]
I applied 79ba1e15d80eba3aff4396f44629eb8960722d36 to xfs_fsck.sh in
5.13.0-1ubuntu2.1 and purposely corrupted the fs.
Booting with fsck.mode=force fsck.repair=yes, fsck.xfs correctly mounts,
replays the log, unmounts, repairs, and mounts the filesystem at boot.
** Also affects: xfsprogs (Ubuntu Noble)
Importance: Undecided
Status: New
** Also affects: xfsprogs (Ubuntu Jammy)
Importance: Undecided
Status: New
** Also affects: xfsprogs (Ubuntu Resolute)
Importance: Undecided
Status: New
** Changed in: xfsprogs (Ubuntu)
Status: Confirmed => Fix Released
** Changed in: xfsprogs (Ubuntu Jammy)
Status: New => In Progress
** Changed in: xfsprogs (Ubuntu Noble)
Status: New => Fix Released
** Changed in: xfsprogs (Ubuntu Resolute)
Status: New => Fix Released
** Changed in: xfsprogs (Ubuntu Jammy)
Assignee: (unassigned) => Seyeong Kim (seyeongkim)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2101888
Title:
fsck.xfs can drop into a rescue shell when fsck.mode=force is set
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/xfsprogs/+bug/2101888/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
