On Mon, Apr 27, 2020 at 09:01:56AM -0700, Igor Dvorzhak wrote:
> Package: e2fsprogs
>
> resize2fs takes extra 90 seconds to resize 2TB boot disk during boot on
> Debian 10 than on Debian 9.
>
> Note that time to create/provision VM instance (gcloud compute instances
> create ...) is the same (around 10 seconds) for both Debian 9 and Debian
> 10, but time to successful SSH is different (see while loop in my test
> command) - this is when VM is actually booted, not when gcloud instances
> create ... returns.
How the root file system is resized on Debian 9 and Debian 10 are very
different, and the difference seems to stem from Google's initramfs scripts.
In Debian 9, the root file system is resized after it is mounted. We
can see this in the kernel messages. In Debian 10, the root file
system is resized in the initramfs, and the root file system isn't
even mounted until 168 seconds after the systme is booted.
There are two scripts in Debian 10's initramfs on GCE, and they are
both marked "Copyright 2018 Google Inc.", and they
scripts/expand-lib.sh, and local-premount/expand_rootfs. These
scripts are not in stock Debian's /usr/share/initramfs-tools
directory, so they are a Google special.
What the bash function resize_filesystem in scripts/expand-lib.sh does
is that it runs e2fsck on the root partition, and then does an
off-line resize. It is the off-line resize which is appearing to take
a long time.
Looking at the off-line resize, it does do a bit more I/O. In
particuar, it's going to do 16k or so extra 4k random writes which
aren't done with an on-line resize. That shouldn't translate to PD
taking 90 seconds to do those writes, though!
That being said, the differences do appear to be in how Google to do
the file system resizing before the file system is mounted. I'm not
sure why someone decided it would be a good idea to do an off-line
resize in Debian 10, but that appears to be a Google decision, not a
Debian decision.
> I also tested this with the newer 1.45.5-2 e2fsprogs version on Debian 10
> (updated from buster-backports repo) and Ubuntu 20.04 LTS. But only Debian
> 10 VM still have this regression, Ubuntu 20.04 LTS doesn't have it, so it
> seems that this is a Debian 10-specific issue.
>
> Are there any configuration options that allow to restore Debian 9 behavior
> in Debian 10 for resize2fs during VM boot time?
No, it looks like it would require making some changes to the GCE's
init scripts in the Debian 10 image. I suspect it means needing to dig around
at:
https://github.com/GoogleCloudPlatform/compute-image-packages
Cheers,
- Ted
