On Thu, 21 Oct 2021 13:31:31 -0500 Glenn Washburn <[email protected]> wrote:
> The gnulib sources are large but more importantly have lots of changes. So > initial checkout of the repository can take a long time when network or > cpu resources are limited. The later is especially acute in a non-KVM QEMU > virtual machine (which can take 40+ minutes compared to <30 seconds with > this change[1]). The problem is specific to bootstrap configurations using > a specific gnulib revision specified by commit hash. In this case, git can > not do a shallow clone by using the --depth option because git does not know > ahead of time how deep the revision is from the tip. So git must clone the > whole repository. > > However, there is an alternate method that requires support from the git > server[2], namely by asking for a specific commit on fetch. Refactor to use > fetch and fallback to fetching the entire repository if fetching by commit > hash fails. > > Currently the git server hosting the official gnulib git repository does not > support fetch by commit hash[3]. However, there are mirrors which do support > this[4], and can be specified by setting the $GNULIB_URL. The issue with the savannah git server got resolved more quickly than anticipated. I suggest cutting out the last paragraph and the last two links below. No need to mention the non-Free service. I can recreate the patch if that's desired as well, but have no issue with a maintainer editing the commit message before committing. Glenn > > [1] https://savannah.nongnu.org/support/index.php?110553#comment1 > [2] https://stackoverflow.com/a/3489576/2108011 > [3] https://savannah.nongnu.org/support/index.php?110553 > [4] https://github.com/coreutils/gnulib > > Signed-off-by: Glenn Washburn <[email protected]> > --- > Hi list, > > This patch can have a dramatic performance improvement in checking out gnulib > sources via bootstrap. These changes use the git -C which was introduced in > 2013 in v1.8.5, so I think its been around long enough to rely on it. Also, > I'm > not subscribed to the list, so please ensure to CC me on any replies to this > patch. > > Thanks, > Glenn > > --- > build-aux/bootstrap | 16 ++++++++++++++-- > 1 file changed, 14 insertions(+), 2 deletions(-) > > diff --git a/build-aux/bootstrap b/build-aux/bootstrap > index 733659850..9f8b918a2 100755 > --- a/build-aux/bootstrap > +++ b/build-aux/bootstrap > @@ -694,9 +694,21 @@ if $use_gnulib; then > shallow= > if test -z "$GNULIB_REVISION"; then > git clone -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth > 2' > + git clone $shallow ${GNULIB_URL:-$default_gnulib_url} "$gnulib_path" > \ > + || cleanup_gnulib > + else > + git fetch -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth > 2' > + mkdir -p "$gnulib_path" > + git -C "$gnulib_path" init > + git -C "$gnulib_path" remote add origin > ${GNULIB_URL:-$default_gnulib_url} > + # Can not do a shallow fetch if fetch by commit hash fails because we > + # do not know how deep to go to get to $GNULIB_REVISION, so we must > get > + # all commits. > + git -C "$gnulib_path" fetch $shallow origin "$GNULIB_REVISION" \ > + || git -C "$gnulib_path" fetch origin \ > + || cleanup_gnulib > + git -C "$gnulib_path" reset --hard FETCH_HEAD > fi > - git clone $shallow ${GNULIB_URL:-$default_gnulib_url} "$gnulib_path" \ > - || cleanup_gnulib > > trap - 1 2 13 15 > fi
