The gnulib sources are large but more importantly have lots of changes. So initial checkout of the repository can take a long time when network or cpu resources are limited. The later is especially acute in a non-KVM QEMU virtual machine (which can take 40+ minutes compared to <30 seconds with this change[1]). The problem is specific to bootstrap configurations using a specific gnulib revision specified by commit hash. In this case, git can not do a shallow clone by using the --depth option because git does not know ahead of time how deep the revision is from the tip. So git must clone the whole repository.
However, there is an alternate method that requires support from the git server[2], namely by asking for a specific commit on fetch. Refactor to use fetch and fallback to fetching the entire repository if fetching by commit hash fails. [1] https://savannah.nongnu.org/support/index.php?110553#comment1 [2] https://stackoverflow.com/a/3489576/2108011 Signed-off-by: Glenn Washburn <developm...@efficientek.com> --- Changes since v1: * Updated commit message to reflect that the official repo now has support for fetch by commit hash. * Update comment in source per Bruno's suggestion. Glenn --- Range-diff against v1: 1: 9f6fe7c4a ! 1: 9d980f00c bootstrap: When a commit hash is specified, do a shallow fetch if possible @@ Commit message fetch and fallback to fetching the entire repository if fetching by commit hash fails. - Currently the git server hosting the official gnulib git repository does not - support fetch by commit hash[3]. However, there are mirrors which do support - this[4], and can be specified by setting the $GNULIB_URL. - [1] https://savannah.nongnu.org/support/index.php?110553#comment1 [2] https://stackoverflow.com/a/3489576/2108011 - [3] https://savannah.nongnu.org/support/index.php?110553 - [4] https://github.com/coreutils/gnulib ## build-aux/bootstrap ## @@ build-aux/bootstrap: if $use_gnulib; then @@ build-aux/bootstrap: if $use_gnulib; then + else + git fetch -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth 2' + mkdir -p "$gnulib_path" ++ # Only want a shallow checkout of $GNULIB_REVISION, but git does not ++ # support cloning by commit hash. So attempt a shallow fetch by commit ++ # hash to minimize the amount of data downloaded and changes needed to ++ # be processed, which can drastically reduce download and processing ++ # time for checkout. If the fetch by commit fails, a shallow fetch can ++ # not be performed because we do not know what the depth of the commit ++ # is without fetching all commits. So fallback to fetching all commits. + git -C "$gnulib_path" init + git -C "$gnulib_path" remote add origin ${GNULIB_URL:-$default_gnulib_url} -+ # Can not do a shallow fetch if fetch by commit hash fails because we -+ # do not know how deep to go to get to $GNULIB_REVISION, so we must get -+ # all commits. + git -C "$gnulib_path" fetch $shallow origin "$GNULIB_REVISION" \ + || git -C "$gnulib_path" fetch origin \ + || cleanup_gnulib build-aux/bootstrap | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/build-aux/bootstrap b/build-aux/bootstrap index 733659850..5315857e0 100755 --- a/build-aux/bootstrap +++ b/build-aux/bootstrap @@ -694,9 +694,25 @@ if $use_gnulib; then shallow= if test -z "$GNULIB_REVISION"; then git clone -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth 2' + git clone $shallow ${GNULIB_URL:-$default_gnulib_url} "$gnulib_path" \ + || cleanup_gnulib + else + git fetch -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth 2' + mkdir -p "$gnulib_path" + # Only want a shallow checkout of $GNULIB_REVISION, but git does not + # support cloning by commit hash. So attempt a shallow fetch by commit + # hash to minimize the amount of data downloaded and changes needed to + # be processed, which can drastically reduce download and processing + # time for checkout. If the fetch by commit fails, a shallow fetch can + # not be performed because we do not know what the depth of the commit + # is without fetching all commits. So fallback to fetching all commits. + git -C "$gnulib_path" init + git -C "$gnulib_path" remote add origin ${GNULIB_URL:-$default_gnulib_url} + git -C "$gnulib_path" fetch $shallow origin "$GNULIB_REVISION" \ + || git -C "$gnulib_path" fetch origin \ + || cleanup_gnulib + git -C "$gnulib_path" reset --hard FETCH_HEAD fi - git clone $shallow ${GNULIB_URL:-$default_gnulib_url} "$gnulib_path" \ - || cleanup_gnulib trap - 1 2 13 15 fi -- 2.27.0
