On Thu, Jan 07, 2021 at 06:17:19PM +0100, Paolo Bonzini wrote: > We are still using quite a bit of bandwidth to run CI jobs, even though > GitLab has switched to gitlab.com to fetch the sources. This is in > part because we are handling submodules ourselves and therefore those > do not use shallow clones.
I presume the impact is going to vary for GitLab vs the other CI systems. For gitlab, any bandwidth used should be entirely within gitlab's own network infra since its pulling repos from gitlab.com/qemu-project For Cirrus/Shippable/Travis the bandwidth used would be more of a potential performance problem. > Observe GitLab's GIT_DEPTH environment variable in > scripts/git-submodule.sh so that CI systems can use a shallow clone > for submodules. Set a depth of 3 for all CI systems, for both the main > repository (using whatever native configuration mechanism is there) > and the submodules. > > Reported-by: Stefan Hajnoczi <[email protected]> > Signed-off-by: Paolo Bonzini <[email protected]> > --- > Completely untested, just to propose the idea. The check-patch and check-dco jobs need the git history depth to be long enough to cover the entire contents of the branch that is being tested vs common ancestor against git master. This could potentially be 100's of commits deep for some of the very largest branches we see merged in qemu Travis has issues with git cloning and concurrent pushes. eg if you push branch A, it schedules a CI job. Then you push branch B before jobs for A have started. When the job for A starts, it will be unable to checkout the commit for branch A's merge because the history will be too shallow. I'm not sure if the other CI systems have similar issues or not. > > .cirrus.yml | 1 + > .gitlab-ci.yml | 3 +++ > .shippable.yml | 2 ++ > .travis.yml | 2 ++ > scripts/git-submodule.sh | 2 +- > 5 files changed, 9 insertions(+), 1 deletion(-) > > diff --git a/.cirrus.yml b/.cirrus.yml > index 6f2a958472..47c2467d04 100644 > --- a/.cirrus.yml > +++ b/.cirrus.yml > @@ -1,5 +1,6 @@ > env: > - CIRRUS_CLONE_DEPTH: 1 > + CIRRUS_CLONE_DEPTH: 3 > + GIT_DEPTH: 3 > > freebsd_12_task: > freebsd_instance: > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 01c9e46410..ff19a9c313 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -7,6 +7,9 @@ stages: > - build > - test > > +variables: > + GIT_DEPTH: 3 > + > include: > - local: '/.gitlab-ci.d/edk2.yml' > - local: '/.gitlab-ci.d/opensbi.yml' > diff --git a/.shippable.yml b/.shippable.yml > index 14350e6de8..f2d59b478e 100644 > --- a/.shippable.yml > +++ b/.shippable.yml > @@ -1,9 +1,11 @@ > language: c > +depth: 3 > git: > submodules: false > env: > global: > - LC_ALL=C > + - GIT_DEPTH=3 > matrix: > - IMAGE=debian-amd64 > TARGET_LIST=x86_64-softmmu,x86_64-linux-user > diff --git a/.travis.yml b/.travis.yml > index f2a101936c..f2a785a42b 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -80,9 +80,11 @@ env: > - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime" > - CCACHE_MAXSIZE=1G > - G_MESSAGES_DEBUG=error > + - GIT_DEPTH=3 > > > git: > + depth: 3 > # we want to do this ourselves > submodules: false > > diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh > index 65ed877aef..2886087a12 100755 > --- a/scripts/git-submodule.sh > +++ b/scripts/git-submodule.sh > @@ -75,7 +75,7 @@ update) > exit 0 > fi > > - $GIT submodule update --init $modules 1>/dev/null > + $GIT submodule update ${GIT_DEPTH:+--depth "$GIT_DEPTH"} --init $modules > 1>/dev/null > test $? -ne 0 && error "failed to update modules" > > $GIT submodule status $modules > "${substat}" > -- > 2.29.2 > > Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
