Johannes Schauer writes ("Re: Bug#868527: [buildd-tools-devel] Bug#868527:
Bug#868527: want sbuild --no-source or something"):
> I just had another idea. Since the issue at hand is, that you need a
> way to transfer the source tree into sbuild and sbuild currently
> only supports source packages, why can dgit not be amended to build
> a temporary source package from the git tree it has in a temporary
> location which it then hands to sbuild or any other builder?
Hi.
Thanks for looking at this bug again. I have been meaning to reply to
your messages for some days. Sorry for the delay, which is partly
caused by family stuff and partly by me traveling a lot (right now I
am in Madrid for the GNU Hackers Meeting...)
I'm afraid this is going to be a long mail.
I don't think Sean is right about the existence of functionality in
dgit which addresses this issue.
dgit is a bidirectional geteway between a dsc-style archive (and .dscs
in general) and git. When it is generating a .dsc, it tries to
generate a sane .dsc. dgit (at least in its existing operation modes)
cannot be used with git trees which are not representable as sane
.dscs.
It is *possible* to convert any git tree into a .dsc which will unpack
into that tree. But in the general case that conversion to a .dsc
requires several additional options to dpkg-source - the hideous rune
which is the subject of this bug report and which we are trying to get
rid of.
The resulting .dsc is insane. I don't want to try to completely
define "sane" in this context, but ISTM that a necessary condition for
sanity of a .dsc is that it is a fixed point under dpkg-source -x &&
dpkg-source -b. With one of these insane .dscs, dpkg-source -x works
but the subsequent dpkg-source -b fails (because it lacks the hideous
override options).
So let me step back a bit. I think it would be worth starting with an
investigation of what the problems are that this bug is about, and
then think about the best solution approach, intended new behaviour
and implementation strategy, and the names of the new stuff.
There are three things that seem to me like they are a significant
nuisance to a dgit user using a dgit view, with the current
arrangements. I fact they affect all users who are using pure git
workflow based on what Debian publishes, unless those users take
special care to maintain quilt patch queues up to date etc. - even if
those users don't ever run dgit (ie they got a git branch from
somewhere). I want users to be able to just use git and not have to
care about quilt etc.
The issues I see using sbuild in this situation are:
1. The need to pass a long, silly-looking and unergonomic option
to sbuild.
2. The fact that a .dsc is left lying around.
3. The fact that the .dsc left lying around is a beartrap: it cannot
be unpacked and repacked without again using weird options, and
must not be uploaded to anything like a Debian-style archive.
There are other possible concerns which I think are far less important
and possibly not important at all, but which I mention here only
because I thought of them (so that if someone else thinks they are
important they can be prompted to say so):
4. I'm not sure if making a 1.0 native gzip -1 dsc is as fast as other
possible approaches for copying the source into the chroot.
5. Possibly unclarity and infelicity surrounding untracked working
tree files and/or uncommitted changes.
6. Whether the .git directory should be passed into the chroot and
thus be present during the build. (I think yes - see below.)
Of these concerns, (3) follows inevitably from (2), in the sense that
if we want to deal with git trees generated from arbitrary packages,
and then modified in aribtrary ways (as envisaged in dgit-user(7)),
the .dsc might be insane, causing (3). Avoiding (2) avoids (3).
So we need to tackle (1) and (2).
It would of course be possible to write some wrapper program which:
- contains the horrid rune
- makes a .dsc somwhere private
- passes that .dsc to sbuild
There are a number of ways in which that would seem suboptimal to me:
- Such a wrapper program might have to interact with sbuild's option
parser (eg try to filter or understand options to sbuild). At the
very least, certain sbuild options don't make sense but maybe we
can just tell users who pass silly combinations and get strange
behaviours to please not do that.
- This would still do the work of building a source package, which
is needless copying etc. This is consideration (4) above.
Generating a .dsc makes us vulnerable to bugs and infelicities
in dpkg-source.
- The generated .dsc would presumably have to be in a directory in
/tmp or a temp directory under .git which is kind of inconvenient
from an implementation point of view (and we might leave it behind
if we don't manage to run our cleanup - ie we become less
crash-only). More generally the overall data flow is more
complicated than it needs to be.
- Thin wrapper programs are confusing to users and should be avoided
where possible. I know Debian packaging workflows are full of thin
wrappers - but this is indeed confusing. For example, any time you
see someone give a talk or a demo, or try to help them with some
packaging problem, you have some good chance that they will use
some wrapper or small utility you haven't heard of. This makes
everything hard to follow. I used to think that this was just me
being ignorant but since incidentally having to become an expert on
Debian git practices I have become convinced that there is so much
of this stuff no-one could be expected to know it all.
So I think it would be best to teach sbuild another way of
transferring source into the chroot, ideally without making an
unnecessary additional copy of it.
To my mind the obvious approach would be to make a new git tree in the
chroot and "git push" to it. (We want to copy the git objects because
some gittish workflows want a working .git directory, for example to
encode the built commitid somewhere in the output. This is obviously
not done in Debian but it would make sense for some downstream or
ad-hoc workflows. This does imply that git would have to be installed
in the chroot, by sbuild presumably.)
I haven't looked at the internals of sbuild at all. Do you think this
would be fairly straightforward ? Is it desirable ?
For the new option name(s) I suggest:
--git-ref=<somthing git-rev-parse understands>
--git equivalent to --git-ref=HEAD
As for naming etc.:
I appreciate that you are seeing this through a dgit lens. I can see
why, and dgit adoption is certainly a reason to think about this again
now. But I don't think this is actually "dgit stuff":
dgit's purpose is to help bridge the gap between source packages and
git. If you do not have an actual desire for source packages then you
should not get involved with dgit. (And if your involvement with
source packages is solely that your upstream publishes dscs, then you
use dgit clone/fetch to access it but then work solely in git.)
Ie, you can start with a pure git tree containing a debian/rules and
so forth, and modify it and build it, and so on, without involving
dgit. dgit is complicated and users who don't need to deal with dscs
shouldn't need to wrestle with it. (Ideally eventually dgit would
render itself obsolete...)
So for this reason I think any new sbuild option should *not* be named
after dgit. Likewise, if we decide that the best approach is the
wrapper script, I think the sbuild wrapper is probably best not being
function of dgit.
If we go for a sbuild wrapper I'm not sure where it should live. I
have been putting various things that aren't to do with dgit.deb or
/usr/bin/dgit in src:dgit because of the test suite I have there. But
testing such a wrapper script doesn't really seem to involve most of
the stuff that the src:dgit test suite is good at. Mostly it would
probably involve stuff that the sbuild test suite probably knows about
already. So maybe it should live in src:sbuild...
HTH.
Regards,
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
