Hi!
On Mon, 2016-06-06 at 15:35:14 -0700, Josh Triplett wrote:
> On Mon, Jun 06, 2016 at 11:44:00PM +0200, Guillem Jover wrote:
> > This is a non-portable Linux-only assumption. dpkg works on other
> > systems besides Linux-based ones.
>
> I'm aware; however, the question is whether other NFS client
> implementations fail to implement flock, and whether dpkg supports doing
> parallel builds on those systems on their less capable NFS
> implementations. What other NFS client implementations does dpkg
> support, and what do those other clients do with flock?
dpkg has been ported and has been reported to work on several Unix
system, some of which have had reportedly bad flock(2) behavior on
NFS. But I'm not entirely sure which ones currently still do that.
And while historically the dpkg-dev tools have not performed any
locking, I'd rather not regress on this now. But see below.
> It's unfortunate that the failure mode for flock on NFS is "silently
> no-op" rather than "fail"; the latter would make it easier to fall back
> to a serial build, for instance. Is there any reliable, portable way to
> detect an NFS filesystem with non-functional locking, or more generally
> a filesystem with non-functional locking?
I don't think there's even a way to detect that a filesystem is NFS! I
looked at that at the time, to just silently run flock(2) in that case,
but all interfaces are very system-specific. Not even command-line tools
to query. :(
> > I'm also unhappy even about the
> > fallback to flock when fcntl is not available, as on other systems
> > the locks might not see each other so that's the worst possible
> > behavior.
>
> Another potential solution to the problem would be to avoid serializing
> this at all, which then avoids the need to lock. What if each run of
> the requisite tool generated a different target control file, for
> instance? Or perhaps debian/files should become debian/files.d? As
> long as the corresponding dpkg tools support reading that format when
> constructing a .deb, migrating to that for parallel builds seems
> reasonable. That might also speed up dpkg-gencontrol, which at the
> moment can be a major bottleneck in the parallel build of a multi-binary
> package.
Actually I think I like this better than any other option, thanks!
It poses multiple problems though, which I'd like to tackle anyway,
because they would improve other parts of the toolchain, and things
I'd like to eventually fix as well:
* Although in principle debian/files is managed internally by dpkg-dev
and there's even a tool (dpkg-distaddfile) to add entries to it, it
has leaked as an internal interface, because debian/rules needs to
clean it up on its clean target. Using a different file/dir might
mean we start leaving cruft behind. :/
* I've just checked and using something like a debian/files.d dir
seems safe, but you never know. I think at this point, perhaps it
would be better to use a new directory for all generated cruft,
such as debian/.build/files.d/ or similar, and then we only need
to remove only that top entry.
* Currently dpkg-gencontrol does some sanity checks when adding a
new entry and overwrites entries with the same
(<pkgname>,<pkgarch>,<pkgtype>) tuples, where same arch includes
all or the exact arch. I think this can be mapped almost by
storing the entries in filenames with the form
pkgname_pkgarch_pkgtype for entries matching a package filename,
and otherwhite the whole filename.
The possible fixes could be:
1. Designate a new directory for temporary build-time artifacts
instead of polluting debian/.
2. Adding a new dpkg-buildclean or whatever that cleans all detritus
generated by dpkg-dev (or even other) tools, so that we can cover
similar issues in the future.
3. Changing debhelper to clean the new files/dir (possibly via the
new dpkg command), which would go a long way already.
I think I'll sleep over this a bit, and bring it up on debian-devel
for comments in case people see any drawbacks.
Thanks,
Guillem
