Hi Richard,
On Wed, Aug 12, 2020 at 10:18:54PM -0500, Richard Laager wrote:
> Package: src:git-buildpackage
> Version: 0.9.20
> Severity: wishlist
> Tags: patch
>
> Feature request:
>
> `gbp import-orig` should strip [~+](dfsg|ds).* or even [~+].* from the
> upstream version (only) when calculating the upstream-vcs-tag.
>
>
> Details:
>
> I'm trying to get `gbp import-orig --uscan` working with a variant of
> the repack-waf script from: https://wiki.debian.org/UnpackWaf
>
> The example here is NTPsec 1.1.9.
>
> The stock repack-waf script takes an input orig tarball of "1.1.9" and
> outputs an orig tarball of "1.1.9+dfsg1". Unfortunately, in this state,
> uscan is not aware that the tarball was repacked, so neither is `gbp
> import-orig`. gbp gets the version from the tarball name, which it gets
> from the <target> element in the `uscan --dehs` output.
>
> If I add oversionmangle=s/$/+dfsg1/ to the opts in debian/watch, then
> uscan is aware that the output will be 1.1.9+dfsg1. This seems like the
> correct thing to do from uscan's perspective, as its man page says
> oversionmangle "should be used to add a suffix such as +dfsg1".
> However, uscan will also symlink the upstream orig tarball with that
> name, so the stock repack-waf script breaks. That can easily be
> addressed by changing repack-waf to use the 1.1.9+dfsg1 name as both
> input and output, which I have done.
>
> In that configuration, gbp gets the upstream version of 1.1.9+dfsg1.
> This leads to tag names like upstream/1.1.9+dfsg1, which is what I have
> been using so far. DEP-14 isn't clear whether it should be
> upstream/1.1.9+dfsg1 or upstream/1.1.9. However, DEP-14 does
> contemplate that "The upstream/<version> tag would be created by the
> package maintainer when needed: for example when
> it does a release based on a Git snapshot". So those tags are not
> expected to correspond exactly with upstream's tags. I think they
> should be the "Debian upstream version", so 1.1.9+dfsg1 is correct.
>
> Further, in bug #546598, Charles Plessy <ple...@debian.org> suggested
> that `gbp import-orig --filter` should automatically add "~dfsg":
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=546598 That bug was
> from 2009. Today, plessy seems to use +dfsg.1, +ds-1, and +dfsg-1
> suffixes. More importantly than those minor differences in suffix
> style, some of those packages currently have numbers greater than 1.
> That to me shows that stripping the +dfsg1 to get just upstream/1.1.9
> would be inappropriate, as that would not scale to +dfsg2 and beyond.
> This is further evidence that upstream/1.1.9+dfsg1 is correct.
I agree that the tag should have '+dfsg' thus upstream/1.1.9+dfsg1.
> However, that still leaves one problem. The --upstream-vcs-tag tag
> format only allows for one substitution which has to be a single
> character substitution. There is no way to strip the +dfsg1. In my
> case, I want to get to the NTPsec_1_1_9 format that upstream uses by
> using: upstream-vcs-tag = NTPsec_%(version%.%_)s
>
> I ran some queries against UDD to find various patterns. Based on that,
> the attached patch series implements this and a bunch more. With this
> change, I was able to import version 1.1.9 of NTPsec using:
>
> gbp import-orig --uscan --upstream-vcs-tag="NTPsec_%(version%.%_)s"
>
>
> The first patch is just a typo fix to a comment nearby. It's unrelated
> to this change.
>
> The second two patches are refactoring changes. Those can be squashed
> together or into the last change, if you prefer. I included them to show
> the refactoring steps individually, in case that helps you review this.
>
> The third change is the meat of this. I have doctests, but I'm not sure
> if those are automatically run or how I integrate them into the
> git-buildpackage tests. I'm also not sure about the function name for
> _upstream_version_from_debian_upstream(). In a previous version, I
> called it _mangle_upstream_version().
I think the approach to incorporate common practice in Debian to figure
this out makes sense. I wonder if we
- should toggle it on by default
- have a way to turn it off (that is use the tag verbatim)
- have a NEWS.Debian entry that explains the change
- have this mentions in the docs (at least the import-orig manpage)
I've applied the first three patches already, thanks for that! Doctest
should be picked up by nostests automatically (you can check by creating
a failing test).
[..snip..]
> From 1c8f2acbcbd3ccf14dc4d7bd99b4d1e92e4c1574 Mon Sep 17 00:00:00 2001
> From: Richard Laager <rlaa...@wiktel.com>
> Date: Wed, 12 Aug 2020 22:03:18 -0500
> Subject: [PATCH 4/4] import-orig: Parse various Debian version patterns
>
> This parses common Debian version patterns into the upstream version.
> For example, 1.1.8+dfsg1 becomes 1.1.8. This strips epochs, handles the
> +really convention, finds git revisions, and strips other + or ~
> patterns.
> ---
> gbp/deb/git.py | 59 +++++++++++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 56 insertions(+), 3 deletions(-)
>
> diff --git a/gbp/deb/git.py b/gbp/deb/git.py
> index 4a17c2d..a5367f4 100644
> --- a/gbp/deb/git.py
> +++ b/gbp/deb/git.py
> @@ -129,6 +129,57 @@ class DebianGitRepository(PkgGitRepository):
> version = "%s:%s" % (epoch, version)
> return version
>
> + @classmethod
> + def _upstream_version_from_debian_upstream(cls, version):
I can't come up with a better name either.
> + """
> + Convert a Debian upstream version to an upstream version.
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "1.1.8+dfsg1")
> + ('1.1.8', None)
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "1:5.5.0~dfsg")
> + ('5.5.0', None)
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "0.0~git20160722.0.0cdb66a")
> + (None, '0cdb66a')
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "3:6.04~git20190206.bf6db5b4+dfsg1")
> + (None, 'bf6db5b4')
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "2.3+really2.2")
> + ('2.2', None)
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "4.99.4+dfsg+really4.10.0+py3")
> + ('4.10.0', None)
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "5.4.1+really5.4.1~repack")
> + ('5.4.1', None)
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "0.0~git20190103.40eba7e+really0.0~git20181023.b4e2780")
> + (None, 'b4e2780')
> + >>> DebianGitRepository._upstream_version_from_debian_upstream(
> + ... "1.1.0+really1.0.0+git20181028.94f6ae3")
> + (None, '94f6ae3')
Maybe the demo case where nothing gets mangled too.
Cheers,
-- Guido
> + """
> +
> + # If version starts with an epoch, remove it.
> + m = re.search(r'^[0-9]+:(.*)', version)
> + if m:
> + version = m.group(1)
> +
> + # If version has "really", keep the version starting after "really".
> + m = re.search(r'really(.*)', version)
> + if m:
> + version = m.group(1)
> +
> + # If version matches a git snapshot pattern, return the git revision.
> + m = re.search(r'[~+]git[0-9]{8}(\.[0-9]+)?\.([0-9a-f]+)', version)
> + if m:
> + return (None, m.group(2))
> +
> + # Strip off anything starting with [+~] and return the rest.
> + return (re.split(r'[+~]', version)[0], None)
> +
> @staticmethod
> def _build_legacy_tag(format, version):
> """
> @@ -377,11 +428,13 @@ class DebianGitRepository(PkgGitRepository):
> """If linking to the upstream VCS get the commit id"""
> if not vcs_tag_format:
> return None
> + (version, revision) =
> self._upstream_version_from_debian_upstream(version)
> try:
> - tag = self.version_to_tag(vcs_tag_format, version)
> - return [self.rev_parse("%s^{}" % tag)]
> + if not revision:
> + revision = self.version_to_tag(vcs_tag_format, version)
> + return [self.rev_parse("%s^{}" % revision)]
> except GitRepositoryError:
> - raise GitRepositoryError("Can't find upstream vcs tag at '%s'" %
> tag)
> + raise GitRepositoryError("Can't find upstream vcs tag/revision
> at '%s'" % revision)
>
>
> # vim:et:ts=4:sw=4:et:sts=4:ai:set list listchars=tab\:»·,trail\:·:
> --
> 2.28.0
>
